Contract Testing with Pact and .NET

What is Contract Testing?

Contract testing is an approach to Integration testing which, in contrary to traditional end-to-end integration testing, does not require components to be deployed together into a test environment.

In a nutshell, contract testing is a strategy to ensure two separate components/services/systems are compatible by testing the messages sent between them. Does this component communicate in the expected format? This is tested both ways, so the consuming component (Aka 'Consumer') is checked that the message sent to an integrated component as per an agreed contract, and the responding component (Aka 'Provider') is checked for it's response. Note these tests are actually run independently from one another so deploying them into a test environment is not required. Think of contract testing as unit testing for integrations.

It is worth pointing out that contract testing is used only for testing integrations and not for testing specific component behaviours. This should be covered in other phases of your test strategy.

Why should I Contract Test?

There are a whole host of reasons why contract testing is so beneficial to a development project. The best arguments for it can be found here Convince me | Pact Docs

However, I'd like to provide a high level overview as to the main weakness in traditional integration test approaches (as summarised by Atlassian in their talk on Pact).

If we categorise our tests we can think of four key types, Mocked Dependency tests (such as Unit Tests), Integration tests, End-to-End tests and Manual tests. All of these test types are valuable but all have a fundamental weakness.

Mocked Dependency tests are cheap to write, fast to run, very reliable and extremely targeted in their approach. All great stuff. However, these types of tests are NOT trustworthy. Why? Because they rely on mocked objects and dependencies, meaning they rely on the assumptions that we as developers and quality engineers have made.

Integration tests on the flip side ARE trustworthy. Having deployed actual components into a real test environment we are not forced to rely on those assumptions mentioned above. However, integration tests are expensive to write, slow to run, can be unreliable and are not sufficiently targeted in their approach. As a result, we end up spending huge amounts of time managing test environments, investigating failures and maintaining test suites. This is what has become known in the Pact community as 'Integration Hell'.

End-to-End tests are essentially integration tests 10X. In other words, the are even more trustworthy than our integration tests as more of our components are deployed and tested. But they also 10X the integration hell. More components means more cost to write, more time to run, less reliability and more time tracking down reasons for failure.

Manual tests have a number of issues which I believe are well known so will not cover here. But in a nutshell, problems with scalability, development bottlenecks etc...

With this in mind, what we actually need in our test approach are some tests that are cheap, fast, reliable, targeted AND trustworthy. Enter contract testing with Pact.

What is Pact?

Take a look at this diagram.

What the diagram demonstrates is how Pact ensures that both the Consumer and Provider adhere to the same contract or pre-determined method of communication.

Pact sits as a mock provider service in Consumer contract tests, listening to requests and responding with the expected response. This means that we can test the consumer in isolation. Pact will also verify whether the incoming request adheres to the contract it has been given. If it doesn't then Pact will fail the test. Note that when you first run your consumer test it will generate a Contract or PACT file (JSON), you do not need to do this yourself.

The team responsible for the Provider component will then pick up the Pact file and Pact will now act as a simulated consumer, sending the request to the Provider and waiting for the response. Once it receives a response from the provider Pact will verify whether the response satisfies the minimum expected response stated in the contract. It is important to point out that we are not required to link the provider up to a real database for these tests to work, they should actually have mocked data so that the test remains truly independent. There is a simple way to do this which is explained in the tutorial project I have linked below.

By utilising Pact we can simply shift left our integration tests running them much earlier in the process and including them alongside unit tests in our pipelines.

What about Pact for .NET?

Pact is a consumer driven contract testing library written in Ruby. If you are working with .NET then I'd recommend using the .NET implementation of Pact simply known as PactNet.

The original Pact library has lots of great features and is constantly being improved upon. PactNet on the other hand is a little further behind on some of its core features. Most major features are covered but chances are you will need to play about in some instances to get exactly what you need.

The documentation can be a little sketchy in some areas too, but overall they have done a good job and most organisations will be able to utilise this version of the library.

Id highly recommend you follow the tutorial laid out in the documentation here: pact-foundation/pact-net. Mock up a dummy consumer and provider API if you can and write both sides of the Pact test so that you can see how the whole flow works. Understanding the provider side of the test will help you write good consumer tests going forward.

Are there any other tools I can utilise whilst contract testing?

There are a couple of great tools I'd try out alongside Pact which I think provide some really nice features.

The first one is Atlassian's Swagger Mock Validator tool which provides you with an alternative to completing the whole Consumer-Provider pact test flow. This tool is a console application written in node.js which essentially picks up a pact file and compares it to a Swagger/OpenAPI specification. In other words, if you are working in a consumer team and your provider team aren't yet convinced by contract testing, you can write your contract tests and validate that your consumer is compatible with the swagger spec. This also means you don't need to wait for the provider team to complete validation on their end.

This approach can also be used on the provider side. You may want to make a change to your spec. The Mock Validator tool can be used to ensure that the changes to your spec are compatible with your consumers by comparing to the pact files. This is a lot quicker than writing the full provider pact tests which take a little more thinking about. It is worth noting however that Atlassian see this very much as an experiment and are not yet fully aware of all the benefits and drawbacks. For source code see: atlassian / swagger-mock-validator — Bitbucket

The second tool I would like to mention is a commercial tool known as PactFlow (I have no commercial relationship, I just really like the tool). This is a paid for version of the open source Pact broker, the place where you store all of your contracts. The service is feature rich and provides a nice UI so that you can run, manage and maintain your pact tests with ease. It also allows you to see a network diagram of all of your integrations and the relevant contracts between each. This tool is probably more for anyone who is serious about rolling out contract testing across their organisation, and in many cases the free version will suffice. But it is definitely worth a look if you want to take your contract testing to the next level. More info see: Pactflow | Contract Testing with Pact at Scale

Conclusion

Contract testing is hugely beneficial to teams who implement it correctly. It isn't a quick fix by any means and can take a while for teams to fully adopt. However, given the right amount of care, contract testing with Pact can save you a huge amount of time and resource. It really is a better way to test your integrations.