Test your integrations at scale — Contract testing with Pact.io

Author: Dr. David Dasenbrook (Senior Consultant at Deloitte | Technology & Transformation)

Contract testing provides the advantages and confidence levels of inter-service integration testing while being faster and more scalable. Instead of writing integration tests that span across services, application teams can write tests in isolation and generate contracts, or “pacts”, from these tests. A contract testing framework such as Pact.io can automatically check whether the generated contracts are compatible with each other. In the well-known testing pyramid, contract tests are located between unit and integration tests: they can be as fast to run and write as unit tests but provide almost the same level of confidence as integration tests.

Contract testing — how it works and why it is useful

Limitations of classical integration testing

Let’s face it — cross-service integration tests are hard to write and execute consistently. Consider two services communicating with each other via a RESTful API. If service A (the consumer) wants to automatically test their integration with service B (the provider), they need at least a running instance of the provider service to perform their requests against. The provider will typically be stateful — meaning its behavior and responses will depend on data currently in the database or in-memory state. To set up the state, service A needs to make further requests to the API, meaning tests will have to cover more or less complete end-to-end cases instead of just single interactions. For example, it may have to create an entity, assert it can be retrieved, and delete it again. Extending the scope of single tests in this way inevitably means a compromise has to be made affecting the number of scenarios being tested. Furthermore, the bigger these scenarios become, the greater the risk of flakiness — an instability in one step of the process leads to the whole test failing.

When there are not just two services communicating with each other, but a whole landscape, things only get worse. In realistic cases, it can be shown [1] that in integration testing, the scaling of resources required as the number of services and connections between services grows is at best quadratic, at worst exponential. This means that in addition to the challenges encountered regarding stability and maintenance of a large integration test suite, one encounters bottlenecks in compute resources.

Consumer-driven contract testing

The basic idea of contract-based testing is simple: Instead of testing complex integrations directly, each service initially writes tests in isolation, using test doubles for the external dependencies. Up to now, this comes down to simple unit testing. The magic ingredient in contract testing is that all the interactions performed against the test double during the test execution are recorded and captured in a contract. Contracts are being shared between services and a contract testing framework can provide guarantees on the correctness of the behavior patterns recorded in them.

Fig. 1: Consumer-driven contract testing

One of the most popular frameworks for contract testing is Pact.io [2]. The pattern it implements is the consumer-driven contract testing approach illustrated in the picture above. Here, in each cross-service interaction, the service either plays the role of a consumer or a provider (but each service will typically be both.) In the case of a REST-API, the service implementing the API would be the provider, while the service initiating the calls would be the consumer. In the consumer-driven approach, consumers write tests asserting particular behavior patterns of their provider (1). As described above, these tests run against a mock provider automatically generated by the Pact.io framework, which additionally captures the consumer’s interactions in a contract or Pact (2). Pacts are then shared with the provider team (3), who write verification tests for them (5). This verification involves replaying the interactions captured in the Pact against a running instance of the provider and asserting that the behavior of the real provider matches the expectations formulated by the consumer (4).

How is this different from schema validation?

A common approach is to describe the interactions provided by an API in a structured document such as an OpenAPI specification. This document can be used by the consumer to generate their API client code and the schema described in this document can be validated against test data the provider returns in actual API invocations.

Contract-testing and API documentation (including schema validation) serve different purposes. In contract testing, as the name suggests, the focus is on testing: Consumers write tests that actually feed the responses received from the provider mock into their API client code and make assertions, while providers are encouraged to run the pact verifications against actual application instances — mocking only small parts of their code to be able to set up well-defined application states for the tests. In contract testing, it is meaningful to talk about code coverage. Applications can write tests covering actual business scenarios, with the inter-service communication replaced by mocks recording the contracts.

Machine-readable API specs and their validation are important for documenting the behavior of interfaces and APIs and keeping the documentation in sync with the actual behavior of the services implementing them. In contrast, contract testing tests real-world scenarios involving communication across interface borders. Both approaches can be used alongside each other.

Pact.io

Pact.io is one of the frameworks most commonly used for contract testing, especially in the consumer-driven approach described above. Pact.io consists of a common formal specification for contracts, language-specific frameworks implementing this specification and enabling developers to write consumer tests and provider verifications, and a broker application that facilitates sharing Pacts between applications. We will look at all of these parts in a simple example case.

Example: Consumer test

To illustrate the concepts discussed until now, we will look at a simple example. Consider a unicorn application implementing a CRUD API for unicorns — with endpoints like POST /unicorn, GET /unicorn/{id} and similar. The consumer team has received the API specification from the provider team and implemented a client for it. Instead of setting up a complicated integration test for the client code in combination with the provider application, they want to leverage contract testing. In the consumer-driven approach, they will now start by writing a consumer test. The Pact framework enables them to use their standard unit test framework to write the test, generate a contract from it, and also provides the provider mock that the tests can run against.

This example will use Java and JUnit. Pact implementations for other languages and frameworks exist.

The consumer test first specifies the expectation of the behavior of one of the API endpoints using a Pact-specific DSL:

@Pact(consumer = "UnicornServiceConsumer")
public V4Pact createUnicornListPact(PactDslWithProvider builder) {
    return builder
            .given("some unicorns exist in the database")
            .uponReceiving("a request for available unicorns")
            .path("/unicorns")
            .method("GET")
            .willRespondWith()
            .status(200)
            .body(buildUnicornListJsonBody())
            .toPact(V4Pact.class);
}

This method is part of a standard JUnit test class and describes one of the interactions with the GET /unicorns endpoint. Using the .given statement, we can declare that we need the provider to be in a particular state for any tests using this endpoint. In this case, we simply state “some unicorns exist in the database”, as we want the endpoint to actually return a list of results instead of an empty list. For testing various edge cases, we might later use different .given statements to set up different situations in the provider. We will come back to this when we explain the provider verification.

For completeness, we also show the buildUnicornListJsonBody() method used here, which describes the expected response body in more detail:

private DslPart buildFruitListJsonBody() {
    return new PactDslJsonBody()
        .minArrayLike("unicorn", 1)
            .stringType("name", "sparkles")
            .stringType("id", "123")
            .stringMatcher(
                "color", 
                "^(red|blue|green|yellow|black|white)$", 
                "green"
            )
        .closeArray();
}

This specifies that we expect the JSON response to contain at least 2 elements with a specific structure (properties “name”, “id” and “color” following specific types or matching a regex). The examples provided for the expected values (“sparkles”, “green” etc.) are then used for the automatic mock generated by the Pact framework to return responses.

Having set up the expectations, we can now write a test case making assertions about it:

@Test
@PactTestFor(pactMethod = "createUnicornListPact")
void testFetchListOfUnicorns(MockServer mockServer) {
    var unicornClient = new UnicornClientImpl(mockServer.getUrl() + "/unicorns");
    var unicorns = unicornClient.fetchAvailableUnicorns();
    unicorns.forEach(unicorn -> 
        assertThat(unicorn.getName()).isInstanceOf(String.class)
    );
}

The Pact framework takes care of injecting an instance of the automatically generated MockServer for us, and we use it to make a request against the unicorns endpoint. Importantly, we are using an instance of the unicorn client also used in the production code for the test case. Thus, the test case covers domain code and increases our confidence in it. Otherwise, we would have just created a mock server and tested it without any reference to the consumer application, producing a contract with very little value. Among other things, the test above would break if, for example, the client was no longer able to correctly deserialize the unicorn object from the JSON response received from the provider - a situation we want to be warned about by a failing test.

Example: Provider verification

The above setup on the consumer side will produce, in addition to the consumer test itself, a Pact file that has to be shared with the team developing the provider. The pact file is a JSON file following a specification published by Pact.io and is independent of the language framework used by the individual applications. For example, a provider written in Go can easily verify a contract published by a consumer written in Java.

The next step in the consumer-driven contract testing process is for the provider team to verify the contract published by the consumer. This provides the missing link in the overall testing architecture, ensuring that the tests the consumer is running against the automatically generated mock are realistic and match the actual behavior of the provider service.

Assuming the unicorn service is again implemented in Java, the implementation of the pact verification is simple:

@TestTemplate
@ExtendWith(PactVerificationInvocationContextProvider.class)
void pactVerificationTestTemplate(PactVerificationContext context) {
    context.verifyInteraction();
}

This is already the main part — the pact framework takes care of discovering the consumer contract, replaying the interactions captured in it against a running instance of the provider service, and verifying that the actual provider responses match the expectations previously specified by the consumer. For this to work however, some more setup is required. When using the Spring Boot framework, to spin up an actual instance of the application listening on a random port, the provider verification test class should be annotated by

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)

We can then obtain the value of the port and tell Pact to use it to perform the verification with a setup method like the following:

@LocalServerPort
int port;

@BeforeEach
void setupPactVerification(PactVerificationContext context) {
    context.setTarget(new HttpTestTarget("localhost", port));
}

Provider states

As mentioned before, for the pact verification to be successful, the provider typically needs to be in a particular state. For example, if the unicorn service has no unicorns in the database, the response of the GET /unicorns call will be empty, which does not fulfill the expectations recorded by the consumer into their pact. We will therefore now come back to the .given statement described in the consumer example - these describe specific states that the provider needs to prepare in order to verify the expectations of the consumer. To this end, the provider needs to implement state setup methods.

Get De.Code – Deloitte Germany Engineering Blog’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In our example, it looks like this:

@State("some unicorns exist in the database")
public void someUnicornsExistInTheDatabaseState() {
    var unicornNames = List.of("sparkles", "rainbow", "twilight");
    for (String name : unicornNames) {
        var unicorn = createTestUnicorn(name);
        unicornService.addUnicorn(name);
    }
}

This assumes that we have injected a unicornService instance into our Pact verification test, which can be used in the provider state setup method to prepare the DB in a particular state, i.e. add some example unicorns to it. The unicorn service used here might even be a partial mock and not use the real database, which speeds up the test execution and eliminates the need to spin up a full DB instance for the provider test.

This provider state method will be called by the Pact framework when it is asked to verify a Pact that contains the statement .given("some unicorns exist in the database").

The pact broker

How can the consumers efficiently share the Pacts they have produced with their providers for them to verify? This is where the Pact broker comes into play. It plays a crucial role in facilitating seamless communication and collaboration between consumers and providers in the contract-testing process. By serving as a central platform, it enables consumers to publish their Pacts, which providers can then retrieve for verification. Providers, in turn, can record their verification results back to the Pact broker. One of the key advantages of utilizing the Pact broker is its ability to efficiently manage and track versions of consumers and providers, deduplicating Pacts in the background to streamline the testing process.

One option is to use Pactflow [3], a SaaS solution from SmartBear, but it’s also possible to self-host a broker by using a Docker image. Interactions with the Pact broker are seamlessly integrated into the Pact SDKs for different development frameworks.

The broker knows about all the Pacts each version of each application has with other applications, and whether it is currently successfully verified or not. From this information, the broker builds a compatibility matrix detailing which versions of applications are currently known to work together. Teams can choose to record their deployments with the Pact broker, i.e., let the broker know which version of their application is currently deployed to which environment. Then, a command-line tool called can-i-deploy can be used to literally ask the question if the application developed by the team is currently safe to deploy.

For example, let’s assume an application A has Pacts with two dependent services, D1 and D2. Version 1 of D1 has verified A’s Pacts and is deployed to the testing environment. Version 1 of D2 has also verified A’s Pacts, but the version currently deployed to testing is version 2, which has not yet verified the Pacts with A. Using can-i-deploy to ask if service A is currently safe to deploy to the environment testing would then yield a negative response.

Pact as a quality gate

can-i-deploy is ideal for use as a quality gate in CI pipelines. As outlined above, it answers the question of whether a particular version of an application is currently compatible with the dependencies deployed in a particular environment, and therefore ensures a complex system landscape is always correctly integrated without any breaking points.

The Pact consumer tests and provider verifications are typically part of the application’s unit test suites and can therefore run together with the standard unit tests or alternatively be split off into a separate step in the CI pipeline. A common way of organizing the steps in a pipeline that includes contract testing would be:

1. Run all unit tests, including all Pact consumer and provider tests.

2. If successful,

• publish consumer Pacts,
• and provider verification results.

3. Call can-i-deploy to query the pact broker about whether it is safe to deploy.

4. If successful, deploy the application.

5. Record the deployment of the new version of the application back to the Pact broker.

The caveat here is that all teams in the application landscape have to be on board with the general structure — if one of the application teams chooses not to record their deployments, the use of can-i-deploy as a quality gate is no longer possible for the dependent teams. Contract testing reduces the complexities inherent in full E2E testing scenarios and enables the teams to write unit-like tests to secure their integrations, but there is no free lunch - alignment among teams and some overall guidance is still required.

Contract testing at scale

Contract testing ideally needs some commitment across the whole organization or IT landscape. When scaling the approach beyond a few services that communicate with each other, it is common to encounter challenges related to buy-in, reporting, communication, and quality. We will discuss some of the issues we have encountered at the enterprise scale and some remedies in the following sections.

When is contract testing a good fit?

It is important to emphasize that contract testing is not a silver bullet. When used efficiently, contract testing can be a valuable tool for getting early developer feedback on service integrations, catching integration-related bugs early, and also get an overview on the state of service integrations across a full landscape. It does not completely replace end-to-end testing however.

There are also a couple of situations where contract testing is not a feasible approach. For example, when developing a public API with unknown consumers, contract testing, especially the consumer-driven approach, is not a suitable choice. This is due to the fact that the approach requires close communication between the consumer and provider teams. Furthermore, while many examples and experience exists for using Pact with protocols and integration patterns such as HTTP / REST, Kafka and MQ, some other technologies like GraphQL or Protobufs are less tested.

Observability

Contract testing is not a fire-and-forget process — tests have to be maintained, verifications that were once successful may fail again over time when providers or consumers introduce new features and changes, new Pacts will be written for use cases previously not covered, and existing Pacts will be removed once the integrations are no longer relevant. While an ongoing commitment on the part of the teams is essential to a successful Pact adoption, it is helpful to also have a high-level overview of the current status of Pacts across an application landscape or even on the enterprise level.

Unfortunately, the Pactflow broker only provides a partial overview. It is great as a single exchange trading point for Pacts and recording their verification status, but it is often difficult to track the status of an integration between two applications over time, let alone extract aggregated data on the Pact adoption status and overall quality. Furthermore, to gain actionable insights from this, it is helpful to make use of other data sources as well — such as an overview of all required Pacts extracted from an enterprise architecture tool. It can therefore be wise to integrate the data provided by the Pact broker into a more far-reaching observability tool capable of aggregating and visualizing multiple data sources. Examples of these tools include Grafana or Datadog. With the help of the REST API provided by Pactflow and customized integrations with the observability tooling of choice, one can build dashboards that provide actionable insights such as “80 percent of all our service integrations are covered by Pacts” or “out of those 80 percent, only 50 percent are currently successfully verified”.

Further challenges

In an enterprise context with a potentially large application landscape, some general guidelines and frameworks have to be provided at a high level, typically by a team of enterprise architects. In the case of contract-testing, this is especially important, since there is no value in just a single application adopting a framework like Pact.io. Instead, it has to be set as a global standard along with further rules and conventions for all applications to follow.

This implies getting all teams on board through a clear communication strategy and providing centrally managed resources, such as tutorials, collections of best practices and in-person workshops. It is important to get buy-in not only from the developers, but also from stakeholders such as product owners and project managers. Introducing Pacts in the development and QA processes is not always a trivial task, therefore it is important for everyone involved to understand the added value. Otherwise, it is easy for development teams to fall into the trap of considering implementing contract testing as just “one more task to do”, leading to Pacts which do not actually ensure that integrations won’t accidentally break, or inefficiently written tests duplicating code or being overly specific.

Additional guidelines to be provided at a higher level include naming conventions for entities publishing and verifying Pacts with the Pactflow broker (so-called “pacticipants”), environments, interaction names, and provider states. Agreeing on simple common patterns reduces friction between teams and eliminates misunderstandings. Even then, a key ingredient for a successful Pact adoption is an open and inclusive communication culture. It has to be easy for development teams to contact each other directly and obtain fast feedback on their contracts. In most cases, email is too slow for this, and it is certainly not advisable to have all communication between application teams happen only via the product owners — developers ideally have a low-barrier to just directly contact their counterparts working on their dependent service applications.

As mentioned above, it is useful to have a good overview of the state of various Pacts for the management or enterprise architecture levels. Successfully validated Pacts are not a guarantee for the compatibility of interfaces, in the same way that passing unit tests ensure that the software is free of bugs. Writing good contract tests takes time, commitment and practice. It is therefore a good idea to establish a good culture of technical knowledge sharing, for example by organizing regular workshops and communities of practice, and establishing dedicated channels for exchange in the company’s communication tools.

Finally, with a very complex web of connected services using Pact to test each other’s compatibility, the complexity of the various components that Pact provides (consumer tests, provider verifications, recording deployments, can-i-deploy) also increases. It is possible (although rare) to end up in a deadlock situation where no application is allowed to deploy any more due to dependencies on other applications that have verified some needed Pacts, but are not deployed yet. If this happens, it might also be a good time to review the underlying architecture, simplify, and refactor. In the end, contract testing only complements good software engineering practices.

Conclusion

Consumer-driven contract testing is an efficient approach to testing integrations between microservices or other applications and verifying their compatibility. It combines the speed and simplicity of unit tests with the added security of traditional integration testing. Getting started with Pact.io, a common framework implementing this approach, is simple. When scaling this to an organizational level, additional challenges come up, but the investment can be well worth the effort.

About the Author

Dr. David Dasenbrook is a software architect at Deloitte. A theoretical physicist by training, he switched to IT consulting and engineering in 2016 and has since helped various clients build, refactor and modernize their application systems. His main focus areas include cloud-based architectures and serverless computing.

References/Additional Resources

1. Proving E2E tests are a scam [online] Available at: https://pactflow.io/blog/proving-e2e-tests-are-a-scam/ [Accessed: 17 March 2025]
2. Pact: Microservices testing made easy [online] Available at: https://pact.io/ [Accessed: 17 March 2025]
3. Comprehensive Contract Testing [online] Available at: https://pactflow.io/ [Accessed: 04 April 2025]