# contract-testing-with-openapi
Source: https://speakeasy.com/blog/contract-testing-with-openapi


import Image from "next/image";
import { Table } from "@/mdx/components";

We've all heard that infernal phrase, "It works on my machine." Scaling any solution to work across many machines can be a challenge. The world of APIs with disparate consumers is no different. What if there was a well-defined contract between them?

But APIs and consumers change over time, which begs the question: How do we ensure our systems stick to their agreements? The answer is contract testing.

In this article, we'll explore contract testing, how it fits into the testing pyramid, and why it's important. We'll also walk through how to implement contract testing using OpenAPI, a popular API specification format.

Finally, we'll discuss how you can generate contract tests automatically using OpenAPI and Speakeasy.

## What is contract testing?

Contract testing verifies that two parties who interact via an API stick to the contract they agreed upon. We'll call these parties a consumer and a provider, based on the direction of the API calls between them. The consumer is the system that makes API calls, and the provider is the system that receives and responds to the API call.

In most cases, the contract between our parties is defined by an API specification, such as OpenAPI.

Even if there is no API specification, there is always, at the very least, an implicit agreement between the consumer and the provider. The consumer expects the provider to respond in a certain way, and the provider expects the consumer to send requests in a certain way.

This explicit or implicit agreement is the contract.

Contract testing, from the consumer's perspective, is about verifying that the provider sticks to the contract. From the provider's perspective, it's about verifying that the consumer sticks to the contract.

## How contract testing differs from unit testing, integration testing, and end-to-end testing

Testing strategy is often represented as a pyramid, with unit testing at the base, followed by other testing methodologies in ascending order of complexity and scope. The idea is that the majority of tests should be at the base of the pyramid, with fewer tests at each subsequent level.

The testing pyramid is a useful model for thinking about how different types of tests fit together in a testing strategy. It helps to ensure that the right types of tests are used in the appropriate proportions, balancing the need for comprehensive testing with the need for fast feedback.

<div className="mx-auto max-w-md">
  <Image
    src="/assets/blog/testing-pyramid.svg"
    alt="Testing pyramid showing unit testing at the base, followed by contract testing, integration testing, and end-to-end testing"
    width={1000}
    height={1000}
  />
</div>

Let's discuss each level of the pyramid in more detail, starting from the base.

### Unit testing: Does this function work as expected?

Unit testing forms the base of the testing pyramid. These tests focus on individual components or functions in isolation, typically mocking any dependencies. They are fast to run and easy to maintain, but don't test how components work together.

In the consumer's context, a unit test might verify whether the function that deserializes a JSON object into a Python object works correctly, without making any external API calls.

Unit tests are essential for catching bugs early in the development process and providing fast feedback to developers. They are also useful for ensuring that code behaves as expected when refactoring or adding new features. However, they don't provide much confidence that the system as a whole works correctly.

### Contract testing: Do we honor the API specification?

Moving up the pyramid, we have contract tests. Contract testing sits between unit testing and integration testing. It focuses specifically on the interactions between the provider and consumer for a given call, ensuring that the API contracts are honored. Contract tests are more complex than unit tests but less complex than integration tests.

A contract test verifies that a consumer can create requests with specific data and correctly handle the provider's expected responses or errors. This might be accomplished with mocked request or response data based on the contract. For example, an API contract test for an order creation endpoint might verify that request data correctly maps integer item IDs to quantities and that the response decodes to an expected success with an integer order ID.

Contract tests are useful for catching issues that arise when the consumer or provider strays from the agreed-upon contract. They provide a level of confidence that the system works as expected when the consumer and provider interact. They are also useful for catching breaking changes early in the development process.

### Integration testing: Do systems work together?

Further up, we find integration tests. These verify that different components of a system work together correctly. They are more complex than unit tests and contract tests, and may involve multiple components or services.

An integration test might verify that the consumer can successfully make an API call to the provider and receive a valid response. This test would involve both the consumer and provider, and would typically run in a test environment that mirrors the production environment.

Because integration tests involve multiple systems, they are useful for catching issues that arise when components interact, such as network issues between two services.

### End-to-end testing: Does the user flow work as expected?

At the top of the pyramid are end-to-end tests. These test the entire user flow and supporting systems from start to finish. They provide the highest level of confidence but are also the slowest to run and most difficult to maintain.

In our API context, an end-to-end test might involve making a series of API calls that represent a complete user journey, verifying that the system behaves correctly at each step. This could include creating a resource, updating it, retrieving it, and finally deleting it, all through the API.

When end-to-end tests fail, it can be challenging to identify the root cause of the failure, as the problem could be in any part of the system. Due to their complexity and cost, they are often used as a final check before deploying to production, rather than as part of the regular development process.

### Testing pyramid summary

Here's a summary of the different types of tests and how they compare:

<Table
  title="Testing Pyramid Comparison"
  columns={[
    { key: "aspect", header: "Aspect" },
    { key: "unit", header: "Unit testing" },
    { key: "contract", header: "Contract testing" },
    { key: "integration", header: "Integration testing" },
    { key: "e2e", header: "End-to-end testing" },
  ]}
  data={[
    {
      aspect: "Scope",
      unit: "Functional logic",
      contract: "Interface logic",
      integration: "Provider implementation",
      e2e: "User journeys",
    },
    {
      aspect: "Speed",
      unit: "Very fast",
      contract: "Fast",
      integration: "Moderate",
      e2e: "Slow",
    },
    {
      aspect: "Complexity",
      unit: "Low",
      contract: "Medium",
      integration: "Medium to high",
      e2e: "High",
    },
    {
      aspect: "Isolation",
      unit: "High",
      contract: "Medium",
      integration: "Low",
      e2e: "Very low",
    },
    {
      aspect: "Typical quantity",
      unit: "Many",
      contract: "Several",
      integration: "Some",
      e2e: "Few",
    },
  ]}
  rowKeyField="aspect"
/>

## Why contract testing is important

Over time, APIs change in response to changing requirements, sometimes in subtle and imperceptible ways. For example, a provider may change the format of a field in a certain response from a string to an integer. This change may seem innocuous to the provider but could have catastrophic effects for the consumer.

Contract testing mitigates this risk by ensuring that any changes to the API contract are detected early in the development process. When the provider updates the API, corresponding contract tests will fail if the update is not backward compatible. This failure acts as an immediate signal that the change needs to be reviewed, preventing breaking changes from reaching production.

Consider this example: A subscription management platform (the provider) has an endpoint `/plan/{id}` that returns a subscription plan based on the plan ID. The consumer expects the response to include an `amount` field, which is an integer representing the cost of the plan. If the provider changes the `amount` field from an integer to a string, the consumer's contract test will fail, alerting the consumer to the breaking change.

In this example, a contract test would catch the breaking change early in the development process, before it reaches production. The consumer and provider can then work together to resolve the issue, ensuring that the API contract is honored.

## How to implement contract testing with OpenAPI

Let's walk through the process of implementing contract testing using [OpenAPI](/openapi/), focusing on both the consumer and provider perspectives.

### Step 1: Create a new project

We'll start by creating a new project for our consumer and provider code. We'll use a simple subscription management API as an example.

In the terminal, run:

```bash
mkdir contract-example
cd contract-example
```

Let's create a new TypeScript project for our SDK and tests:

In the terminal, run:

```bash
npm install --save-dev typescript
npx tsc --init
```

Select the default options when prompted.

### Step 2: Define the OpenAPI specification

We'll start by writing an OpenAPI specification for our API. In most cases, the provider will define the OpenAPI specification, as they are responsible for the implementation of the API.

Here's a basic example of an OpenAPI specification. Save it as `subscriptions.yaml` in the root of your project.

The document contains a `/plan/{id}` endpoint, which has a single `GET` operation that retrieves a subscription plan by ID. It expects an integer `id` parameter in the path.

We'll focus on the `200` response for now. The response should be a JSON object with the subscription plan details, as defined in the `Subscription` schema. The schema includes fields for `id`, `name`, `amount`, and `currency`.

```yaml filename="subscriptions.yaml"
openapi: 3.1.0
info:
  title: Subscription Management API
  version: 1.0.0
servers:
  - url: http://127.0.0.1:4010
    description: Local server
tags:
  - name: subscription
    description: Subscription management
security:
  - api_key: []
paths:
  /plan/{id}:
    get:
      operationId: getPlanById
      tags:
        - subscription
      summary: Get a subscription plan by ID
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: integer
          examples:
            basic:
              value: 1
            premium:
              value: 2
      responses:
        "200":
          description: Successful response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Subscription"
              examples:
                basic:
                  value:
                    id: 1
                    name: "Basic"
                    amount: 100
                    currency: "USD"
                premium:
                  value:
                    id: 2
                    name: "Premium"
                    amount: 200
                    currency: "USD"
components:
  schemas:
    Subscription:
      type: object
      properties:
        id:
          type: integer
        name:
          type: string
        amount:
          type: integer
        currency:
          type: string
      example:
        id: 1
        name: "Basic"
        amount: 100
        currency: "USD"
  securitySchemes:
    api_key:
      type: apiKey
      name: X-API-Key
      in: header
```

This example OpenAPI specification only defines the happy path for the `/plan/{id}` endpoint. In a real-world scenario, you would define additional paths, operations, and error responses to cover all possible scenarios.

### Step 3: Create an SDK with Speakeasy

We'll use Speakeasy to create a TypeScript SDK from the OpenAPI specification. If you don't have Speakeasy installed, you can install it from the [Introduction to Speakeasy](/docs/introduction/introduction#getting-started) guide.

With Speakeasy installed, run:

```bash
speakeasy quickstart
```

When prompted, select the `subscriptions.yaml` file and choose TypeScript as the target language. We decided on `Billing` as the SDK name, and `billing` as the package name.

### Step 4: Add tests

Now that we have an SDK, we can write tests to verify that the SDK handles the API responses correctly.

Let's install the necessary dependencies:

```bash
npm i --save-dev vitest
```

Create a new `tests` directory in the root of your project. Then, create a new file, `tests/subscription.test.ts`:

```typescript filename="tests/subscription.test.ts"
import { expect, test } from "vitest";
import { Billing } from "../billing-typescript/src/index.ts";

test("Subscription Get Plan By Id Basic", async () => {
  const billing = new Billing({
    apiKey: process.env["BILLING_API_KEY"] ?? "",
  });
  const result = await billing.subscription.getPlanById({
    id: 1,
  });
  expect(result).toBeDefined();
  expect(result).toEqual({
    id: 1,
    name: "Basic",
    amount: 100,
    currency: "USD",
  });
});
```

Add the following script to your `package.json`:

```json filename="package.json"
{
  "scripts": {
    "test": "vitest run"
  }
}
```

Now you can run the tests:

```bash
npm run test
```

This should run the test and verify that the SDK correctly handles the API responses, but since we haven't started a server yet, the test will fail.

### Step 5: Start a mock server

We'll use Prism to start a mock server that serves responses based on the OpenAPI specification.

Add Prism as a dev dependency:

```bash
npm install --save-dev @stoplight/prism-cli
```

Then, add a new script to your `package.json`:

```json filename="package.json"
{
  "scripts": {
    "test": "vitest run",
    "mock": "prism mock subscriptions.yaml"
  }
}
```

In a new terminal window, run:

```bash
npm run mock
```

This will start a mock server at `http://127.0.0.1:4010`.

### Step 6: Run the tests

Now that the mock server is running, you can run the tests again:

```bash
npm run test
```

This time, Prism returns a `401` status code because we haven't provided an API key. Let's run the test with the `BILLING_API_KEY` set to `test`:

```bash
export BILLING_API_KEY=test
npm run test
```

```txt
$ vitest run

 RUN  v2.1.1 /Users/speakeasy/contract-example

 ✓ tests/subscription.test.ts (1)
   ✓ Subscription Get Plan By Id Basic

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  07:28:38
   Duration  630ms (transform 191ms, setup 0ms, collect 199ms, tests 150ms, environment 0ms, prepare 86ms)
```

The test should now pass, verifying that the SDK correctly handles the API response.

### Step 7: Test for correctness

We've validated that the SDK can correctly handle an API response by interacting with a mock server, but we haven't confirmed whether the response conforms to the contract. To make this a true contract test, let's verify that both the consumer and provider behaviors align with the agreed-upon OpenAPI specification.

We'll add a contract-validation step to the test, then use Ajv, a JSON Schema validator, to validate the response against the OpenAPI schema.

```bash
npm install --save-dev ajv ajv-errors ajv-formats yaml
```

Create a new file, `validateSchema.ts`:

```typescript filename="validateSchema.ts"
import { readFileSync } from "fs";
import Ajv from "ajv";
import addErrors from "ajv-errors";
import addFormats from "ajv-formats";
import yaml from "yaml";

// Load and parse the OpenAPI specification
const openApiSpec = yaml.parse(
  readFileSync("./subscriptions.yaml", "utf8"),
) as any;

// Initialize Ajv with formats and error messages
const ajv = new Ajv({ allErrors: true, strict: false });
addFormats(ajv);
addErrors(ajv);

// Compile the schema for the Subscription response
const subscriptionSchema = {
  ...openApiSpec.components.schemas.Subscription,
};

const validate = ajv.compile(subscriptionSchema);

export const validateSubscription = (data: any) => {
  const isValid = validate(data);
  if (!isValid) {
    console.error(validate.errors);
    throw new Error("Validation failed");
  }
};
```

Update the test to include the contract-validation step:

```typescript filename="tests/subscription.test.ts" mark=3,19
import { expect, expectTypeOf, test } from "vitest";
import { Billing } from "../billing-typescript/src/index.ts";
import { validateSubscription } from "../validateSchema.ts";

test("Subscription Get Plan By Id Basic", async () => {
  const billing = new Billing({
    apiKey: process.env["BILLING_API_KEY"] ?? "",
  });
  const result = await billing.subscription.getPlanById({
    id: 1,
  });
  expect(result).toBeDefined();
  expect(result).toEqual({
    id: 1,
    name: "Basic",
    amount: 100,
    currency: "USD",
  });
  validateSubscription(result); // Contract validation
});
```

Now when you run the tests, the contract validation will ensure that the response from the mock server matches the OpenAPI specification.

```bash
npm run test
```

## Generating contract tests automatically with OpenAPI

Manually writing contract tests can be a time-consuming and error-prone process. If you're starting with an OpenAPI document as your contract, you may be able to automatically generate tests that conform to your contract.

By generating contract tests, you reduce the risk of human error, save significant development time, and ensure that tests are always kept up to date.

The biggest advantage of automated test generation is the assurance that your tests are based on the API specification. This means that all aspects of the API contract, from endpoint paths and methods to data types and constraints, are accurately represented in the generated tests.

A drawback of basing tests on OpenAPI documents is that the OpenAPI Specification does not currently have built-in support for test generation. Although examples of requests and responses allow test case generation, there are still challenges in linking request and response pairs to each other. These are problems we're working hard to overcome at Speakeasy.

## Speakeasy test generation

At Speakeasy, we enable developers to automatically test their APIs and SDKs by creating comprehensive test suites. Shipping automated tests as part of your SDKs will enable your team to make sure that the interfaces your users prefer, your SDKs, are always compatible with your API. We ensure your APIs and SDKs stick to the contract so that you can focus on shipping features and evolving your API with confidence.

The process of adding tests with Speakeasy is straightforward: Add detailed examples to your OpenAPI document, or describe tests in a simple and well-documented YAML specification that lives in your SDK project. Speakeasy will regenerate your tests when they need to change, and you can run the tests as part of development or CI/CD workflows.

## Speakeasy test generation roadmap

Looking ahead, Speakeasy's testing roadmap includes broader language support, advanced server mocking, ability to run contract tests on past versions of the API and SDK, and using the Arazzo specification to string together multiple contract tests. With these features, you'll be able to monitor the health of all your SDKs and APIs in one place.

We're also working on support for behavior-driven development (BDD) and end-to-end (E2E) testing by embracing OpenAPI and the recently published Arazzo specification for complex testing workflows.