# Custom end-to-end API contract tests with Arazzo
Source: https://speakeasy.com/docs/sdks/sdk-contract-testing/custom-contract-tests

import { Callout, CodeWithTabs } from "@/mdx/components";

<Callout title="Beta" type="warning">

We highly recommend fully setting up [SDK tests](/docs/sdk-contract-testing) in all your SDK repositories before exploring custom contract tests.

</Callout>



You can use Speakeasy to create custom end-to-end contract tests that run against a real API.

This document guides you through writing more complex tests using the Arazzo Specification, as well as through the key configuration features for these tests, which include:

- Server URLs
- Security credentials
- Environment variable-provided values

[Arazzo](/openapi/arazzo) is a simple, human-readable, and extensible specification for defining API workflows. Arazzo powers custom test generation, allowing you to define rich tests capable of:

- Testing multiple operations
- Testing different inputs
- Validating that the correct response is returned
- Running against a real API or mock server
- Configuring setup and teardown routines for complex end-to-end (E2E) tests

The Arazzo Specification allows you to define sequences of API operations and their dependencies for contract testing, enabling you to validate whether your API behaves correctly across multiple interconnected endpoints and complex workflows.

When a `.speakeasy/tests.arazzo.yaml` file is found in your SDK repo, the Arazzo workflow is used to generate tests for each of the workflows defined in the file.

## Prerequisites

Before generating tests, ensure that you meet the [testing feature prerequisites](/docs/sdk-contract-testing#prerequisites).

## Writing custom end-to-end tests

The following is an example Arazzo document defining a simple E2E test for the lifecycle of a user resource in the example API:

```yaml
arazzo: 1.0.0
info:
  title: Test Suite
  summary: E2E tests for the SDK and API.
  version: 0.0.1
sourceDescriptions:
  - name: The API
    url: https://example.com/openapi.yaml
    type: openapi
workflows:
  - workflowId: user-lifecycle
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload:
            {
              "email": "Trystan_Crooks@hotmail.com",
              "first_name": "Trystan",
              "last_name": "Crooks",
              "age": 32,
              "postal_code": 94110,
              "metadata":
                {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true,
                },
            }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/postal_code == 94110
        outputs:
          id: $response.body#/id
      - stepId: get
        operationId: getUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94110
        outputs:
          user: $response.body
          age: $response.body#/age
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        requestBody:
          contentType: application/json
          payload: $steps.get.outputs.user
          replacements:
            - target: /postal_code
              value: 94107
            - target: /age
              value: $steps.get.outputs.age
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94107
        outputs:
          email: $response.body#/email
          first_name: $response.body#/first_name
          last_name: $response.body#/last_name
          metadata: $response.body#/metadata
      - stepId: updateAgain
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        requestBody:
          contentType: application/json
          payload:
            {
              "id": "$steps.create.outputs.id",
              "email": "$steps.update.email",
              "first_name": "$steps.update.first_name",
              "last_name": "$steps.update.last_name",
              "age": 33,
              "postal_code": 94110,
              "metadata": "$steps.update.metadata",
            }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == "application/json"
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 33
          - condition: $response.body#/postal_code == 94110
      - stepId: delete
        operationId: deleteUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200
```

This workflow defines four steps, each of which feeds into the next:

1. Create a user
2. Retrieve that user via its new ID
3. Update the user
4. Delete the user

This is possible because the workflow defines outputs for certain steps that serve as inputs for the following steps.

The workflow generates the test shown below:

<CodeWithTabs
  tabs={[
    {
      label: "TypeScript",
      language: "typescript",
      code: `// src/__tests__/sdk.test.ts

  import { assert, expect, it, test } from "vitest";
  import { SDK } from "../index.js";
  import { assertDefined } from "./assertions.js";
  import { createTestHTTPClient } from "./testclient.js";

  test("Sdk User Lifecycle", async () => {
    const sdk = new SDK({
      serverURL: process.env["TEST_SERVER_URL"] ?? "http://localhost:18080",
      httpClient: createTestHTTPClient("user-lifecycle"),
    });

    const createResult = await sdk.createUser({
      email: "Trystan_Crooks@hotmail.com",
      firstName: "Trystan",
      lastName: "Crooks",
      age: 32,
      postalCode: "94110",
      metadata: {
        allergies: "none",
        additionalProperties: {
          color: "red",
          height: "182",
          weight: "77",
          is_smoking: "true",
        },
      },
    });
    expect(createResult.httpMeta.response.status).toBe(200);
    expect(createResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
    expect(createResult.user?.postalCode).toBeDefined();
    expect(createResult.user?.postalCode).toEqual("94110");

    const getResult = await sdk.getUser(assertDefined(createResult.user?.id));
    expect(getResult.httpMeta.response.status).toBe(200);
    expect(getResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
    expect(getResult.user?.firstName).toBeDefined();
    expect(getResult.user?.firstName).toEqual("Trystan");
    expect(getResult.user?.lastName).toBeDefined();
    expect(getResult.user?.lastName).toEqual("Crooks");
    expect(getResult.user?.age).toBeDefined();
    expect(getResult.user?.age).toEqual(32);
    expect(getResult.user?.postalCode).toBeDefined();
    expect(getResult.user?.postalCode).toEqual("94110");

    const user = assertDefined(getResult.user);
    user.postalCode = "94107";
    user.age = getResult.user?.age;
    const updateResult = await sdk.updateUser(
      assertDefined(createResult.user?.id),
      assertDefined(getResult.user),
    );
    expect(updateResult.httpMeta.response.status).toBe(200);
    expect(updateResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
    expect(updateResult.user?.firstName).toBeDefined();
    expect(updateResult.user?.firstName).toEqual("Trystan");
    expect(updateResult.user?.lastName).toBeDefined();
    expect(updateResult.user?.lastName).toEqual("Crooks");
    expect(updateResult.user?.age).toBeDefined();
    expect(updateResult.user?.age).toEqual(32);
    expect(updateResult.user?.postalCode).toBeDefined();
    expect(updateResult.user?.postalCode).toEqual("94107");

    const updateAgainResult = await sdk.updateUser(
      assertDefined(createResult.user?.id),
      {
        id: assertDefined(createResult.user?.id),
        email: assertDefined(updateResult.user?.email),
        firstName: updateResult.user?.firstName,
        lastName: updateResult.user?.lastName,
        age: 33,
        postalCode: "94110",
        metadata: updateResult.user?.metadata,
      },
    );
    expect(updateAgainResult.httpMeta.response.status).toBe(200);
    expect(updateAgainResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
    expect(updateAgainResult.user?.firstName).toBeDefined();
    expect(updateAgainResult.user?.firstName).toEqual("Trystan");
    expect(updateAgainResult.user?.lastName).toBeDefined();
    expect(updateAgainResult.user?.lastName).toEqual("Crooks");
    expect(updateAgainResult.user?.age).toBeDefined();
    expect(updateAgainResult.user?.age).toEqual(33);
    expect(updateAgainResult.user?.postalCode).toBeDefined();
    expect(updateAgainResult.user?.postalCode).toEqual("94110");

    const deleteResult = await sdk.deleteUser(
      assertDefined(createResult.user?.id),
    );
    expect(deleteResult.httpMeta.response.status).toBe(200);
  });`,
    },
    {
      label: "Python",
      language: "python",
      code: `# tests/test_sdk.py

  import io
  import openapi
  from openapi import SDK
  import os
  import pytest
  from tests.test_client import create_test_http_client


  def test_sdk_user_lifecycle():
      with SDK(
          server_url=os.getenv("TEST_SERVER_URL", "http://localhost:18080"),
          client=create_test_http_client("user-lifecycle"),
      ) as sdk:
          assert sdk is not None

          create_res = sdk.create_user(
              request=openapi.BaseUser(
                  email="Trystan_Crooks@hotmail.com",
                  first_name="Trystan",
                  last_name="Crooks",
                  age=32,
                  postal_code="94110",
                  metadata=openapi.Metadata(
                      allergies="none",
                      **{
                          "color": "red",
                          "height": "182",
                          "weight": "77",
                          "is_smoking": "true",
                      },
                  ),
              )
          )
          assert create_res is not None
          assert create_res.email == "Trystan_Crooks@hotmail.com"
          assert create_res.postal_code is not None
          assert create_res.postal_code == "94110"

          get_res = sdk.get_user(id=create_res.id)
          assert get_res is not None
          assert get_res.email == "Trystan_Crooks@hotmail.com"
          assert get_res.first_name is not None
          assert get_res.first_name == "Trystan"
          assert get_res.last_name is not None
          assert get_res.last_name == "Crooks"
          assert get_res.age is not None
          assert get_res.age == 32
          assert get_res.postal_code is not None
          assert get_res.postal_code == "94110"

          user = get_res
          user.postal_code = "94107"
          user.age = get_res.age

          update_res = sdk.update_user(id=create_res.id, user=user)
          assert update_res is not None
          assert update_res.email == "Trystan_Crooks@hotmail.com"
          assert update_res.first_name is not None
          assert update_res.first_name == "Trystan"
          assert update_res.last_name is not None
          assert update_res.last_name == "Crooks"
          assert update_res.age is not None
          assert update_res.age == 32
          assert update_res.postal_code is not None
          assert update_res.postal_code == "94107"

          update_again_res = sdk.update_user(
              id=create_res.id,
              user=openapi.User(
                  id=create_res.id,
                  email=update_res.email,
                  first_name=update_res.first_name,
                  last_name=update_res.last_name,
                  age=33,
                  postal_code="94110",
                  metadata=update_res.metadata,
              ),
          )
          assert update_again_res is not None
          assert update_again_res.email == "Trystan_Crooks@hotmail.com"
          assert update_again_res.first_name is not None
          assert update_again_res.first_name == "Trystan"
          assert update_again_res.last_name is not None
          assert update_again_res.last_name == "Crooks"
          assert update_again_res.age is not None
          assert update_again_res.age == 33
          assert update_again_res.postal_code is not None
          assert update_again_res.postal_code == "94110"

          sdk.delete_user(id=create_res.id)`,
    },
    {
      label: "Go",
      language: "go",
      code: `// tests/sdk_test.go

  package sdk_test

  import (
    "context"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"
    "openapi"
    "openapi/internal/utils"
    "os"
    "testing"
  )

  func TestSDK_UserLifecycle(t *testing.T) {
    ctx := context.Background()

    s := openapi.New(
      openapi.WithServerURL(utils.GetEnv("TEST_SERVER_URL", "http://localhost:18080")),
      openapi.WithClient(createTestHTTPClient("user-lifecycle")),
    )

    createRes, err := s.CreateUser(ctx, openapi.BaseUser{
      Email:      "Trystan_Crooks@hotmail.com",
      FirstName:  openapi.String("Trystan"),
      LastName:   openapi.String("Crooks"),
      Age:        openapi.Float64(32),
      PostalCode: openapi.String("94110"),
      Metadata: &openapi.Metadata{
        Allergies: openapi.String("none"),
        AdditionalProperties: map[string]string{
          "color":      "red",
          "height":     "182",
          "weight":     "77",
          "is_smoking": "true",
        },
      },
    })
    require.NoError(t, err)
    assert.Equal(t, 200, createRes.HTTPMeta.Response.StatusCode)
    assert.Equal(t, "Trystan_Crooks@hotmail.com", createRes.User.Email)
    assert.NotNil(t, createRes.User.PostalCode)
    assert.Equal(t, openapi.String("94110"), createRes.User.PostalCode)

    getRes, err := s.GetUser(ctx, createRes.User.ID)
    require.NoError(t, err)
    assert.Equal(t, 200, getRes.HTTPMeta.Response.StatusCode)
    assert.Equal(t, "Trystan_Crooks@hotmail.com", getRes.User.Email)
    assert.NotNil(t, getRes.User.FirstName)
    assert.Equal(t, openapi.String("Trystan"), getRes.User.FirstName)
    assert.NotNil(t, getRes.User.LastName)
    assert.Equal(t, openapi.String("Crooks"), getRes.User.LastName)
    assert.NotNil(t, getRes.User.Age)
    assert.Equal(t, openapi.Float64(32), getRes.User.Age)
    assert.NotNil(t, getRes.User.PostalCode)
    assert.Equal(t, openapi.String("94110"), getRes.User.PostalCode)

    user := *getRes.User
    user.PostalCode = openapi.String("94107")
    user.Age = getRes.User.Age

    updateRes, err := s.UpdateUser(ctx, createRes.User.ID, user)
    require.NoError(t, err)
    assert.Equal(t, 200, updateRes.HTTPMeta.Response.StatusCode)
    assert.Equal(t, "Trystan_Crooks@hotmail.com", updateRes.User.Email)
    assert.NotNil(t, updateRes.User.FirstName)
    assert.Equal(t, openapi.String("Trystan"), updateRes.User.FirstName)
    assert.NotNil(t, updateRes.User.LastName)
    assert.Equal(t, openapi.String("Crooks"), updateRes.User.LastName)
    assert.NotNil(t, updateRes.User.Age)
    assert.Equal(t, openapi.Float64(32), updateRes.User.Age)
    assert.NotNil(t, updateRes.User.PostalCode)
    assert.Equal(t, openapi.String("94107"), updateRes.User.PostalCode)

    updateAgainRes, err := s.UpdateUser(ctx, createRes.User.ID, openapi.User{
      ID:         createRes.User.ID,
      Email:      updateRes.User.Email,
      FirstName:  updateRes.User.FirstName,
      LastName:   updateRes.User.LastName,
      Age:        openapi.Float64(33),
      PostalCode: openapi.String("94110"),
      Metadata:   updateRes.User.Metadata,
    })
    require.NoError(t, err)
    assert.Equal(t, 200, updateAgainRes.HTTPMeta.Response.StatusCode)
    assert.Equal(t, "Trystan_Crooks@hotmail.com", updateAgainRes.User.Email)
    assert.NotNil(t, updateAgainRes.User.FirstName)
    assert.Equal(t, openapi.String("Trystan"), updateAgainRes.User.FirstName)
    assert.NotNil(t, updateAgainRes.User.LastName)
    assert.Equal(t, openapi.String("Crooks"), updateAgainRes.User.LastName)
    assert.NotNil(t, updateAgainRes.User.Age)
    assert.Equal(t, openapi.Float64(33), updateAgainRes.User.Age)
    assert.NotNil(t, updateAgainRes.User.PostalCode)
    assert.Equal(t, openapi.String("94110"), updateAgainRes.User.PostalCode)

    deleteRes, err := s.DeleteUser(ctx, createRes.User.ID)
    require.NoError(t, err)
    assert.Equal(t, 200, deleteRes.HTTPMeta.Response.StatusCode)
  }`,
}
]}
/>

## Input and outputs

### Inputs

There are various ways to provide an input for a step, including:

- Defining it in the workflow
- Referencing it in a previous step
- Including it as an inline value

**Workflow inputs**

You can provide input parameters to the workflow using the `inputs` field, which is a JSON Schema object that defines a property for each input that the workflow wants to expose. These [workflow inputs](/openapi/arazzo#workflow-object) can be used by any step defined in the workflow.

Test generation can use any of the examples defined for a property in an `inputs` JSON schema as a literal value, which it then uses as an input for the test. Because tests are non-interactive and cannot ask users for input, the test generation randomly generates values for the inputs if no examples are defined.

```yaml
arazzo: 1.0.0