# ...
Source: https://speakeasy.com/blog/e2e-testing-arazzo

import { Callout, Table } from "@/mdx/components";

We've previously written about [the importance of building contract & integration tests](/post/contract-testing-with-openapi) to comprehensively cover your API's endpoints, but there's still a missing piece to the puzzle. Real users don't consume your API one endpoint at a time - they implement complex workflows that chain multiple API calls together.

That's why reliable **end-to-end API tests** are an important component of the testing puzzle. For your APIs most common workflows, you need to ensure that the entire process works as expected, not just the individual parts.

In this tutorial, we'll build a test generator that turns Arazzo specifications into executable end-to-end tests. You'll learn how to:

- Generate tests that mirror real user interactions with your API
- Keep tests maintainable, even as your API evolves
- Validate complex workflows across multiple API calls
- Catch integration issues before they reach production

We'll use a simple "Build-a-bot" API as our example, but the principles and code you'll learn apply to any REST API.

## Arazzo? What & Why

Arazzo is a specification that describes how API calls should be sequenced to achieve specific outcomes. Think of it as OpenAPI for workflows - while OpenAPI describes what your API can do, Arazzo describes how to use it effectively.

Arazzo was designed to bridge the gap between API reference documentation and real-world usage patterns. Fortunately for us, it also makes a perfect fit for generating end-to-end test suites that validate complete user workflows rather than isolated endpoints.

By combining these specifications, we can generate tests that validate not just the correctness of individual endpoints, but the entire user journey.

<Callout title="Arazzo?" type="info">
  Arazzo roughly translates to "tapestry" in Italian. Get it? A tapestry of API
  calls "woven" together to create a complete user experience. We're still
  undecided about how to pronounce it, though. The leading candidates are
  "ah-RAT-so" (like fatso) and "ah-RAHT-zoh" (almost like pizza, but with a
  rat). There is a minor faction pushing for "ah-razzo" as in razzle-dazzle.
  We'll let you decide.
</Callout>

Let's look at a simplified (and mostly invalid) illustrative example. Imagine a typical e-commerce API workflow:

```yaml filename="arazzo.yaml"
arazzo: 1.0.0
workflowId: purchaseProduct
sourceDescriptions:
  - url: ./openapi.yaml
steps:
  - stepId: authenticate
    operationId: loginUser
    # post login details
    # response contains auth token
    # if successful, go to checkInventory
  - stepId: checkInventory
    operationId: getProductsStock
    # with auth token from previous step:
    # get stock levels of multiple products
    # response contains product IDs and stock levels
    # if stock levels are sufficient, go to createOrder
  - stepId: createOrder
    operationId: submitOrder
    # with auth token from first step
    # and product IDs and quantities from previous step
    # post an order that is valid based on stock levels
    # response contains order ID
    # if successful, go to getOrder

```

Arazzo allows us to define these workflows, and specify how each step should handle success and failure conditions, as well as how to pass data between steps and even between workflows.

## From specification to implementation

The example above illustrates the concept, but let's dive into a working implementation. We'll use a simplified but functional example that you can download and run yourself. Our demo implements a subset of the Arazzo specification, focusing on the most immediately valuable features for E2E testing.

We'll use the example of an API called Build-a-bot, which allows users to create and manage their own robots. You can substitute this with your own OpenAPI document, or use the Build-a-bot API to follow along.

If you want a deep dive on the Arazzo specification, check out the [Arazzo specification docs](/openapi/arazzo).

## Setting up the development environment

First, clone the demo repository:

```bash
git clone https://github.com/speakeasy-api/e2e-testing-arazzo.git
cd e2e-testing-arazzo
```

You'll need [Deno v2](https://docs.deno.com/runtime/) installed. On macOS and Linux, you can install Deno using the following command:

```bash
curl -fsSL https://deno.land/install.sh | sh
```

The repository contains:

- A simple API server built with [@oak/acorn](https://oakserver.org/acorn) that serves as the Build-a-bot API (in `packages/server/server.ts`)
- An Arazzo specification file (`arazzo.yaml`)
- An OpenAPI specification file (`openapi.yaml`)
- The test generator implementation (`packages/arazzo-test-gen/generator.ts`)
- Generated E2E tests (`tests/generated.test.ts`)
- An SDK created by Speakeasy to interact with the Build-a-bot API (`packages/sdk`)

### Running the Demo

To run the demo, start the API server:

```bash
deno task server
```

Deno will install the server's dependencies, then start the server on `http://localhost:8080`. You can test the server by visiting `http://localhost:8080/v1/robots`, which should return a `401 Unauthorized` error:

```json
{
  "status": 401,
  "error": "Unauthorized",
  "message": "Header x-api-key is required"
}
```

Next, in a new terminal window, generate the E2E tests:

```bash
deno task dev
```

After installing dependencies, this command will generate the E2E tests in `tests/generated.test.ts` and watch for changes to the Arazzo specification file.

You can run the tests in a new terminal window:

```bash
deno task test
```

This command will run the generated tests against the API server:

```txt
> deno task test
Task test deno test --allow-read --allow-net --allow-env --unstable tests/
running 1 test from ./tests/generated.test.ts
Create, assemble, and activate a new robot ...
  Create a new robot design session ... ok (134ms)
  Add parts to the robot ... ok (2ms)
  Assemble the robot ... ok (1ms)
  Configure robot features ... ok (2ms)
  Activate the robot ... ok (3ms)
  Get the robot details ... ok (1ms)
Create, assemble, and activate a new robot ... ok (143ms)

ok | 1 passed (6 steps) | 0 failed (147ms)
```

Beautiful, everything works! Let's see how we got here.

## Building an Arazzo test generator

Let's start with the overall structure of the test generator.

### Project structure

The test generator is a Deno project that consists of several modules, each with a specific responsibility:

- `generator.ts`: The main entry point that orchestrates the test generation process. It reads the Arazzo and OpenAPI specifications, validates their compatibility, and generates test cases.

- `readArazzoYaml.ts` and `readOpenApiYaml.ts`: Handle parsing and validation of the Arazzo and OpenAPI specifications respectively. They ensure the specifications are well-formed and contain all required fields.

- `expressionParser.ts`: A parser for runtime expressions like `$inputs.BUILD_A_BOT_API_KEY` and `$steps.createRobot.outputs.robotId`. These expressions are crucial for passing data between steps and accessing workflow inputs.

- `successCriteria.ts`: Processes the success criteria for each step, including status code validation, regex patterns, direct comparisons, and JSONPath expressions.

- `generateTestCase.ts`: Takes the parsed workflow and generates the actual test code, including setup, execution, and validation for each step.

- `security.ts`: Handles security-related aspects like API key authentication and other security schemes defined in the OpenAPI specification.

- `utils.ts`: Contains utility functions for common operations like JSON pointer resolution and type checking.

The project also includes a `runtime-expression` directory containing the grammar definition for runtime expressions:

- `runtimeExpression.peggy`: A Peggy grammar file that defines the syntax for runtime expressions
- `runtimeExpression.js`: The generated parser from the grammar
- `runtimeExpression.d.ts`: TypeScript type definitions for the parser

Let's dive deeper into each of these components to understand how they work together to generate effective E2E tests.

### Parsing until you parse out

While our project says "test generator" on the tin, the bulk of our work will go into parsing different formats. To generate tests from an Arazzo document, we need to parse:

1. The Arazzo document
2. The OpenAPI document
3. Conditions in the Arazzo success criteria
4. Runtime expressions in the success criteria, outputs, and parameters
5. Regular expressions in the success criteria
6. JSONPath expressions in the success criteria
7. JSON pointers in the runtime expressions

We won't cover all of these in detail, but we'll touch on each to get a sense of the complexity involved and the tools we use to manage it.

### Parsing the Arazzo specification

The first step in our test generator is parsing the Arazzo specification in `readArazzoYaml.ts`. This module reads the Arazzo YAML file and should ideally validate its structure against the Arazzo specification.

For our demo, we didn't implement full validation, instead parsing the YAML file into a JavaScript object. We then use TypeScript interfaces to define the expected structure of the Arazzo document:

```typescript
export interface ArazzoDocument {
  arazzo: string;
  info: ArazzoInfo;
  sourceDescriptions: Array<ArazzoSourceDescription>;
  workflows: Array<ArazzoWorkflow>;
  components: Record<string, unknown>;
}

export interface ArazzoWorkflow {
  workflowId: string;
  description: string;
  inputs: {
    type: string;
    properties: Record<string, { type: string; description: string }>;
  };
  steps: Array<ArazzoStep>;
}

export interface ArazzoStep {
  stepId: string;
  description: string;
  operationId: string;
  parameters?: Array<ArazzoParameter>;
  requestBody?: ArazzoRequestBody;
  successCriteria: Array<ArazzoSuccessCriterion>;
  outputs?: Record<string, string>;
}
```

These TypeScript interfaces help with autocompletion, type checking, and documentation, making it easier to work with the parsed Arazzo document in the rest of our code.

The real complexity comes in validating that the parsed document follows all the rules in the Arazzo specification. For example:

- Each `workflowId` must be unique within the document
- Each `stepId` must be unique within its workflow
- An `operationId` must reference a valid operation in the OpenAPI document
- Runtime expressions must follow the correct syntax
- Success criteria must use valid JSONPath or regex patterns

We don't validate all these rules in our demo, but in production, we'd use [Zod](https://zod.dev/) or [Valibot](https://github.com/fabian-hiller/valibot) to enforce these constraints at runtime and provide helpful error messages when the document is invalid.

The OpenAPI team hasn't finalized the Arazzo specification's JSON Schema yet, but once they do, we can use it to validate the Arazzo document against the schema with tools like [Ajv](https://ajv.js.org/).

Speakeasy also provides a [command-line interface](/docs/speakeasy-cli/getting-started) for linting Arazzo documents:

```bash