# Expects arazzo.yaml in the current directory
speakeasy lint arazzo
```

### Parsing the OpenAPI specification

The OpenAPI specification's path is gathered from the Arazzo document. In our test, we simply use the first `sourceDescription` to find the OpenAPI document path. But in a production generator, we'd need to handle multiple `sourceDescriptions` and ensure the OpenAPI document is accessible.

We parse the OpenAPI document in `readOpenApiYaml.ts` and use TypeScript interfaces from the [`npm:openapi-types`](https://www.npmjs.com/package/openapi-types) package to define the expected structure of the OpenAPI document.

We won't cover the OpenAPI parsing in detail, but it's similar to the Arazzo parsing: Read the YAML file, parse it into a JavaScript object, and type check it against TypeScript interfaces.

For OpenAPI, writing a custom validator is more complex due to the specification's size and complexity. We recommend validating against the [official OpenAPI 3.1.1 JSON Schema](https://spec.openapis.org/oas/v3.1.1.html) using [Ajv](https://ajv.js.org/), or Speakeasy's own OpenAPI linter:

```bash
speakeasy lint openapi --schema openapi.yaml
```

### Parsing success criteria

This is where things get interesting. Success criteria in Arazzo are a list of conditions that must be met for a step to be considered successful. Each criterion can be one of the following types:

- `simple`: Selects a value with a runtime expression and compares it to an expected value
- `jsonpath`: Selects a value using a JSONPath expression and compares it to an expected value
- `regex`: Validates a value against a regular expression pattern
- `xpath`: Selects a value using an XPath expression and compares it to an expected value, used for XML documents

In our demo, we don't implement the `xpath` type, but we do cover the other three. Here's an example of a success criterion in the Arazzo document:

```yaml
successCriteria:
  - condition: $statusCode == 201
  - condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i
    context: $response.body#/robotId
    type: regex
  - condition: $response.body#/model == "humanoid"
  - condition: $response.body#/name == "MyFirstRobot"
  - condition: $response.body#/status == "designing"
  - context: $response.body#/links
    condition: $.length == 5
    type: jsonpath
```

The `condition` field is required, and contains the expression to evaluate, while the `context` field specifies the part of the response to evaluate. The `type` field indicates the type of validation to perform.

If no `type` is specified, the success criterion is treated as a `simple` comparison, where the `condition` is evaluated directly.

### Evaluating simple criteria

Here's an example of how we parse a `simple` success criterion:

```yaml
condition: $statusCode == 201
```

We split this simple condition into:

- Left-hand side: `$statusCode` - Runtime expression to evaluate
- Operator: `==` - Comparison operator or assertion
- Right-hand side: `201` - Expected value

We'll evaluate the runtime expression `$statusCode` and compare it to the expected value `201`. If the comparison is true, the criterion passes; otherwise, it fails.

Runtime expressions can also reference other variables, like `$inputs.BUILD_A_BOT_API_KEY` or `$steps.createRobot.outputs.robotId`, or fields in the response body, like `$response.body#/model`.

We'll cover runtime expressions in more detail later.

### Evaluating JSONPath criteria

For JSONPath criteria, we use the `jsonpath` type and a JSONPath expression to select a value from the response:

```yaml
context: $response.body#/links
condition: $.length == 5
type: jsonpath
```

Let's break down the JSONPath criterion:

- `context`: `$response.body#/links` - Runtime expression to select the `links` array from the response body
- `condition`: `$.length == 5` - JSONPath expression compared to an expected value
- `type`: `jsonpath` - Indicates the criterion type

We further need to break down the condition into:

- Left-hand side: `$.length` - JSONPath expression to evaluate
- Operator: `==` - Comparison operator
- Right-hand side: `5` - Expected value

We evaluate the JSONPath expression `$.length` and compare it to the expected value `5`. If the comparison is true, the criterion passes.

### Evaluating regex criteria

For regex criteria, we use the `regex` type and a regular expression pattern to validate a value:

```yaml
context: $response.body#/robotId
condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i
type: regex
```

Let's break down the regex criterion:

- `context`: `$response.body#/robotId` - Runtime expression to select the `robotId` field from the response body
- `condition`: `/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i` - Regular expression pattern to validate the value as a UUID v4
- `type`: `regex` - Indicates the criterion type

We evaluate the runtime expression `$response.body#/robotId` against the regular expression pattern. If the value matches the pattern, the criterion passes.

In our implementation, we use TypeScript's `factory.createRegularExpressionLiteral` to create a regular expression literal from the pattern string. This ensures that the pattern is properly escaped and formatted as a valid JavaScript regular expression.

The generated test code looks something like this:

```typescript
assertMatch(
  response.body.robotId,
  new RegExp(/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i),
  'robotId should be a valid UUID v4'
);
```

This code uses Deno's built-in `assertMatch` function to validate that the `robotId` matches the UUID v4 pattern. If the value doesn't match, the test fails with a helpful error message.

### Parsing runtime expressions

Runtime expressions are used throughout the Arazzo specification to reference variables, fields in the response body, or outputs from previous steps. They follow a specific syntax defined in the Arazzo specification as an ABNF (augmented Backus–Naur form) grammar.

To parse runtime expressions, we use a parser generated from the ABNF grammar. In our demo, this is a two-step process. First, we use the [abnf](https://www.npmjs.com/package/abnf) npm package to generate a Peggy grammar file from the ABNF grammar:

```bash
cd packages/arazzo-test-gen
abnf_gen runtime-expression/runtimeExpression.abnf
```

This generates a `runtime-expression/runtimeExpression.peggy` file that defines the syntax for runtime expressions. We then use the [peggy](https://www.npmjs.com/package/peggy) npm package to generate a parser from the Peggy grammar:

```bash
cd packages/arazzo-test-gen
peggy --dts --output runtime-expression/runtimeExpression.js --format es runtime-expression/runtimeExpression.peggy
```

This generates a `runtime-expression/runtimeExpression.js` file that contains the parser for runtime expressions. We also generate TypeScript type definitions in `runtime-expression/runtimeExpression.d.ts`.

The parser reads a runtime expression like `$response.body#/robotId` and breaks it down into tokens. We then evaluate the tokens to resolve the expression at runtime.

### Evaluating runtime expressions

Once we've parsed a runtime expression, we need to evaluate it to get the value it references. For example, given the expression `$response.body#/robotId`, we need to extract the `robotId` field from the response body.

The `evaluateRuntimeExpression` function in `utils.ts` handles this evaluation. Here's an example of how it works:

```typescript
switch (root) {
  case "$statusCode": {
    // Handle $statusCode expressions
    result = factory.createPropertyAccessExpression(
      factory.createIdentifier("response"),
      factory.createIdentifier("status"),
    );
    break;
  }
  case "$response.": {
    // Handle $request and $response expressions
    const data = factory.createIdentifier("data");
    // use parseRef to handle everything after $response.body
    const pointer = parsePointer(expression.slice(15));
    result = pointer.length > 0
      ? factory.createPropertyAccessExpression(data, pointer.join("."))
      : data;
    break;
  }
  // Handle other cases ...
}
```

Here, we handle two types of runtime expressions: `$statusCode` and `$response.body`. We extract the `status` field from the `response` object for `$statusCode`, and the `body` object from the `response` object for `$response.body`.

We use the TypeScript compiler API to generate an abstract syntax tree (AST) that represents the expression. This AST is then printed to a string and saved as a source file that Deno can execute.

### Supported runtime expressions

In our demo, we support a limited set of runtime expressions:

- `$statusCode`: The HTTP status code of the response
- `$steps.stepId.outputs.field`: The output of a previous step
- `$response.body#/path/to/field`: A field in the response body selected by a JSON pointer

Arazzo supports many more runtime expressions, for example:

<Table
  title="Expression Reference"
  columns={[
    { key: "expression", header: "Expression" },
    { key: "description", header: "Description" }
  ]}
  data={[
    { expression: "`$url`", description: "The full URL of the request" },
    { expression: "`$method`", description: "The HTTP method of the request" },
    { expression: "`$statusCode`", description: "The HTTP status code of the response" },
    { expression: "`$request.header.{name}`", description: "The value of the specified request header" },
    { expression: "`$request.query.{name}`", description: "The value of the specified query parameter from the request URL" },
    { expression: "`$request.path.{name}`", description: "The value of the specified path parameter from the request URL" },
    { expression: "`$request.body`", description: "The entire request body" },
    { expression: "`$request.body#/path/to/property`", description: "The value of the specified JSON pointer path from the request body" },
    { expression: "`$response.header.{name}`", description: "The value of the specified response header" },
    { expression: "`$response.body`", description: "The entire response body" },
    { expression: "`$response.body#/path/to/property`", description: "The value of the specified JSON pointer path from the response body" },
    { expression: "`$inputs.{name}`", description: "The value of the specified workflow input" },
    { expression: "`$outputs.{name}`", description: "The value of the specified workflow output" },
    { expression: "`$steps.{stepId}.{outputName}`", description: "The value of the specified output from the step with ID `{stepId}`" },
    { expression: "`$workflows.{id}.{inputName}`", description: "The value of the specified input from the workflow with ID `{id}`" },
    { expression: "`$workflows.{id}.{outputName}`", description: "The value of the specified output from the workflow with ID `{id}`" }
  ]}
  rowKeyField="expression"
/>

### Parsing regular expressions

Regular expressions in Arazzo are used to validate string values against patterns. They're particularly useful for validating IDs, dates, and other structured strings.

In our implementation, we handle regex patterns in the `parseRegexCondition` function:

```typescript
function parseRegexCondition(
  condition: string,
  usedAssertions: Set<string>,
  context: string,
): Expression {
  usedAssertions.add("assertMatch");
  return factory.createCallExpression(
    factory.createIdentifier("assertMatch"),
    undefined,
    [
      evaluateRuntimeExpression(context),
      factory.createNewExpression(
        factory.createIdentifier("RegExp"),
        undefined,
        [factory.createRegularExpressionLiteral(condition)],
      ),
      factory.createStringLiteral(condition),
    ],
  );
}
```

This function takes three parameters:

- `condition`: The regex pattern to match against
- `usedAssertions`: A set to track which assertion functions we've used
- `context`: The runtime expression that selects the value to validate

The function generates code that:

1. Evaluates the context expression to get the value to validate
2. Creates a new RegExp object from the pattern
3. Uses Deno's `assertMatch` function to validate the value against the pattern

The generated code looks like this:

```typescript
assertMatch(
  response.body.robotId,
  new RegExp(/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i),
  'robotId should be a valid UUID v4'
);
```

This approach has several advantages:

- It preserves the original pattern's flags (like `i` for case-insensitive matching).
- It provides clear error messages when validation fails.
- It integrates well with Deno's testing framework.

In a production implementation, we'd want to add:

- Validation of the regex pattern syntax
- Support for named capture groups
- Error handling for malformed patterns
- Performance optimizations like pattern caching

But for our demo, this simple implementation is sufficient to show how regex validation works in Arazzo.

### Parsing JSONPath expressions

JSONPath expressions are a powerful way to query JSON data. In Arazzo, we use them in success criteria to select objects or values from complex response structures. While JSON Pointer (which we'll cover next) is great for accessing specific values, JSONPath shines when you need to:

- Validate arrays (for example, checking array length)
- Filter elements (for example, finding items matching a condition)
- Access deeply nested data with wildcards
- Aggregate values (for example, counting matches)

Here's how our test generator handles JSONPath expressions:

```typescript
function parseJsonPathExpression(path: string, context: string): Expression {
  return factory.createCallExpression(
    factory.createIdentifier("JSONPath"),
    undefined,
    [
      factory.createObjectLiteralExpression(
        [
          factory.createPropertyAssignment(
            factory.createIdentifier("wrap"),
            factory.createFalse(),
          ),
          factory.createPropertyAssignment(
            factory.createIdentifier("path"),
            factory.createStringLiteral(path),
          ),
          factory.createPropertyAssignment(
            factory.createIdentifier("json"),
            evaluateRuntimeExpression(context),
          ),
        ],
        true,
      ),
    ],
  );
}
```

This function generates code that evaluates a JSONPath expression against a context object (usually the response body). For example, given this success criterion:

```yaml
successCriteria:
  - context: $response.body#/links
    condition: $.length == 5
    type: jsonpath
```

Our generator creates a test that:

1. Extracts the `links` array from the response body using a JSON pointer
2. Evaluates the JSONPath expression `$.length` against this array
3. Compares the result to the expected value `5`

The generated test code looks something like this:

```typescript
assertEquals(
  JSONPath({
    wrap: false,
    path: "$.length",
    json: response.body.links
  }),
  5,
  "links array should contain exactly 5 elements"
);
```

JSONPath is particularly useful for validating:

- Array operations: `$.length`, `$[0]`, `$[(@.length-1)]`
- Deep traversal: `$..name` (all name properties at any depth)
- Filtering: `$[?(@.status=="active")]` (elements where status is `active`)
- Wildcards: `$.*.name` (name property of all immediate children)

A few things to keep in mind when using JSONPath:

1. JSONPath isn't well standardized, so different implementations vary widely. Arazzo makes provisions for this by allowing us to specify the JSONPath version in the test specification.
2. Even though we can specify a version, we still need to be cautious when using advanced features. Some features might not be supported by the chosen JSONPath library.
3. Check the [JSONPath comparison](https://cburgmer.github.io/json-path-comparison/) page to see how different libraries handle various features, and decide which features are safe to use.

### Parsing JSON Pointers

While JSONPath is great for complex queries, [JSON Pointer (RFC 6901)](https://datatracker.ietf.org/doc/html/rfc6901) is perfect for directly accessing specific values in a JSON document. In Arazzo, we use JSON Pointers in runtime expressions to extract values from responses and pass them to subsequent steps.

Here's how our test generator handles JSON Pointers:

```typescript
function evaluateRuntimeExpression(expression: string): Expression {
  // ...
  case "$response.": {
    const data = factory.createIdentifier("data");
    // Parse everything after $response.body
    const pointer = parsePointer(expression.slice(15));
    result = pointer.length > 0
      ? factory.createPropertyAccessExpression(data, pointer.join("."))
      : data;
    break;
  }
  // ...
}
```

This function parses runtime expressions that use JSON Pointers. For example, given this output definition:

```yaml
outputs:
  robotId: $response.body#/robotId
```

Our generator creates code that:

1. Takes the part after `#` as the JSON Pointer (`/robotId`)
2. Converts the pointer segments into property access expressions
3. Generates code to extract the value

The generated test code looks something like this:

```typescript
// During test setup
const context = {};
// ...
// In the first test
const data = response.json();
// Generated because of outputs: { robotId: $response.body#/robotId } in the Arazzo document
// highlight-next-line
context["createRobot.outputs.robotId"] = data.robotId;
// ...
// In a subsequent test
const robotId = context["createRobot.outputs.robotId"];
const response = await fetch(`${serverUrl}/v1/robots/${robotId}/assemble`, {
  // ...
});
```

## Generating end-to-end tests

Now that we understand how to parse Arazzo documents, let's look at how we generate executable tests from them. Our generator creates type-safe test code using TypeScript's factory methods rather than string templates, providing better error detection and maintainability.

### Test structure

The generator creates a test suite for each workflow in the Arazzo document. Each step in the workflow becomes a test case that executes sequentially.

Let's explore the structure of a generated test case.

We start by setting up a test suite for the workflow, using the Arazzo workflow `description` as the suite name.

```typescript filename="generated.test.ts" focus=9
!from ./generated.test.ts.txt
```

Next we define the `serverUrl`, `apiKey`, and `context` variables. The `serverUrl` points to the API server. We use the `servers` list in the OpenAPI document to determine the server URL.

We also set up the `apiKey` for authentication. In our demo, we use a hardcoded API key, but in a real-world scenario, we'd likely get this after authenticating with the API.

We'll use the `context` object to store values extracted from the response body for use in subsequent steps.

```typescript filename="generated.test.ts" focus=10:12
!from ./generated.test.ts.txt
```

For each step in the workflow, we generate a test case that executes the step and validates the success criteria.

Our first step is to create a new robot design session.

```typescript filename="generated.test.ts" focus=13
!from ./generated.test.ts.txt
```

The HTTP method and path are extracted from the OpenAPI document using the `operationId` from the Arazzo step.

```typescript filename="generated.test.ts" focus=14:15
!from ./generated.test.ts.txt
```

We set up the request headers, including the `x-api-key` header for authentication.

```typescript filename="generated.test.ts" focus=16:19
!from ./generated.test.ts.txt
```

The request body is set up using the `requestBody` object from the Arazzo step.

```typescript filename="generated.test.ts" focus=20
!from ./generated.test.ts.txt
```

We extract the response body as JSON.

```typescript filename="generated.test.ts" focus=22
!from ./generated.test.ts.txt
```

We assert the success criteria for the step.

```typescript filename="generated.test.ts" focus=23:50
!from ./generated.test.ts.txt
```

Finally, we extract the outputs from the step and store them in the `context` object for use in subsequent steps.

```typescript filename="generated.test.ts" focus=51
!from ./generated.test.ts.txt
```

This structure repeats for each step in the workflow, creating a series of test cases that execute the workflow sequentially. The generated tests validate the API's behavior at each step, ensuring that the workflow progresses correctly.

## Future development and improvements

Our generated tests are a good start, but they might not be truly end-to-end if we don't consider the interfaces our users interact with to access the API.

### Testing with SDKs

In our demo, we use the `fetch` API to interact with the Build-a-bot API. While this is a common approach, it's not always the most user-friendly. Developers often prefer SDKs that provide a more idiomatic interface to the API.

To make our tests more end-to-end, we could use the SDK Speakeasy created from the OpenAPI document to interact with the API.

Since the SDK is generated from the OpenAPI document, with names and methods derived from the API's tags and operation IDs, we could use Arazzo to validate the SDK's behavior against the API's capabilities.

For example, we could:

1. Get the `operationId` from the Arazzo step and derive the corresponding SDK method import.
2. Call the SDK method with the required parameters.
3. Validate the response against the success criteria.
4. Extract the outputs from the response and store them in the `context` object.
5. Repeat for each step in the workflow.

This approach would provide a more realistic end-to-end test, validating the SDK's behavior against the API's capabilities.

### Handling authentication

In our demo, we use a hard-coded API key for authentication. In a real-world scenario, we'd likely need to authenticate with the API to get a valid API key.

OpenAPI also supports more advanced authentication schemes like OAuth 2.0, JWT, and API key in headers, query parameters, or cookies. Our test generator should handle these schemes to ensure the tests are realistic and cover all authentication scenarios.

Arazzo can point to the security schemes in the OpenAPI document, allowing us to extract the required authentication parameters and set them up in the test suite.

### Hardening the parsers against vulnerabilities

Our parsers are simple and work well for the demo, but they lack robust error handling and edge case coverage.

For example, JSONPath-plus, the library we use for JSONPath, recently fixed a remote code execution vulnerability. We should ensure our parser is up to date and secure against similar vulnerabilities, or limit the JSONPath features we support to reduce the attack surface.

This applies to parsers in general, and the risk is even higher when parsing user input and generating code from it.

Deno provides some protection by limiting access to the filesystem and network by default, but the nature of API testing means we need to access the network and read files.

## Where to next?

The Arazzo specification, although released as v1.0.0, is in active development. The OpenAPI team is working on a JSON Schema for Arazzo, which will provide a formal definition of the specification's structure and constraints.

We found the specification slightly ambiguous in places, but the team is [active on GitHub](https://github.com/OAI/Arazzo-Specification/issues) and open to feedback and contributions. If you're interested in API testing, Arazzo is a great project to get involved with.

At Speakeasy, we're building tools to make API testing easier and more effective. Our TypeScript, Python, and Go SDK generators can already generate tests from OpenAPI documents, and we're working on integrating Arazzo support. Our CLI can already lint Arazzo documents, and we'll have more to share soon.

We're excited to see how Arazzo evolves and how it can help developers build robust, end-to-end tests for their APIs.