# Running Evals in CI
[EN] Source: https://mastra.ai/docs/scorers/evals-legacy/running-in-ci

:::info Scorers
This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
:::

Running evals in your CI pipeline helps bridge this gap by providing quantifiable metrics for measuring agent quality over time.

## Setting up CI integration

We support any testing framework that supports ESM modules. For example, you can use [Vitest](https://vitest.dev/), [Jest](https://jestjs.io/) or [Mocha](https://mochajs.org/) to run evals in your CI/CD pipeline.

```typescript copy showLineNumbers title="src/mastra/agents/index.test.ts"
import { describe, it, expect } from "vitest";
import { evaluate } from "@mastra/evals";
import { ToneConsistencyMetric } from "@mastra/evals/nlp";
import { myAgent } from "./index";

describe("My Agent", () => {
  it("should validate tone consistency", async () => {
    const metric = new ToneConsistencyMetric();
    const result = await evaluate(myAgent, "Hello, world!", metric);

    expect(result.score).toBe(1);
  });
});
```

You will need to configure a testSetup and globalSetup script for your testing framework to capture the eval results. It allows us to show these results in your mastra dashboard.

## Framework configuration

### Vitest setup

Add these files to your project to run evals in your CI/CD pipeline:

```typescript copy showLineNumbers title="globalSetup.ts"
import { globalSetup } from "@mastra/evals";

export default function setup() {
  globalSetup();
}
```

```typescript copy showLineNumbers title="testSetup.ts"
import { beforeAll } from "vitest";
import { attachListeners } from "@mastra/evals";

beforeAll(async () => {
  await attachListeners();
});
```

```typescript copy showLineNumbers title="vitest.config.ts"
import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    globalSetup: "./globalSetup.ts",
    setupFiles: ["./testSetup.ts"],
  },
});
```

## Storage configuration

To store eval results in Mastra Storage and capture results in the Mastra dashboard:

```typescript copy showLineNumbers title="testSetup.ts"
import { beforeAll } from "vitest";
import { attachListeners } from "@mastra/evals";
import { mastra } from "./your-mastra-setup";

beforeAll(async () => {
  // Store evals in Mastra Storage (requires storage to be enabled)
  await attachListeners(mastra);
});
```

With file storage, evals persist and can be queried later. With memory storage, evals are isolated to the test process.


---
title: "Textual Evals | Scorers"
description: "Understand how Mastra uses LLM-as-judge methodology to evaluate text quality."
---