# SummarizationMetric
[EN] Source: https://mastra.ai/reference/evals/summarization

:::info Scorers
This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
:::

The `SummarizationMetric` evaluates how well an LLM's summary captures the original text's content while maintaining factual accuracy. It combines two aspects: alignment (factual correctness) and coverage (inclusion of key information), using the minimum scores to ensure both qualities are necessary for a good summary.

## Basic Usage

```typescript
import { openai } from "@ai-sdk/openai";
import { SummarizationMetric } from "@mastra/evals/llm";

// Configure the model for evaluation
const model = openai("gpt-4o-mini");

const metric = new SummarizationMetric(model);

const result = await metric.measure(
  "The company was founded in 1995 by John Smith. It started with 10 employees and grew to 500 by 2020. The company is based in Seattle.",
  "Founded in 1995 by John Smith, the company grew from 10 to 500 employees by 2020.",
);

console.log(result.score); // Score from 0-1
console.log(result.info); // Object containing detailed metrics about the summary
```

## Constructor Parameters

<PropertiesTable
  content={[
    {
      name: "model",
      type: "LanguageModel",
      description: "Configuration for the model used to evaluate summaries",
      isOptional: false,
    },
    {
      name: "options",
      type: "SummarizationMetricOptions",
      description: "Configuration options for the metric",
      isOptional: true,
      defaultValue: "{ scale: 1 }",
    },
  ]}
/>

### SummarizationMetricOptions

<PropertiesTable
  content={[
    {
      name: "scale",
      type: "number",
      description: "Maximum score value",
      isOptional: true,
      defaultValue: "1",
    },
  ]}
/>

## measure() Parameters

<PropertiesTable
  content={[
    {
      name: "input",
      type: "string",
      description: "The original text to be summarized",
      isOptional: false,
    },
    {
      name: "output",
      type: "string",
      description: "The generated summary to evaluate",
      isOptional: false,
    },
  ]}
/>

## Returns

<PropertiesTable
  content={[
    {
      name: "score",
      type: "number",
      description: "Summarization score (0 to scale, default 0-1)",
    },
    {
      name: "info",
      type: "object",
      description: "Object containing detailed metrics about the summary",
      properties: [
        {
          type: "string",
          parameters: [
            {
              name: "reason",
              type: "string",
              description:
                "Detailed explanation of the score, including both alignment and coverage aspects",
            },
          ],
        },
        {
          type: "number",
          parameters: [
            {
              name: "alignmentScore",
              type: "number",
              description: "Alignment score (0 to 1)",
            },
          ],
        },
        {
          type: "number",
          parameters: [
            {
              name: "coverageScore",
              type: "number",
              description: "Coverage score (0 to 1)",
            },
          ],
        },
      ],
    },
  ]}
/>

## Scoring Details

The metric evaluates summaries through two essential components:

1. **Alignment Score**: Measures factual correctness
   - Extracts claims from the summary
   - Verifies each claim against the original text
   - Assigns "yes", "no", or "unsure" verdicts

2. **Coverage Score**: Measures inclusion of key information
   - Generates key questions from the original text
   - Check if the summary answers these questions
   - Checks information inclusion and assesses comprehensiveness

### Scoring Process

1. Calculates alignment score:
   - Extracts claims from summary
   - Verifies against source text
   - Computes: `supported_claims / total_claims`

2. Determines coverage score:
   - Generates questions from source
   - Checks summary for answers
   - Evaluates completeness
   - Calculates: `answerable_questions / total_questions`

Final score: `min(alignment_score, coverage_score) * scale`

### Score interpretation

(0 to scale, default 0-1)

- 1.0: Perfect summary - completely factual and covers all key information
- 0.7-0.9: Strong summary with minor omissions or slight inaccuracies
- 0.4-0.6: Moderate quality with significant gaps or inaccuracies
- 0.1-0.3: Poor summary with major omissions or factual errors
- 0.0: Invalid summary - either completely inaccurate or missing critical information

## Example with Analysis

```typescript
import { openai } from "@ai-sdk/openai";
import { SummarizationMetric } from "@mastra/evals/llm";

// Configure the model for evaluation
const model = openai("gpt-4o-mini");

const metric = new SummarizationMetric(model);

const result = await metric.measure(
  "The electric car company Tesla was founded in 2003 by Martin Eberhard and Marc Tarpenning. Elon Musk joined in 2004 as the largest investor and became CEO in 2008. The company's first car, the Roadster, was launched in 2008.",
  "Tesla, founded by Elon Musk in 2003, revolutionized the electric car industry starting with the Roadster in 2008.",
);

// Example output:
// {
//   score: 0.5,
//   info: {
//     reason: "The score is 0.5 because while the coverage is good (0.75) - mentioning the founding year,
//           first car model, and launch date - the alignment score is lower (0.5) due to incorrectly
//           attributing the company's founding to Elon Musk instead of Martin Eberhard and Marc Tarpenning.
//           The final score takes the minimum of these two scores to ensure both factual accuracy and
//           coverage are necessary for a good summary."
//     alignmentScore: 0.5,
//     coverageScore: 0.75,
//   }
// }
```

## Related

- [Faithfulness Metric](./faithfulness)
- [Completeness Metric](./completeness)
- [Contextual Recall Metric](./contextual-recall)
- [Hallucination Metric](./hallucination)


---
title: "Reference: TextualDifferenceMetric | Evals"
description: Documentation for the Textual Difference Metric in Mastra, which measures textual differences between strings using sequence matching.
---