# Evaluate your agent

> Evaluate your LLM applications confidently. Learn the five steps to assess complex LLM chains or agents effectively.

Evaluating your LLM application allows you to have confidence in the performance of your LLM application. In this guide, we will walk through the process of evaluating complex applications like LLM chains or agents.

<Tip>
  In this guide, we will focus on evaluating complex LLM applications. If you
  are looking at evaluating single prompts you can refer to the [Evaluate A
  Prompt](/evaluation/evaluate_prompt) guide.
</Tip>

The evaluation is done in five steps:

1. Add tracing to your LLM application
2. Define the evaluation task
3. Choose the `Dataset` that you would like to evaluate your application on
4. Choose the metrics that you would like to evaluate your application with
5. Create and run the evaluation experiment

## Running an offline evaluation

### 1. (Optional) Add tracking to your LLM application

While not required, we recommend adding tracking to your LLM application. This allows you to have
full visibility into each evaluation run. In the example below we will use a combination of the
`track` decorator and the `track_openai` function to trace the LLM application.

<CodeBlocks>
  ```python title="Python" language="python"
  from opik import track
  from opik.integrations.openai import track_openai
  import openai

  openai_client = track_openai(openai.OpenAI())

  # This method is the LLM application that you want to evaluate
  # Typically this is not updated when creating evaluations
  @track
  def your_llm_application(input: str) -> str:
      response = openai_client.chat.completions.create(
          model="gpt-3.5-turbo",
          messages=[{"role": "user", "content": input}],
      )

      return response.choices[0].message.content
  ```
</CodeBlocks>

<Tip>
  Here we have added the `track` decorator so that this trace and all its nested
  steps are logged to the platform for further analysis.
</Tip>

### 2. Define the evaluation task

Once you have added instrumentation to your LLM application, we can define the evaluation
task. The evaluation task takes in as an input a dataset item and needs to return a
dictionary with keys that match the parameters expected by the metrics you are using. In
this example we can define the evaluation task as follows:

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  import { EvaluationTask } from "opik";
  import { OpenAI } from "openai";

  // Define dataset item type
  type DatasetItem = {
  input: string;
  expected: string;
  };

  const llmTask: EvaluationTask<DatasetItem> = async (datasetItem) => {
      const { input } = datasetItem;

      const openai = new OpenAI();
      const response = await openai.chat.completions.create({
          model: "gpt-4o",
          messages: [
              { role: "system", content: "You are a coding assistant" },
              { role: "user", content: input }
          ],
      });

      return { output: response.choices[0].message.content };

  };

  ```

  ```python title="Python" language="python"
  def evaluation_task(x):
      return {
          "output": your_llm_application(x['user_question'])
      }
  ```
</CodeBlocks>

<Warning>
  If the dictionary returned does not match with the parameters expected by the
  metrics, you will get inconsistent evaluation results.
</Warning>

### 3. Choose the evaluation Dataset

In order to create an evaluation experiment, you will need to have a Dataset that includes
all your test cases.

If you have already created a Dataset, you can use the get or create dataset methods to
fetch it.

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  import { Opik } from "opik";

  const client = new Opik();
  const dataset = await client.getOrCreateDataset<DatasetItem>("Example dataset");

  // Opik deduplicates items that are inserted into a dataset so we can insert them
  // for multiple times
  await dataset.insert([
      {
          input: "Hello, world!",
          expected: "Hello, world!"
      },
      {
          input: "What is the capital of France?",
          expected: "Paris"
      },
  ]);
  ```

  ```python title="Python" language="python"
  from opik import Opik

  client = Opik()
  dataset = client.get_or_create_dataset(name="Example dataset")

  # Opik deduplicates items that are inserted into a dataset so we can insert them
  # for multiple times
  dataset.insert([
      {
          "input": "Hello, world!",
          "expected_output": "Hello, world!"
      },
      {
          "input": "What is the capital of France?",
          "expected_output": "Paris"
      },
  ])
  ```
</CodeBlocks>

### 4. Choose evaluation metrics

Opik provides a set of built-in evaluation metrics that you can choose from. These are broken down into two main categories:

1. Heuristic metrics: These metrics that are deterministic in nature, for example `equals` or `contains`
2. LLM-as-a-judge: These metrics use an LLM to judge the quality of the output; typically these are used for detecting `hallucinations` or `context relevance`

In the same evaluation experiment, you can use multiple metrics to evaluate your application:

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  import { ExactMatch } from "opik";

  const exact_match_metric = new ExactMatch();

  ```

  ```python title="Python" language="python"
  from opik.evaluation.metrics import Hallucination

  hallucination_metric = Hallucination()
  ```
</CodeBlocks>

<Tip>
  Each metric expects the data in a certain format. You will need to ensure that
  the task you have defined in step 2 returns the data in the correct format.
</Tip>

### 5. Run the evaluation

Now that we have the task we want to evaluate, the dataset to evaluate on, and the metrics
we want to evaluate with, we can run the evaluation:

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript" maxLines=1000
  import { EvaluationTask, Opik, ExactMatch, evaluate } from "opik";
  import { OpenAI } from "openai";

  // Define dataset item type
  type DatasetItem = {
      input: string;
      expected: string;
  };

  // Define the evaluation task
  const llmTask: EvaluationTask<DatasetItem> = async (datasetItem) => {
      const { input } = datasetItem;

      const openai = new OpenAI();
      const response = await openai.chat.completions.create({
          model: "gpt-4o",
          messages: [
              { role: "system", content: "You are a coding assistant" },
              { role: "user", content: input }
          ],
      });

      return { output: response.choices[0].message.content };
  };

  // Get or create the dataset - items are automatically deduplicated
  const client = new Opik();
  const dataset = await client.getOrCreateDataset<DatasetItem>("Example dataset");
  await dataset.insert([
      {
          input: "Hello, world!",
          expected: "Hello, world!"
      },
      {
          input: "What is the capital of France?",
          expected: "Paris"
      },
  ]);

  // Define the metric
  const exact_match_metric = new ExactMatch();

  // Run the evaluation
  const result = await evaluate({
      dataset,
      task: llmTask,
      scoringMetrics: [exact_match_metric],
      experimentName: "Example Evaluation",
  });
  console.log(`Experiment ID: ${result.experimentId}`);
  console.log(`Experiment Name: ${result.experimentName}`);
  console.log(`Total test cases: ${result.testResults.length}`);
  ```

  ```python title="Python" language="python" maxLines=1000
  from opik import Opik, track
  from opik.evaluation import evaluate
  from opik.evaluation.metrics import Equals, Hallucination
  from opik.integrations.openai import track_openai
  import openai

  # Define the task to evaluate
  openai_client = track_openai(openai.OpenAI())

  MODEL = "gpt-3.5-turbo"

  @track
  def your_llm_application(input: str) -> str:
      response = openai_client.chat.completions.create(
          model=MODEL,
          messages=[{"role": "user", "content": input}],
      )
      return response.choices[0].message.content

  # Define the evaluation task
  def evaluation_task(x):
      return {
          "output": your_llm_application(x['input'])
      }

  # Create a simple dataset
  client = Opik()
  dataset = client.get_or_create_dataset(name="Example dataset")
  dataset.insert([
      {"input": "What is the capital of France?"},
      {"input": "What is the capital of Germany?"},
  ])

  # Define the metrics
  hallucination_metric = Hallucination()

  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      experiment_config={
          "model": MODEL
      }
  )
  ```
</CodeBlocks>

<Tip>
  You can use the `experiment_config` parameter to store information about your
  evaluation task. Typically we see teams store information about the prompt
  template, the model used and model parameters used to evaluate the
  application.
</Tip>

### 6. Analyze the evaluation results

Once the evaluation is complete, you will get a link to the Opik UI where you can analyze the
evaluation results. In addition to being able to deep dive into each test case, you will also
be able to compare multiple experiments side by side.

<Frame>
  <img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2026-01-30T17:33:07.444Z/img/evaluation/evaluation_results.png" />
</Frame>

## Advanced usage

### Missing arguments for scoring methods

When you face the `opik.exceptions.ScoreMethodMissingArguments` exception, it means that the dataset
item and task output dictionaries do not contain all the arguments expected by the scoring method.
The way the evaluate function works is by merging the dataset item and task output dictionaries and
then passing the result to the scoring method. For example, if the dataset item contains the keys
`user_question` and `context` while the evaluation task returns a dictionary with the key `output`,
the scoring method will be called as `scoring_method.score(user_question='...', context= '...', output= '...')`.
This can be an issue if the scoring method expects a different set of arguments.

You can solve this by either updating the dataset item or evaluation task to return the missing
arguments or by using the `scoring_key_mapping` parameter of the `evaluate` function. In the example
above, if the scoring method expects `input` as an argument, you can map the `user_question` key to
the `input` key as follows:

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  evaluation = evaluate({
      dataset,
      task: evaluation_task,
      scoringMetrics: [hallucination_metric],
      scoringKeyMapping: {"input": "user_question"},
  })
  ```

  ```python title="Python" language="python"
  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      scoring_key_mapping={"input": "user_question"},
  )
  ```
</CodeBlocks>

### Linking prompts to experiments

The [Opik prompt library](/prompt_engineering/prompt_management) can be used to version your prompt templates.

When creating an Experiment, you can link the Experiment to a specific prompt version:

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  import { Opik, evaluate, evaluatePrompt } from 'opik';
  import { Hallucination } from 'opik';

  const client = new Opik();

  // Create a prompt
  const prompt = await client.createPrompt({
      name: "My prompt",
      prompt: "Translate to French: {{input}}",
  });

  // Link prompt to evaluation experiment
  await evaluatePrompt({
      dataset: myDataset,
      messages: [
          { role: "user", content: "Translate to French: {{input}}" },
      ],
      model: "gpt-4o",
      scoringMetrics: [new Hallucination()],
      prompts: [prompt],
  });
  ```

  ```python title="Python" language="python"
  import opik

  # Create a prompt
  prompt = opik.Prompt(
      name="My prompt",
      prompt="..."
  )

  # Run the evaluation
  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      prompts=[prompt],
  )
  ```
</CodeBlocks>

The experiment will now be linked to the prompt allowing you to view all experiments that use a specific prompt:

<Frame>
  <img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2026-01-30T17:33:07.444Z/img/evaluation/linked_prompt.png" />
</Frame>

### Logging traces to a specific project

You can use the `project_name` parameter of the `evaluate` function to log evaluation traces to a specific project:

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  const evaluation = await evaluate({
      dataset,
      task: evaluation_task,
      scoringMetrics: [hallucination_metric],
      projectName: "hallucination-detection",
  })
  ```

  ```python title="Python" language="python"
  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      project_name="hallucination-detection",
  )
  ```
</CodeBlocks>

### Evaluating a subset of the dataset

You can use the `nb_samples` parameter to specify the number of samples to use for the evaluation. This is useful if you only want to evaluate a subset of the dataset.

<CodeBlocks>
  ```typescript title="TypeScript" language="typescript"
  const evaluation = await evaluate({
      dataset,
      task: evaluation_task,
      scoringMetrics: [hallucination_metric],
      nbSamples: 10,
  })
  ```

  ```python title="Python" language="python"
  evaluation = evaluate(
      experiment_name="My experiment",
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      nb_samples=10,
  )
  ```
</CodeBlocks>

### Sampling the dataset for evaluation

You can use the `dataset_sampler` parameter to specify the instance of dataset sampler to use for sampling the dataset.
This is useful if you want to sample the dataset differently than the default sampling strategy (accept all items).

For example, you can use the `RandomDatasetSampler` to sample the dataset randomly:

<CodeBlocks>
  ```python title="Python" language="python"
  from opik.evaluation import samplers

  evaluation = evaluate(
      experiment_name="My experiment",
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      dataset_sampler=samplers.RandomDatasetSampler(max_samples=10),
  )
  ```
</CodeBlocks>

In the example above, the evaluation will sample 10 random items from the dataset.

Also, you can implement your own dataset sampler by extending the `BaseDatasetSampler` and overriding the `sample` method.

<CodeBlocks>
  ```python title="Python" language="python"
  import re
  from typing import List

  from opik.api_objects.dataset import dataset_item
  from opik.evaluation import samplers

  class MyDatasetSampler(samplers.BaseDatasetSampler):

      def __init__(self, filter_string: str, field_name: str) -> None:
          self.filter_regex = re.compile(filter_string)
          self.field_name = field_name

      def sample(self, dataset: List[dataset_item.DatasetItem]) -> List[dataset_item.DatasetItem]:
          # Sample items from the dataset that match the filter string in the 'field_name' field
          return [item for item in filter(lambda x: self.filter_regex.search(x[self.field_name]), dataset)]

  # Example usage
  evaluation = evaluate(
      experiment_name="My experiment",
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      dataset_sampler=MyDatasetSampler(filter_string="\\.*SUCCESS\\.*", field_name="output"),
  )

  ```
</CodeBlocks>

Implementing your own dataset sampler is useful if you want to implement a custom sampling strategy. For instance,
you can implement a dataset sampler that samples the dataset using some filtering criteria as in the example above.

### Analyzing the evaluation results

The `evaluate` function returns an `EvaluationResult` object that contains the evaluation results.
You can create aggregated statistics for each metric by calling its `aggregate_evaluation_scores` method:

<CodeBlocks>
  ```python title="Python" language="python"
      evaluation = evaluate(
          experiment_name="My experiment",
          dataset=dataset,
          task=evaluation_task,
          scoring_metrics=[hallucination_metric],
      )

      # Retrieve and print the aggregated scores statistics (mean, min, max, std) per metric
      scores = evaluation.aggregate_evaluation_scores()
      for metric_name, statistics in scores.aggregated_scores.items():
          print(f"{metric_name}: {statistics}")

  ```
</CodeBlocks>

Aggregated statistics can help analyze evaluation results and are useful for comparing the
performance of different models or different versions of the same model, for example.

### Computing experiment-level metrics

In addition to per-item metrics, you can compute experiment-level aggregate metrics that are calculated across all test results. These experiment scores are displayed in the Opik UI alongside feedback scores and can be used for sorting and filtering experiments.

Experiment scores are computed after all test results are collected. You define experiment score functions that take a list of `TestResult` objects and return a list of `ScoreResult` objects representing aggregate metrics.

<CodeBlocks>
  ```python title="Python" language="python"
  from typing import List
  from opik.evaluation import evaluate, test_result
  from opik.evaluation.metrics import Hallucination, score_result

  # Define an experiment score function
  def compute_hallucination_max(
      test_results: List[test_result.TestResult],
  ) -> List[score_result.ScoreResult]:
      """Compute the maximum hallucination score across all test results."""
      hallucination_scores = [
          result.score_results[0].value 
          for result in test_results 
          if result.score_results and len(result.score_results) > 0
      ]
      
      if not hallucination_scores:
          return []
      
      return [
          score_result.ScoreResult(
              name="hallucination_metric (max)",
              value=max(hallucination_scores),
              reason=f"Maximum hallucination score across {len(hallucination_scores)} test cases"
          )
      ]

  # Run evaluation with experiment scores
  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[Hallucination()],
      experiment_scoring_functions=[compute_hallucination_max],
      experiment_name="My experiment"
  )

  # Access experiment scores from the result
  print(f"Experiment scores: {evaluation.experiment_scores}")
  ```
</CodeBlocks>

<Tip>
  Experiment scores are displayed in the Opik UI in the experiments table alongside feedback scores. They can be used for sorting and filtering experiments, making it easy to compare experiments based on aggregate metrics.
</Tip>

You can define multiple experiment score functions to compute different aggregate metrics:

<CodeBlocks>
  ```python title="Python" language="python"
  from typing import List
  from opik.evaluation import evaluate, test_result
  from opik.evaluation.metrics import Equals, score_result

  def compute_accuracy_stats(
      test_results: List[test_result.TestResult],
  ) -> List[score_result.ScoreResult]:
      """Compute accuracy statistics across all test results."""
      accuracy_scores = [
          result.score_results[0].value 
          for result in test_results 
          if result.score_results and len(result.score_results) > 0
      ]
      
      if not accuracy_scores:
          return []
      
      return [
          score_result.ScoreResult(
              name="accuracy (mean)",
              value=sum(accuracy_scores) / len(accuracy_scores),
              reason=f"Mean accuracy across {len(accuracy_scores)} test cases"
          ),
          score_result.ScoreResult(
              name="accuracy (min)",
              value=min(accuracy_scores),
              reason=f"Minimum accuracy across {len(accuracy_scores)} test cases"
          ),
          score_result.ScoreResult(
              name="accuracy (max)",
              value=max(accuracy_scores),
              reason=f"Maximum accuracy across {len(accuracy_scores)} test cases"
          ),
      ]

  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[Equals()],
      experiment_scoring_functions=[compute_accuracy_stats],
      experiment_name="My experiment"
  )
  ```
</CodeBlocks>

<Warning>
  Experiment score functions receive all test results after evaluation completes. Make sure your functions handle edge cases like empty test results or missing score values gracefully.
</Warning>

### Python SDK

#### Using async evaluation tasks

The `evaluate` function does not support `async` evaluation tasks, if you pass
an async task you will get an error similar to:

```python wordWrap
Input should be a valid dictionary [type=dict_type, input_value='<coroutine object kyc_qu...ng_task at 0x3336d0a40>', input_type=str]
```

As it might not always be possible to convert all your LLM logic to not rely on async logic,
we recommend using `asyncio.run` within the evaluation task:

```python
import asyncio

async def your_llm_application(input: str) -> str:
    return "Hello, World"

def evaluation_task(x):
    # your_llm_application here is an async function
    result = asyncio.run(your_llm_application(x['input']))
    return {
        "output": result
    }
```

This should solve the issue and allow you to run the evaluation.

<Tip>
  If you are running in a Jupyter notebook, you will need to add the following line to the top of your notebook:

  ```python
  import nest_asyncio
  nest_asyncio.apply()
  ```

  otherwise you might get the error `RuntimeError: asyncio.run() cannot be called from a running event loop`
</Tip>

<Warning>
  The `evaluate` function uses multi-threading under the hood to speed up the evaluation run. Using both
  `asyncio` and multi-threading can lead to unexpected behavior and hard to debug errors.

  If you run into any issues, you can disable the multi-threading in the SDK by setting `task_threads` to 1:

  ```python
  evaluation = evaluate(
      dataset=dataset,
      task=evaluation_task,
      scoring_metrics=[hallucination_metric],
      task_threads=1
  )
  ```
</Warning>

#### Disabling threading

In order to evaluate datasets more efficiently, Opik uses multiple background threads to evaluate the dataset. If this is causing issues, you can disable these by setting `task_threads` and `scoring_threads` to `1` which will lead Opik to run all calculations in the main thread.

#### Passing additional arguments to `evaluation_task`

Sometimes your evaluation task needs extra context besides the dataset item (commonly referred to as `x`). For example, you may want to pass a model name, a system prompt, or a pre-initialized client.
Since `evaluate` calls the task as `task(x)` for each dataset item, the recommended pattern is to create a wrapper (or use `functools.partial`) that closes over any additional arguments.

Using a wrapper function:

```python