# Evaluate agent trajectories

> Evaluate agent trajectories to optimize tool selection and reasoning paths, ensuring efficient agent behavior before production with Opik.

Evaluating agents requires more than checking the final output. You need to assess The
**trajectory** — the steps your agent takes to reach an answer, including tool selection, reasoning
chains, and intermediate decisions.

Agent trajectory evaluation helps you catch tool selection errors, identify inefficient reasoning
paths, and optimize agent behavior before it reaches production.

<Frame>
  <img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2026-01-30T17:33:07.444Z/img/evaluation/agent_trajectory.png" alt="Agent trajectory showing multiple steps and tool calls" caption="Agent trajectory showing multiple steps and tool calls" />
</Frame>

## Prerequisites

Before evaluating agent trajectories, you need:

1. **Opik SDK installed and configured** — See [Quickstart](/quickstart) for setup
2. **Agent with observability enabled** — Your agent must be instrumented with Opik tracing
3. **Test dataset** — Examples with expected agent behavior

If your agent isn't traced yet, see [Log Traces](/tracing/log_traces) to add observability first.

### Installing the Opik SDK

To install the Opik Python SDK you can run the following command:

```bash
pip install opik
```

Then you can configure the SDK by running the following command:

```bash
opik configure
```

This will prompt you for your API key and workspace or your instance URL if you are self-hosting.

### Adding observability to your agent

In order to be able to evaluate the agent's trajectory, you need to add tracing to your agent. This
will allow us to capture the agent's trajectory and evaluate it.

<CodeBlocks>
  ```python title="LangChain" language="python" {2,4,22} maxLines=25
  from langchain.agents import create_agent
  from opik.integrations.langchain import OpikTracer

  opik_tracer = OpikTracer()

  def get_weather(city: str) -> str:
      """Get weather for a given city."""
      return f"It's always sunny in {city}!"

  agent = create_agent(
      model="openai:gpt-4o",
      tools=[get_weather],
      system_prompt="You are a helpful assistant"
  )

  # Run the agent
  agent.invoke(
      {"messages": [{
          "role": "user",
          "content": "what is the weather in sf"
      }]},
      config={"callbacks": [opik_tracer]}
  )
  ```

  ```python title="OpenAI" language="python" {4,5,7,24,29} maxLines=25
  import json
  import openai

  from opik import track
  from opik.integrations.openai import track_openai

  openai_client = track_openai(openai.OpenAI())

  # Define tools
  tools = [
      {
          "type": "function",
          "function": {
              "name": "get_weather",
              "description": "Get weather for a given city.",
              "parameters": {
                  "type": "object",
                  "properties": {"city": {"type": "string"}},
              }
          }
      }
  ]

  @track(type="tool")
  def get_weather(city: str) -> str:
      """Get weather for a given city."""
      return f"It's always sunny in {city}!"

  @track
  def agent_with_tools(user_input: str):
      messages = [{"role": "user", "content": user_input}]

      while True:
          response = openai_client.chat.completions.create(
              model="gpt-4o",
              messages=messages,
              tools=tools,
              tool_choice="auto"
          )

          if response.choices[0].finish_reason == "tool_calls":
              messages.append(response.choices[0].message)

              for tool_call in response.choices[0].message.tool_calls:    
                  tool_name = tool_call.function.name
                  tool_args = json.loads(tool_call.function.arguments)
                  tool_result = get_weather(tool_args["city"])
                  user_input = f"The weather in {tool_args['city']} is {tool_result}"

                  messages.append({
                      "role": "tool",
                      "content": tool_result,
                      "tool_call_id": tool_call.id
                  })
          else:
              break
          
      return messages
  ```
</CodeBlocks>

<Tip>
  If you're using specific agent frameworks like CrewAI, LangGraph, or OpenAI Agents, check our
  [integrations](/integrations/overview) for framework-specific setup instructions.
</Tip>

## Evaluating your agent's trajectory

In order to evaluate the agent's trajectory, we will need to create a dataset, define an evaluation
metric and then run the evaluation.

### Creating a dataset

We are going to create a dataset with a set of user questions and some expected tools that the
agent should be calling:

```python
from opik import Opik

client = Opik()
dataset = client.get_or_create_dataset(name="agent_tool_selection")
dataset.insert([
    {
        "input": "What is 25 * 17?",
        "expected_tool": []
    },
    {
        "input": "What is the weather in SF?",
        "expected_tool": ["get_weather"]
    },
    {
        "input": "What is the weather in NY?",
        "expected_tool": ["get_weather"]
    }
])
```

<Tip>
  The format of dataset items is very flexible, you can include any fields you want in each item.
</Tip>

### Defining the evaluation metric

In this task, we are going to measure `Strict Tool Adherence` which measures the agent's adherence
to the expected tools in the same order as they are expected.

The key to this metric is the use of the optional `task_span` parameter, this is available for all
custom metrics and can be used to access the agent's trajectory:

<CodeBlocks>
  ```python title="Strict Tool Adherence Metric" maxLines=1000
  from opik.evaluation.metrics import BaseMetric, score_result
  from opik.message_processing.emulation.models import SpanModel
  from typing import List

  class StrictToolAdherenceMetric(BaseMetric):
      def __init__(self, name: str = "strict_tool_adherence"):
          self.name = name

      def find_tools(self, task_span):
          """Find all tool spans in the SpanModel hierarchy."""
          tools_used = []
          
          def extract_tools_from_spans(spans):
              """Recursively extract tools from spans list."""
              for span in spans:
                  # Check if this span is a tool
                  if span.type == "tool" and span.name:
                      tools_used.append(span.name)
                  
                  # Recursively check nested spans
                  if span.spans:
                      extract_tools_from_spans(span.spans)
          
          # Start the recursive search from the top level spans
          if task_span.spans:
              extract_tools_from_spans(task_span.spans)
          
          return tools_used

      def score(self, task_span: SpanModel,
                expected_tool: List[str], **kwargs):
          # Find tool calls in trajectory
          tool_used = self.find_tools(task_span)

          if tool_used == expected_tool:
              return score_result.ScoreResult(
                  value=1.0,
                  name=self.name,
                  reason=f"Correct: used {tool_used}"
              )
          else:
              return score_result.ScoreResult(
                  value=0.0,
                  name=self.name,
                  reason=f"Used {tool_used}, expected {expected_tool}"
              )
              
  ```

  ```python title="Tool Adherence Metric" maxLines=1000
  from opik.evaluation.metrics import BaseMetric, score_result
  from opik.message_processing.emulation.models import SpanModel
  from typing import List

  class ToolAdherenceMetric(BaseMetric):
      def __init__(self, name: str = "tool_adherence"):
          self.name = name

      def find_tools(self, task_span):
          """Recursively find all tool spans in the SpanModel hierarchy."""
          tools_used = []
          
          def extract_tools_from_spans(spans):
              """Recursively extract tools from spans list."""
              for span in spans:
                  # Check if this span is a tool
                  if span.type == "tool" and span.name:
                      tools_used.append(span.name)
                  
                  # Recursively check nested spans
                  if span.spans:
                      extract_tools_from_spans(span.spans)
          
          # Start the recursive search from the task_span's spans
          if task_span.spans:
              extract_tools_from_spans(task_span.spans)
          
          return tools_used

      def score(self, task_span: SpanModel,
                expected_tool: List[str], **kwargs):
          # Find tool calls in trajectory
          tool_used = self.find_tools(task_span)

          if set(tool_used) == set(expected_tool):
              return score_result.ScoreResult(
                  value=1.0,
                  name=self.name,
                  reason=f"Correct: used {tool_used}"
              )
          else:
              return score_result.ScoreResult(
                  value=0.0,
                  name=self.name,
                  reason=f"Used {tool_used}, expected {expected_tool}"
              )
  ```
</CodeBlocks>

### Running the evaluation

Let's define our evaluation task that will run our agent and return the assistant's response:

<CodeBlocks>
  ```python title="LangChain" maxLines=1000
  def evaluation_task(dataset_item: dict) -> dict:
      res = agent.invoke(
          {"messages": [{
              "role": "user",
              "content": dataset_item["input"]
          }]},
          config={"callbacks": [opik_tracer]}
      )
      
      return {"output": res['messages'][-1].content}
  ```

  ```python title="OpenAI" maxLines=1000
  def evaluation_task(dataset_item: dict) -> dict:
      res = agent_with_tools(dataset_item["input"])
      return {"output": messages[-1]['content']}
  ```
</CodeBlocks>

Now that we have our dataset and metric, we can run the evaluation:

```python title="Running the evaluation" maxLines=1000
from opik.evaluation import evaluate