# Retrieval evaluation using LLM-as-a-judge via Pydantic AI

> This page contains a tutorial on how to evaluate retrieval systems using LLMs as judges via Pydantic AI.

<CookbookHeader href="https://github.com/cohere-ai/cohere-developer-experience/blob/main/notebooks/guides/retrieval_eval_pydantic_ai.ipynb" />

We'll explore how to evaluate retrieval systems using Large Language Models (LLMs) as judges.Retrieval evaluation is a critical component in building high-quality information retrieval systems, and recent advancements in LLMs have made it possible to automate this evaluation process.

**What we'll cover**

* How to query the Wikipedia API
* How to implement and compare two different retrieval approaches:
  * The original search results from the Wikipedia API
  * Using Cohere's reranking model to rerank the search results
* How to set up an LLM-as-a-judge evaluation framework using Pydantic AI

**Tools We'll Use**

* **Cohere's API**: For reranking search results and providing evaluation models
* **Wikipedia's API**: As our information source
* **Pydantic AI**: For creating evaluation agents

This tutorial demonstrates a methodology for comparing different retrieval systems objectively. By the end, you'll have an example you can adapt to evaluate your own retrieval systems across different domains and use cases.

## Setup

First, let's import the necessary libraries.

```python PYTHON
%pip install -U cohere pydantic-ai
```

```python PYTHON
import requests
import cohere
import pandas as pd
from pydantic_ai import Agent
from pydantic_ai.models import KnownModelName
from collections import Counter

import os
co = cohere.ClientV2(os.getenv("COHERE_API_KEY"))
```

```python PYTHON
import nest_asyncio
nest_asyncio.apply()
```

## Perform Wikipedia search

Next, we implement a function to query Wikipedia for relevant information based on user input. The `search_wikipedia()` function allows us to retrieve a specified number of Wikipedia search results, extracting their titles, snippets, and page IDs.

This will provide us with the dataset for our retrieval evaluation experiment, where we'll compare different approaches to finding and ranking relevant information.

We'll use a small dataset of 10 questions about geography to test the Wikipedia search.

```python PYTHON
import requests

def search_wikipedia(query, limit=10):
    url = "https://en.wikipedia.org/w/api.php"
    params = {
        'action': 'query',
        'list': 'search',
        'srsearch': query,
        'format': 'json',
        'srlimit': limit
    }

    response = requests.get(url, params=params)
    data = response.json()
    
    # Format the results
    results = []
    for item in data['query']['search']:
        results.append({
            "title": item["title"],
            "snippet": item["snippet"].replace("<span class=\"searchmatch\">", "").replace("</span>", ""),
        })
            
    return results
```

```python PYTHON