# app/main.py
Source: https://speakeasy.com/mcp/tool-design/designing-rag-tools-for-llms

Retrieval-Augmented Generation (RAG) and the Model Context Protocol (MCP) are often positioned as alternatives — with RAG enabling semantic searches and MCP allowing API actions — but they can be complementary. You can use RAG to search your knowledge base efficiently and MCP to standardize how LLMs access that search.

This guide shows you how to design RAG tools specifically for LLMs. It demonstrates the input patterns that work, the output structures LLMs need, and the best design choices for RAG tools.

## RAG overview

RAG is an architecture pattern for semantic search. It combines information retrieval with text generation, allowing LLMs to search external databases or sources for relevant context before generating an answer.

This usually works by breaking documents into chunks, converting those chunks into vectors, storing them in a database, and then retrieving information based on the semantic similarity to user queries.

## Why MCP servers should have a RAG tool

MCP servers provide tools that LLMs interact with to perform actions such as searching databases, calling APIs, and updating records. RAG provides LLMs with additional context by semantically searching the knowledge base. MCP gives LLMs capabilities by connecting them to a system.

For example, a RAG tool could enable your enterprise AI chatbot to answer questions from your user guides and documentation, and MCP tools could help customer support agents retrieve a [user's license information](/mcp/using-mcp/use-cases/customer-support) or create a new support ticket. In this example, RAG handles knowledge retrieval and MCP handles your system actions.

### The problem with MCP resources

MCP servers provide three primitives: tools, resources, and prompts. [MCP resources](/mcp/core-concepts/resources) are designed to give context to LLMs. These resources can be images, guides, or PDFs. MCP resources seem like the natural choice for searching documentation — you expose your docs, and the LLM accesses them.

But the problem is scale. MCP resources dump the entire collection or document into the context window with no processing. If MCP dumps a 100-page product guide in the LLM context, it risks bloating the context and immediately hitting context limits, which could cause timeouts, refusals, or hallucinations. Most LLM clients, including Claude Desktop and ChatGPT, don't index resources from MCP servers due to rate limits and context window issues.

In our [RAG vs MCP](/blog/rag-vs-mcp) blog, we compared an RAG implementation to an MCP implementation for searching Django documentation. RAG used 12,405 tokens and found the answer in 7.64 seconds. MCP used more than double that number of tokens (30,044) and took over four times longer (33.28 seconds) than RAG, but still failed to find the answer because the relevant content fell beyond the first 50 pages it could fit in the context window.

### How RAG tools solve context bloating

This is where RAG tools come in handy. Instead of an LLM loading, managing, and searching multiple MCP resources, it can call a RAG tool with a natural language query. The tool handles embedding, vector search, and relevance filtering, and returns only the chunks most relevant to the search. The LLM gets precisely what it needs without managing the search infrastructure.

RAG tools also enable features that don't work with static resources, including: 

- **Relevance scoring:** LLMs can request more context when scores are low.
- **Metadata filtering:** LLMs can search for specific versions or sections of a resource.
- **Context management:** You can implement automatic token budgeting.

The following diagram illustrates how this architecture works in practice:

![RAG tool architecture: User queries Claude Desktop, which calls Gram's MCP server, which calls your FastAPI, which queries the RAG service and ChromaDB](/assets/mcp/tool-design/designing-rag-tools-for-llms/illustration.png)

## RAG input parameters

A well-designed RAG tool needs three types of parameters: the search query itself, result controls, and quality filters. If an LLM uses incorrect parameters, it could fail to express what it needs or be flooded with irrelevant results.

### The query parameter

The query parameter should actually be a natural language query, not a list of keywords, because the RAG system uses embeddings for semantic search, and the embedding models (`all-MiniLM-L6-v2` or `text-embedding-3-small` for OpenAI) are trained on natural language sentences, not keyword lists. When a user asks, *"How do I work with curved geometries in Django's GIS module?"* the LLM immediately parses the intent (implementation guidance), identifies the domain (Django GIS geometry handling), and understands the context (a how-to question).

Forcing the LLM to translate the natural language prompt into structured keywords like `["django", "gis", "curve"]` with filters like `{"type": "tutorial"}` throws away semantic understanding. The LLM would have to decide which words were keywords and which were context, map natural language to your filter taxonomy, and lose the semantic relationships that make embeddings work. This would give you worse search results and waste tokens.

### The result count control

LLMs understand and manage their context windows. In the tool parameters, let the LLM specify how many results it needs. Cap results at `10` to prevent context overflow. Make this parameter optional with a documented default (a default of `3` results works well).

### Quality filtering

Not all search results are equally relevant, so you should allow the LLM to filter by quality.

For example, when you query a vector database like [ChromaDB](https://docs.trychroma.com/docs/overview/getting-started?lang=typescript#next-steps), it can (if configured) return results ranked by cosine similarity, a score measuring how close the query embedding is to each document embedding. A score of `1.0` means the query and embedding have identical semantic meanings, `0.5` means they are somewhat related, and `0.0` means they are unrelated.

This keeps low-quality results out of the LLM's context window entirely. When Claude asks for `min_score=0.7`, the RAG tool enforces this at retrieval time and filters out anything below that threshold.

The LLM uses these scores to adjust its strategy. If it receives two results with scores of `0.72` and `0.71`, it knows the match is marginal, and it may lower the threshold to `min_score=0.6` for a broader search. If it gets ten results, all above `0.9`, it knows the search is highly targeted.

## How to design a RAG tool

If you're exposing RAG capabilities via multiple endpoints, rather use a single endpoint. 

When you have numerous guides or documentation sets to index, you may be tempted to use separate tools or endpoints, but if you're designing RAG for an enterprise with dozens of products and documentation sets, exposing too many tools to the LLM could result in a tool explosion and cause context bloating. The LLM may face decision paralysis, leading to incorrect tool choices or hallucinations.

Instead, use a single search tool with a `collection` parameter for specifying which documentation set it should search (for example, `collection="user-guide"` or `collection="api-reference"`).

### Response format

LLMs need results in a format they can immediately use, such as the following:

```json
{
    "results": [
        {
            "content": "The actual documentation text...",
            "source": "https://docs.djangoproject.com/en/5.2/ref/contrib/gis/",
            "score": 0.87
        },
        {
            "content": "More documentation text...",
            "source": "https://docs.djangoproject.com/en/5.2/releases/5.2/",
            "score": 0.82
        }
    ],
    "total_found": 2,
    "tokens_estimate": 1847
}
```

The response format will vary depending on your case, but you should follow these best practices:

- **Use flat results arrays:** Don't nest results in complex structures because the LLM iterates through them sequentially.
- **Return content first:** Put the actual text in `content`, not `text`, `document`, or `chunk`.
- **Include sources:** The LLM needs to cite its sources. URLs, page numbers, or document IDs work.
- **Expose scores:** Let the LLM judge result quality. If all scores are below `0.6`, it knows the search was weak and might rephrase the query.
- **Provide token estimates:** This is critical for context management. The LLM needs to determine whether it can fit these results, along with its reasoning, in the context window. Divide the total number of characters by four for a rough estimate (this works well for English documentation).
- **Avoid returning too much data to the LLM:**

   ```json
   // ❌ Bad: too much metadata
   {
       "results": [
           {
               "content": "...",
               "metadata": {
                   "chunk_id": "abc123",
                   "embedding_model": "all-MiniLM-L6-v2",
                   "embedding_dimensions": 384,
                   "created_at": "2025-01-15T10:23:45Z",
                   "database_shard": "shard-3",
                   "index_version": "v2.1"
               }
           }
       ]
   }
   ```

### Error responses for RAG tools

When searches fail, LLMs need actionable errors. Compare the following versions of an error:

```json
// ❌ Bad: Generic error
{
    "error": "Search failed",
    "code": 400
}

// ✅ Good: Actionable error
{
    "error": "no_results_found",
    "message": "No documentation found for 'Djago GIS features'",
    "attempted_query": "Djago GIS features"
}
```

The second version tells the LLM what went wrong (a typo in "Django") and echoes the query so the LLM can verify the search.

## How to build a Django documentation RAG MCP server

Let's build a Django documentation search API and expose it as an MCP tool through Gram. This example extends the RAG implementation from the RAG vs MCP post by wrapping it in a REST API with the correct input/output design for LLM consumption.

You can find the complete project in the [Speakeasy Examples repository](https://github.com/speakeasy-api/examples/tree/main/rag-mcp-example), in the `complete` directory. Clone the project and use the code in the `base` folder to follow the instructions below.

### Set up the project

Clone and install the dependencies:

```bash
git clone https://github.com/speakeasy-api/examples.git
cd examples/rag-mcp-example/base
uv sync
```

Download the [Django 5.2.8 documentation PDF](https://app.readthedocs.org/projects/django/downloads/pdf/5.2.x/) and save it in the `base` directory as `django.pdf`. Run the indexing script to build the ChromaDB collection:

```bash
uv run python scripts/build_rag_index.py
```

### Define the search interface

First, define the schemas in the `app/main.py` file:

```python


import logging
from typing import List, Optional
from pathlib import Path
from pydantic import BaseModel, Field
from fastapi import FastAPI
from fastapi.openapi.utils import get_openapi
from sentence_transformers import SentenceTransformer
import chromadb