# Prompt Caching
Model prompts often contain repetitive content, such as system prompts and tool definitions.
Prompt caching automatically reuses computation from recent requests when they share a common prefix, delivering significant cost savings and improved response times while maintaining data privacy through volatile-only storage that expires automatically.

Prompt caching works automatically on all your API requests with no code changes required and no additional fees.

## How It Works

1. **Prefix Matching**: When you send a request, the system examines and identifies matching prefixes from recently processed requests stored temporarily in volatile memory. Prefixes can include system prompts, tool definitions, few-shot examples, and more.

2. **Cache Hit**: If a matching prefix is found, cached computation is reused, dramatically reducing latency and token costs by 50% for cached portions.

3. **Cache Miss**: If no match exists, your prompt is processed normally, with the prefix temporarily cached for potential future matches.

4. **Automatic Expiration**: All cached data automatically expires within a few hours, which helps ensure privacy while maintaining the benefits.

Prompt caching works automatically on all your API requests to supported models with no code changes required and no additional fees. Groq tries to maximize cache hits, but this is not guaranteed. Pricing discount will only apply on successful cache hits.

Cached tokens do not count towards your rate limits. However, cached tokens are subtracted from your limits after processing, so it's still possible to hit your limits if you are sending a large number of input tokens in parallel requests.

## Supported Models

Prompt caching is currently only supported for the following models:


| Model ID                        | Model                          |
|---------------------------------|--------------------------------|
| moonshotai/kimi-k2-instruct-0905                  | Kimi K2 |
| openai/gpt-oss-20b                  | GPT-OSS 20B |
| openai/gpt-oss-120b                  | GPT-OSS 120B |
| openai/gpt-oss-safeguard-20b                  | GPT-OSS-Safeguard 20B |


We're starting with a limited selection of models and will roll out prompt caching to more models soon.

## Pricing

Prompt caching is provided at no additional cost. There is a 50% discount for cached input tokens.

## Structuring Prompts for Optimal Caching

Cache hits are only possible for exact prefix matches within a prompt. To realize caching benefits, you need to think strategically about prompt organization:

### Optimal Prompt Structure
Place static content like instructions and examples at the beginning of your prompt, and put variable content, such as user-specific information, at the end. This maximizes the length of the reusable prefix across different requests.

If you put variable information (like timestamps or user IDs) at the beginning, even identical system instructions later in the prompt won't benefit from caching because the prefixes won't match.

<br />

**Place static content first:**
- System prompts and instructions
- Few-shot examples
- Tool definitions
- Schema definitions
- Common context or background information

<br />

**Place dynamic content last:**
- User-specific queries
- Variable data
- Timestamps
- Session-specific information
- Unique identifiers

### Example Structure

```text
[SYSTEM PROMPT - Static]
[TOOL DEFINITIONS - Static]  
[FEW-SHOT EXAMPLES - Static]
[COMMON INSTRUCTIONS - Static]
[USER QUERY - Dynamic]
[SESSION DATA - Dynamic]
```

This structure maximizes the likelihood that the static prefix portion will match across different requests, enabling cache hits while keeping user-specific content at the end.

## Prompt Caching Examples


### How Prompt Caching Works in Multi-Turn Conversations

In this example, we demonstrate how to use prompt caching in a multi-turn conversation.

During each turn, the system automatically caches the longest matching prefix from previous requests. The system message and conversation history that remain unchanged between requests will be cached, while only new user messages and assistant responses need fresh processing.

This approach is useful for maintaining context in ongoing conversations without repeatedly processing the same information.

**For the first request:**
- `prompt_tokens`: Number of tokens in the system message and first user message
- `cached_tokens`: 0 (no cache hit on first request)

<br />

**For subsequent requests within the cache lifetime:**
- `prompt_tokens`: Total number of tokens in the entire conversation (system message + conversation history + new user message)
- `cached_tokens`: Number of tokens in the system message and previous conversation history that were served from cache

<br />

When set up properly, you should see increasing cache efficiency as the conversation grows, with the system message and earlier conversation turns being served from cache while only new content requires processing.

### How Prompt Caching Works with Large Context

In this example, we demonstrate caching large static content like legal documents, research papers, or extensive context that remains constant across multiple queries.

The large legal document in the system message represents static content that benefits significantly from caching. Once cached, subsequent requests with different questions about the same document will reuse the cached computation for the document analysis, processing only the new user questions.

This approach is particularly effective for document analysis, research assistance, or any scenario where you need to ask multiple questions about the same large piece of content.

**For the first request:**
- `prompt_tokens`: Total number of tokens in the system message (including the large legal document) and user message
- `cached_tokens`: 0 (no cache hit on first request)

<br />

**For subsequent requests within the cache lifetime:**
- `prompt_tokens`: Total number of tokens in the system message (including the large legal document) and user message
- `cached_tokens`: Number of tokens in the entire cached system message (including the large legal document)

<br />

The caching efficiency is particularly high in this scenario since the large document (which may be thousands of tokens) is reused across multiple requests, while only small user queries (typically dozens of tokens) need fresh processing.

### How Prompt Caching Works with Tool Definitions

In this example, we demonstrate caching tool definitions.

All tool definitions, including their schemas, descriptions, and parameters, are cached as a single prefix when they remain consistent across requests. This is particularly valuable when you have a comprehensive set of tools that you want to reuse across multiple requests without re-processing them each time.

The system message and all tool definitions form the static prefix that gets cached, while user queries remain dynamic and are processed fresh for each request.

This approach is useful when you have a consistent set of tools that you want to reuse across multiple requests without re-processing them each time.

**For the first request:**
- `prompt_tokens`: Total number of tokens in the system message, tool definitions, and user message
- `cached_tokens`: 0 (no cache hit on first request)

<br />

**For subsequent requests within the cache lifetime:**
- `prompt_tokens`: Total number of tokens in the system message, tool definitions, and user message
- `cached_tokens`: Number of tokens in all cached tool definitions and system prompt

<br />

Tool definitions can be quite lengthy due to detailed parameter schemas and descriptions, making caching particularly beneficial for reducing both latency and costs when the same tool set is used repeatedly.

## Requirements and Limitations

### Caching Requirements
- **Exact Prefix Matching**: Cache hits require exact matches of the beginning of your prompt
- **Minimum Prompt Length**: The minimum cacheable prompt length varies by model, ranging from 128 to 1024 tokens depending on the specific model used

To check how much of your prompt was cached, see the response [usage fields](#response-usage-structure).

### What Can Be Cached

- **Complete message arrays** including system, user, and assistant messages
- **Tool definitions** and function schemas
- **System instructions** and prompt templates
- **One-shot** and **few-shot examples**
- **Structured output schemas**
- **Large static content** like legal documents, research papers, or extensive context that remains constant across multiple queries
- **Image inputs**, including image URLs and base64-encoded images

### Limitations

- **Exact Matching**: Even minor changes in cached portions prevent cache hits and force a new cache to be created
- **No Manual Control**: Cache clearing and management is automatic only

## Tracking Cache Usage

You can monitor how many tokens are being served from cache by examining the `usage` field in your API response. The response includes detailed token usage information, including how many tokens were cached.

### Response Usage Structure

```json
{
  "id": "chatcmpl-...",
  "model": "moonshotai/kimi-k2-instruct",
  "usage": {
      "queue_time": 0.026959759,
      "prompt_tokens": 4641,
      "prompt_time": 0.009995497,
      "completion_tokens": 1817,
      "completion_time": 5.57691751,
      "total_tokens": 6458,
      "total_time": 5.586913007,
      "prompt_tokens_details": {
          "cached_tokens": 4608
      }
  },
  ... other fields
}
```

### Understanding the Fields

- **`prompt_tokens`**: Total number of tokens in your input prompt
- **`cached_tokens`**: Number of input tokens that were served from cache (within `prompt_tokens_details`)
- **`completion_tokens`**: Number of tokens in the model's response
- **`total_tokens`**: Sum of prompt and completion tokens

<br />

In the example above, out of 4641 prompt tokens, 4608 tokens (99.3%) were served from cache, resulting in significant cost savings and improved response time.

### Calculating Cache Hit Rate

To calculate your cache hit rate:

```
Cache Hit Rate = cached_tokens / prompt_tokens × 100%
```

For the example above: `4608 / 4641 × 100% = 99.3%`

A higher cache hit rate indicates better prompt structure optimization leading to lower latency and more cost savings.

## Troubleshooting
- Verify that sections that you want to cache are identical between requests
- Check that calls are made within the cache lifetime (a few hours). Calls that are too far apart will not benefit from caching.
- Ensure that `tool_choice`, tool usage, and image usage remain consistent between calls
- Validate that you are caching at least the [minimum number of tokens](#caching-requirements) through the [usage fields](#response-usage-structure).

<br />

Changes to cached sections, including `tool_choice` and image usage, will invalidate the cache and require a new cache to be created. Subsequent calls will use the new cache.

## Frequently Asked Questions

### How is data privacy maintained with caching?

All cached data exists only in volatile memory and automatically expires within a few hours. No prompt or response content is ever stored in persistent storage or shared between organizations.

### Does caching affect the quality or consistency of responses?

No. Prompt caching only affects the processing of the input prompt, not the generation of responses. The actual model inference and response generation occur normally, maintaining identical output quality whether caching is used or not.

### Can I disable prompt caching?

Prompt caching is automatically enabled and cannot be manually disabled. This helps customers benefit from reduced costs and latency. Prompts are not stored in persistent storage.

### How do I know if my requests are benefiting from caching?

You can track cache usage by examining the `usage` field in your API responses. Cache hits are not guaranteed, but Groq tries to maximize them. See the [Tracking Cache Usage](#tracking-cache-usage) section above for detailed information on how to monitor cached tokens and calculate your cache hit rate.

### Are there any additional costs for using prompt caching?

No. Prompt caching is provided at no additional cost and can help to reduce your costs by 50% for cached tokens while improving response times.

### Does caching affect rate limits?

Cached tokens do not count toward your rate limits.

### Can I manually clear or refresh caches?

No manual cache management is available. All cache expiration and cleanup happens automatically.

### Does the prompt caching discount work with batch requests?

Batch requests can still benefit from prompt caching, but the prompt caching discount does not stack with the batch discount. [Batch requests](/docs/batch) already receive a 50% discount on all tokens, and while caching functionality remains active, no additional discount is applied to cached tokens in batch requests.

---

## Firecrawl + Groq: AI-Powered Web Scraping & Data Extraction