# Understanding and Optimizing Latency on Groq
### Overview
Latency is a critical factor when building production applications with Large Language Models (LLMs). This guide helps you understand, measure, and optimize latency across your Groq-powered applications, providing a comprehensive foundation for production deployment.

## Understanding Latency in LLM Applications
### Key Metrics in Groq Console

Your Groq Console [dashboard](/dashboard) contains pages for metrics, usage, logs, and more. When you view your Groq API request logs, you'll see important data regarding your API requests. The following are ones relevant to latency that we'll call out and define:
<br/>
- **Time to First Token (TTFT)**: Time from API request sent to first token received from the model
- **Latency**: Total server time from API request to completion
- **Input Tokens**: Number of tokens provided to the model (e.g. system prompt, user query, assistant message), directly affecting TTFT
- **Output Tokens**: Number of tokens generated, impacting total latency
- **Tokens/Second**: Generation speed of model outputs

### The Complete Latency Picture
The users of the applications you build with APIs in general experience total latency that includes:
<br/>
`User-Experienced Latency = Network Latency + Server-side Latency`
<br/>
<small>
Server-side Latency is <a href="https://console.groq.com/dashboard/logs" target="_blank">shown in the console</a>.
</small>
<br/>

**Important**: Groq Console metrics show server-side latency only. Client-side network latency measurement examples are provided in the Network Latency Analysis section below.
<br/>
We recommend visiting [Artificial Analysis](https://artificialanalysis.ai/providers/groq) for third-party performance benchmarks across all models hosted on GroqCloud, including end-to-end response time.

## How Input Size Affects TTFT

Input token count is the primary driver of TTFT performance. Understanding this relationship allows developers to optimize prompt design and context management for predictable latency characteristics.

### The Scaling Pattern

TTFT demonstrates linear scaling characteristics across input token ranges:

- **Minimal inputs (100 tokens)**: Consistently fast TTFT across all model sizes
- **Standard contexts (1K tokens)**: TTFT remains highly responsive
- **Large contexts (10K tokens)**: TTFT increases but remains competitive
- **Maximum contexts (100K tokens)**: TTFT increases to process all the input tokens

### Model Architecture Impact on TTFT

Model architecture fundamentally determines input processing characteristics, with parameter count, attention mechanisms, and specialized capabilities creating distinct performance profiles.
<br/>
**Parameter Scaling Patterns**:
- **8B models**: Minimal TTFT variance across context lengths, optimal for latency-critical applications
- **32B models**: Linear TTFT scaling with manageable overhead for balanced workloads
- **70B and above**: Exponential TTFT increases at maximum context, requiring context management
<br/>
**Architecture-Specific Considerations**:
- **Reasoning models**: Additional computational overhead for chain-of-thought processing increases baseline latency by 10-40%
- **Mixture of Experts (MoE)**: Router computation adds fixed latency cost but maintains competitive TTFT scaling
- **Vision-language models**: Image encoding preprocessing significantly impacts TTFT independent of text token count

### Model Selection Decision Tree
<br/>
```python
# Model Selection Logic

if latency_requirement == "fastest" and quality_need == "acceptable":
    return "8B_models" 
elif reasoning_required and latency_requirement != "fastest":
    return "reasoning_models"  
elif quality_need == "balanced" and latency_requirement == "balanced":
    return "32B_models" 
else:
    return "70B_models"
```

## Output Token Generation Dynamics

Sequential token generation represents the primary latency bottleneck in LLM inference. Unlike parallel input processing, each output token requires a complete forward pass through the model, creating linear scaling between output length and total generation time. Token generation demands significantly higher computational resources than input processing due to the autoregressive nature of transformer architectures.

### Architectural Performance Characteristics

Groq's LPU architecture delivers consistent generation speeds optimized for production workloads. Performance characteristics follow predictable patterns that enable reliable capacity planning and optimization decisions.
<br/>

**Generation Speed Factors**:
- **Model size**: Inverse relationship between parameter count and generation speed
- **Context length**: Quadratic attention complexity degrades speeds at extended contexts
- **Output complexity**: Mathematical reasoning and structured outputs reduce effective throughput

### Calculating End-to-End Latency
<br/>
```
Total Latency = TTFT + Decoding Time + Network Round Trip
```
<br/>
<br/>
Where:
- **TTFT** = Queueing Time + Prompt Prefill Time
- **Decoding Time** = Output Tokens / Generation Speed
- **Network Round Trip** = Client-to-server communication overhead
## Infrastructure Optimization

### Network Latency Analysis

Network latency can significantly impact user-experienced performance. If client-measured total latency substantially exceeds server-side metrics returned in API responses, network optimization becomes critical.
<br/>
**Diagnostic Approach**:

**Response Header Analysis**:

The `x-groq-region` header confirms which datacenter processed your request, enabling latency correlation with geographic proximity. This information helps you understand if your requests are being routed to the optimal datacenter for your location.

### Context Length Management

As shown above, TTFT scales with input length. End users can employ several prompting strategies to optimize context usage and reduce latency:

- **Prompt Chaining**: Decompose complex tasks into sequential subtasks where each prompt's output feeds the next. This technique reduces individual prompt length while maintaining context flow. Example: First prompt extracts relevant quotes from documents, second prompt answers questions using those quotes. Improves transparency and enables easier debugging.

- **Zero-Shot vs Few-Shot Selection**: For concise, well-defined tasks, zero-shot prompting ("Classify this sentiment") minimizes context length while leveraging model capabilities. Reserve few-shot examples only when task-specific patterns are essential, as examples consume significant tokens.

- **Strategic Context Prioritization**: Place critical information at prompt beginning or end, as models perform best with information in these positions. Use clear separators (triple quotes, headers) to structure complex prompts and help models focus on relevant sections.

<br/>
For detailed implementation strategies and examples, consult the [Groq Prompt Engineering Documentation](/docs/prompting) and [Prompting Patterns Guide](/docs/prompting/patterns).

## Groq's Processing Options

### Service Tier Architecture

Groq offers three service tiers that influence latency characteristics and processing behavior:
<br/>

**On-Demand Processing** (`"service_tier":"on_demand"`): For real-time applications requiring guaranteed processing, the standard API delivers:
- Industry-leading low latency with consistent performance
- Streaming support for immediate perceived response
- Controlled rate limits to ensure fairness and consistent experience

**Flex Processing** (`"service_tier":"flex"`): [Flex Processing](/docs/flex-processing) optimizes for throughput with higher request volumes in exchange for occasional failures. Flex processing gives developers 10x their current rate limits, as system capacity allows, with rapid timeouts when resources are constrained.

_Best for_: High-volume workloads, content pipelines, variable demand spikes.
<br/>
**Auto Processing** (`"service_tier":"auto"`): Auto Processing uses on-demand rate limits initially, then automatically falls back to flex tier processing if those limits are exceeded. This provides optimal balance between guaranteed processing and high throughput.

_Best for_: Applications requiring both reliability and scalability during demand spikes.

### Processing Tier Selection Logic

```python
# Processing Tier Selection Logic  

if real_time_required and throughput_need != "high":
    return "on_demand"  
elif throughput_need == "high" and cost_priority != "critical":
    return "flex"  
elif real_time_required and throughput_need == "variable":
    return "auto"  
elif cost_priority == "critical":
    return "batch"  
else:
    return "on_demand" 
```

### Batch Processing

[Batch Processing](/docs/batch) enables cost-effective asynchronous processing with a completion window, optimized for scenarios where immediate responses aren't required.
<br/> 
**Batch API Overview**: The Groq Batch API processes large-scale workloads asynchronously, offering significant advantages for high-volume use cases:

- **Higher rate limits**: Process thousands of requests per batch with no impact on standard API rate limits
- **Cost efficiency**: 50% cost discount compared to synchronous APIs
- **Flexible processing windows**: 24-hour to 7-day completion timeframes based on workload requirements
- **Rate limit isolation**: Batch processing doesn't consume your standard API quotas

<br/>
**Latency Considerations**: While batch processing trades immediate response for efficiency, understanding its latency characteristics helps optimize workload planning:

- **Submission latency**: Minimal overhead for batch job creation and validation
- **Queue processing**: Variable based on system load and batch size
- **Completion notification**: Webhook or polling-based status updates
- **Result retrieval**: Standard API latency for downloading completed outputs

<br/>
**Optimal Use Cases**: Batch processing excels for workloads where processing time flexibility enables significant cost and throughput benefits: large dataset analysis, content generation pipelines, model evaluation suites, and scheduled data enrichment tasks.

## Streaming Implementation

### Server-Sent Events Best Practices

Implement streaming to improve perceived latency:
<br/>

**Streaming Implementation**:

```python
import os
from groq import Groq

def stream_response(prompt):
    client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
    stream = client.chat.completions.create(
        model="meta-llama/llama-4-scout-17b-16e-instruct",
        messages=[{"role": "user", "content": prompt}],
        stream=True
    )
    
    for chunk in stream:
        if chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content

# Example usage with concrete prompt
prompt = "Write a short story about a robot learning to paint in exactly 3 sentences."
for token in stream_response(prompt):
    print(token, end='', flush=True)
```

```javascript
import Groq from "groq-sdk";

async function streamResponse(prompt) {
    const groq = new Groq({
        apiKey: process.env.GROQ_API_KEY
    });
    
    const stream = await groq.chat.completions.create({
        model: "meta-llama/llama-4-scout-17b-16e-instruct",
        messages: [{ role: "user", content: prompt }],
        stream: true
    });

    for await (const chunk of stream) {
        if (chunk.choices[0]?.delta?.content) {
            process.stdout.write(chunk.choices[0].delta.content);
        }
    }
}

// Example usage with concrete prompt
const prompt = "Write a short story about a robot learning to paint in exactly 3 sentences.";
streamResponse(prompt);
```
<br/>
**Key Benefits**:
- Users see immediate response initiation
- Better user engagement and experience
- Error handling during generation

_Best for_: Interactive applications requiring immediate feedback, user-facing chatbots, real-time content generation where perceived responsiveness is critical.

## Next Steps

Go over to our [Production-Ready Checklist](/docs/production-readiness/production-ready-checklist) and start the process of getting your AI applications scaled up to all your users with consistent performance.
<br/>
Building something amazing? Need help optimizing? Our team is here to help you achieve production-ready performance at scale. Join our [developer community](https://community.groq.com)!

---

## Security Onboarding