# Sampling: Overview on our sampling settings

Here, we will discuss the sampling settings that influence the output of Large Language Models (LLMs). This guide covers parameters such as **Temperature**, **N**, **Top P**, **Presence Penalty**, and **Frequency Penalty**, and explains how to adjust them. Whether you aim to generate creative content or ensure accurate responses, understanding these settings is key.

Let's explore each parameter and learn how to fine-tune LLM outputs effectively.

<details>

<summary><b>N Completions</b></summary>

## N Completions

**N** represents the number of completions to return for each request. This parameter is useful when you want to generate multiple responses for a single input. Each completion will be a unique response generated by the model, providing a variety of outputs to choose from.

### Key Points

- **Multiple Responses**: By setting `N` to a value greater than 1, you can get multiple responses for the same input.
- **Cost Efficiency**: Input tokens are only billed once, regardless of the number of completions requested. This makes it cost-effective to explore different possibilities.

### Example

Here's an example of how to use the `N` parameter in the API:

```py

from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
model = "ministral-3b-latest"

client = Mistral(api_key=api_key)

chat_response = client.chat.complete(
    model=model,
    messages=[
        {
            "role": "user",
            "content": "What is the best mythical creature? Answer with a single word.",
        },
    ],
    temperature = 1, # Increasing randomness and diversity of the output, this is required to be higher than 0 to have diverse outputs
    n = 10 # Number of completions
)

for i, choice in enumerate(chat_response.choices):
    print(choice.message.content)
```

### Output

```
Phoenix.
Dragon
Dragon
Unicorn
Unicorn
Phoenix
Unicorn
Dragon
Dragon.
Unicorn
```

In this example, the model generates 10 responses for the same input prompt. This allows you to see a variety of possible answers and choose the one that best fits your needs.

</details>

<details>

<summary><b>Temperature</b></summary>

## Temperature

**Temperature** in Large Language Models (LLMs) controls output diversity. Lower values make the model more deterministic, focusing on likely responses for accuracy. Higher values increase creativity and diversity. During text generation, LLMs predict tokens with associated probabilities using a softmax function. Temperature scales these probabilities: higher temperatures flatten the distribution, making outputs more varied, while lower temperatures amplify differences, favoring more likely tokens.

## Visualization

To better understand the underlying principle and impact it has on the probability distribution, here is a visualisation of the Temperature with a simple prompt:
    *"What is the best mythical creature? Answer with a single word."*

<div style={{ textAlign: 'center' }}>
  <img src="/img/barplot.png" alt="Example Image" width="800"/>

  <sub><sup>Barplot example comparing the distribution with different `Temperature` values and the top 5 tokens using Mistral 7B at 4 bits precision.</sup></sub>
</div>

**Temperature** significantly affects the probability distribution in LLMs. At a Temperature of 0, the model always outputs the most likely token, e.g., "**Dragon**". Increasing the Temperature to 0.2 introduces variability, allowing for tokens like "**Un**" (as in "**Un**icorn"). Further increases reveal more diverse tokens: the third token might still be "**Drag**" (for "**Drag**on"), but the fourth could start "**Peg**asus", and the fifth, "**Phoenix**". Higher Temperatures make less likely tokens more probable, enhancing the diversity of the model's output.

## API
You can set a temperature value easily via our clients, let's experiment with our API.
```py

from mistralai import Mistral

api_key = os.environ["MISTRAL_API_KEY"]
model = "ministral-3b-latest"

client = Mistral(api_key=api_key)

chat_response = client.chat.complete(
    model = model,
    messages = [
        {
            "role": "user",
            "content": "What is the best mythical creature? Answer with a single word.",
        },
    ],
    temperature = 0.1,
    n = 10
)

for i, choice in enumerate(chat_response.choices):
    print(choice.message.content)
```
```
Dragon
Dragon
Dragon
Dragon
Dragon
Dragon
Dragon
Dragon
Dragon
Dragon
```
The model answered mostly with Dragon! Lets try with a higher temperature to try to have more diverse outputs, let's set it to `temperature = 1`.
```
Unicorn
Dragon
Phoenix
Unicorn
Dragon
Phoenix.
Dragon.
Phoenix
Dragon
Unicorn.
```

The outputs ended much more diverse, the model answering with a different creature more frequently, we have "Dragon", "Unicorn" and "Phoenix".

## The Best Temperature

There's no one-size-fits-all Temperature for all use cases, but some guidelines can help you find the best for your applications.

### Determinism

- **Requirements**: Tasks needing consistent, accurate responses, such as Mathematics, Classification, Healthcare, or Reasoning.
- **Temperature**: Use very low values, sometimes not null to add slight uniqueness.

For example, a classification agent should use a Temperature of 0 to always pick the best token. A math chat assistant might use very low Temperature values to avoid repetition while maintaining accuracy.

### Creativity

- **Requirements**: Tasks needing diverse, unique text, like brainstorming, writing novels, creating slogans, or roleplaying.
- **Temperature**: Use high values, but avoid excessively high Temperatures to prevent randomness and nonsense outputs.

Consider the trade-off: higher Temperatures increase creativity but may decrease quality and accuracy.

</details>

<details>

<summary><b>Top P</b></summary>