You may now close this window.
" self.wfile.write(response_html.encode()) threading.Thread(target=httpd.shutdown).start() except Exception: self.send_response(500) self.end_headers() server_address = ("localhost", CALLBACK_PORT) httpd = HTTPServer(server_address, OAuthCallbackHandler) threading.Thread(target=httpd.serve_forever).start() redirect_url = f"http://localhost:{CALLBACK_PORT}/oauth/callback" return httpd, redirect_url, auth_response ``` #### Step 3: Define Server URL and Create MCP Client We define the URL for the remote MCP server and create an MCP client to connect to it. ```python async def main(): # Initialize the Mistral client with your API key api_key = os.environ["MISTRAL_API_KEY"] client = Mistral(api_key) # Define the URL for the remote MCP server server_url = "https://mcp.linear.app/sse" mcp_client = MCPClientSSE(sse_params=SSEServerParams(url=server_url)) ``` #### Step 4: Handle Authentication We handle the authentication process, including setting up a callback event and event loop, checking if authentication is required, and managing the OAuth flow. ```python # Set up a callback event and event loop callback_event = asyncio.Event() event_loop = asyncio.get_event_loop() # Check if authentication is required if await mcp_client.requires_auth(): # Set up a callback server and handle OAuth flow httpd, redirect_url, auth_response = run_callback_server( callback_func=lambda: event_loop.call_soon_threadsafe(callback_event.set) ) try: # Build OAuth parameters and get the login URL oauth_params = await build_oauth_params( mcp_client.base_url, redirect_url=redirect_url ) mcp_client.set_oauth_params(oauth_params=oauth_params) login_url, state = await mcp_client.get_auth_url_and_state(redirect_url) # Open the login URL in a web browser print("Please go to this URL and authorize the application:", login_url) webbrowser.open(login_url, new=2) await callback_event.wait() # Exchange the authorization code for a token mcp_client = MCPClientSSE( sse_params=SSEServerParams(url=server_url), oauth_params=oauth_params, ) token = await mcp_client.get_token_from_auth_response( auth_response["url"], redirect_url=redirect_url, state=state ) mcp_client.set_auth_token(token) except Exception as e: print(f"Error during authentication: {e}") finally: httpd.shutdown() httpd.server_close() ``` #### Step 5: Create a Run Context and Register MCP Client We create a Run Context for the agent and register the MCP client with it. ```python # Create a run context for the agent async with RunContext( model=MODEL, ) as run_ctx: # Register the MCP client with the run context await run_ctx.register_mcp_client(mcp_client=mcp_client) ``` #### Step 6: Run the Agent and Print Results Finally, we run the agent with a query and print the results. ```python # Run the agent with a query run_result = await client.beta.conversations.run_async( run_ctx=run_ctx, inputs="Tell me which projects do I have in my workspace?", ) # Print the final response print(f"Final Response: {run_result.output_as_text}") if __name__ == "__main__": asyncio.run(main()) ```
2. (alternative) Manually edit config.json
- Click on the gear icon in the bottom right corner of the Continue window to open `~/.continue/config.json` (MacOS) / `%userprofile%\.continue\config.json` (Windows)
- Log in and request a Codestral API key on Mistral AI's La Plateforme [here](https://console.mistral.ai/codestral)
- To use Codestral as your model for both `autocomplete` and `chat`, replace `[API_KEY]` with your Mistral API key below and add it to your `config.json` file:
```json title="~/.continue/config.json"
{
"models": [
{
"title": "Codestral",
"provider": "mistral",
"model": "codestral-latest",
"apiKey": "[API_KEY]"
}
],
"tabAutocompleteModel": {
"title": "Codestral",
"provider": "mistral",
"model": "codestral-latest",
"apiKey": "[API_KEY]"
}
}
```
If you run into any issues or have any questions, please join our Discord and post in `#help` channel [here](https://discord.gg/EfJEfdFnDQ)
## Codestral Embed API
To generate code embeddings using Mistral AI's embeddings API, we can make a request to the API endpoint and specify the embedding model `codestral-embed`, along with providing a list of input texts. The API will then return the corresponding embeddings as numerical vectors, which can be used for further analysis or processing in NLP applications.
We also provide `output_dtype` and `output_dimension` parameters that allow you to control the type and dimensional size of your embeddings.
### Output DType
`output_dtype` allows you to select the precision and format of the embeddings, enabling you to obtain embeddings with your desired level of numerical accuracy and representation.
The accepted dtypes are:
- **float** (default): A list of 32-bit (4-byte) single-precision floating-point numbers. Provides the highest precision and retrieval accuracy.
- **int8**: A list of 8-bit (1-byte) integers ranging from -128 to 127.
- **uint8**: A list of 8-bit (1-byte) integers ranging from 0 to 255.
- **binary**: A list of 8-bit integers that represent bit-packed, quantized single-bit embedding values using the `int8` type. The length of the returned list of integers is 1/8 of `output_dimension`. This type uses the offset binary method.
- **ubinary**: Similar to `binary`, but uses the `uint8` type for bit-packed, quantized single-bit embedding values.
### Output Dimension
`output_dimension` allows you to select a specific size for the embedding, enabling you to obtain an embedding of your chosen dimension, **defaults to 1536** and has a **maximum value of 3072**.
For any integer target dimension n, you can choose to retain the first n dimensions. These dimensions are ordered by relevance, and the first n are selected for a smooth trade-off between quality and cost.
### t-SNE embeddings visualization
We mentioned previously that our embeddings have 1536 dimensions, which makes them impossible to visualize directly. Thus, in order to visualize our embeddings, we can use a dimensionality reduction technique such as t-SNE to project our embeddings into a lower-dimensional space that is easier to visualize.
In this example, we transform our embeddings to 2 dimensions and create a 2D scatter plot showing the relationships among embeddings of different problems.
```python
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=0).fit_transform(np.array(df['embeddings'].to_list()))
ax = sns.scatterplot(x=tsne[:, 0], y=tsne[:, 1], hue=np.array(df['Name'].to_list()))
sns.move_legend(ax, 'upper left', bbox_to_anchor=(1, 1))
```
## Retrieval
Our embedding model excels in retrieval tasks, as it is trained with retrieval in mind. Embeddings are also incredibly helpful in implementing retrieval-augmented generation (RAG) systems, which use retrieved relevant information from a knowledge base to generate responses. At a high-level, we embed a knowledge base, whether it is a local directory, text files, or internal wikis, into text embeddings and store them in a vector database. Then, based on the user's query, we retrieve the most similar embeddings, which represent the relevant information from the knowledge base. Finally, we feed these relevant embeddings to a large language model to generate a response that is tailored to the user's query and context. If you are interested in learning more about how RAG systems work and how to implement a basic RAG, check out our [previous guide](/guides/rag) on this topic.
## Cookbooks
For more information and guides on how to make use of our embedding sdk, we have the following cookbooks:
- [Embedding Cookbook](https://colab.research.google.com/github/mistralai/cookbook/blob/main/mistral/embeddings/code_embedding.ipynb)
- [Dequantization Cookbook](https://colab.research.google.com/github/mistralai/cookbook/blob/main/mistral/embeddings/dequantization.ipynb)
[Embeddings Overview]
Source: https://docs.mistral.ai/docs/capabilities/embeddings/embeddings_overview
## Mistral Embed API
To generate text embeddings using Mistral AI's embeddings API, we can make a request to the API endpoint and specify the embedding model `mistral-embed`, along with providing a list of input texts. The API will then return the corresponding embeddings as numerical vectors, which can be used for further analysis or processing in NLP applications.
### t-SNE embeddings visualization
We mentioned previously that our embeddings have 1024 dimensions, which makes them impossible to visualize directly. Thus, in order to visualize our embeddings, we can use a dimensionality reduction technique such as t-SNE to project our embeddings into a lower-dimensional space that is easier to visualize.
In this example, we transform our embeddings to 2 dimensions and create a 2D scatter plot showing the relationships among embeddings of different diseases.
```python
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=0).fit_transform(np.array(df['embeddings'].to_list()))
ax = sns.scatterplot(x=tsne[:, 0], y=tsne[:, 1], hue=np.array(df['label'].to_list()))
sns.move_legend(ax, 'upper left', bbox_to_anchor=(1, 1))
```
### Comparison with fastText
We can compare it with fastText, a popular open-source embeddings model. However, when examining the t-SNE embeddings plot, we notice that fastText embeddings fail to create clear separations between data points with matching labels.
```python
fasttext.util.download_model('en', if_exists='ignore') # English
ft = fasttext.load_model('cc.en.300.bin')
df['fasttext_embeddings'] = df['text'].apply(lambda x: ft.get_word_vector(x).tolist())
tsne = TSNE(n_components=2, random_state=0).fit_transform(np.array(df['fasttext_embeddings'].to_list()))
ax = sns.scatterplot(x=tsne[:, 0], y=tsne[:, 1], hue=np.array(df['label'].to_list()))
sns.move_legend(ax, 'upper left', bbox_to_anchor=(1, 1))
```
## Classification
Text embeddings can be used as input features in machine learning models, such as classification and clustering. In this example, we use a classification model to predict the disease labels from the embeddings of disease description text.
```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
---
# Create a train / test split
train_x, test_x, train_y, test_y = train_test_split(
df["embeddings"], df["label"], test_size=0.2
)
---
# Normalize features
scaler = StandardScaler()
train_x = scaler.fit_transform(train_x.to_list())
test_x = scaler.transform(test_x.to_list())
---
# Train a classifier and compute the test accuracy
---
# For a real problem, C should be properly cross validated and the confusion matrix analyzed
clf = LogisticRegression(random_state=0, C=1.0, max_iter=500).fit(
train_x, train_y.to_list()
)
---
# clf = LogisticRegression(random_state=0, C=1.0, max_iter=1000, solver='sag').fit(train_x, train_y)
print(f"Precision: {100*np.mean(clf.predict(test_x) == test_y.to_list()):.2f}%")
```
Output
```
Precision: 98.75%
```
After we trained the classifier with our embeddings data, we can try classify other text:
```python
---
# Classify a single example
text = "I've been experiencing frequent headaches and vision problems."
clf.predict([get_text_embedding([text])])
```
Output
```
'Migraine'
```
### Comparison with fastText
Additionally, let's take a look at the performance using fastText embeddings in this classification task. It appears that the classification model achieves better performance with Mistral AI Embeddings model as compared to using fastText embeddings.
```python
---
# Create a train / test split
train_x, test_x, train_y, test_y = train_test_split(
df["fasttext_embeddings"], df["label"], test_size=0.2
)
---
# Normalize features
scaler = StandardScaler()
train_x = scaler.fit_transform(train_x.to_list())
test_x = scaler.transform(test_x.to_list())
---
# Train a classifier and compute the test accuracy
---
# For a real problem, C should be properly cross validated and the confusion matrix analyzed
clf = LogisticRegression(random_state=0, C=1.0, max_iter=500).fit(
train_x, train_y.to_list()
)
---
# clf = LogisticRegression(random_state=0, C=1.0, max_iter=1000, solver='sag').fit(train_x, train_y)
print(f"Precision: {100*np.mean(clf.predict(test_x) == test_y.to_list()):.2f}%")
```
Output
```
Precision: 86.25%
```
## Clustering
What if we don't have disease labels? One approach to gain insights from the data is through clustering. Clustering is an unsupervised machine learning technique that groups similar data points together based on their similarity with respect to certain features. In the context of text embeddings, we can use the distance between each embedding as a measure of similarity, and group together data points with embeddings that are close to each other in the high-dimensional space.
Since we already know there are 24 clusters, let's use the K-means clustering with 24 clusters. Then we can inspect a few examples and verify whether the examples in a single cluster are similar to one another. For example, take a look at the first three rows of cluster 23. We can see that they look very similar in terms of symptoms.
```python
from sklearn.cluster import KMeans
model = KMeans(n_clusters=24, max_iter=1000)
model.fit(df['embeddings'].to_list())
df["cluster"] = model.labels_
print(*df[df.cluster==23].text.head(3), sep='\n\n')
```
Output:
```
I have been feeling extremely tired and weak, and I've also been coughing a lot with difficulty breathing. My fever is very high, and I'm producing a lot of mucus when I cough.
I've got a cough that won't go away, and I'm exhausted. I've been coughing up thick mucous and my fever is also pretty high.
I have a persistent cough and have been feeling quite fatigued. My fever is through the roof, and I'm having trouble breathing. When I cough, I also cough up a lot of mucous.
```
## Retrieval
Our embedding model excels in retrieval tasks, as it is trained with retrieval in mind. Embeddings are also incredibly helpful in implementing retrieval-augmented generation (RAG) systems, which use retrieved relevant information from a knowledge base to generate responses. At a high-level, we embed a knowledge base, whether it is a local directory, text files, or internal wikis, into text embeddings and store them in a vector database. Then, based on the user's query, we retrieve the most similar embeddings, which represent the relevant information from the knowledge base. Finally, we feed these relevant embeddings to a large language model to generate a response that is tailored to the user's query and context. If you are interested in learning more about how RAG systems work and how to implement a basic RAG, check out our [previous guide](/guides/rag) on this topic.
[Classifier Factory]
Source: https://docs.mistral.ai/docs/capabilities/finetuning/classifier-factory
In various domains and enterprises, classification models play a crucial role in enhancing efficiency, improving user experience, and ensuring compliance. These models serve diverse purposes, including but not limited to:
- **Moderation**: Classification models are essential for moderating services and classifying unwanted content. For instance, our [moderation service](../../guardrailing/#moderation-api) helps in identifying and filtering inappropriate or harmful content in real-time, ensuring a safe and respectful environment for users.
- **Intent Detection**: These models help in understanding user intent and behavior. By analyzing user interactions, they can predict the user's next actions or needs, enabling personalized recommendations and improved customer support.
- **Sentiment Analysis**: Emotion and sentiment detection models analyze text data to determine the emotional tone behind words. This is particularly useful in social media monitoring, customer feedback analysis, and market research, where understanding public sentiment can drive strategic decisions.
- **Data Clustering**: Classification models can group similar data points together, aiding in data organization and pattern recognition. This is beneficial in market segmentation, where businesses can identify distinct customer groups for targeted marketing campaigns.
- **Fraud Detection**: In the financial sector, classification models help in identifying fraudulent transactions by analyzing patterns and anomalies in transaction data. This ensures the security and integrity of financial systems.
- **Spam Filtering**: Email services use classification models to filter out spam emails, ensuring that users receive only relevant and safe communications.
- **Recommendation Systems**: Classification models power recommendation engines by categorizing user preferences and suggesting relevant products, movies, or content based on past behavior and preferences.
By leveraging classification models, organizations can make data-driven decisions, improve operational efficiency, and deliver better products and services to their customers.
For this reason, we designed a friendly and easy way to make your own classifiers. Leveraging our small but highly efficient models and training methods, the Classifier Factory is both available directly in [la plateforme](https://console.mistral.ai/build/finetuned-models) and our API.
## Dataset Format
Data must be stored in JSON Lines (`.jsonl`) files, which allow storing multiple JSON objects, each on a new line.
We provide two endpoints:
- `v1/classifications`: To classify raw text.
- `v1/chat/classifications`: To classify chats and multi-turn interactions.
There are 2 main kinds of classification models:
- Single Target
- Multi-Target
### 1. Single Target
For single label classification, data must have the label name and the value for that corresponding label. Example:
Function calling allows Mistral models to connect to external tools. By integrating Mistral models with external tools such as user defined functions or APIs, users can easily build applications catering to specific use cases and practical problems. In this guide, for instance, we wrote two functions for tracking payment status and payment date. We can use these two tools to provide answers for payment-related queries.
Alternatively, if you are already in La Plateforme, click on your name in the bottom left section,
followed by selecting "Create or join workspace".
You can create your workspace for your own or your organization.
## Switch to a workspace
You can switch between your personal workspace and your organization workspace.
## Invite members to your organization
To invite members to your organization, navigate to "Workspace - Members"
and click "Invite a new member".
[La Plateforme]
Source: https://docs.mistral.ai/docs/deployment/laplateforme/overview
[platform_url]: https://console.mistral.ai/
[deployment_img]: /img/deployment.png
[deployment_url]: https://console.mistral.ai/
Mistral AI currently provides three types of access to Large Language Models:
- **La Plateforme**: We provide API endpoints through [La Plateforme][platform_url] providing pay-as-you-go access to our latest models.
- **Cloud**: You can access Mistral AI models via your preferred [cloud platforms](/deployment/cloud/overview/).
- **Self-deployment**: You can self-deploy our open-weights models on your own on-premise infrastructure. Our open weights models are available under the [Apache 2.0](https://github.com/apache/.github/blob/main/LICENSE) License, available on [Hugging Face](https://huggingface.co/mistralai) or directly from [the documentation](/getting-started/models/weights).
[![deployment_img]][deployment_url]
### API Access with the La Plateforme
You will need to activate payments on your account to enable your API keys in the [La Plateforme][platform_url]. Check out the [Quickstart](/getting-started/quickstart/) guide to get started with your first Mistral API request.
Explore the capabilities of our models:
- [Completion](/capabilities/completion)
- [Embeddings](/capabilities/embeddings/overview)
- [Function calling](/capabilities/function_calling)
- [JSON mode](/capabilities/structured-output/json_mode)
- [Guardrailing](/capabilities/guardrailing)
### Cloud-based deployments
For a comprehensive list of options to deploy and consume Mistral AI models on the cloud, head on to the **[cloud deployment section](/deployment/cloud/overview)**.
### Raw model weights
Raw model weights can be used in several ways:
- For self-deployment, on cloud or on premise, using either [TensorRT-LLM](/deployment/self-deployment/trt) or [vLLM](/deployment/self-deployment/vllm), head on to **[Deployment](/deployment/self-deployment/skypilot)**
- For research, head-on to our [reference implementation repository](https://github.com/mistralai/mistral-src),
- For local deployment on consumer grade hardware, check out the [llama.cpp](https://github.com/ggerganov/llama.cpp) project or [Ollama](https://ollama.ai/).
[Pricing]
Source: https://docs.mistral.ai/docs/deployment/laplateforme/pricing
:::note[ ]
Please refer to the [pricing page](https://mistral.ai/pricing#api-pricing) for detailed information on costs.
:::
[Rate limit and usage tiers]
Source: https://docs.mistral.ai/docs/deployment/laplateforme/tier
:::note[ ]
Please visit https://admin.mistral.ai/plateforme/limits for detailed information on the current rate limit and usage tiers for your workspace.
:::
## How does rate limits rate work?
To prevent misuse and manage the capacity of our API, we have implemented limits on how much a workspace can utilize the Mistral API.
We offer two types of rate limits:
- Requests per second (RPS)
- Tokens per minute/month
Key points to note:
- Rate limits are set at the workspace level.
- Limits are defined by usage tier, where each tier is associated with a different set of rate limits.
- In case you need to raise your usage limits, please feel free to contact us by utilizing the support button, providing details about your specific use case.
## Usage tiers
You can view the rate and usage limits for your workspace under the [limits](https://admin.mistral.ai/plateforme/limits) section on la Plateforme.
We offer various tiers on the platform, including a **free API tier** with restrictive rate limits. The free API tier is designed to allow you to try and explore our API. For actual projects and production use, we recommend upgrading to a higher tier.
[Deploy with Cerebrium]
Source: https://docs.mistral.ai/docs/deployment/self-deployment/cerebrium
[Cerebrium](https://www.cerebrium.ai/) is a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications. They offer Serverless GPU's with low cold start times with over 12 varieties of GPU chips that auto scale and you only pay for the compute you use.
## Setup Cerebrium
First, we install Cerebrium and login to get authenticated.
```bash
pip install cerebrium
cerebrium login
```
Then let us create our first project
```bash
cerebrium init mistral-vllm
```
## Setup Environment and Hardware
You set up your environment and hardware in the **cerebrium.toml** file that was created using the init function above. Here we are using a Ampere A10 GPU etc.
You can read more [here](https://docs.cerebrium.ai/cerebrium/environments/custom-images)
```toml
[cerebrium.deployment]
name = "mistral-vllm"
python_version = "3.11"
docker_base_image_url = "debian:bookworm-slim"
include = "[./*, main.py, cerebrium.toml]"
exclude = "[.*]"
[cerebrium.hardware]
cpu = 2
memory = 14.0
compute = "AMPERE_A10"
gpu_count = 1
provider = "aws"
region = "us-east-1"
[cerebrium.dependencies.pip]
sentencepiece = "latest"
torch = ">=2.0.0"
vllm = "latest"
transformers = ">=4.35.0"
accelerate = "latest"
xformers = "latest"
```
## Setup inference
Running code in Cerebrium is like writing normal python with no special syntax. In your **main.py** specify the following:
```python
from vllm import LLM, SamplingParams
from huggingface_hub import login
from cerebrium import get_secret
---
# Your huggingface token (HF_AUTH_TOKEN) should be stored in your project secrets on your Cerebrium dashboard
login(token=get_secret("HF_AUTH_TOKEN"))
---
# Initialize the model
llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.3", dtype="bfloat16", max_model_len=20000, gpu_memory_utilization=0.9)
```
We need to add our Hugging Face token to our [Cerebrium Secrets](https://docs.cerebrium.ai/cerebrium/environments/using-secrets) since using the Mistral model requires authentication. Please make sure the Huggingface token you added, has WRITE permissions. On first deploy, it will download the model and store it on disk therefore for subsequent calls it will load the model from disk.
Add the following to your main.py:
```python
def run(prompt: str, temperature: float = 0.8, top_p: float = 0.75, top_k: int = 40, max_tokens: int = 256, frequency_penalty: int = 1):
sampling_params = SamplingParams(
temperature=temperature,
top_p=top_p,
top_k=top_k,
max_tokens=max_tokens,
frequency_penalty=frequency_penalty
)
outputs = llm.generate([item.prompt], sampling_params)
generated_text = []
for output in outputs:
generated_text.append(output.outputs[0].text)
return {"result": generated_text}
```
Every function in Cerebrium is callable through and API endpoint. Code at the top most layer (ie: not in a function) is instantiated only when the container is spun up the first time so for subsequent calls, it will simply run the code defined in the function you call.
Our final main.py should look like this:
```python
from vllm import LLM, SamplingParams
from huggingface_hub import login
from cerebrium import get_secret
---
# Your huggingface token (HF_AUTH_TOKEN) should be stored in your project secrets on your Cerebrium dashboard
login(token=get_secret("HF_AUTH_TOKEN"))
---
# Initialize the model
llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.3", dtype="bfloat16", max_model_len=20000, gpu_memory_utilization=0.9)
def run(prompt: str, temperature: float = 0.8, top_p: float = 0.75, top_k: int = 40, max_tokens: int = 256, frequency_penalty: int = 1):
sampling_params = SamplingParams(
temperature=temperature,
top_p=top_p,
top_k=top_k,
max_tokens=max_tokens,
frequency_penalty=frequency_penalty
)
outputs = llm.generate([item.prompt], sampling_params)
generated_text = []
for output in outputs:
generated_text.append(output.outputs[0].text)
return {"result": generated_text}
```
## Run on the cloud
```bash
cerebrium deploy
```
You will see your application deploy, install pip packages and download the model. Once completed it will output a CURL request you can use to call your endpoint. Just remember to end
the url with the function you would like to call - in this case /run.
```CURL
curl --location --request POST 'https://api.cortex.cerebrium.ai/v4/p-
[Quickstart]
Source: https://docs.mistral.ai/docs/getting-started/quickstart
[platform_url]: https://console.mistral.ai/
:::tip[ ]
Looking for La Plateforme? Head to [console.mistral.ai][platform_url]
:::
## Account setup
- To get started, create a Mistral account or sign in at https://console.mistral.ai.
- Then, navigate to your "Organization" settings at https://admin.mistral.ai.
- To add your payment information and activate payments on your account, find the [billing](https://admin.mistral.ai/organization/billing) section under Administration.
- You can now manage all your [Workspaces](https://admin.mistral.ai/organization/workspaces) and Organization via this page.
- Return to https://console.mistral.ai once everything is settled.
- After that, go to the [API keys](https://console.mistral.ai/api-keys) page under your Workspace and create a new API key by clicking "Create new key". Make sure to copy the API key, save it securely, and do not share it with anyone.
## Getting started with Mistral AI API
Mistral AI API provides a seamless way for developers to integrate Mistral's state-of-the-art
models into their applications and production workflows with just a few lines of code.
Our API is currently available through [La Plateforme][platform_url].
You need to activate payments on your account to enable your API keys.
After a few moments, you will be able to use our `chat` endpoint:
## RAG from scratch
This section aims to guide you through the process of building a basic RAG from scratch. We have two goals: firstly, to offer users a comprehensive understanding of the internal workings of RAG and demystify the underlying mechanisms; secondly, to empower you with the essential foundations needed to build an RAG using the minimum required dependencies.
### Import needed packages
The first step is to install the packages `mistralai` and `faiss-cpu` and import the needed packages:
```python
from mistralai import Mistral
from getpass import getpass
api_key= getpass("Type your API Key")
client = Mistral(api_key=api_key)
```
### Get data
In this very simple example, we are getting data from an essay written by Paul Graham:
```python
response = requests.get('https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt')
text = response.text
```
We can also save the essay in a local file:
```python
f = open('essay.txt', 'w')
f.write(text)
f.close()
```
### Split document into chunks
In a RAG system, it is crucial to split the document into smaller chunks so that it's more effective to identify and retrieve the most relevant information in the retrieval process later. In this example, we simply split our text by character, combine 2048 characters into each chunk, and we get 37 chunks.
```python
chunk_size = 2048
chunks = [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)]
len(chunks)
```
**Output**
```
37
```
#### Considerations:
- **Chunk size**: Depending on your specific use case, it may be necessary to customize or experiment with different chunk sizes and chunk overlap to achieve optimal performance in RAG. For example, smaller chunks can be more beneficial in retrieval processes, as larger text chunks often contain filler text that can obscure the semantic representation. As such, using smaller text chunks in the retrieval process can enable the RAG system to identify and extract relevant information more effectively and accurately. However, it's worth considering the trade-offs that come with using smaller chunks, such as increasing processing time and computational resources.
- **How to split**: While the simplest method is to split the text by character, there are other options depending on the use case and document structure. For example, to avoid exceeding token limits in API calls, it may be necessary to split the text by tokens. To maintain the cohesiveness of the chunks, it can be useful to split the text by sentences, paragraphs, or HTML headers. If working with code, it's often recommended to split by meaningful code chunks for example using an Abstract Syntax Tree (AST) parser.
### Create embeddings for each text chunk
For each text chunk, we then need to create text embeddings, which are numeric representations of the text in the vector space. Words with similar meanings are expected to be in closer proximity or have a shorter distance in the vector space.
To create an embedding, use Mistral AI's embeddings API endpoint and the embedding model `mistral-embed`. We create a `get_text_embedding` to get the embedding from a single text chunk and then we use list comprehension to get text embeddings for all text chunks.
```python
def get_text_embedding(input):
embeddings_batch_response = client.embeddings.create(
model="mistral-embed",
inputs=input
)
return embeddings_batch_response.data[0].embedding
text_embeddings = np.array([get_text_embedding(chunk) for chunk in chunks])
```
### Load into a vector database
Once we get the text embeddings, a common practice is to store them in a vector database for efficient processing and retrieval. There are several vector database to choose from. In our simple example, we are using an open-source vector database Faiss, which allows for efficient similarity search.
With Faiss, we instantiate an instance of the Index class, which defines the indexing structure of the vector database. We then add the text embeddings to this indexing structure.
```python
d = text_embeddings.shape[1]
index = faiss.IndexFlatL2(d)
index.add(text_embeddings)
```
#### Considerations:
- **Vector database**: When selecting a vector database, there are several factors to consider including speed, scalability, cloud management, advanced filtering, and open-source vs. closed-source.
### Create embeddings for a question
Whenever users ask a question, we also need to create embeddings for this question using the same embedding models as before.
```python
question = "What were the two main things the author worked on before college?"
question_embeddings = np.array([get_text_embedding(question)])
```
#### Considerations:
- **Hypothetical Document Embeddings (HyDE)**: In some cases, the user's question might not be the most relevant query to use for identifying the relevant context. Instead, it maybe more effective to generate a hypothetical answer or a hypothetical document based on the user's query and use the embeddings of the generated text to retrieve similar text chunks.
### Retrieve similar chunks from the vector database
We can perform a search on the vector database with `index.search`, which takes two arguments: the first is the vector of the question embeddings, and the second is the number of similar vectors to retrieve. This function returns the distances and the indices of the most similar vectors to the question vector in the vector database. Then based on the returned indices, we can retrieve the actual relevant text chunks that correspond to those indices.
```python
D, I = index.search(question_embeddings, k=2) # distance, index
retrieved_chunk = [chunks[i] for i in I.tolist()[0]]
```
#### Considerations:
- **Retrieval methods**: There are a lot different retrieval strategies. In our example, we are showing a simple similarity search with embeddings. Sometimes when there is metadata available for the data, it's better to filter the data based on the metadata first before performing similarity search. There are also other statistical retrieval methods like TF-IDF and BM25 that use frequency and distribution of terms in the document to identify relevant text chunks.
- **Retrieved document**: Do we always retrieve individual text chunk as it is? Not always.
- Sometime, we would like to include more context around the actual retrieved text chunk. We call the actual retrieved text chunk "child chunk" and our goal is to retrieve a larger "parent chunk" that the "child chunk" belongs to.
- On occasion, we might also want to provide weights to our retrieve documents. For example, a time-weighted approach would help us retrieve the most recent document.
- One common issue in the retrieval process is the "lost in the middle" problem where the information in the middle of a long context gets lost. Our models have tried to mitigate this issue. For example, in the passkey task, our models have demonstrated the ability to find a "needle in a haystack" by retrieving a randomly inserted passkey within a long prompt, up to 32k context length. However, it is worth considering experimenting with reordering the document to determine if placing the most relevant chunks at the beginning and end leads to improved results.
### Combine context and question in a prompt and generate response
Finally, we can offer the retrieved text chunks as the context information within the prompt. Here is a prompt template where we can include both the retrieved text and user question in the prompt.
```python
prompt = f"""
Context information is below.
---------------------
{retrieved_chunk}
---------------------
Given the context information and not prior knowledge, answer the query.
Query: {question}
Answer:
"""
```
Then we can use the Mistral chat completion API to chat with a Mistral model (e.g., mistral-medium-latest) and generate answers based on the user question and the context of the question.
```python
def run_mistral(user_message, model="mistral-large-latest"):
messages = [
{
"role": "user", "content": user_message
}
]
chat_response = client.chat.complete(
model=model,
messages=messages
)
return (chat_response.choices[0].message.content)
run_mistral(prompt)
```
**Output:**
```
'The two main things the author worked on before college were writing and programming. They wrote short stories and tried writing programs on an IBM 1401 in 9th grade.'
```
#### Considerations:
- **Prompting techniques**: Most of the prompting techniques can be used in developing a RAG system as well. For example, we can use few-shot learning to guide the model's answers by providing a few examples. Additionally, we can explicitly instruct the model to format answers in a certain way.
## RAG Examples
Find multiple RAG cookbooks exploring diverse topics and solutions in our community-driven [cookbook](https://github.com/mistralai/cookbook).
Among them you can find how to perform...
- **RAG with LangChain**: Visit our [community cookbook examples](https://github.com/mistralai/cookbook/tree/main/third_party/langchain) to discover how to use LangChain's LangGraph with the Mistral API. These cookbooks cover various implementations, including adaptive RAG, corrective RAG, and self-RAG, showcasing how to integrate LangChain's capabilities for enhanced retrieval-augmented generation.
- **RAG with LlamaIndex**: Visit our [community cookbook example](https://github.com/mistralai/cookbook/blob/main/third_party/LlamaIndex/llamaindex_agentic_rag.ipynb) to learn how to use LlamaIndex with the Mistral API to perform complex queries over multiple documents using a ReAct agent, an autonomous LLM-powered agent capable of using tools.
- **RAG with Haystack**: Visit our [community cookbook example](https://github.com/mistralai/cookbook/blob/main/third_party/Haystack/haystack_chat_with_docs.ipynb) to explore how to use Haystack with the Mistral API for chat functionalities with documents.
[Ambassador]
Source: https://docs.mistral.ai/docs/guides/contribute/ambassador
---
# Welcome to the Mistral AI Ambassador Program!
As our Mistral AI community continues to grow, we are looking for Mistral experts who are passionate about our models and offerings, and who are committed to giving back to the community and supporting fellow members.
---
# ➡️ Apply ➡️
Applications for the Summer 2025 cohort are now open and will be accepted until July 1, 2025. If selected, you will be contacted by the end of August 2025 to discuss next steps and possibly participate in an interview with additional questions.
### ✍ [fill out the application here](https://docs.google.com/forms/d/e/1FAIpQLSdBSiRzm2xBpMszB_9fBixJNyKdGnPMj99DtZbagHMdHgkGUg/viewform) ✍
Our team will review each application, evaluating candidates based on the following criteria. We accept applications on a quarterly basis.
- **Passion**: Genuine passion for Mistral AI.
- **Expertise**: Knowledge and experience in AI, machine learning, or a related field.
- **Mistral advocacy**: Has previously advocated for Mistral AI, either through community engagement, blog posts, public speaking, video tutorials, or other means.
- **Commitment**: Willingness to commit to the program for at least 6 months.
---
# 🤠 Meet our current Ambassadors 🤠
Thank you to each and every one of you, including those who prefer not to be named, for contributing to our community!
Many companies face the challenge of evaluating whether a Large Language Model (LLM) is suitable for their specific use cases and determining which LLMs offer the best performance. It is essential to assess whether one model can effectively replace another and ensure that the chosen LLM meets the companies’ unique requirements. However, the process of evaluating LLMs can be complex and confusing. This guide provides a comprehensive approach to evaluating LLMs for various applications. We will start by discussing the goal of evaluation, the creation of evaluation datasets, and then we will dive into three primary evaluation methods with detailed Python code walkthroughs in notebooks.
- What exact task do you want to solve?
- How to create an evaluation dataset?
- How to evaluate?
- Metrics-based evaluation
- LLM-based evaluation
- Human-based evaluation
## What is your task?
Before we get started with the evaluation process, it is important to think about what exact task you want to solve with the LLM. It is crucial that the task is precisely defined.
Some bad examples:
- Be a helpful assistant (What does “helpful” mean in your context?)
- Convert the text into code (What type of text and code?)
- Improve the quality of the documents (What is high vs. low quality?)
Some good examples:
- Be an assistant for customer complaints. Make sure to answer complaints in a polite and helpful way and give concise answers that don’t exceed 3 sentences.
- Make sure to notify the user if you do not know the answer.
- Convert pseudo code into Python code
- Rephrase the documents by improving all spelling and grammatical errors and give the text a more professional tone
- Extract all relevant information from medical records (which use case would be subject to specific applicable regulations)
- Summarize various types of documents, such as legal agreements, news articles, or scientific papers
Different goals and use cases may require different evaluation strategies. Some use cases may prioritize accuracy, while others may emphasize brevity and helpfulness. Therefore, it is crucial to know exactly what task you want to solve before starting an evaluation process.
## How to create an evaluation dataset?
There are numerous public benchmarks available for evaluating Large Language Models (LLMs) such as MMLU, GSMK8, and others. The first question to consider is whether these existing benchmarks can be used for your specific use cases. If applicable, then start with the existing benchmark dataset.
However, we often observe that existing benchmarks are academic in nature and may not cover real-world customer use cases or only address a limited scope. Therefore, it is often preferable to create your own evaluation dataset.
Once you have established your goals and determined whether existing benchmarks are suitable, you can proceed to create custom evaluation datasets tailored to your specific use cases.
### Labeled data
In many cases, it is important to create labeled data. For example, if your task is to use an LLM to extract information from medical records subject to specific applicable regulations, you can use human annotation to label the data and get the golden answer including all the information you would like to extract, e.g., `{"age": 60, "gender": "male", "diagnosis": "diabetes", "weight": 210, "smoking": "yes"}`.
How much data annotation do you need? It depends on the task. You should always prioritize quality over quantity. If you have high quality data, 100 test cases can be enough to provide a meaningful signal.
Additionally, It is recommended to have a discriminative benchmark, meaning that it’s neither too easy, nor too hard. If every model being tested achieves over 90% accuracy, the differences between them might not be meaningful, indicating the need for a better benchmark. The goal is to create data that can differentiate models effectively. Creating discriminative data can be challenging, so focus on your specific task. Potentially multiple models perform well enough for your needs. A good evaluation dataset should lead to performances that are in agreement with model sizes, ie, larger models perform on par or better than smaller counterparts (unless the smaller models have been specifically trained for your purposes).
Here are some examples of various prompts and golden answers across different capabilities:
| Task | Prompt/Context| Golden Answer/Criteria|
|--------------------|:--------------------|:--------------------|
| Information extraction | A 60-year-old male patient, Mr. Johnson, presented with symptoms of increased thirst, frequent urination, fatigue, and unexplained weight loss. Upon evaluation, he was diagnosed with diabetes, confirmed by elevated blood sugar levels. Mr. Johnson's weight is 210 lbs. He has been prescribed Metformin to be taken twice daily with meals. It was noted during the consultation that the patient is a current smoker.| `{"age": 60, "gender": "male", "diagnosis": "diabetes", "weight": 210, "smoking": "yes"}` |
|Python code generation|Write a python function to check whether the given number is odd or not using bitwise operator.|`assert is_odd(5) == True` |
|Summarization| Summarize the following into a short paragraph for 3rd graders: "Water is a chemical compound composed of two hydrogen atoms bonded to one oxygen atom, with the chemical formula H2O. The hydrogen atoms are bonded to the oxygen atom via polar covalent bonds, which means that the electrons in the bond are not shared equally between the atoms, resulting in a slight negative charge on the oxygen atom and a slight positive charge on the hydrogen atoms. This gives water molecules a bent, or V-shaped, structure, and allows them to form hydrogen bonds with other water molecules, which are responsible for many of water's unique properties, such as its high boiling point and surface tension." |Water is a special substance that is made up of two tiny parts called atoms, specifically, two hydrogen atoms and one oxygen atom. These atoms are stuck together in a way that creates a bent shape, which makes water have some amazing properties. For example, water can stick to itself, which helps it stay together as a liquid and not just turn into a gas. This also makes water have a high boiling point, which means it needs a lot of heat to turn into a gas. Water's unique properties make it an important part of our lives and our planet.|
|Multilingual|Translate this college homework assignment into advanced Spanish for an advanced writing course: "Write a personal essay of 500 words about a crucial moment in your family, describing the event, its impact on family relationships, and the emotions and thoughts that surrounded it. Your essay should have a clear structure, with an introduction, body, and conclusion, and use expressive and precise language. It will be evaluated based on clarity, coherence, depth, and originality."|Escriba un ensayo personal de 500 palabras sobre un momento crucial en su familia, describiendo el evento, su impacto en las relaciones familiares, y las emociones y pensamientos que lo rodearon. Su ensayo debe tener una estructura clara, con una introducción, un cuerpo y una conclusión, y utilizar un lenguaje expresivo y preciso. Será evaluado sobre la base de claridad, coherencia, profundidad y originalidad.|
|Chat|You are an 8th grade tutor for a Latin class, your job is to help students memorize their weekly word list test. You don’t give the answers to your students but instead help them get to the answer themselves. Begin your tutoring session by asking your student what word list do they need help memorizing. Remember to be encouraging and ask students to "show their work" i.e. articulate how they came up with that answer.|Hello there! I hope you're having a great day. As your Latin tutor, I'm here to help you with your weekly word list test. So, let's get started! Could you please tell me which word list you need help memorizing this week? Remember, the goal is not just to remember the words, but also to understand their meanings and how to use them in context.
You can fine-tune all Mistral’s models via Mistral API. Follow the steps below using Mistral's fine-tuning API.
### Prepare dataset
In this example, let’s use the [ultrachat_200k dataset](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k). We load a chunk of the data into Pandas Dataframes, split the data into training and validation, and save the data into the required `jsonl` format for fine-tuning.
```py
df = pd.read_parquet('https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k/resolve/main/data/test_gen-00000-of-00001-3d4cd8309148a71f.parquet')
df_train=df.sample(frac=0.995,random_state=200)
df_eval=df.drop(df_train.index)
df_train.to_json("ultrachat_chunk_train.jsonl", orient="records", lines=True)
df_eval.to_json("ultrachat_chunk_eval.jsonl", orient="records", lines=True)
```
### Reformat dataset
If you upload this `ultrachat_chunk_train.jsonl` to Mistral API, you might encounter an error message “Invalid file format” due to data formatting issues. To reformat the data into the correct format, you can download the reformat_data.py script and use it to validate and reformat both the training and evaluation data:
```bash
---
# download the validation and reformat script
wget https://raw.githubusercontent.com/mistralai/mistral-finetune/main/utils/reformat_data.py
---
# validate and reformat the training data
python reformat_data.py ultrachat_chunk_train.jsonl
---
# validate the reformat the eval data
python reformat_data.py ultrachat_chunk_eval.jsonl
```
:::important[ ]
This `reformat_data.py` script is tailored for the UltraChat data and may not be universally applicable to other datasets. Please modify this script and reformat your data accordingly.
:::
After running the script, few cases were removed from the training data.
```
Skip 3674th sample
Skip 9176th sample
Skip 10559th sample
Skip 13293th sample
Skip 13973th sample
Skip 15219th sample
```
Let’s inspect one of these cases. There are two issues with this use case:
- one of the assistant messages is an empty string;
- the last message is not an assistant message.
### Upload dataset
We can then upload both the training data and evaluation data to the Mistral Client, making them available for use in fine-tuning jobs.
## End-to-end example with open-source `mistral-finetune`
We have also open sourced fine-tuning codebase mistral-finetune allowing you to fine-tune Mistral’s open-weights models (Mistral 7B, Mixtral 8x7B, Mixtral 8x22B).
To see an end-to-end example of how to install mistral-finetune, prepare and validate your dataset, define your training configuration, fine-tune using Mistral-LoRA, and run inference, please refer to the README file provided in the Mistral-finetune repo: https://github.com/mistralai/mistral-finetune/tree/main or follow this example:
[get data from hugging face]
Source: https://docs.mistral.ai/docs/guides/finetuning_sections/_04_faq
## FAQ
### How to validate data format?
- Mistral API: We currently validate each file when you upload the dataset.
- `mistral-finetune`: You can run the [data validation script](https://github.com/mistralai/mistral-finetune/blob/main/utils/validate_data.py) to validate the data and run the [reformat data script](https://github.com/mistralai/mistral-finetune/blob/main/utils/reformat_data.py) to reformat the data to the right format:
```bash
# download the reformat script
wget https://raw.githubusercontent.com/mistralai/mistral-finetune/main/utils/reformat_data.py
# download the validation script
wget https://raw.githubusercontent.com/mistralai/mistral-finetune/main/utils/validate_data.py
# reformat data
python reformat_data.py data.jsonl
# validate data
python validate_data.py data.jsonl
```
However, it's important to note that these scripts might not detect all problematic cases. Therefore, you may need to manually validate and correct any unique edge cases in your data.
### What's the size limit of the training data?
While the size limit for an individual training data file is 512MB, there's no limitation on the number of files you can upload. You can upload multiple files and reference them when creating the job.
### What's the size limit of the validation data?
The size limit for the validation data is 1MB. As a rule of thumb:
`validation_set_max_size = min(1MB, 5% of training data)`
### What happens if I try to create a job that already exists?
At job creation, you will receive a `409 Conflict` error in case a similar job is already running / validated / queued. This mechanism helps avoid inadvertently creating duplicate jobs, saving resources and preventing redundancy.
### What if I upload an already existing file?
If a file is uploaded and matches an existing file in both content and name, the pre-existing file is returned instead of creating a new one.
### How many epochs are in the training process?
A general rule of thumb is: Num epochs = max_steps / file_of_training_jsonls_in_MB. For instance, if your training file is 100MB and you set max_steps=1000, the training process will roughly perform 10 epochs.
### Where can I find information on cost/ ETA / number of tokens / number of passes over each files?
Mistral API: When you create a fine-tuning job, you should automatically see these info with the default `auto_start=False` argument.
Note that the `dry_run=True` argument will be removed in September.
`mistral-finetune`: You can use the following script to find out: https://github.com/mistralai/mistral-finetune/blob/main/utils/validate_data.py. This script accepts a .yaml training file as input and returns the number of tokens the model is being trained on.
### How to estimate cost of a fine-tuning job?
For Mistral API, you can use the `auto_start=False` argument as mentioned in the previous question.
### What is the recommended learning rate?
For LoRA fine-tuning, we recommend 1e-4 (default) or 1e-5.
Note that the learning rate we define is the peak learning rate, instead of a flat learning rate. The learning rate follows a linear warmup and cosine decay schedule. During the warmup phase, the learning rate is linearly increased from a small initial value to a larger value over a certain number of training steps. After the warmup phase, the learning rate is decayed using a cosine function.
### Is the fine-tuning API compatible with OpenAI data format?
Yes, we support OpenAI format.
### What if my file size is larger than 500MB and I get the error message `413 Request Entity Too Large`?
You can split your data file into chunks. Here is an example:
**Pros:**
* LangSmith is compatible with both the LangChain ecosystem and external systems.
* Deployment option coming soon.
* It offers a broad range of observable areas, serving as an all-in-one platform.
**Mistral integration Example:**
* All of the [langchain notebooks](https://github.com/mistralai/cookbook/tree/main/third_party/langchain) in the Mistral cookbook include LangSmith integration.
Here is an example tracking traces, input, output, documents, tokens, and status when we run the [corrective RAG example](https://github.com/mistralai/cookbook/blob/main/third_party/langchain/corrective_rag_mistral.ipynb) from the Mistral cookbook.
### Integration with 🪢 Langfuse
[Langfuse](https://langfuse.com) ([GitHub](https://github.com/langfuse/langfuse)) is an open-source platform for LLM engineering. It provides tracing and monitoring capabilities for AI applications, helping developers debug, analyze, and optimize their products. Langfuse integrates with various tools and frameworks via native integrations, OpenTelemetry, and SDKs.
**Pros:**
* Most used open-source LLMOps platform ([blog post](https://langfuse.com/blog/2024-11-most-used-oss-llmops))
* Model and framework agnostic
* Built for production
* Incrementally adoptable, start with one feature and expand to the full platform over time
* API-first, all features are available via API for custom integrations
* Optionally, Langfuse can be easily self-hosted
**Mistral integration example:**
* [Step-by-step guide](https://langfuse.com/docs/integrations/mistral-sdk) on tracing Mistral models with Langfuse.
* [Cookbook](https://langfuse.com/guides/cookbook/integration_llama_index_posthog_mistral) on building a RAG application with Mistral and LlamaIndex and trace the steps with Langfuse.
_[Public example trace in Langfuse](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/a3360c6f-24ad-455c-aae7-eb9d5c6f5dac?observation=767f8ac1-0c7d-412f-8fd8-2642acb267c6&display=preview)_
### Integration with Arize Phoenix
Phoenix is an open-source observability library designed for experimentation, evaluation, and troubleshooting. It is designed to support agents, RAG pipelines, and other LLM applications.
**Pros:**
* Open-source ([Github](https://github.com/Arize-ai/phoenix)), and built on OpenTelemetry
* Can be [self-hosted](https://docs.arize.com/phoenix/setup/environments#container), accessed via [cloud](https://docs.arize.com/phoenix/hosted-phoenix), or run directly in a [notebook](https://docs.arize.com/phoenix/setup/environments#notebooks)
* Provides a [Mistral integration](https://docs.arize.com/phoenix/tracing/integrations-tracing/mistralai) to automatically trace Client.chat and Agent.chat calls
* Strong analytical platform, with a copilot agent to help debug your application
**Mistral integration Example:**
Here is an [example notebook](https://github.com/mistralai/cookbook/blob/main/third_party/Phoenix/arize_phoenix_tracing.ipynb) that shows how to trace Mistral chat.complete and tool calls in Phoenix.
### Integration with Weights and Biases
Weights & Biases is an end-to-end AI developer platform for ML and LLM workflows used for both fine-tuning and LLM application building. Use W&B Weave to evaluate, monitor, and iterate on GenAI applications, and W&B Models as a system of record to train, fine-tune, and manage AI models.
**Pros:**
* Platform for both LLM app development and fine-tuning
* Integrated with [Mistral API](https://weave-docs.wandb.ai/guides/integrations/mistral/)
* Get started by adding one line: `weave.init('my-project')`
* Automatically tracks inputs, output, context, errors, evaluation metrics & traces
* Integrated with [Mistral fine-tuning service](/guides/finetuning/#integration-with-weights-and-biases)
* Track training metrics while fine-tuning
* Compare training experiments
**Mistral integration Example:**
To get you started you can check our recent webinar "Fine-tuning an LLM judge to reduce hallucination" and the [cookbook](https://github.com/mistralai/cookbook/tree/main/third_party/wandb).
### Integration with PromptLayer
PromptLayer is a platform for prompt management, collaboration, monitoring, and evaluation. Good for hackers and production teams alike.
**Pros:**
* No-code CMS for prompt management and versioning
* Native support for Mistral
* Prompts are model agnostic by default
* Simple prompt tracking and observability
**Mistral integration:**
### Integration with AgentOps
AgentOps is an open-source observability and DevTool platform for AI Agents. It helps developers build, evaluate, and monitor AI agents.
**Pros:**
* Open-source
* Designed for observing agents
* Allow for time travel
* Integrates with CrewAI, AutoGen, & LangChain
**Mistral integration Example:**
[https://github.com/mistralai/cookbook/blob/main/third\_party/CAMEL\_AI/camel\_roleplaying\_scraper.ipynb](https://github.com/mistralai/cookbook/blob/main/third_party/CAMEL_AI/camel_roleplaying_scraper.ipynb)
### Integration with phospho
[phospho](https://phospho.ai/) is a text analytics platform that makes it easy to get answers, take decisions and reduce churn by data mining user messages.
**Pros:**
- Open-source ([github](https://github.com/phospho-app)) platform
- No code clustering and analytics
- Customizable dashboards
- Many integrations with other observability frameworks, languages, APIs…
**Mistral integration example:**
- Check out the [phospho notebooks](https://github.com/mistralai/cookbook/tree/main/third_party/phospho) in the Mistral cookbook.
### Integration with MLflow
MLflow is a unified, end-to-end, open source MLOps platform for both traditional ML and GenAI applications, providing comprehensive tracing capabilities to monitor and analyze the execution of GenAI applications.
**Pros:**
* Open-source ([Github](https://github.com/mlflow/mlflow))
* Add Mistral integration with one line: `mlflow.mistral.autolog()` and get full tracing of chat and embedding calls.
* Can be [run locally or self-hosted](https://mlflow.org/docs/latest/getting-started/intro-quickstart/index.html), or used via one of the available [Managed MLflow services](https://mlflow.org/docs/latest/introduction/index.html#running-mlflow-anywhere)
* Provides complete model evaluation, versioning, and deployment capabilities, in addition to tracing and experiment tracking.
**Mistral integration Example:**
Here is an [example notebook](https://github.com/mistralai/cookbook/blob/main/third_party/MLflow/mistral-mlflow-tracing.ipynb).
[Other resources]
Source: https://docs.mistral.ai/docs/guides/other-resources
Visit the [Mistral AI Cookbook](https://github.com/mistralai/cookbook) for additional inspiration,
where you'll find example code, community contributions, and demonstrations of integrations with third-party tools, including:
- [LlamaIndex \<\> MistralAI Cookbooks on agents and advanced RAG](https://github.com/mistralai/cookbook/tree/main/third_party/LlamaIndex)
- [LangChain \<\> MistralAI Cookbooks on advanced RAG](https://github.com/mistralai/cookbook/tree/main/third_party/langchain)
[Prefix]
Source: https://docs.mistral.ai/docs/guides/prefix
---
# Prefix: Use Cases
Prefixes are one feature that can easily be game-changing for many use cases and scenarios, while the concept is simple, the possibilities are endless.
We will now dig into a few different cool examples and explore prefixes
hidden potential!
Essentially, prefixes enable a high level of instruction following and
adherence or define the model's response more effectively with less
effort.
For all of the following examples, we will need to set up our client.
Let's import the required package and then create the client with your
API key!
How to make a model always answer in a specific language regardless of input:
## Classification
Mistral models can easily **categorize text** into distinct classes. Take a customer support bot for a bank as an illustration: we can establish a series of predetermined categories within the prompt and then instruct Mistral AI models to categorize the customer's question accordingly.
In the following example, when presented with the customer inquiry, Mistral AI models correctly categorizes it as "country support":
| User | I am inquiring about the availability of your cards in the EU, as I am a resident of France and am interested in using your cards. |
| Assistant | country support |
| User | Dear mortgage lender, What's your 30-year fixed-rate APR, how is it compared to the 15-year fixed rate? Regards, Anna |
| Assistant |
Dear Anna, Thank you for your inquiry regarding our fixed-rate mortgage options. Our current 30-year fixed-rate interest rate is 6.403% with an Annual Percentage Rate (APR) of 6.484%. In comparison, our 15-year fixed-rate interest rate is 5.705% with an APR of 5.848%. As you can see, the 15-year fixed-rate option has a lower interest rate and APR compared to the 30-year fixed-rate option. This means that you would pay less in interest over the life of the loan if you choose the 15-year option. However, the monthly payments would be higher due to the shorter loan term. Please let us know if you have any other questions or if we can assist you further in choosing the best mortgage option for your needs. Best regards, Lender Customer Support |
| Assistant | ``` { "Summaries": [ { "Summary": "The author discusses their early experiences with programming and writing, starting with writing short stories and programming on an IBM 1401 in 9th grade. They then moved on to working with microcomputers, building their own from a Heathkit, and eventually convincing their father to buy a TRS-80 in 1980. They wrote simple games, a program to predict rocket flight trajectories, and a word processor.", "Confidence": 0.9 }, { "Summary": "The author began college as a philosophy major, but found it to be unfulfilling and switched to AI. They were inspired by a novel and a PBS documentary, as well as the potential for AI to create intelligent machines like those in the novel. Despite this excitement, they eventually realized that the traditional approach to AI was flawed and shifted their focus to Lisp.", "Confidence": 0.85 }, { "Summary": "The author briefly worked at Interleaf, where they found that their Lisp skills were highly valued. They eventually left Interleaf to return to RISD, but continued to work as a freelance Lisp hacker. While at RISD, they started painting still lives in their bedroom at night, which led to them applying to art schools and eventually attending the Accademia di Belli Arti in Florence.", "Confidence": 0.9 } ] } ``` |
| Assistant | ``` Summary 1: The essay discusses the author's experiences with writing and programming before college, his initial struggles with learning Fortran on an IBM 1401, and the shift towards microcomputers that led to his interest in AI. He also touches on his decision to pursue a degree in AI and the disillusionment he felt when he realized that AI as practiced at the time was a hoax. Summary 2: The author tells the story of his journey into programming, starting with writing short stories before discovering the world of computers. He describes his early experiences with an IBM 1401 and how it was like a mini Bond villain's lair. He also talks about his fascination with microcomputers and how it led him to learn Lisp and become interested in AI. However, he eventually became disillusioned with AI and decided to focus on Lisp instead. Summary 3: The essay recounts the author's experiences with writing and programming before college, including his early attempts at writing short stories and his struggles with learning to program on an IBM 1401. He discusses how the arrival of microcomputers changed everything and sparked his interest in AI. However, he eventually realized that AI, as practiced at the time, was a hoax, and he turned his attention to Lisp. He also touches on his decision to pursue a degree in AI despite his disillusionment with the field. Evaluation: I believe that Summary 1 is the best summary because it provides a clear and concise overview of the author's experiences with writing and programming before college, the shift towards microcomputers, and his eventual disillusionment with AI. It covers the key points of the essay without getting bogged down in unnecessary details. Summary 2 is also good, but it focuses more on the author's early experiences with computers and less on his later experiences with AI. Summary 3 is the weakest summary because it does not capture the full scope of the author's experiences and does not mention his decision to pursue a degree in AI despite his disillusionment with the field. ``` |
| Assistant | ``` Summary 1: The author writes about their experiences with programming and writing before college. They started programming using Fortran on an IBM 1401 and wrote short stories. They later got into AI, but became disillusioned with it while in graduate school. The author then turned their focus to Lisp, eventually writing a book about it called "On Lisp". They also tried pursuing art, but found it difficult to make a living from it. Summary 2: The author describes their experiences with programming in college and later in the workforce. They learned Lisp while in college, which expanded their concept of what a program could do. After graduation, the author worked at Interleaf, a company that made document creation software. They also did some consulting and writing on the side, eventually writing a book called "On Lisp". The author enjoyed working at Interleaf, but found that the Lisp they used was just a thin layer on top of a larger C cake. Summary 3: The author tells the story of their early experiences with programming and writing. They started writing short stories before college, but they were not very good. The first programs they wrote were on an IBM 1401 using Fortran. In college, they got into AI, but found it disappointing. The author then turned their focus to Lisp, which they found to be more interesting. They also tried pursuing art, but found it difficult to make a living from it. The author eventually wrote a book called "On Lisp" and worked at Interleaf, a company that made document creation software. They enjoyed their time at Interleaf, but found that the Lisp they used was just a thin layer on top of a larger C cake. ``` |
| Assistant | ``` Summary 3 provides the most complete and accurate summary of the essay. It covers the author's experiences with writing and programming before college, their disappointment with AI in college, and their eventual focus on Lisp and work at Interleaf. Summary 1 is missing some key details, such as the author's work at Interleaf and their writing of "On Lisp". Summary 2 focuses too much on the author's time at Interleaf and neglects their early experiences with writing and programming before college. Overall, Summary 3 is the best choice as it is the most complete and accurate summary of the essay. ``` |
Barplot example comparing the distribution with different `Temperature` values and the top 5 tokens using Mistral 7B at 4 bits precision.
Tokenization is a fundamental step in LLMs. It is the process of breaking down text into smaller subword units, known as tokens. We recently open-sourced our tokenizer at Mistral AI. This guide will walk you through the fundamentals of tokenization, details about our open-source tokenizers, and how to use our tokenizers in Python.
## What is tokenization?
Tokenization is the first step and the last step of text processing and modeling. Texts need to be represented as numbers in our models so that our model can understand. Tokenization breaks down text into tokens, and each token is assigned a numerical representation, or index, which can be used to feed into a model. In a typical LLM workflow:
- We first encode the input text into tokens using a tokenizer. Each unique token is assigned a specific index number in the tokenizer’s vocabulary.
- Once the text is tokenized, these tokens are passed through the model, which typically includes an embedding layer and transformer blocks. The embedding layer converts the tokens into dense vectors that capture semantic meanings. Check out our [embedding guide](/capabilities/embeddings/overview) for details. The transformer blocks then process these embedding vectors to understand the context and generate results.
- The last step is decoding, which detokenize output tokens back to human-readable text. This is done by mapping the tokens back to their corresponding words using the tokenizer’s vocabulary.
Most people only tokenize text.
Our first release contains tokenization.
Our tokenizers go beyond the usual text \<-\> tokens,
adding parsing of tools and structured conversation.
We also release the validation and normalization code that is used in our API.
Specifically, we use control tokens, which are special tokens to indicate different types of elements.
These tokens are not treated as strings and are added directly to the code.
Note that we are still iterating on the tokenizer. Things may change and this is the current state of things.
We have released three versions of our tokenizers powering different sets of models.
- v1: `mistral-embed`, `open-mixtral-8x7b`
- v2: `mistral-small-2402` (deprecated), `mistral-large-2402`
- v3: `open-mixtral-8x22b`, `mistral-large-latest`, `mistral-small-latest`, `open-mistral-7b`
- v3 (tekken): `open-mistral-nemo`, `ministral-8b-latest`
This guide will focus on our latest v3 (tekken) tokenizer and v3 tokenizer.
## v3 (tekken) tokenizer
There are several tokenization methods used in Natural Language Processing (NLP) to convert raw text into tokens such as word-level tokenization, character-level tokenization, and subword-level tokenization including the Byte-Pair Encoding (BPE).
Our newest tokenizer, tekken, uses the Byte-Pair Encoding (BPE) with [Tiktoken](https://github.com/openai/tiktoken).
Tekken was trained on more than 100 languages and compresses natural language text and
source code more efficiently than the SentencePiece tokeniser used in previous Mistral models.
In particular, it is ~30% more efficient at compressing source code in Chinese, Italian,
French, German, Spanish, and Russian. It is also 2x and 3x more efficient at compressing
Korean and Arabic, respectively. Compared to the Llama 3 tokeniser,
Tekken proved more proficient in compressing text for approximately 85% of all languages.
### Our tokenization vocabulary
Our tokenization vocabulary is released in the https://github.com/mistralai/mistral-common/tree/main/tests/data folder. Let’s take a look at the vocabulary of our v3 tekken tokenizer.
#### Vocabulary size
Our vocabulary consists of 130k vocab + 1k control tokens.
#### Control tokens
Our vocabulary starts with 14 control tokens, which are special tokens we use in the encoding process to represent specific instructions or indicators:
```
