# Create a t-SNE model and transform the data
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)
vis_dims = tsne.fit_transform(matrix)

colors = ["red", "darkorange", "gold", "turquiose", "darkgreen"]
x = [x for x,y in vis_dims]
y = [y for x,y in vis_dims]
color_indices = df.Score.values - 1

colormap = matplotlib.colors.ListedColormap(colors)
plt.scatter(x, y, c=color_indices, cmap=colormap, alpha=0.3)
plt.title("Amazon ratings visualized in language using t-SNE")
```

Embedding as a text feature encoder for ML algorithms

[Regression_using_embeddings.ipynb](https://cookbook.openai.com/examples/regression_using_embeddings)

An embedding can be used as a general free-text feature encoder within a machine
learning model. Incorporating embeddings will improve the performance of any
machine learning model, if some of the relevant inputs are free text. An
embedding can also be used as a categorical feature encoder within a ML model.
This adds most value if the names of categorical variables are meaningful and
numerous, such as job titles. Similarity embeddings generally perform better
than search embeddings for this task.

We observed that generally the embedding representation is very rich and
information dense. For example, reducing the dimensionality of the inputs using
SVD or PCA, even by 10%, generally results in worse downstream performance on
specific tasks.

This code splits the data into a training set and a testing set, which will be
used by the following two use cases, namely regression and classification.

```python
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    list(df.ada_embedding.values),
    df.Score,
    test_size = 0.2,
    random_state=42
)
```

#### Regression using the embedding features

Embeddings present an elegant way of predicting a numerical value. In this
example we predict the reviewerâs star rating, based on the text of their
review. Because the semantic information contained within embeddings is high,
the prediction is decent even with very few reviews.

We assume the score is a continuous variable between 1 and 5, and allow the
algorithm to predict any floating point value. The ML algorithm minimizes the
distance of the predicted value to the true score, and achieves a mean absolute
error of 0.39, which means that on average the prediction is off by less than
half a star.

```python
from sklearn.ensemble import RandomForestRegressor

rfr = RandomForestRegressor(n_estimators=100)
rfr.fit(X_train, y_train)
preds = rfr.predict(X_test)
```

Classification using the embedding features

[Classification_using_embeddings.ipynb](https://cookbook.openai.com/examples/classification_using_embeddings)

This time, instead of having the algorithm predict a value anywhere between 1
and 5, we will attempt to classify the exact number of stars for a review into 5
buckets, ranging from 1 to 5 stars.

After the training, the model learns to predict 1 and 5-star reviews much better
than the more nuanced reviews (2-4 stars), likely due to more extreme sentiment
expression.

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, accuracy_score

clf = RandomForestClassifier(n_estimators=100)
clf.fit(X_train, y_train)
preds = clf.predict(X_test)
```

Zero-shot classification

[Zero-shot_classification_with_embeddings.ipynb](https://cookbook.openai.com/examples/zero-shot_classification_with_embeddings)

We can use embeddings for zero shot classification without any labeled training
data. For each class, we embed the class name or a short description of the
class. To classify some new text in a zero-shot manner, we compare its embedding
to all class embeddings and predict the class with the highest similarity.

```python
from openai.embeddings_utils import cosine_similarity, get_embedding

df= df[df.Score!=3]
df['sentiment'] = df.Score.replace({1:'negative', 2:'negative', 4:'positive', 5:'positive'})

labels = ['negative', 'positive']
label_embeddings = [get_embedding(label, model=model) for label in labels]

def label_score(review_embedding, label_embeddings):
    return cosine_similarity(review_embedding, label_embeddings[1]) - cosine_similarity(review_embedding, label_embeddings[0])

prediction = 'positive' if label_score('Sample Review', label_embeddings) > 0 else 'negative'
```

Obtaining user and product embeddings for cold-start recommendation

[User_and_product_embeddings.ipynb](https://cookbook.openai.com/examples/user_and_product_embeddings)

We can obtain a user embedding by averaging over all of their reviews.
Similarly, we can obtain a product embedding by averaging over all the reviews
about that product. In order to showcase the usefulness of this approach we use
a subset of 50k reviews to cover more reviews per user and per product.

We evaluate the usefulness of these embeddings on a separate test set, where we
plot similarity of the user and product embedding as a function of the rating.
Interestingly, based on this approach, even before the user receives the product
we can predict better than random whether they would like the product.

![Boxplot grouped by Score](https://cdn.openai.com/API/docs/images/embeddings-boxplot.png)

```python
user_embeddings = df.groupby('UserId').ada_embedding.apply(np.mean)
prod_embeddings = df.groupby('ProductId').ada_embedding.apply(np.mean)
```

Clustering

[Clustering.ipynb](https://cookbook.openai.com/examples/clustering)

Clustering is one way of making sense of a large volume of textual data.
Embeddings are useful for this task, as they provide semantically meaningful
vector representations of each text. Thus, in an unsupervised way, clustering
will uncover hidden groupings in our dataset.

In this example, we discover four distinct clusters: one focusing on dog food,
one on negative reviews, and two on positive reviews.

![Clusters identified visualized in language 2d using t-SNE](https://cdn.openai.com/API/docs/images/embeddings-cluster.png)

```python
import numpy as np
from sklearn.cluster import KMeans

matrix = np.vstack(df.ada_embedding.values)
n_clusters = 4

kmeans = KMeans(n_clusters = n_clusters, init='k-means++', random_state=42)
kmeans.fit(matrix)
df['Cluster'] = kmeans.labels_
```

## FAQ

### How can I tell how many tokens a string has before I embed it?

In Python, you can split a string into tokens with OpenAI's tokenizer tiktoken.

Example code:

```python
import tiktoken

def num_tokens_from_string(string: str, encoding_name: str) -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

num_tokens_from_string("tiktoken is great!", "cl100k_base")
```

For third-generation embedding models like `text-embedding-3-small`, use the
`cl100k_base` encoding.

More details and example code are in the OpenAI Cookbook guide how to count
tokens with tiktoken.

### How can I retrieve K nearest embedding vectors quickly?

For searching over many vectors quickly, we recommend using a vector database.
You can find examples of working with vector databases and the OpenAI API in our
Cookbook on GitHub.

### Which distance function should I use?

We recommend cosine similarity. The choice of distance function typically
doesn't matter much.

OpenAI embeddings are normalized to length 1, which means that:

- Cosine similarity can be computed slightly faster using just a dot product
- Cosine similarity and Euclidean distance will result in the identical rankings

### Can I share my embeddings online?

Yes, customers own their input and output from our models, including in the case
of embeddings. You are responsible for ensuring that the content you input to
our API does not violate any applicable law or our Terms of Use.

### Do V3 embedding models know about recent events?

No, the `text-embedding-3-large` and `text-embedding-3-small` models lack
knowledge of events that occurred after September 2021. This is generally not as
much of a limitation as it would be for text generation models but in certain
edge cases it can reduce performance.