# Topic Modeling System for AI Papers

> This page discusses how to create a topic-modeling system for papers focused on AI papers.

<CookbookHeader href="https://github.com/cohere-ai/cohere-developer-experience/blob/main/notebooks/guides/Topic_Modeling_AI_Papers.ipynb" />

Natural Language Processing (NLP) is a key area of machine learning focused on analyzing and understanding text data. One popular NLP application is topic modeling, which uses unsupervised learning and clustering algorithms to group similar texts and extract underlying topics from document collections. This approach enables automatic document organization, efficient information retrieval, and content filtering.

Here, you'll learn how to use Cohere’s NLP tools to perform semantic search and clustering of AI Papers, which could help you discover trends in AI. You'll:

* Scrape the most recent page of the ArXiv page for AI, with the output being a list of recently published AI papers.
* Use Cohere’s Embed Endpoint to generate word embeddings using your list of AI papers.
* Visualize the embeddings and proceed to perform topic modeling.
* Use a tool to find the papers most relevant to a query you provide.

To follow along with this tutorial, you need to be familiar with Python and have python version 3.6+ installed, and you'll need to have a Cohere account. Everything that follows can be tested with a Google Colab notebook.

First, you need to install the python dependencies required to run the project. Use pip to install them using the command below.

```python PYTHON
pip install requests beautifulsoup4 cohere altair clean-text numpy pandas scikit-learn > /dev/null
```

And we'll also initialize the Cohere client.

```python PYTHON
import cohere

api_key = '<api_key>'
co = cohere.ClientV2(api_key="YOUR API KEY")
```

With that done, we'll import the required libraries to make web requests, process the web content, and perform our topic-modeling functions.

```python PYTHON
## Getting the web content
import requests
from bs4 import BeautifulSoup

## Processing the content 
import pandas as pd
import numpy as np

## Handling the underlying NLP
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
```

Next, make an HTTP request to the source website that has an archive of the AI papers.

```python PYTHON
URL = "https://arxiv.org/list/cs.AI/new"
page = requests.get(URL)
```

## Setting up the Functions We Need

In this section, we'll walk through some of the Python code we'll need for our topic modeling project.

### Getting and Processing ArXiv Papers.

This `make_raw_df` function scrapes paper data from a given URL, pulling out titles and abstracts. It uses `BeautifulSoup` to parse the HTML content, extracting titles from elements with class `"list-title mathjax"` and abstracts from paragraph elements with class `"mathjax"`. Finally, it organizes this data into a pandas dataframe with two columns - "titles" and "texts" - where each row represents the information from a single paper.

```python PYTHON
def make_raw_df(url):
    response=requests.get(url)
    soup=BeautifulSoup(response.content, "html.parser")

    titles=list()
    texts=list()

    # Extract titles from <div class="list-title mathjax">
    title_tags = soup.find_all(class_="list-title mathjax")
    for title_tag in title_tags:
        titles.append(title_tag.text.strip())  # Remove leading/trailing whitespace

    # Extract abstracts from <p class="mathjax">
    abstract_tags = soup.find_all('p', class_="mathjax")#, tag="p")
    for abstract_tag in abstract_tags:
        texts.append(abstract_tag.text.strip())

    df = pd.DataFrame({"titles": titles, "texts": texts})
    return df
```

### Generating embeddings

[Word embedding](https://docs.cohere.com/docs/embeddings) is a technique for learning a numerical representation of words. You can use these embeddings to:

• Cluster large amounts of text\
• Match a query with other similar sentences\
• Perform classification tasks like sentiment classification

All of which we will do today.

Cohere’s platform provides an [Embed endpoint](https://docs.cohere.com/reference/embed) that returns text embeddings. An embedding is a list of floating-point numbers, and it captures the semantic meaning of the represented text. Models used to create these embeddings are available in several; small models are faster while large models offer better performance.

In the `get_embeddings`, `make_clusters`, and `create_cluster_names` functions defined below, we'll generate embeddings from the papers, use principal component analysis to create axes for later plotting efforts, use KMeans clustering to group the embedded papers into broad topics, and create a 'short name' that captures the essence of each cluster. This short name will make our Altair plot easier to read.

```python PYTHON
def get_embeddings(text,model='embed-v4.0'):
  output = co.embed(
                model=model,
                texts=[text],
                input_type="classification",
                embedding_types=["float"],)
  return output.embeddings.float_[0]