# Delete any existing records in the collection
collection.delete_many({})
```

```

DeleteResult({'n': 63, 'electionId': ObjectId('7fffffff000000000000002b'), 'opTime': {'ts': Timestamp(1721913981, 63), 't': 43}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1721913981, 63), 'signature': {'hash': b'cU;+\xe3\xbdRc\t\x80\xad\x03\x16\x11\x18\xe6s\xebF\x01', 'keyId': 7353740577831124994}}, 'operationTime': Timestamp(1721913981, 63)}, acknowledged=True)

```

## Step 5: Data Ingestion

MongoDB's Document model and its compatibility with Python dictionaries offer several benefits for data ingestion.

* Document-oriented structure:
  * MongoDB stores data in JSON-like documents: BSON(Binary JSON).
  * This aligns naturally with Python dictionaries, allowing for seamless data representation using key value pair data structures.
* Schema flexibility:
  * MongoDB is schema-less, meaning each document in a collection can have a different structure.
  * This flexibility matches Python's dynamic nature, allowing you to ingest varied data structures without predefined schemas.
* Efficient ingestion:
  * The similarity between Python dictionaries and MongoDB documents allows for direct ingestion without complex transformations.
  * This leads to faster data insertion and reduced processing overhead.

![](https://files.buildwithfern.com/cohere.docs.buildwithfern.com/71df7983b662ecd6c9b374492f10b9d8fd3c451d62b9f0c56eb25901d21faca8/assets/images/rag-cohere-mongodb-4.png)

```python
documents = dataset_df.to_dict("records")
collection.insert_many(documents)

print("Data ingestion into MongoDB completed")
```

Data ingestion into MongoDB completed

## Step 6: MongoDB Query language and Vector Search

**Query flexibility**

MongoDB's query language is designed to work well with document structures, making it easy to query and manipulate ingested data using familiar Python-like syntax.

**Aggregation Pipeline**

MongoDB's aggregation pipelines is a powerful feature of the MongoDB Database that allows for complex data processing and analysis within the database.
Aggregation pipeline can be thought of similarly to pipelines in data engineering or machine learning, where processes operate sequentially, each stage taking an input, performing operations, and providing an output for the next stage.

**Stages**

Stages are the building blocks of an aggregation pipeline.
Each stage represents a specific data transformation or analysis operation.
Common stages include:

* `$match`: Filters documents (similar to WHERE in SQL)
* `$group`: Groups documents by specified fields
* `$sort`: Sorts the documents
* `$project`: Reshapes documents (select, rename, compute fields)
* `$limit`: Limits the number of documents
* `$unwind`: Deconstructs array fields
* `$lookup`: Performs left outer joins with other collections

![](https://files.buildwithfern.com/cohere.docs.buildwithfern.com/a047d301a9d4c8a7849ed421df6678bc5c92c39a75f8e98f76569a742780d1ad/assets/images/rag-cohere-mongodb-5.png)

```python
def vector_search(user_query, collection):
    """
    Perform a vector search in the MongoDB collection based on the user query.

    Args:
    user_query (str): The user's query string.
    collection (MongoCollection): The MongoDB collection to search.

    Returns:
    list: A list of matching documents.
    """

    # Generate embedding for the user query
    query_embedding = get_embedding(
        user_query, input_type="search_query"
    )

    if query_embedding is None:
        return "Invalid query or embedding generation failed."

    # Define the vector search pipeline
    vector_search_stage = {
        "$vectorSearch": {
            "index": "vector_index",
            "queryVector": query_embedding,
            "path": "embedding",
            "numCandidates": 150,  # Number of candidate matches to consider
            "limit": 5,  # Return top 4 matches
        }
    }

    unset_stage = {
        "$unset": "embedding"  # Exclude the 'embedding' field from the results
    }

    project_stage = {
        "$project": {
            "_id": 0,  # Exclude the _id field
            "company": 1,  # Include the plot field
            "reports": 1,  # Include the title field
            "combined_attributes": 1,  # Include the genres field
            "score": {
                "$meta": "vectorSearchScore"  # Include the search score
            },
        }
    }

    pipeline = [vector_search_stage, unset_stage, project_stage]

    # Execute the search
    results = collection.aggregate(pipeline)
    return list(results)
```

## Step 7:  Add the Cohere Reranker

Cohere rerank functions as a second stage search that can improve the precision of your first stage search results

![](https://files.buildwithfern.com/cohere.docs.buildwithfern.com/53dba5826ab9f86b921f08e72f67efee195ef84cc91bce31119c6ffc5f849293/assets/images/rag-cohere-mongodb-6.png)

```python
def rerank_documents(query: str, documents, top_n: int = 3):
    # Perform reranking with Cohere ReRank Model
    try:
        response = co.rerank(
            model="rerank-english-v3.0",
            query=query,
            documents=documents,
            top_n=top_n,
            rank_fields=["company", "reports", "combined_attributes"],
        )

        # Extract the top reranked documents
        top_documents_after_rerank = []
        for result in response.results:
            original_doc = documents[result.index]
            top_documents_after_rerank.append(
                {
                    "company": original_doc["company"],
                    "combined_attributes": original_doc[
                        "combined_attributes"
                    ],
                    "reports": original_doc["reports"],
                    "vector_search_score": original_doc["score"],
                    "relevance_score": result.relevance_score,
                }
            )

        return top_documents_after_rerank

    except Exception as e:
        print(f"An error occurred during reranking: {e}")
        # Return top N documents without reranking
        return documents[:top_n]
```

```python
import pprint

query = "What companies have negative market reports or negative sentiment that might deter from investment in the long term"

get_knowledge = vector_search(query, collection)
pd.DataFrame(get_knowledge).head()
```

<div>
  <table border="1" class="dataframe fern-table">
    <thead>
      <tr>
        <th />

        <th>
          reports
        </th>

        <th>
          company
        </th>

        <th>
          combined_attributes
        </th>

        <th>
          score
        </th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <th>
          0
        </th>

        <td>
          \[{'author': 'Jordan Garcia, Senior Tech Analys...
        </td>

        <td>
          GreenEnergy Corp
        </td>

        <td>
          GreenEnergy Corp Information Technology 2023 G...
        </td>

        <td>
          0.659524
        </td>
      </tr>

      <tr>
        <th>
          1
        </th>

        <td>
          \[{'author': 'Morgan Smith, Technology Sector L...
        </td>

        <td>
          BioTech Therapeutics
        </td>

        <td>
          BioTech Therapeutics Information Technology 20...
        </td>

        <td>
          0.646300
        </td>
      </tr>

      <tr>
        <th>
          2
        </th>

        <td>
          \[{'author': 'Casey Davis, Technology Sector Le...
        </td>

        <td>
          RenewableEnergy Innovations
        </td>

        <td>
          RenewableEnergy Innovations Information Techno...
        </td>

        <td>
          0.645224
        </td>
      </tr>

      <tr>
        <th>
          3
        </th>

        <td>
          \[{'author': 'Morgan Johnson, Technology Sector...
        </td>

        <td>
          QuantumSensor Corp
        </td>

        <td>
          QuantumSensor Corp Information Technology 2023...
        </td>

        <td>
          0.644383
        </td>
      </tr>

      <tr>
        <th>
          4
        </th>

        <td>
          \[{'author': 'Morgan Williams, Senior Tech Anal...`
        </td>

        <td>
          BioEngineering Corp
        </td>

        <td>
          BioEngineering Corp Information Technology 202...
        </td>

        <td>
          0.643690
        </td>
      </tr>
    </tbody>
  </table>
</div>

```python
reranked_documents = rerank_documents(query, get_knowledge)
pd.DataFrame(reranked_documents).head()
```

<div>
  <table border="1" class="dataframe fern-table">
    <thead>
      <tr>
        <th />

        <th>
          company
        </th>

        <th>
          combined_attributes
        </th>

        <th>
          reports
        </th>

        <th>
          vector_search_score
        </th>

        <th>
          relevance_score
        </th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <th>
          0
        </th>

        <td>
          GreenEnergy Corp
        </td>

        <td>
          GreenEnergy Corp Information Technology 2023 G...
        </td>

        <td>
          \[{'author': 'Jordan Garcia, Senior Tech Analys...
        </td>

        <td>
          0.659524
        </td>

        <td>
          0.000147
        </td>
      </tr>

      <tr>
        <th>
          1
        </th>

        <td>
          BioEngineering Corp
        </td>

        <td>
          BioEngineering Corp Information Technology 202...
        </td>

        <td>
          \[{'author': 'Morgan Williams, Senior Tech Anal...
        </td>

        <td>
          0.643690
        </td>

        <td>
          0.000065
        </td>
      </tr>

      <tr>
        <th>
          2
        </th>

        <td>
          QuantumSensor Corp
        </td>

        <td>
          QuantumSensor Corp Information Technology 2023...
        </td>

        <td>
          \[{'author': 'Morgan Johnson, Technology Sector...
        </td>

        <td>
          0.644383
        </td>

        <td>
          0.000054
        </td>
      </tr>
    </tbody>
  </table>
</div>

## Step 8: Handling User Queries

```python
def format_documents_for_chat(documents):
    return [
        {
            "company": doc["company"],
            # "reports": doc['reports'],
            "combined_attributes": doc["combined_attributes"],
        }
        for doc in documents
    ]