# Connect Your Data with Firecrawl (/connect-your-data/firecrawl)

Connect websites to your agents using Firecrawl



## Overview

Firecrawl is a web scraping and web crawling platform that extracts clean content from web pages and converts it to markdown or structured JSON, ready for embedding and use in RAG pipelines.

With Firecrawl you can connect your agents to:

* **Websites**: Website crawling and indexing for extracting clean content from web pages
* **Web pages**: Individual page scraping with automatic content extraction

## RAG pipeline workflow

Here's what a complete RAG pipeline looks like to connect your websites to your agents:

<Steps>
  <Step>
    **Scrape web content with Firecrawl** - Extract clean markdown from websites
  </Step>

  <Step>
    **Save the clean markdown** - Store scraped content locally
  </Step>

  <Step>
    **Index the documents in Pinecone** - Using Pinecone Assistant SDK
  </Step>

  <Step>
    **Agent queries via MCP server** - Retrieves relevant content using semantic search
  </Step>
</Steps>

## Getting started

### Prerequisites

Before we get started, make sure you have the following:

* A [Firecrawl account](https://firecrawl.dev/)
* A [Pinecone account](https://app.pinecone.io/)
* [uv](https://docs.astral.sh/uv/) installed
* A python virtual environment running

### Step 1: Set up Firecrawl and collect data

Install Firecrawl and retrieve your API key from [firecrawl.dev](https://firecrawl.dev):

```bash
uv pip install firecrawl-py python-dotenv
```

Save your API key to a `.env` file:

```bash title=".env"
FIRECRAWL_API_KEY=fc-YOUR-KEY-HERE
```

The following script uses Firecrawl to explore a website's structure, identify all available pages, and convert each page's content into markdown files.

```python
from firecrawl import Firecrawl
from dotenv import load_dotenv
from pathlib import Path

load_dotenv()
app = Firecrawl()