# Mastering Firecrawl's Crawl Endpoint: A Complete Web Scraping Guide

![Mastering Firecrawl's Crawl Endpoint: A Complete Web Scraping Guide image](https://www.firecrawl.dev/images/blog/crawl-masterclass/images/mastering-crawl.jpg)

## Introduction

Web scraping and data extraction have become essential tools as businesses race to convert unprecedented amounts of online data into LLM-friendly formats. Firecrawl’s powerful web scraping API streamlines this process with enterprise-grade automation and scalability features.

This comprehensive guide focuses on Firecrawl’s most powerful feature - the `/crawl` endpoint, which enables automated website scraping at scale. You’ll learn how to:

- Recursively traverse website sub-pages
- Handle dynamic JavaScript-based content
- Bypass common web scraping blockers
- Extract clean, structured data for AI/ML applications

Want to follow along with our python notebook version of this post? [Check it out here!](https://github.com/firecrawl/firecrawl/blob/main/examples/mastering-the-crawl-endpoint/mastering-the-crawl-endpoint.ipynb)

## Table of Contents

- [Introduction](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#introduction)
- [Web Scraping vs Web Crawling: Understanding the Key Differences](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#web-scraping-vs-web-crawling-understanding-the-key-differences)
  - [What’s the Difference?](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#whats-the-difference)
  - [How Firecrawl Combines Both](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#how-firecrawl-combines-both)
- [Step-by-Step Guide to Web Crawling with Firecrawl’s API](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#step-by-step-guide-to-web-crawling-with-firecrawls-api)
- [Performance & Limits](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#performance--limits)
- [Asynchronous Web Crawling with Firecrawl: Efficient Large-Scale Data Collection](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#asynchronous-web-crawling-with-firecrawl-efficient-large-scale-data-collection)
  - [Asynchronous programming in a nutshell](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#asynchronous-programming-in-a-nutshell)
  - [Using `async_crawl_url` method](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#using-async_crawl_url-method)
  - [Benefits of asynchronous crawling](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#benefits-of-asynchronous-crawling)
- [How to Save and Store Web Crawling Results](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#how-to-save-and-store-web-crawling-results)
  - [Local file storage](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#local-file-storage)
- [Building AI-Powered Web Crawlers with Firecrawl and LangChain Integration](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#building-ai-powered-web-crawlers-with-firecrawl-and-langchain-integration)
- [Conclusion](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl#conclusion)

## Web Scraping vs Web Crawling: Understanding the Key Differences

### What’s the Difference?

_Web scraping_ refers to extracting specific data from individual web pages like a Wikipedia article or a technical tutorial. It is primarily used when you need specific information from pages with _known URLs_.

_Web crawling_, on the other hand, involves systematically browsing and discovering web pages by following links. It focuses on website navigation and URL discovery.

For example, to build a chatbot that answers questions about Stripe’s documentation, you would need:

1. Web crawling to discover and traverse all pages in Stripe’s documentation site
2. Web scraping to extract the actual content from each discovered page

### How Firecrawl Combines Both

Firecrawl’s `/crawl` endpoint combines both capabilities:

1. URL analysis: Identifies links through sitemap or page traversal
2. Recursive traversal: Follows links to discover sub-pages
3. Content scraping: Extracts clean content from each page
4. Results compilation: Converts everything to structured data

When you pass the URL `https://docs.stripe.com/api` to the endpoint, it automatically discovers and crawls all documentation sub-pages. The endpoint returns the content in your preferred format - whether that’s markdown, HTML, screenshots, links, or metadata.

## Step-by-Step Guide to Web Crawling with Firecrawl’s API

Firecrawl is a web scraping engine exposed as a REST API. You can use it from the command line using cURL or using one of its language SDKs for Python, Node, Go, or Rust. For this tutorial, we’ll focus on its Python SDK.

To get started:

1. Sign up at [firecrawl.dev](https://firecrawl.dev/) and copy your API key
2. Save the key as an environment variable:

```bash
export FIRECRAWL_API_KEY='fc-YOUR-KEY-HERE'

```

Or use a dot-env file:

```bash
touch .env
echo "FIRECRAWL_API_KEY='fc-YOUR-KEY-HERE'" >> .env

```

Then use the Python SDK:

```python
from firecrawl import FirecrawlApp
from dotenv import load_dotenv

load_dotenv()
app = FirecrawlApp()

```

Once your API key is loaded, the `FirecrawlApp` class uses it to establish a connection with the Firecrawl API engine.

First, we will crawl the [https://books.toscrape.com/](https://books.toscrape.com/) website, which is built for web-scraping practice:

![Homepage of Books to Scrape website showing a grid layout of book covers, prices and ratings - a popular web scraping practice site](https://www.firecrawl.dev/images/blog/crawl-masterclass/images/books_to_scrape_homepage.png)

Instead of writing dozens of lines of code with libraries like `beautifulsoup4` or `lxml` to parse HTML elements, handle pagination and data retrieval, Firecrawl’s `crawl_url` endpoint lets you accomplish this in a single line:

```python
base_url = "https://books.toscrape.com/"
crawl_result = app.crawl_url(url=base_url)

```

The result is a dictionary with the following keys:

```python
crawl_result.keys()

```

```text
dict_keys(['success', 'status', 'completed', 'total', 'creditsUsed', 'expiresAt', 'data'])

```

First, we are interested in the status of the crawl job:

```python
crawl_result['status']

```

```text
'completed'

```

If it is completed, let’s see how many pages were scraped:

```python
crawl_result['total']

```

```text
1195

```

Almost 1200 pages (it took about 70 seconds on my machine; the speed vary based on your connection speed). Let’s look at one of the elements of the `data` list:

```python
sample_page = crawl_result['data'][10]
markdown_content = sample_page['markdown']

print(markdown_content[:500])

```

```text
- [Home](../../../../index.html)
- [Books](../../books_1/index.html)
- Womens Fiction