# Print the total number of pages crawled
print(f"Total pages crawled: {url_control_result['total']}")

```

```out
Total pages crawled: 99

```

Note: When a page has pagination (e.g. pages 2, 3, 4), these paginated pages are not counted as additional depth levels when using `maxDepth`.

### Performance & Limits

The `limit` parameter, which we’ve used in previous examples, is essential for controlling the scope of web crawling. It sets a maximum number of pages that will be scraped, which is particularly important when crawling large websites or when external links are enabled. Without this limit, the crawler could potentially traverse an endless chain of connected pages, consuming unnecessary resources and time.

While the limit parameter helps control the breadth of crawling, you may also need to ensure the quality and completeness of each page crawled. To make sure all desired content is scraped, you can enable a waiting period to let pages fully load. For example, some websites use JavaScript to handle dynamic content, have iFrames for embedding content or heavy media elements like videos or GIFs:

```python
stripe_crawl_result = app.crawl_url(
    url="https://docs.stripe.com/api",
    params={
        "limit": 5,
        "scrapeOptions": {
            "formats": ["markdown", "html"],
            "waitFor": 1000,  # wait for a second for pages to load
            "timeout": 10000,  # timeout after 10 seconds
        },
    },
)

```

The above code also sets the `timeout` parameter to 10000 milliseconds (10 seconds), which ensures that if a page takes too long to load, the crawler will move on rather than getting stuck.

Note: `waitFor` duration applies to all pages the crawler encounters.

All the while, it is important to keep the limits of your plan in mind:

| Plan | /scrape (requests/min) | /crawl (requests/min) | /search (requests/min) |
| --- | --- | --- | --- |
| Free | 10 | 1 | 5 |
| Hobby | 20 | 3 | 10 |
| Standard | 100 | 10 | 50 |
| Growth | 1000 | 50 | 500 |

## Asynchronous Web Crawling with Firecrawl: Efficient Large-Scale Data Collection

Even after following the tips and best practices from the previous section, the crawling process can be significantly long for large websites with thousands of pages. To handle this efficiently, Firecrawl provides asynchronous crawling capabilities that allow you to start a crawl and monitor its progress without blocking your application. This is particularly useful when building web applications or services that need to remain responsive while crawling is in progress.

### Asynchronous programming in a nutshell

First, let’s understand asynchronous programming with a real-world analogy:

Asynchronous programming is like a restaurant server taking multiple orders at once. Instead of waiting at one table until the customers finish their meal before moving to the next table, they can take orders from multiple tables, submit them to the kitchen, and handle other tasks while the food is being prepared.

In programming terms, this means your code can initiate multiple operations (like web requests or database queries) and continue executing other tasks while waiting for responses, rather than processing everything sequentially.

This approach is particularly valuable in web crawling, where most of the time is spent waiting for network responses - instead of freezing the entire application while waiting for each page to load, async programming allows you to process multiple pages concurrently, dramatically improving efficiency.

### Using `async_crawl_url` method

Firecrawl offers an intuitive asynchronous crawling method via `async_crawl_url`:

```python
app = FirecrawlApp()

crawl_status = app.async_crawl_url("https://docs.stripe.com")

print(crawl_status)

```

```python
{'success': True, 'id': 'c4a6a749-3445-454e-bf5a-f3e1e6befad7', 'url': 'https://api.firecrawl.dev/v1/crawl/c4a6a749-3445-454e-bf5a-f3e1e6befad7'}

```

It accepts the same parameters and scrape options as `crawl_url` but returns a crawl status dictionary.

We are mostly interested in the crawl job `id` and can use it to check the status of the process using `check_crawl_status`:

```python
checkpoint = app.check_crawl_status(crawl_status['id'])

print(len(checkpoint['data']))

```

```python
29

```

`check_crawl_status` returns the same output as `crawl_url` but only includes the pages scraped so far. You can run it multiple times and see the number of scraped pages increasing.

If you want to cancel the job, you can use `cancel_crawl` passing the job id:

```python
final_result = app.cancel_crawl(crawl_status['id'])

print(final_result)

```

```python
{'status': 'cancelled'}

```

### Benefits of asynchronous crawling

There are many advantages of using the `async_crawl_url` over `crawl_url`:

- You can create multiple crawl jobs without waiting for each to complete.
- You can monitor progress and manage resources more effectively.
- Perfect for batch processing or parallel crawling tasks.
- Applications can remain responsive while crawling happens in background
- Users can monitor progress instead of waiting for completion
- Allows for implementing progress bars or status updates
- Easier to integrate with message queues or job schedulers
- Can be part of larger automated workflows
- Better suited for microservices architectures

In practice, you almost always use asynchronous crawling for large websites.

## How to Save and Store Web Crawling Results

When crawling large websites, it’s important to save the results persistently. Firecrawl provides the crawled data in a structured format that can be easily saved to various storage systems. Let’s explore some common approaches.

### Local file storage

The simplest approach is saving to local files. Here’s how to save crawled content in different formats:

```python
import json
from pathlib import Path

def save_crawl_results(crawl_result, output_dir="firecrawl_output"):
    # Create output directory if it doesn't exist
    Path(output_dir).mkdir(parents=True, exist_ok=True)

    # Save full results as JSON
    with open(f"{output_dir}/full_results.json", "w") as f:
        json.dump(crawl_result, f, indent=2)

    # Save just the markdown content in separate files
    for idx, page in enumerate(crawl_result["data"]):
        # Create safe filename from URL
        filename = (
            page["metadata"]["url"].split("/")[-1].replace(".html", "") or f"page_{idx}"
        )

        # Save markdown content
        if "markdown" in page:
            with open(f"{output_dir}/{filename}.md", "w") as f:
                f.write(page["markdown"])

```

Here is what the above function does:

1. Creates an output directory if it doesn’t exist
2. Saves the complete crawl results as a JSON file with proper indentation
3. For each crawled page:
   - Generates a filename based on the page URL
   - Saves the markdown content to a separate .md file

```python
app = FirecrawlApp()

crawl_result = app.crawl_url(url="https://docs.stripe.com/api", params={"limit": 10})

save_crawl_results(crawl_result)

```

It is a basic function that requires modifications for other scraping formats.

### Database storage

For more complex applications, you might want to store the results in a database. Here’s an example using SQLite:

```python
import sqlite3

def save_to_database(crawl_result, db_path="crawl_results.db"):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    # Create table if it doesn't exist
    cursor.execute(
        """
        CREATE TABLE IF NOT EXISTS pages (
            url TEXT PRIMARY KEY,
            title TEXT,
            content TEXT,
            metadata TEXT,
            crawl_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """
    )

    # Insert pages
    for page in crawl_result["data"]:
        cursor.execute(
            "INSERT OR REPLACE INTO pages (url, title, content, metadata) VALUES (?, ?, ?, ?)",
            (
                page["metadata"]["url"],
                page["metadata"]["title"],
                page.get("markdown", ""),
                json.dumps(page["metadata"]),
            ),
        )

    conn.commit()

    print(f"Saved {len(crawl_result['data'])} pages to {db_path}")
    conn.close()

```

The function creates a SQLite database with a `pages` table that stores the crawled data. For each page, it saves the URL (as primary key), title, content (in markdown format), and metadata (as JSON). The crawl date is automatically added as a timestamp. If a page with the same URL already exists, it will be replaced with the new data. This provides a persistent storage solution that can be easily queried later.

```python
save_to_database(crawl_result)

```

```python
Saved 9 pages to crawl_results.db

```

Let’s query the database to double-check:

```python