# Schedule the scraper to run every hour
schedule.every().hour.do(save_firecrawl_news_data)

while True:
    schedule.run_pending()
    time.sleep(1)

```

You can start the schedule with:

```bash
python scrape_scheduler.py

```

> **Tip**: For debugging purposes, start with a shorter interval like 60 seconds before implementing the hourly schedule.

The scheduler will continue running until you terminate the main terminal process executing the `scrape_scheduler.py` script. Thanks to Firecrawl’s AI-powered HTML parsing and layout adaptation capabilities, the scraper is quite resilient to website changes and has a low probability of breaking.

Nevertheless, web scraping can be unpredictable, so it’s recommended to review [the exception handling](https://schedule.readthedocs.io/en/stable/exception-handling.html) section of the `schedule` documentation to handle potential errors gracefully.

* * *

The `schedule` library provides a simple and intuitive way to run periodic tasks like web scrapers. While it lacks some advanced features of other scheduling methods, it’s a great choice for basic scheduling needs and getting started with automated scraping. Just remember to implement proper error handling for production use.

### Using Python’s built-in tools to automate web scrapers

In this section, we will explore a few other local scheduling methods that have the advantage of being built into Python or the operating system, making them more reliable and robust than third-party libraries. These methods also provide better error handling, logging capabilities, and system-level control over the scheduling process.

#### How to automate a web scraper with `asyncio`?

`asyncio` is a Python library for running code concurrently - executing multiple tasks at the same time by switching between them when one is waiting. It’s built into Python and helps schedule tasks efficiently. Here’s why it’s great for web scrapers:

1. It can do other work while waiting for web requests to complete.
2. You can run multiple scrapers at the same time with precise timing control.
3. It uses less computer resources than regular multitasking.
4. It handles errors well with `try/except` blocks.

Let’s see how to use `asyncio` for scheduling scrapers:

```python
import asyncio
import time
from firecrawl_scraper import save_firecrawl_news_data

async def schedule_scraper(interval_hours: float = 1):
    while True:
        try:
            print(f"Starting scrape at {time.strftime('%Y-%m-%d %H:%M:%S')}")
            # Run the scraper
            filename = save_firecrawl_news_data()
            print(f"Data saved to {filename}")

        except Exception as e:
            print(f"Error during scraping: {e}")

        # Wait for the specified interval
        await asyncio.sleep(interval_hours * 3600)  # Convert hours to seconds

async def main():
    # Create tasks for different scheduling intervals
    tasks = [\
        schedule_scraper(interval_hours=1),  # Run every hour\
        # Add more tasks with different intervals if needed\
        # schedule_scraper(interval_hours=0.5),  # Run every 30 minutes\
        # schedule_scraper(interval_hours=2),    # Run every 2 hours\
    ]

    # Run all tasks concurrently
    await asyncio.gather(*tasks)

if __name__ == "__main__":
    # Run the async scheduler
    asyncio.run(main())

```

Let’s break down what’s happening above:

The `schedule_scraper()` function is an `async` function that runs indefinitely in a loop. For each iteration, it:

1. Runs the scraper and saves the data
2. Handles any errors that occur during scraping
3. Waits for the specified interval using `asyncio.sleep()`

The `main()` function sets up concurrent execution of multiple scraper tasks with different intervals. This allows running multiple scrapers simultaneously without blocking each other.

This asyncio-based approach has several advantages over the `schedule` library:

1. True concurrency: Multiple scrapers can run simultaneously without blocking each other, unlike `schedule` which runs tasks sequentially.

2. Precise Timing: `asyncio.sleep()` provides more accurate timing control compared to schedule’s `run_pending()` approach.

3. Resource Efficiency: `asyncio` uses cooperative multitasking which requires less system resources than `schedule`’s threading-based approach.

4. Better Error Handling: `Async/await` makes it easier to implement proper error handling and recovery

5. Flexibility: You can easily add or remove scraper tasks and modify their intervals without affecting other tasks


The code structure also makes it simple to extend functionality by adding more concurrent tasks or implementing additional error handling logic.

#### How to automate a web scraper with cron jobs?

Cron is a time-based job scheduler in Unix-like operating systems (Linux, macOS). Think of it as a digital scheduler or calendar that can automatically run programs at specified times. A cron job is simply a task that you schedule to run at specific intervals.

For web scraping, cron jobs are incredibly useful because they let you automate your scraper to run at predetermined times. For example, you could set up a cron job to:

- Run your scraper every hour to collect real-time data
- Execute scraping tasks during off-peak hours (like 2 AM) to minimize server load
- Collect data at specific times when websites update their content

The scheduling format uses five time fields: minute, hour, day of month, month, and day of week. For instance:

- `0 * * * *` means “run every hour at minute 0”
- `0 0 * * *` means “run at midnight every day”
- `*/15 * * * *` means “run every 15 minutes”

Cron jobs are especially reliable for web scraping because they’re built into the operating system, use minimal resources, and continue running even after system reboots.

So, let’s run the `save_firecrawl_news_data()` function on a schedule using cron. First, we will create a dedicated script for the cron job named `cron_scraper.py`:

```python