# firecrawl_scraper.py
import json
from firecrawl import FirecrawlApp
from dotenv import load_dotenv
from pydantic import BaseModel, Field
from typing import List
from datetime import datetime

load_dotenv()
BASE_URL = "https://news.ycombinator.com/"

```

After the imports, we are calling `load_dotenv()` so that our Firecrawl API key is loaded from the `.env` file. Then, we are defining a new variable containing the URL we will scrape.

Next, we create a Pydantic model to specify the information we want to scrape from each Hacker News post:

```python
class NewsItem(BaseModel):
    title: str = Field(description="The title of the news item")
    source_url: str = Field(description="The URL of the news item")
    author: str = Field(
        description="The URL of the post author's profile concatenated with the base URL."
    )
    rank: str = Field(description="The rank of the news item")
    upvotes: str = Field(description="The number of upvotes of the news item")
    date: str = Field(description="The date of the news item.")

```

Pydantic models are Python classes that provide data validation and serialization capabilities. They allow you to define the structure and types of your data using Python type hints, while automatically handling validation, serialization, and documentation.

In the context of our Firecrawl scraper, the `NewsItem` model defines the exact structure of data we want to extract from each Hacker News post. Each field in the model ( `title`, `source_url`, `author`, etc.) specifies what data should be scraped and includes a description of what that field represents.

This model is crucial for Firecrawl because it uses the model’s schema to understand exactly what data to extract from the webpage. When we pass this model to Firecrawl, it will automatically attempt to find and extract data matching these field definitions from the HTML structure of Hacker News.

For example, when Firecrawl sees we want a “title” field, it will look for elements on the page that are likely to contain post titles based on their HTML structure and content. The `Field` descriptions help provide additional context about what each piece of data represents.

Next, we create another model called `NewsData` that contains a list of `NewsItem` objects. This model will serve as a container for all the news items we scrape from Hacker News. The `news_items` field is defined as a List of `NewsItem` objects, which means it can store multiple news items in a single data structure.

```python
class NewsData(BaseModel):
    news_items: List[NewsItem]

```

Without this second model, our scraper will return not one but all news items.

Now, we define a new function that will run Firecrawl based on the scraping schema we just defined:

```python
def get_firecrawl_news_data():
    app = FirecrawlApp()

    data = app.scrape_url(
        BASE_URL,
        params={
            "formats": ["extract"],
            "extract": {"schema": NewsData.model_json_schema()},
        },
    )

    return data

```

This function initializes a FirecrawlApp instance and uses it to scrape data from Hacker News. It passes the `BASE_URL` and parameters specifying that we want to extract data according to our `NewsData` schema. The schema tells Firecrawl exactly what fields to look for and extract from each news item on the page. The function returns the scraped data which will contain a list of news items matching our defined structure.

Let’s quickly test it:

```python
data = get_firecrawl_news_data()

print(type(data))

```

```python
<class 'dict'>

```

Firecrawl always returns the scraped data in a dictionary. Let’s look at the keys:

```python
data['metadata']

```

```python
{
    'title': 'Hacker News',
    'language': 'en',
    'ogLocaleAlternate': [],
    'referrer': 'origin',
    'viewport': 'width=device-width, initial-scale=1.0',
    'sourceURL': 'https://news.ycombinator.com/',
    'url': 'https://news.ycombinator.com/',
    'statusCode': 200
}

```

The first key is the metadata field containing basic page information. We are interested in the `extract` field which contains the data scraped by the engine:

```python
data['extract']['news_items'][0]

```

```python
{
    'title': "Send someone you appreciate an official 'Continue and Persist' Letter",
    'source_url': 'https://ContinueAndPersist.org',
    'author': 'https://news.ycombinator.com/user?id=adnanaga',
    'rank': '1',
    'upvotes': '1122',
    'date': '17 hours ago'
}

```

The `extract` field contains a dictionary with a list of scraped news items. We can see above that when printing the first item, it includes all the fields we defined in our `NewsItem` Pydantic model, including title, source URL, author, rank, upvotes and date.

```python
len(data['extract']['news_items'])

```

```python
30

```

The output shows 30 news items, confirming that our scraper successfully extracted all posts from the first page of Hacker News. This matches the site’s standard layout which displays exactly 30 posts per page.

Now, let’s create a new function that saves this data to a JSON file:

```python
def save_firecrawl_news_data():
    # Get the data
    data = get_firecrawl_news_data()
    # Format current date for filename
    date_str = datetime.now().strftime("%Y_%m_%d_%H_%M")
    filename = f"firecrawl_hacker_news_data_{date_str}.json"

    # Save the news items to JSON file
    with open(filename, "w") as f:
        json.dump(data["extract"]["news_items"], f, indent=4)

    return filename

```

`save_firecrawl_news_data()` handles saving the scraped Hacker News data to a JSON file. It first calls `get_firecrawl_news_data()` to fetch the latest data from Hacker News. Then, it generates a filename using the current timestamp in the format `YYYY_MM_DD_HH_MM`. The data is saved to this timestamped JSON file with proper indentation, and the filename is returned. This allows us to maintain a historical record of the scraped data with clear timestamps indicating when each scrape occurred.

Finally, add a `__main__` block to the `firecrawl_scraper.py` script to allow running the scraper directly from the command line:

```python
if __name__ == "__main__":
    save_firecrawl_news_data()

```

The complete scraper script is available in [our GitHub repository](https://github.com/firecrawl/firecrawl/blob/main/examples/hacker_news_scraper/firecrawl_scraper.py). For reference, we also provide [a BeautifulSoup implementation of the same scraper](https://github.com/firecrawl/firecrawl/blob/main/examples/hacker_news_scraper/bs4_scraper.py).

## Local Web Scraping Automation Methods

In this section, we will explore how to run the scraper from the previous section on schedule using local tools like the Python `schedule` library and cron.

### The basics of the Python `schedule` library

`schedule` is a job scheduling library written for humans ( [from the documentation](https://schedule.readthedocs.io/en/stable/index.html)). It runs any Python function or callable periodically using intuitive syntax.

To get started, please install it with `pip`:

```bash
pip install schedule

```

Then, scheduling a Python function is as easy as shown in the codeblock below:

```python
import schedule
import time

def job():
    current_time = time.strftime("%H:%M:%S")
    print(f"{current_time}: I'm working...")