# scrapy runspider hackernews_spider.py -o results.json

```

This code defines a simple Scrapy spider that crawls Hacker News. The spider starts at the homepage, extracts story titles from each page, and could optionally follow pagination links (currently commented out). The spider uses CSS selectors to find and extract content, demonstrating Scrapy’s built-in parsing capabilities. The results can be exported to JSON using Scrapy’s command line interface.

What sets Scrapy apart is its architecture. Instead of making one request at a time like BS4, Scrapy can handle multiple requests simultaneously, making it much faster for large projects. It also includes built-in features that you’d otherwise need to build yourself.

Scrapy’s key components include:

Spider middleware for customizing request/response handling, item pipelines for processing and storing data, and automatic request queuing and scheduling. It provides built-in support for exporting data in formats like JSON, CSV, and XML. The framework also includes robust error handling with retry mechanisms and a command-line interface for project management.

## 4\. Head-to-Head Comparison

Let’s break down how BS4 and Scrapy compare in key areas that matter most for web scraping projects.

### Performance

When it comes to speed and efficiency, Scrapy has a clear advantage. Its ability to handle multiple requests at once means it can scrape hundreds of pages while BS4 is still working on its first dozen. Think of BS4 as a solo worker, carefully processing one page at a time, while Scrapy is like a team of workers tackling many pages simultaneously.

Memory usage tells a similar story. BS4 is lightweight and uses minimal memory for single pages, making it perfect for small projects. However, Scrapy’s smart memory management shines when dealing with large websites, efficiently handling thousands of pages without slowing down your computer.

### Ease of Use

BS4 takes the lead in simplicity. You can start scraping with just 4-5 lines of code and basic Python knowledge. Here’s a quick comparison:

BS4:

```python
from bs4 import BeautifulSoup
import requests

response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "html.parser")
titles = soup.find_all("h1")

```

Scrapy:

```python
import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'
    start_urls = ['https://example.com']

    def parse(self, response):
        titles = response.css('h1::text').getall()
        yield {'titles': titles}