# Scraping Company Data and Funding Information in Bulk With Firecrawl and Claude

![Scraping Company Data and Funding Information in Bulk With Firecrawl and Claude image](https://www.firecrawl.dev/images/blog/company-data-scraping/company-data-scraping.jpg)

## Introduction

In today’s data-driven business world, having access to accurate information about companies and their funding history is incredibly valuable. There are several online databases that track startups, investments, and company growth, containing details about millions of businesses, their funding rounds, and investors. While many of these platforms offer APIs, they can be expensive and out of reach for many users. This tutorial will show you how to build a web scraper that can gather company and funding data from public sources using Python, Firecrawl, and Claude.

This guide is designed for developers who want to collect company data efficiently and ethically. By the end of this tutorial, you’ll have a working tool that can extract company details, funding rounds, and investor information from company profiles across the web.

Here is the preview of the app:

![Screenshot of the Crunchbase scraping app interface showing company input fields, a scraping button, and results display area with download option](https://www.firecrawl.dev/images/blog/company-data-scraping/app_demo.png)

The application provides two input methods for users - they can either upload a file containing company names or enter them directly as text. Behind the scenes, Firecrawl automatically scrapes relevant company information from public databases like Crunchbase. This scraped data is then processed by Claude, an AI assistant that generates concise company summaries. The results are displayed in a clean Streamlit interface, complete with a download option that exports all findings to a CSV file for further analysis.

## Table of Contents

1. Introduction
2. Setting up the Development Environment
3. Prerequisite: Scraping with Firecrawl
4. Building the Funding Data Scraper
   - Step 1: Adding brief app information
   - Step 2: Adding components for company name input
   - Step 3: Building a scraping class
   - Step 4: Adding a scraping button
   - Step 5: Creating a download button
   - Step 6: Generating summaries
   - Step 7: Deployment
5. Conclusion

**Time to Complete:** ~60 minutes
**Prerequisites:**

- Python 3.10+
- Basic Python knowledge
- API keys for Firecrawl and Claude

**Important Note:** This tutorial demonstrates web scraping for educational purposes. Always review and comply with websites’ terms of service and implement appropriate rate limiting in production environments.

## Setting up the Development Environment

Let’s start by setting up our development environment and installing the necessary dependencies.

1. **Create a working directory**

First, create a working directory:

```bash
mkdir company-data-scraper
cd company-data-scraper

```

2. **Install dependencies**

We’ll use Poetry for dependency management. If you haven’t installed Poetry yet:

```bash
curl -sSL https://install.python-poetry.org | python3 -

```

Then, initialize it inside the current working directory:

```bash
poetry init

```

Type “^3.10” when asked for the Python version but, don’t specify the dependencies interactively.

Next, install the project dependencies with the `add` command:

```bash
poetry add streamlit firecrawl-py pandas pydantic openpyxl python-dotenv anthropic

```

3. **Build the project structure**

```bash
mkdir data src
touch .gitignore README.md .env src/{app.py,models.py,scraper.py}

```

The created files serve the following purposes:

- `data/` \- Directory to store input files and scraped results
- `src/` \- Source code directory containing the main application files
- `.gitignore` \- Specifies which files Git should ignore
- `README.md` \- Project documentation and setup instructions
- `.env` \- Stores sensitive configuration like API keys
- `src/app.py` \- Main Streamlit application and UI code
- `src/models.py` \- Data models and validation logic
- `src/scraper.py` \- Web scraping and data collection functionality

4. **Configure environment variables**

This project requires two accounts of third-party services:

- [Firecrawl](https://firecrawl.dev/) for AI-powered web scraping
- [Anthropic (Claude)](https://console.anthropic.com/) for summarizing scraped data

Click on the hyperlinks above to create your accounts and copy/generate your API keys. Then, Inside the `.env` file in the root directory, add your API keys:

```plaintext
FIRECRAWL_API_KEY=your_api_key_here
ANTHROPIC_API_KEY=your_api_key_here

```

The `.env` file is used to store sensitive configuration like API keys securely.The `python-dotenv` package will automatically load these environment variables when the app starts. It should never be committed to version control so add the following line to your `.gitignore` file:

```plaintext
.env

```

5. **Start the app UI**

Run the Streamlit app (which is blank just now) to ensure everything is working:

```bash
poetry run streamlit run src/app.py

```

You should see the Streamlit development server start up and your default browser open to the app’s interface. Keep this tab open to see the changes we make to the app in the next steps.

Now that we have our development environment set up, let’s cover how Firecrawl works, which is a prerequisite to building our app.

## Prerequisite: Scraping with Firecrawl

The biggest challenge with any application that scrapes websites is maintenance. Since websites regularly update their layout and underlying HTML/CSS code, traditional scrapers break easily, making the entire app useless. Firecrawl solves this exact problem by allowing you to scrape websites using natural language.

Instead of writing complex CSS selectors and XPath expressions that need constant maintenance, you can simply describe what data you want to extract in plain English. Firecrawl’s AI will figure out how to get that data from the page, even if the website’s structure changes. This makes our scraper much more reliable and easier to maintain over time.

Here is a simple Firecrawl workflow we will later use in the app to scrape company information:

```python
from firecrawl import FirecrawlApp
from pydantic import BaseModel, Field
from dotenv import load_dotenv

load_dotenv()

```

```plaintext
True

```

We import `FirecrawlApp` to interact with the Firecrawl API for web scraping. `BaseModel` and `Field` from pydantic help us define structured data models with validation. The `load_dotenv` function loads environment variables from our `.env` file to securely access API keys.

```python