# from scraper import CrunchbaseScraper

load_dotenv()

```

The imports above serve the following purposes:

- `streamlit`: Provides the web interface components and app framework
- `pandas`: Used for data manipulation and CSV file handling
- `anthropic`: Client library for accessing Claude AI capabilities
- `typing.List`: Type hint for lists to improve code readability
- `dotenv`: Loads environment variables from `.env` file for configuration

Currently, the `CrunchbaseScraper` class is commented out since we are yet to write it.

Next, we create a `main` function that holds the core UI components:

```python
def main():
    st.title("Crunchbase Company Data Scraper")
    st.write(
        """
        Extract detailed company information from Crunchbase including funding data,
        employee counts, industries, and more. Upload a file with company names or
        enter them manually below.
        """
    )

```

Right now, the function gives brief info about the app’s purpose. To run the app, add the following `main` block to the end of `src/app.py`:

```python
if __name__ == "__main__":
    main()

```

You should see the change in the Streamlit development server.

### Step 2: Adding components for company name input

In this step, we add a new function to `src/app.py`:

```python
def load_companies(file) -> List[str]:
    """Load company names from uploaded file"""
    companies = []
    for line in file:
        company = line.decode("utf-8").strip()
        if company:  # Skip empty lines
            companies.append(company)
    return companies

```

The `load_companies` function takes a file object as input and parses it line by line, extracting company names. It decodes each line from bytes to UTF-8 text, strips whitespace, and skips any empty lines. The function returns a list of company names that can be used for scraping Crunchbase data.

Now, make the following changes to the `main` function:

```python
def main():
    st.title("Crunchbase Company Data Scraper")
    st.write(
        """
        Extract detailed company information from Crunchbase including funding data,
        employee counts, industries, and more. Upload a file with company names or
        enter them manually below.
        """
    )

    # File upload option
    uploaded_file = st.file_uploader(
        "Upload a text file with company names (one per line)", type=["txt"]
    )

    # Manual input option
    st.write("### Or Enter Companies Manually")
    manual_input = st.text_area(
        "Enter company names (one per line)",
        height=150,
        help="Enter each company name on a new line",
    )

```

In this version, we’ve added two main ways for users to input company names: file upload and manual text entry. The file upload component accepts `.txt` files and for manual entry, users can type or paste company names directly into a text area, with each name on a new line. This provides flexibility for users whether they have a prepared list or want to enter names ad-hoc.

Furthermore, add these two blocks of code after the input components:

```python
def main():
    ...

    companies = []

    if uploaded_file:
        companies = load_companies(uploaded_file)
        st.write(f"Loaded {len(companies)} companies from file")
    elif manual_input:
        companies = [line.strip() for line in manual_input.split("\n") if line.strip()]
        st.write(f"Found {len(companies)} companies in input")

```

This code block processes the user input to create a list of company names. When a file is uploaded, it uses the `load_companies()` function to read and parse the file contents. For manual text input, it splits the input text by newlines and strips whitespace to extract company names. In both cases, it displays a message showing how many companies were found. The companies list will be used later for scraping data from funding data sources.

### Step 3: Building a scraping class with Firecrawl

Let’s take a look at the snapshot of the final UI once again:

![Screenshot of the Crunchbase scraping app interface showing company input fields, a scraping button, and results display area with download option](https://www.firecrawl.dev/images/blog/company-data-scraping/app_demo.png)

In this step, we implement the backend process that happens when a user clicks on “Start scraping” button. To do so, we use Firecrawl like we outlined in the prerequisites section. First, go to `src/models.py` script to write the data model we are going to use to scrape company and funding information:

```python
from pydantic import BaseModel
from typing import List, Optional

class CompanyData(BaseModel):
    name: str
    about: Optional[str]
    employee_count: Optional[str]
    financing_type: Optional[str]
    industries: List[str] = []
    headquarters: List[str] = []
    founders: List[str] = []
    founded_date: Optional[str]
    operating_status: Optional[str]
    legal_name: Optional[str]
    stock_symbol: Optional[str]
    acquisitions: List[str] = []
    investments: List[str] = []
    exits: List[str] = []
    total_funding: Optional[str]
    contacts: List[str] = []

```

This data model is more detailed and tries to extract as much information as possible from given sources. Now, switch to `src/scraper.py` where we implement a class called `CrunchbaseScraper`:

```python
from firecrawl import FirecrawlApp
from models import CompanyData
from typing import List, Dict

class CrunchbaseScraper:
    def __init__(self):
        self.app = FirecrawlApp()

    def scrape_companies(self, urls: List[str]) -> List[Dict]:
        """Scrape multiple Crunchbase company profiles"""
        schema = CompanyData.model_json_schema()

        try:
            data = self.app.batch_scrape_urls(
                urls,
                params={
                    "formats": ["extract"],
                    "extract": {
                        "prompt": """Extract information from given pages based on the schema provided.""",
                        "schema": schema,
                    },
                },
            )

            return [res["extract"] for res in data["data"]]

        except Exception as e:
            print(f"Error while scraping companies: {str(e)}")
            return []

```

Let’s break down how the class works.

When the class is initialized, it creates an instance of `FirecrawlApp`. The main method `scrape_companies` takes a list of URLs and returns a list of dictionaries containing the scraped data. It works by:

1. Getting the JSON schema from our `CompanyData` model to define the structure
2. Using `batch_scrape_urls` to process multiple URLs at once
3. Configuring the scraper to use the “extract” format with our schema
4. Providing a prompt that instructs the scraper how to extract the data
5. Handling any errors that occur during scraping

Error handling ensures the script continues running even if individual URLs fail, returning an empty list in case of errors rather than crashing.

Now, the only thing left to do to finalize the scraping feature is to add the “Start Scraping” button to the UI.

### Step 4: Adding a button to start scraping

In this step, return to `src/app.py` and add the following code block to the very end of the `main()` function:

```python
def main():
    ...

    if companies and st.button("Start Scraping"):
        scraper = CrunchbaseScraper()

        with st.spinner("Scraping company data from Crunchbase..."):
            try:
                # Convert company names to Crunchbase URLs
                urls = [\
                    f"https://www.crunchbase.com/organization/{name.lower().replace(' ', '-')}"\
                    for name in companies\
                ]

                results = scraper.scrape_companies(urls)

            except Exception as e:
                st.error(f"An error occurred: {str(e)}")

```

This code block builds on the previous functionality by adding the core scraping logic. When the “Start Scraping” button is clicked (and companies have been provided), it:

1. Creates a new instance of our `CrunchbaseScraper` class
2. Shows a loading spinner to indicate scraping is in progress
3. Converts the company names into proper Crunchbase URLs by:
   - Converting to lowercase
   - Replacing spaces with hyphens
   - Adding the base Crunchbase URL prefix
4. Calls the `scrape_companies` method we created earlier to fetch the data

The try-except block ensures any scraping errors are handled gracefully rather than crashing the application. This is important since web scraping can be unpredictable due to network issues, rate limiting, and so on.

To finish this step, uncomment the single import at the top of `src/app.py` so that they look like this:

```python
import streamlit as st
import pandas as pd
import anthropic
from typing import List
from scraper import CrunchbaseScraper
from dotenv import load_dotenv

load_dotenv()

```

### Step 5: Creating a download button for the scraped results

Now, we must create a button to download the scraped results as a CSV file. To do so, add the following code block after the scraping part:

```python
def main():
    ...

    if companies and st.button("Start Scraping"):
        scraper = CrunchbaseScraper()

        with st.spinner("Scraping company data from Crunchbase..."):
            try:
                ...
                results = scraper.scrape_companies(urls)

                # THIS PART IS NEW
                df = pd.DataFrame(results)
                csv = df.to_csv(index=False)

                # Create download button
                st.download_button(
                    "Download Results (CSV)",
                    csv,
                    "crunchbase_data.csv",
                    "text/csv",
                    key="download-csv",
                )

            except Exception as e:
                st.error(f"An error occurred: {str(e)}")

```

In the new lines of code, we convert the results to a Pandas dataframe and use its `to_csv()` function to save the dataframe as a CSV file. The method returns a filename, which we pass to the `st.download_button` method along with other details.

### Step 6: Generating a summary of scraped results

After scraping the raw company data, we can use Claude to generate concise summaries that highlight key insights. Let’s add this functionality to our app. First, create a new function in `src/app.py` to handle the summarization:

```python
def generate_company_summary(company_data: dict) -> str:
    """Generate a summary of the company data"""
    client = anthropic.Anthropic()

    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1000,
        system="You are a company & funding data expert. Summarize the given company data by the user in a few sentences.",
        messages=[\
            {"role": "user", "content": [{"type": "text", "text": str(company_data)}]}\
        ],
    )

    return message.content[0].text

```

Now, update the scraping section in the `main()` function to include the summary generation after the download button:

```python
def main():
    ...

    if companies and st.button("Start Scraping"):
        scraper = CrunchbaseScraper()

        with st.spinner("Scraping company data from Crunchbase..."):
            try:
                ...

                # Give summary of each company
                for company in results:
                    summary = generate_company_summary(company)
                    st.write(f"### Summary of {company['name']}")
                    st.write(summary)

            except Exception as e:
                st.error(f"An error occurred: {str(e)}")

```

This implementation:

1. Creates a new `generate_company_summary()` function that:

- Formats the scraped company data into readable text
- Uses Claude to analyze the data and generate insights
- Returns a structured summary highlighting key patterns.

2. Updates the main scraping workflow to:

- Generate the summary after scraping is complete
- Display the insights for each company after the download button

The summary can provide context about the scraped data, helping users get the gist of the scraped data.

### Step 7: Deploying the app to Streamlit Cloud

Now that our app is working locally, let’s deploy it to Streamlit Cloud so others can use it. First, we need to prepare our project for deployment.

1. **Create a requirements.txt**

Since Streamlit Cloud doesn’t support Poetry directly, we need to convert our dependencies to a `requirements.txt` file. Run this command in your terminal:

```bash
poetry export -f requirements.txt --output requirements.txt --without-hashes

```

2. **Create a GitHub repository**

Initialize a Git repository and push your code to GitHub:

```python
git init
git add .
git commit -m "Initial commit"
git branch -M main
git remote add origin https://github.com/yourusername/company-data-scraper.git
git push -u origin main

```

3. **Add secrets to Streamlit Cloud**

Visit [share.streamlit.io](https://share.streamlit.io/) and connect your GitHub account. Then:

1. Click “New app”
2. Select your repository and branch
3. Set the main file path as src/app.py
4. Click “Advanced settings” and add your environment variables:

- `FIRECRAWL_API_KEY`
- `ANTHROPIC_API_KEY`

4. **Update imports for deployment**

Sometimes local imports need adjustment for Streamlit Cloud. Ensure your imports in src/app.py use relative paths:

```python
from .models import CompanyData
from .scraper import CrunchbaseScraper

```

5. **Add a .streamlit/config.toml file**

Create a `.streamlit` directory and add a `config.toml` file for custom theme settings:

```python
[theme]
primaryColor = "#FF4B4B"
backgroundColor = "#FFFFFF"
secondaryBackgroundColor = "#F0F2F6"
textColor = "#262730"
font = "sans serif"

[server]
maxUploadSize = 5

```

6. **Create a README.md file**

Add a README.md file to help users understand your app:

```markdown