# Initialize FirecrawlApp
app = FirecrawlApp(api_key=firecrawl_api_key)

```

### Step 3: Define the Grok-2 API Interaction Function

We need a function to interact with the Grok-2 API.

```python
def grok_completion(prompt):
    url = "https://api.x.ai/v1/chat/completions"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {grok_api_key}"
    }
    data = {
        "messages": [\
            {"role": "system", "content": "You are a helpful assistant."},\
            {"role": "user", "content": prompt}\
        ],
        "model": "grok-2",
        "stream": False,
        "temperature": 0
    }
    response = requests.post(url, headers=headers, json=data)
    response_data = response.json()
    return response_data['choices'][0]['message']['content']

```

### Step 4: Identify Relevant Pages on the Website

Define a function to find pages related to our objective.

```python
def find_relevant_pages(objective, url):
    prompt = f"Based on the objective '{objective}', suggest a 1-2 word search term to locate relevant information on the website."
    search_term = grok_completion(prompt).strip()
    map_result = app.map_url(url, params={"search": search_term})
    return map_result.get("links", [])

```

### Step 5: Extract Data from the Pages

Create a function to scrape the pages and extract the required data.

```python
def extract_data_from_pages(links, objective):
    for link in links[:3]:  # Limit to top 3 links
        scrape_result = app.scrape_url(link, params={'formats': ['markdown']})
        content = scrape_result.get('markdown', '')
        prompt = f"""Given the following content, extract the information related to the objective '{objective}' in JSON format. If not found, reply 'Objective not met'.

Content: {content}

Remember:
- Only return JSON if the objective is met.
- Do not include any extra text.
"""
        result = grok_completion(prompt).strip()
        if result != "Objective not met":
            try:
                data = json.loads(result)
                return data
            except json.JSONDecodeError:
                continue  # Try the next link if JSON parsing fails
    return None

```

### Step 6: Implement the Main Function

Combine everything into a main function.

```python
def main():
    url = input("Enter the website URL to crawl: ")
    objective = input("Enter your data extraction objective: ")

    print("\nFinding relevant pages...")
    links = find_relevant_pages(objective, url)

    if not links:
        print("No relevant pages found.")
        return

    print("Extracting data from pages...")
    data = extract_data_from_pages(links, objective)

    if data:
        print("\nData extracted successfully:")
        print(json.dumps(data, indent=2))
    else:
        print("Could not find data matching the objective.")

if __name__ == "__main__":
    main()

```

### Step 7: Run the Script

Save your script and run it from the command line.

```bash
python web_crawler.py

```

**Example Interaction:**

```
Enter the website URL to crawl: https://example.com
Enter your data extraction objective: Retrieve the list of services offered.

Finding relevant pages...
Extracting data from pages...

Data extracted successfully:
{
  "services": [\
    "Web Development",\
    "SEO Optimization",\
    "Digital Marketing"\
  ]
}

```

## Conclusion

In this tutorial, we’ve successfully set up Grok-2, obtained an API key, and built a web crawler using Firecrawl. This powerful combination allows you to automate the process of extracting structured data from websites, making it a valuable tool for various applications.

## Next Steps

- **Explore More Features:** Check out the Grok-2 and Firecrawl documentation to learn about additional functionalities.
- **Enhance Error Handling:** Improve the script with better error handling and logging.
- **Customize Data Extraction:** Modify the extraction logic to suit different objectives or data types.

## References

- [x.ai Grok-2 API Documentation](https://docs.x.ai/docs)
- [Firecrawl Python Library Documentation](https://docs.firecrawl.dev/sdks/python)
- [x.ai Cloud Console](https://accounts.x.ai/cloud-console)
- [GitHub Repository with Full Code](https://github.com/firecrawl/firecrawl/tree/main/examples/grok_web_crawler)

Article updated recently

[🔥](https://www.firecrawl.dev/)

## Ready to _Build?_

Start scraping web data for your AI apps today.

No credit card needed.

Get Started

## About the Author

[![Nicolas Camara image](https://www.firecrawl.dev/nick-img.jpeg)\\
Nicolas Camara@nickscamara\_](https://x.com/nickscamara_)

Nicolas Camara is the Chief Technology Officer (CTO) at Firecrawl.
He previously built and scaled Mendable, one of the pioneering "chat with your documents" apps,
which had major Fortune 500 customers like Snapchat, Coinbase, and MongoDB.
Prior to that, Nicolas built SideGuide, the first code-learning tool inside VS Code,
and grew a community of 50,000 users. Nicolas studied Computer Science and has over 10 years of experience in building software.

### More articles by Nicolas Camara

[Using OpenAI's Realtime API and Firecrawl to Talk with Any Website\\
\\
Build a real-time conversational agent that interacts with any website using OpenAI's Realtime API and Firecrawl.](https://www.firecrawl.dev/blog/How-to-Talk-with-Any-Website-Using-OpenAIs-Realtime-API-and-Firecrawl) [Extract website data using LLMs\\
\\
Learn how to use Firecrawl and Groq to extract structured data from a web page in a few lines of code.](https://www.firecrawl.dev/blog/data-extraction-using-llms) [Getting Started with Grok-2: Setup and Web Crawler Example\\
\\
A detailed guide on setting up Grok-2 and building a web crawler using Firecrawl.](https://www.firecrawl.dev/blog/grok-2-setup-and-web-crawler-example) [Launch Week I / Day 6: LLM Extract (v1)\\
\\
Extract structured data from your web pages using the extract format in /scrape.](https://www.firecrawl.dev/blog/launch-week-i-day-6-llm-extract) [Launch Week I / Day 7: Crawl Webhooks (v1)\\
\\
New /crawl webhook support. Send notifications to your apps during a crawl.](https://www.firecrawl.dev/blog/launch-week-i-day-7-webhooks) [OpenAI Swarm Tutorial: Create Marketing Campaigns for Any Website\\
\\
A guide to building a multi-agent system using OpenAI Swarm and Firecrawl for AI-driven marketing strategies](https://www.firecrawl.dev/blog/openai-swarm-agent-tutorial) [Build a 'Chat with website' using Groq Llama 3\\
\\
Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.](https://www.firecrawl.dev/blog/chat-with-website)

## Install BeautifulSoup Easily
Introducing /extract - Get web data with a prompt [Try now](https://www.firecrawl.dev/extract)

Aug 9, 2024

•

[![Eric Ciarla image](https://www.firecrawl.dev/eric-img.jpeg)Eric Ciarla](https://x.com/ericciarla)