# Get logger for the scraper module
logger = logging.getLogger(__name__)

```

Then, we define the core data structure for documentation pages:

```python
class DocPage(BaseModel):
    title: str = Field(description="Page title")
    content: str = Field(description="Main content of the page")
    url: str = Field(description="Page URL")

```

The `DocPage` model represents a single documentation page with three essential fields:

- `title`: The page’s heading or title
- `content`: The main markdown content of the page
- `url`: Direct link to the original page

This model is used by both the scraper to structure extracted content and the RAG system to process documentation for the vector store.

The main scraper class handles all documentation collection:

```python
class DocumentationScraper:
    def init(self):
        self.app = FirecrawlApp()

```

The `DocumentationScraper` initializes a connection to Firecrawl and provides three main methods for documentation collection:

1. `get_documentation_links`: Discovers all documentation pages from a base URL:

```python
def get_documentation_links(self, base_url: str) -> list[str]:
    """Get all documentation page links from a given base URL."""
    logger.info(f"Getting documentation links from {base_url}")
    initial_crawl = self.app.crawl_url(
        base_url,
        params={
            "scrapeOptions": {"formats": ["links"]},
        },
    )
    all_links = []
    for item in initial_crawl["data"]:
        all_links.extend(item["links"])
    filtered_links = set(
        [link.split("#")[0] for link in all_links if link.startswith(base_url)]
    )
    logger.info(f"Found {len(filtered_links)} unique documentation links")
    return list(filtered_links)

```

This method:

- Uses Firecrawl’s link extraction mode to find all URLs
- Filters for links within the same documentation domain
- Removes duplicate URLs and anchor fragments
- Returns a clean list of documentation page URLs

2. `scrape_documentation`: Processes all documentation pages into structured content:

```python
def scrape_documentation(self, base_url: str, limit: int = None):
    """Scrape documentation pages from a given base URL."""
    logger.info(f"Scraping doc pages from {base_url}")

    filtered_links = self.get_documentation_links(base_url)
    if limit:
        filtered_links = filtered_links[:limit]

    try:
        logger.info(f"Scraping {len(filtered_links)} documentation pages")
        crawl_results = self.app.batch_scrape_urls(filtered_links)
    except Exception as e:
        logger.error(f"Error scraping documentation pages: {str(e)}")
        return []

    doc_pages = []
    for result in crawl_results["data"]:
        if result.get("markdown"):
            doc_pages.append(
                DocPage(
                    title=result.get("metadata", {}).get("title", "Untitled"),
                    content=result["markdown"],
                    url=result.get("metadata", {}).get("url", ""),
                )
            )
        else:
            logger.warning(
                f"Failed to scrape {result.get('metadata', {}).get('url', 'unknown URL')}"
            )

    logger.info(f"Successfully scraped {len(doc_pages)} pages out of {len(filtered_links)} URLs")
    return doc_pages

```

This method:

- Gets all documentation links using the previous method
- Optionally limits the number of pages to scrape
- Uses Firecrawl’s batch scraping to efficiently process multiple pages
- Converts raw scraping results into structured `DocPage` objects
- Handles errors and provides detailed logging

3. `save_documentation_pages`: Stores scraped content as markdown files:

```python
def save_documentation_pages(self, doc_pages: List[DocPage], docs_dir: str):
    """Save scraped documentation pages to markdown files."""
    Path(docs_dir).mkdir(parents=True, exist_ok=True)

    for page in doc_pages:
        url_path = page.url.replace("https://docs.firecrawl.dev", "")
        safe_filename = url_path.strip("/").replace("/", "-")
        filepath = os.path.join(docs_dir, f"{safe_filename}.md")

        with open(filepath, "w", encoding="utf-8") as f:
            f.write("---\n")
            f.write(f"title: {page.title}\n")
            f.write(f"url: {page.url}\n")
            f.write("---\n\n")
            f.write(page.content)

    logger.info(f"Saved {len(doc_pages)} pages to {docs_dir}")

```

This method:

- Creates a documentation directory if needed
- Converts URLs to safe filenames
- Saves each page as a markdown file with YAML frontmatter
- Preserves original titles and URLs for reference

Finally, the class provides a convenience method to handle the entire scraping workflow:

```python
def pull_docs(self, base_url: str, docs_dir: str, n_pages: int = None):
    doc_pages = self.scrape_documentation(base_url, n_pages)
    self.save_documentation_pages(doc_pages, docs_dir)

```

This scraper component is used by:

- The Streamlit interface ( `app.py`) for initial documentation collection
- The RAG system ( `rag.py`) for processing documentation into the vector store
- The command-line interface for testing and manual scraping

The use of Firecrawl’s AI capabilities allows the scraper to handle diverse documentation layouts without custom selectors, while the structured output ensures consistency for downstream processing.

### 2\. Implementing RAG with Ollama - `src/rag.py`

The RAG (Retrieval Augmented Generation) component is the core of our documentation assistant, handling document processing, embedding generation, and question answering. Let’s examine each part in detail:

First, we import the necessary LangChain components:

```python
from langchain_chroma import Chroma
from langchain_community.document_loaders import DirectoryLoader
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

```

These imports provide:

- `Chroma`: Vector database for storing embeddings
- `DirectoryLoader`: Utility for loading markdown files from a directory
- `ChatPromptTemplate`: Template system for LLM prompts
- `ChatOllama` and `OllamaEmbeddings`: Local LLM and embedding models
- `RecursiveCharacterTextSplitter`: Text chunking utility

The main RAG class initializes all necessary components:

```python
class DocumentationRAG:
    def __init__(self):
        # Initialize embeddings and vector store
        self.embeddings = OllamaEmbeddings(model="nomic-embed-text")
        self.vector_store = Chroma(
            embedding_function=self.embeddings, persist_directory="./chroma_db"
        )

        # Initialize LLM
        self.llm = ChatOllama(model="deepseek-r1:14b")

        # Text splitter for chunking
        self.text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=1000, chunk_overlap=200, add_start_index=True
        )

```

The initialization:

1. Creates an embedding model using Nomic’s text embeddings
2. Sets up a Chroma vector store with persistent storage
3. Initializes the DeepSeek R1 14B model for question answering
4. Configures a text splitter with 1000-character chunks and 200-character overlap

The prompt template defines how the LLM should process questions:

```python
        # RAG prompt template
        self.prompt = ChatPromptTemplate.from_template(
            """
            You are an expert documentation assistant. Use the following documentation context
            to answer the question. If you don't know the answer, just say that you don't
            have enough information. Keep the answer concise and clear.

            Context: {context}
            Question: {question}

            Answer:"""
        )

```

This template:

- Sets the assistant’s role and behavior
- Provides placeholders for context and questions
- Encourages concise and clear responses

The document loading method handles reading markdown files:

```python
    def load_docs_from_directory(self, docs_dir: str):
        """Load all markdown documents from a directory"""
        markdown_docs = DirectoryLoader(docs_dir, glob="*.md").load()
        return markdown_docs

```

This method:

- Uses `DirectoryLoader` to find all markdown files
- Automatically handles file reading and basic preprocessing
- Returns a list of Document objects

The document processing method prepares content for the vector store:

```python
    def process_documents(self, docs_dir: str):
        """Process documents and add to vector store"""
        # Clear existing documents
        self.vector_store = Chroma(
            embedding_function=self.embeddings, persist_directory="./chroma_db"
        )

        # Load and process new documents
        documents = self.load_docs_from_directory(docs_dir)
        chunks = self.text_splitter.split_documents(documents)
        self.vector_store.add_documents(chunks)

```

This method:

1. Reinitializes the vector store to clear existing documents
2. Loads new documents from the specified directory
3. Splits documents into manageable chunks
4. Generates and stores embeddings in the vector database

Finally, the query method handles question answering:

```python
    def query(self, question: str) -> tuple[str, str]:
        """Query the documentation"""
        # Get relevant documents
        docs = self.vector_store.similarity_search(question, k=3)

        # Combine context
        context = "\n\n".join([doc.page_content for doc in docs])

        # Generate response
        chain = self.prompt | self.llm
        response = chain.invoke({"context": context, "question": question})

        # Extract chain of thought between <think> and </think>
        chain_of_thought = response.content.split("<think>")[1].split("</think>")[0]

        # Extract response
        response = response.content.split("</think>")[1].strip()

        return response, chain_of_thought

```

The query process:

1. Performs semantic search to find the 3 most relevant document chunks
2. Combines the chunks into a single context string
3. Creates a LangChain chain combining the prompt and LLM
4. Generates a response with chain-of-thought reasoning
5. Extracts and returns both the final answer and reasoning process

This RAG component is used by:

- The Streamlit interface ( `app.py`) for handling user questions
- The command-line interface for testing and development
- Future extensions that need documentation Q&A capabilities

The implementation uses LangChain’s abstractions to create a modular and maintainable system while keeping all AI components running locally through Ollama.

### 3\. Building a clean UI with Streamlit - `src/app.py`

The Streamlit interface brings together the scraping and RAG components into a user-friendly web application. Let’s break down each component:

First, we set up basic configuration and utilities:

```python
import glob
import logging
from pathlib import Path

import streamlit as st
from dotenv import load_dotenv
from rag import DocumentationRAG
from scraper import DocumentationScraper