# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    handlers=[logging.StreamHandler()],
)
logger = logging.getLogger(__name__)

```

These imports and configurations:

- Set up logging for debugging and monitoring
- Import our custom RAG and scraper components
- Load environment variables for configuration

Helper functions handle documentation management:

```python
def get_existing_docs():
    """Get all documentation directories with -docs suffix"""
    docs_dirs = glob.glob("*-docs")
    return [Path(dir_path).name for dir_path in docs_dirs]

def get_doc_page_count(docs_dir: str) -> int:
    """Get number of markdown files in a documentation directory"""
    return len(list(Path(docs_dir).glob("*.md")))

```

These utilities:

- Find all documentation directories with “-docs” suffix
- Count pages in each documentation set
- Support the UI’s documentation selection features

The scraping configuration section handles documentation collection:

```python
def scraping_config_section():
    """Create the documentation scraping configuration section"""
    st.markdown("### Configure Scraping")
    base_url = st.text_input(
        "Documentation URL",
        placeholder="https://docs.firecrawl.dev",
        help="The base URL of the documentation to scrape",
    )

    docs_name = st.text_input(
        "Documentation Name",
        placeholder="Firecrawl-docs",
        help="Name of the directory to store documentation",
    )

    n_pages = st.number_input(
        "Number of Pages",
        min_value=0,
        value=0,
        help="Limit the number of pages to scrape (0 for all pages)",
    )

    st.info(
        "💡 Add '-docs' suffix to the documentation name. "
        "Set pages to 0 to scrape all available pages."
    )

    if st.button("Start Scraping"):
        if not base_url or not docs_name:
            st.error("Please provide both URL and documentation name")
        elif not docs_name.endswith("-docs"):
            st.error("Documentation name must end with '-docs'")
        else:
            with st.spinner("Scraping documentation..."):
                try:
                    scraper = DocumentationScraper()
                    n_pages = None if n_pages == 0 else n_pages
                    scraper.pull_docs(base_url, docs_name, n_pages=n_pages)
                    st.success("Documentation scraped successfully!")
                except Exception as e:
                    st.error(f"Error scraping documentation: {str(e)}")

```

This section:

- Provides input fields for documentation URL and name
- Allows limiting the number of pages to scrape
- Handles validation and error reporting
- Shows progress during scraping
- Uses our `DocumentationScraper` class for content collection

The documentation selection interface manages switching between docs:

```python
def documentation_select_section():
    """Create the documentation selection section"""
    st.markdown("### Select Documentation")
    existing_docs = get_existing_docs()

    if not existing_docs:
        st.caption("No documentation found yet")
        return None

    # Create options with page counts
    doc_options = [f"{doc} ({get_doc_page_count(doc)} pages)" for doc in existing_docs]

    selected_doc = st.selectbox(
        "Choose documentation to use as context",
        options=doc_options,
        help="Select which documentation to use for answering questions",
    )

    if selected_doc:
        # Extract the actual doc name without page count
        st.session_state.current_doc = selected_doc.split(" (")[0]
        return st.session_state.current_doc
    return None

```

This component:

- Lists available documentation sets
- Shows page counts for each set
- Updates session state when selection changes
- Handles the case of no available documentation

The chat interface consists of two main functions that work together to create the interactive Q&A experience:

First, we initialize the necessary session state:

```python
def initialize_chat_state():
    """Initialize session state for chat"""
    if "messages" not in st.session_state:
        st.session_state.messages = []
    if "rag" not in st.session_state:
        st.session_state.rag = DocumentationRAG()

```

This initialization:

- Creates an empty message list if none exists
- Sets up the RAG system for document processing and querying
- Uses Streamlit’s session state to persist data between reruns

The main chat interface starts with basic setup:

```python
def chat_interface():
    """Create the chat interface"""
    st.title("Documentation Assistant")

    # Check if documentation is selected
    if "current_doc" not in st.session_state:
        st.info("Please select a documentation from the sidebar to start chatting.")
        return

```

This section:

- Sets the page title
- Ensures documentation is selected before proceeding
- Shows a helpful message if no documentation is chosen

Document processing is handled next:

```python
    # Process documentation if not already processed
    if (
        "docs_processed" not in st.session_state
        or st.session_state.docs_processed != st.session_state.current_doc
    ):
        with st.spinner("Processing documentation..."):
            st.session_state.rag.process_documents(st.session_state.current_doc)
            st.session_state.docs_processed = st.session_state.current_doc

```

This block:

- Checks if the current documentation needs processing
- Shows a loading spinner during processing
- Updates the session state after processing
- Prevents unnecessary reprocessing of the same documentation

Message display is handled by iterating through the chat history:

```python
    # Display chat messages
    for message in st.session_state.messages:
        with st.chat_message(message["role"]):
            st.markdown(message["content"])
            if "chain_of_thought" in message:
                with st.expander("View reasoning"):
                    st.markdown(message["chain_of_thought"])

```

This section:

- Shows each message with appropriate styling based on role
- Displays the main content using markdown
- Creates expandable sections for reasoning chains
- Maintains visual consistency in the chat

Finally, the input handling and response generation:

```python
    # Chat input
    if prompt := st.chat_input("Ask a question about the documentation"):
        # Add user message
        st.session_state.messages.append({"role": "user", "content": prompt})

        with st.chat_message("user"):
            st.markdown(prompt)

        # Generate and display response
        with st.chat_message("assistant"):
            with st.spinner("Thinking..."):
                response, chain_of_thought = st.session_state.rag.query(prompt)
                st.markdown(response)
                with st.expander("View reasoning"):
                    st.markdown(chain_of_thought)

        # Store assistant response
        st.session_state.messages.append({
            "role": "assistant",
            "content": response,
            "chain_of_thought": chain_of_thought,
        })

```

This section:

1. Captures user input:
   - Uses Streamlit’s chat input component
   - Stores the message in session state
   - Displays the message immediately
2. Generates response:
   - Shows a “thinking” spinner during processing
   - Queries the RAG system for an answer
   - Displays the response with expandable reasoning
3. Updates chat history:
   - Stores both response and reasoning
   - Maintains the conversation flow
   - Preserves the interaction for future reference

The entire chat interface creates a seamless experience by:

- Managing state effectively
- Providing immediate feedback
- Showing processing status
- Maintaining conversation context
- Exposing the AI’s reasoning process

Finally, the main application structure:

```python
def sidebar():
    """Create the sidebar UI components"""
    with st.sidebar:
        st.title("Documentation Scraper")
        scraping_config_section()
        documentation_select_section()

def main():
    initialize_chat_state()
    sidebar()
    chat_interface()

if __name__ == "__main__":
    main()

```

This structure:

- Organizes UI components into sidebar and main area
- Initializes necessary state on startup
- Provides a clean entry point for the application

The Streamlit interface brings together all components into a cohesive application that:

- Makes documentation scraping accessible to non-technical users
- Provides immediate feedback during operations
- Maintains conversation history
- Shows the AI’s reasoning process
- Handles errors gracefully

## How to Increase System Performance

There are several ways to optimize the performance of this documentation assistant. The following sections explore key areas for potential improvements:

### 1\. Optimize document chunking

In `rag.py`, we currently use a basic chunking strategy:

```python
self.text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    add_start_index=True
)

```

We can improve this by:

- Using semantic chunking that respects document structure
- Adjusting chunk size based on content type (e.g., larger for API docs)
- Implementing custom splitting rules for documentation headers
- Adding metadata to chunks for better context preservation

Example improved configuration:

```python
self.text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1500,  # Larger chunks for more context
    chunk_overlap=300,  # Increased overlap for better coherence
    separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],  # Respect markdown structure
    add_start_index=True,
    length_function=len,
    is_separator_regex=False
)

```

### 2\. Enhance vector search

The current similarity search in `rag.py` is basic:

```python
docs = self.vector_store.similarity_search(question, k=3)

```

We can improve retrieval by:

- Increasing `k`, i.e. the number of chunks returned
- Implementing hybrid search (combining semantic and keyword matching)
- Using Maximum Marginal Relevance (MMR) for diverse results
- Adding metadata filtering based on document sections
- Implementing re-ranking of retrieved chunks

Example enhanced retrieval:

```python
def query(self, question: str) -> tuple[str, str]:
    # Get relevant documents with MMR
    docs = self.vector_store.max_marginal_relevance_search(
        question,
        k=5,  # Retrieve more candidates
        fetch_k=20,  # Consider larger initial set
        lambda_mult=0.7  # Diversity factor
    )

    # Filter and re-rank results
    filtered_docs = [\
        doc for doc in docs\
        if self._calculate_relevance_score(doc, question) > 0.7\
    ]

    # Use top 3 most relevant chunks
    context = "\n\n".join([doc.page_content for doc in filtered_docs[:3]])

```

### 3\. Implement caching

The current implementation reprocesses documentation on every selection:

```python
if (
    "docs_processed" not in st.session_state
    or st.session_state.docs_processed != st.session_state.current_doc
):
    with st.spinner("Processing documentation..."):
        st.session_state.rag.process_documents(st.session_state.current_doc)

```

We can improve this by:

- Implementing persistent vector storage with versioning
- Caching processed embeddings
- Adding incremental updates for documentation changes

Example caching implementation:

```python
from hashlib import md5
import pickle

class CachedDocumentationRAG(DocumentationRAG):
    def process_documents(self, docs_dir: str):
        cache_key = self._get_cache_key(docs_dir)
        cache_path = f"cache/{cache_key}.pkl"

        if os.path.exists(cache_path):
            with open(cache_path, 'rb') as f:
                self.vector_store = pickle.load(f)
        else:
            super().process_documents(docs_dir)
            os.makedirs("cache", exist_ok=True)
            with open(cache_path, 'wb') as f:
                pickle.dump(self.vector_store, f)

```

### 4\. Optimize model loading

Currently, we initialize models in `__init__`:

```python
def __init__(self):
    self.embeddings = OllamaEmbeddings(model="nomic-embed-text")
    self.llm = ChatOllama(model="deepseek-r1:14b")

```

We can improve this by:

- Implementing lazy loading of models
- Using smaller models for initial responses
- Adding model quantization options
- Implementing model caching

Example optimized initialization:

```python
class OptimizedDocumentationRAG:
    def __init__(self, use_small_model=True):
        self._embeddings = None
        self._llm = None
        self._use_small_model = use_small_model

    @property
    def llm(self):
        if self._llm is None:
            model_size = "7b" if self._use_small_model else "14b"
            self._llm = ChatOllama(
                model=f"deepseek-r1:{model_size}",
                temperature=0.1,  # Lower temperature for docs
                num_ctx=2048  # Reduced context for faster inference
            )
        return self._llm

```

These optimizations can significantly improve:

- Response latency
- Memory usage
- Processing throughput
- User experience

Remember to benchmark performance before and after implementing these changes to measure their impact. Also, consider your specific use case - some optimizations might be more relevant depending on factors like user load, documentation size, and hardware constraints.

## Conclusion

This local documentation assistant demonstrates how modern AI technologies can be combined to create powerful, practical tools for technical documentation. By using DeepSeek’s language capabilities, Firecrawl’s AI-powered scraping, and the RAG architecture, we’ve built a system that makes documentation more accessible and interactive. The application’s modular design, with clear separation between scraping, RAG implementation, and user interface components, provides a solid foundation for future enhancements and adaptations to different documentation needs.

Most importantly, this implementation shows that sophisticated AI applications can be built entirely with local components, eliminating privacy concerns and reducing operational costs. The combination of Streamlit’s intuitive interface, LangChain’s flexible abstractions, and Ollama’s local AI models creates a seamless experience that feels like a cloud service but runs entirely on your machine. Whether you’re a developer learning a new framework, a technical writer maintaining documentation, or a team lead looking to improve documentation accessibility, this assistant provides a practical solution that can be customized and extended to meet your specific needs.

Article updated recently

[🔥](https://www.firecrawl.dev/)

## Ready to _Build?_

Start scraping web data for your AI apps today.

No credit card needed.

Get Started

## About the Author

[![Bex Tuychiev image](https://www.firecrawl.dev/bex.jpg)\\
Bex Tuychiev@bextuychiev](https://x.com/bextuychiev)

Bex is a Top 10 AI writer on Medium and a Kaggle Master with over 15k followers. He loves writing detailed guides, tutorials, and notebooks on complex data science and machine learning topics

### More articles by Bex Tuychiev

[Building an Automated Price Tracking Tool\\
\\
Build an automated e-commerce price tracker in Python. Learn web scraping, price monitoring, and automated alerts using Firecrawl, Streamlit, PostgreSQL.](https://www.firecrawl.dev/blog/automated-price-tracking-tutorial-python) [Web Scraping Automation: How to Run Scrapers on a Schedule\\
\\
Learn how to automate web scraping in Python using free tools like schedule, asyncio, cron jobs and GitHub Actions. This comprehensive guide covers local and cloud-based scheduling methods to run scrapers reliably in 2025.](https://www.firecrawl.dev/blog/automated-web-scraping-free-2025) [Automated Data Collection - A Comprehensive Guide\\
\\
Learn how to build robust automated data collection systems using modern tools and best practices. This guide covers everything from selecting the right tools to implementing scalable collection pipelines.](https://www.firecrawl.dev/blog/automated-data-collection-guide) [BeautifulSoup4 vs. Scrapy - A Comprehensive Comparison for Web Scraping in Python\\
\\
Learn the key differences between BeautifulSoup4 and Scrapy for web scraping in Python. Compare their features, performance, and use cases to choose the right tool for your web scraping needs.](https://www.firecrawl.dev/blog/beautifulsoup4-vs-scrapy-comparison) [How to Build an Automated Competitor Price Monitoring System with Python\\
\\
Learn how to build an automated competitor price monitoring system in Python that tracks prices across e-commerce sites, provides real-time comparisons, and maintains price history using Firecrawl, Streamlit, and GitHub Actions.](https://www.firecrawl.dev/blog/automated-competitor-price-scraping) [Scraping Company Data and Funding Information in Bulk With Firecrawl and Claude\\
\\
Learn how to build a web scraper in Python that gathers company details, funding rounds, and investor information from public sources like Crunchbase using Firecrawl and Claude for automated data collection and analysis.](https://www.firecrawl.dev/blog/crunchbase-scraping-with-firecrawl-claude) [How to Create Custom Instruction Datasets for LLM Fine-tuning\\
\\
Learn how to build high-quality instruction datasets for fine-tuning large language models (LLMs). This guide covers when to create custom datasets, best practices for data collection and curation, and a practical example of building a code documentation dataset.](https://www.firecrawl.dev/blog/custom-instruction-datasets-llm-fine-tuning) [Data Enrichment: A Complete Guide to Enhancing Your Data Quality\\
\\
Learn how to enrich your data quality with a comprehensive guide covering data enrichment tools, best practices, and real-world examples. Discover how to leverage modern solutions like Firecrawl to automate data collection, validation, and integration for better business insights.](https://www.firecrawl.dev/blog/complete-guide-to-data-enrichment)

## Structured Web Data Extraction
Introducing /extract - Get web data with a prompt [Try now](https://www.firecrawl.dev/extract)

January 20, 2025

•

[![Eric Ciarla image](https://www.firecrawl.dev/eric-img.jpeg)Eric Ciarla](https://x.com/ericciarla)