# Source: https://docs.crewai.com/en/observability/mlflow.md

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.crewai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# MLflow Integration

> Quickly start monitoring your Agents with MLflow.

# MLflow Overview

[MLflow](https://mlflow.org/) is an open-source platform to assist machine learning practitioners and teams in handling the complexities of the machine learning process.

It provides a tracing feature that enhances LLM observability in your Generative AI applications by capturing detailed information about the execution of your application’s services.
Tracing provides a way to record the inputs, outputs, and metadata associated with each intermediate step of a request, enabling you to easily pinpoint the source of bugs and unexpected behaviors.

<img src="https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow-tracing.gif?s=be88ff36aec776c005102164d804322a" alt="Overview of MLflow crewAI tracing usage" data-og-width="1144" width="1144" data-og-height="720" height="720" data-path="images/mlflow-tracing.gif" data-optimize="true" data-opv="3" />

### Features

* **Tracing Dashboard**: Monitor activities of your crewAI agents with detailed dashboards that include inputs, outputs and metadata of spans.
* **Automated Tracing**: A fully automated integration with crewAI, which can be enabled by running `mlflow.crewai.autolog()`.
* **Manual Trace Instrumentation with minor efforts**: Customize trace instrumentation through MLflow's high-level fluent APIs such as decorators, function wrappers and context managers.
* **OpenTelemetry Compatibility**: MLflow Tracing supports exporting traces to an OpenTelemetry Collector, which can then be used to export traces to various backends such as Jaeger, Zipkin, and AWS X-Ray.
* **Package and Deploy Agents**: Package and deploy your crewAI agents to an inference server with a variety of deployment targets.
* **Securely Host LLMs**: Host multiple LLM from various providers in one unified endpoint through MFflow gateway.
* **Evaluation**: Evaluate your crewAI agents with a wide range of metrics using a convenient API `mlflow.evaluate()`.

## Setup Instructions

<Steps>
  <Step title="Install MLflow package">
    ```shell  theme={null}
    # The crewAI integration is available in mlflow>=2.19.0
    pip install mlflow
    ```
  </Step>

  <Step title="Start MFflow tracking server">
    ```shell  theme={null}
    # This process is optional, but it is recommended to use MLflow tracking server for better visualization and broader features.
    mlflow server
    ```
  </Step>

  <Step title="Initialize MLflow in Your Application">
    Add the following two lines to your application code:

    ```python  theme={null}
    import mlflow

    mlflow.crewai.autolog()

    # Optional: Set a tracking URI and an experiment name if you have a tracking server
    mlflow.set_tracking_uri("http://localhost:5000")
    mlflow.set_experiment("CrewAI")
    ```

    Example Usage for tracing CrewAI Agents:

    ```python  theme={null}
    from crewai import Agent, Crew, Task
    from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource
    from crewai_tools import SerperDevTool, WebsiteSearchTool

    from textwrap import dedent

    content = "Users name is John. He is 30 years old and lives in San Francisco."
    string_source = StringKnowledgeSource(
        content=content, metadata={"preference": "personal"}
    )

    search_tool = WebsiteSearchTool()


    class TripAgents:
        def city_selection_agent(self):
            return Agent(
                role="City Selection Expert",
                goal="Select the best city based on weather, season, and prices",
                backstory="An expert in analyzing travel data to pick ideal destinations",
                tools=[
                    search_tool,
                ],
                verbose=True,
            )

        def local_expert(self):
            return Agent(
                role="Local Expert at this city",
                goal="Provide the BEST insights about the selected city",
                backstory="""A knowledgeable local guide with extensive information
            about the city, it's attractions and customs""",
                tools=[search_tool],
                verbose=True,
            )


    class TripTasks:
        def identify_task(self, agent, origin, cities, interests, range):
            return Task(
                description=dedent(
                    f"""
                    Analyze and select the best city for the trip based
                    on specific criteria such as weather patterns, seasonal
                    events, and travel costs. This task involves comparing
                    multiple cities, considering factors like current weather
                    conditions, upcoming cultural or seasonal events, and
                    overall travel expenses.
                    Your final answer must be a detailed
                    report on the chosen city, and everything you found out
                    about it, including the actual flight costs, weather
                    forecast and attractions.

                    Traveling from: {origin}
                    City Options: {cities}
                    Trip Date: {range}
                    Traveler Interests: {interests}
                """
                ),
                agent=agent,
                expected_output="Detailed report on the chosen city including flight costs, weather forecast, and attractions",
            )

        def gather_task(self, agent, origin, interests, range):
            return Task(
                description=dedent(
                    f"""
                    As a local expert on this city you must compile an
                    in-depth guide for someone traveling there and wanting
                    to have THE BEST trip ever!
                    Gather information about key attractions, local customs,
                    special events, and daily activity recommendations.
                    Find the best spots to go to, the kind of place only a
                    local would know.
                    This guide should provide a thorough overview of what
                    the city has to offer, including hidden gems, cultural
                    hotspots, must-visit landmarks, weather forecasts, and
                    high level costs.
                    The final answer must be a comprehensive city guide,
                    rich in cultural insights and practical tips,
                    tailored to enhance the travel experience.

                    Trip Date: {range}
                    Traveling from: {origin}
                    Traveler Interests: {interests}
                """
                ),
                agent=agent,
                expected_output="Comprehensive city guide including hidden gems, cultural hotspots, and practical travel tips",
            )


    class TripCrew:
        def __init__(self, origin, cities, date_range, interests):
            self.cities = cities
            self.origin = origin
            self.interests = interests
            self.date_range = date_range

        def run(self):
            agents = TripAgents()
            tasks = TripTasks()

            city_selector_agent = agents.city_selection_agent()
            local_expert_agent = agents.local_expert()

            identify_task = tasks.identify_task(
                city_selector_agent,
                self.origin,
                self.cities,
                self.interests,
                self.date_range,
            )
            gather_task = tasks.gather_task(
                local_expert_agent, self.origin, self.interests, self.date_range
            )

            crew = Crew(
                agents=[city_selector_agent, local_expert_agent],
                tasks=[identify_task, gather_task],
                verbose=True,
                memory=True,
                knowledge={
                    "sources": [string_source],
                    "metadata": {"preference": "personal"},
                },
            )

            result = crew.kickoff()
            return result


    trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")
    result = trip_crew.run()

    print(result)
    ```

    Refer to [MLflow Tracing Documentation](https://mlflow.org/docs/latest/llms/tracing/index.html) for more configurations and use cases.
  </Step>

  <Step title="Visualize Activities of Agents">
    Now traces for your crewAI agents are captured by MLflow.
    Let's visit MLflow tracking server to view the traces and get insights into your Agents.

    Open `127.0.0.1:5000` on your browser to visit MLflow tracking server.

    <Frame caption="MLflow Tracing Dashboard">
      <img src="https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=0685aeb79319ad21ff842053ce5303c9" alt="MLflow tracing example with crewai" data-og-width="3024" width="3024" data-og-height="1712" height="1712" data-path="images/mlflow1.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?w=280&fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=f45f73922d8ea58f0e72c16c0e22d977 280w, https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?w=560&fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=5025f431d9c88bb7a094ab032aeaa753 560w, https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?w=840&fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=385879dd49ee957a4c25c5f94a9766e1 840w, https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?w=1100&fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=c779a19494038367ecb7f1caefe1014c 1100w, https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?w=1650&fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=ff45740a0aaab934952c2928edbdcbc6 1650w, https://mintcdn.com/crewai/qVjgZHKAyEOgSSUS/images/mlflow1.png?w=2500&fit=max&auto=format&n=qVjgZHKAyEOgSSUS&q=85&s=af14478b932df5a9f9261514f38d5e14 2500w" />
    </Frame>
  </Step>
</Steps>