# Langfuse > title: A/B Testing of LLM Prompts ## Pages - [A/B Testing of LLM Prompts](a-b-testing.md): [Langfuse Prompt Management](/docs/prompts/get-started) enables A/B testing by allowing you to label different versio... - [Administration Overview (self-hosted)](administration.md): Self-hosted Langfuse provides comprehensive administration capabilities to manage your deployment, users, organizatio... - [Advanced features of the Langfuse SDKs](advanced-features.md): Use these methods to harden your Langfuse instrumentation, protect sensitive data, and adapt the SDKs to your specifi... - [Advanced usage of the Langfuse Python SDK](advanced-usage.md): The Python SDK provides advanced usage options for your application. This includes data masking, logging, sampling, f... - [Agent Graphs](agent-graphs.md): import { GhDiscussionsPreview } from "@/components/gh-discussions/GhDiscussionsPreview"; - [Annotation Queues](annotation-queues.md): Annotation Queues are a manual [evaluation method](/docs/evaluation/core-concepts#evaluation-methods) which is build ... - [Human Annotation for LLM apps](annotation.md): Human Annotation is a manual evaluation method. It is used to collaboratively annotate traces, sessions and observati... - [Ask AI](ask-ai.md): The Langfuse AI assistant helps you find answers about Langfuse's features, integrations, and best practices. It's tr... - [Audit Logs](audit-logs.md): availability={{ - [Single Sign-On (SSO)](authentication-and-sso.md): By default, Langfuse supports email/password and social logins (Sign in with Google, GitHub, Microsoft). - [Automated Access Provisioning (self-hosted)](automated-access-provisioning.md): Optionally, you can configure automated access provisioning for new users. - [Deploy Langfuse on AWS with Terraform](aws.md): This guide will walk you through the steps to deploy Langfuse on AWS using the official Terraform module ([langfuse/l... - [Deploy Langfuse on Azure with Terraform](azure.md): This guide will walk you through the steps to deploy Langfuse on Azure using the official Terraform module ([langfuse... - [Background Migrations (self-hosted)](background-migrations.md): Langfuse uses background migrations to perform long-running changes within the storage components when [upgrading](/s... - [Backup Strategies for Langfuse](backups.md): This guide covers backup strategies for self-hosted Langfuse deployments. - [Billable Units](billable-units.md): Langfuse Cloud [pricing](/pricing) is based on the number of ingested units per billing period. Units are either [tra... - [S3 / Blob Storage (self-hosted)](blobstorage.md): This is a deep dive into the configuration of S3. Follow one of the [deployment guides](/self-hosting#deployment-opti... - [Cache (Redis/Valkey) (self-hosted)](cache.md): This is a deep dive into Redis/Valkey configuration. Follow one of the [deployment guides](/self-hosting#deployment-o... - [Caching in Client SDKs](caching.md): Langfuse prompts are cached client-side in the SDKs, so **there's no latency impact after the first use** and no avai... - [ClickHouse (self-hosted)](clickhouse.md): This is a deep dive into ClickHouse configuration. Follow one of the [deployment guides](/self-hosting#deployment-opt... - [Comments](comments.md): Comments enable teams to add contextual notes and discussions directly to traces, observations, sessions, and prompts... - [Prompt Composability](composability.md): import { FaqPreview } from "@/components/faq/FaqPreview"; - [Langfuse Evaluation Concepts](concepts.md): - **Scores** are a flexible data object that can be used to store any evaluation metric and link it to oth... - [Prompt Config](config.md): The prompt`config`in Langfuse is an **optional arbitrary JSON object** attached to each prompt, that can be used by... - [Configuration via Environment Variables (self-hosted)](configuration.md): Langfuse (self-hosted) has extensive configuration options via environment variables. These need to be passed to all ... - [Application Containers (self-hosted)](containers.md): This is a deep dive into the configuration of the application containers. Follow one of the [deployment guides](/self... - [Concepts](core-concepts.md): This page digs into the different concepts of evaluations, and what's available in Langfuse. - [Corrected Outputs](corrections.md): Corrections allow you to capture improved versions of LLM outputs directly in trace and observation views. Domain exp... - [Custom Base Path (self-hosted)](custom-base-path.md): By default, Langfuse is served on the root path of a domain, e.g.`` - [Custom Dashboards](custom-dashboards.md): Transform your LLM application data into actionable insights with Langfuse custom dashboards. - [Custom Scores](custom-scores.md): Custom Scores are the most flexible way to implement evaluation workflows using Langfuse. As any other evaluation met... - [Data Deletion](data-deletion.md): There may be use-cases where you want to remove selected data from Langfuse, like erroneously created traces in a dev... - [Evaluation Data Model](data-model.md): This page describes the data model for evaluation-related objects in Langfuse. For an overview of how these objects w... - [Data Retention](data-retention.md): availability={{ - [Datasets](datasets.md): A dataset is a collection of inputs and expected outputs and is used to test your application. Both [UI-based](/docs/... - [Example Project](demo.md): import { Button } from "@/components/ui/button"; - [Self-hosting Langfuse v2](deployment-guide.md): This guide covers Langfuse v2. For Langfuse v3, see the [v3 - [Deployment Strategies (self-hosted)](deployment-strategies.md): Use this guide to learn how to manage Langfuse effectively. It covers strategies for handling multiple projects and e... - [Docker Compose (self-hosted)](docker-compose.md): This guide will walk you through deploying Langfuse locally or on a VM using Docker Compose. - [Langfuse Docs MCP Server](docs-mcp.md): The Langfuse Docs MCP server exposes the Langfuse docs to AI agents. - [Langfuse Documentation](docs.md): Langfuse is an **open-source LLM engineering platform** ([GitHub](https://github.com/langfuse/langfuse)) that helps t... - [Encryption (self-hosted)](encryption.md): Security and privacy are core design objectives at Langfuse. The Langfuse Team runs Langfuse in production on Langfus... - [Environments](environments.md): Environments allow you to organize your traces, observations, and scores from different contexts such as production, ... - [Evaluations with the Langfuse Python SDK](evaluation.md): The Python SDK provides ways to evaluate your application. You can add custom scores to your traces and observations,... - [Experiments via SDK](experiments-via-sdk.md): Experiments via SDK are used to programmatically loop your applications or prompts through a dataset and optionally a... - [Experiments via UI](experiments-via-ui.md): You can execute Experiments via UI (also called Prompt Experiments) in the Langfuse UI to test different prompt versi... - [Export Data from UI](export-from-ui.md): Langfuse is [open-source](/open-source) and data tracked with Langfuse is open. Export your observability data for an... - [Export via Blob Storage Integration](export-to-blob-storage.md): availability={{ - [Export for Fine-Tuning](fine-tuning.md): Langfuse is [open-source](/open-source) and data tracked with Langfuse is open. You can easily [trace](/docs/tracing)... - [Prompt Folders](folders.md): Prompts can be organized into virtual folders to group prompts with similar purposes. - [Deploy Langfuse on GCP with Terraform](gcp.md): This guide will walk you through the steps to deploy Langfuse on GCP using the official Terraform module ([langfuse/l... - [Get Started with Tracing](get-started.md): This guide walks you through ingesting your first trace into Langfuse. If you're looking to understand what tracing i... - [GitHub Integration for Langfuse Prompts](github-integration.md): There are two methods to integrate Langfuse prompts with GitHub: - [Glossary](glossary.md): import { Glossary } from "@/components/Glossary"; - [Guaranteed Availability of Prompts](guaranteed-availability.md): Implementing this is usually not necessary as it adds complexity to your application. The Langfuse Prompt Management ... - [Headless Initialization (self-hosted)](headless-initialization.md): By default, you need to create a user account, organization and project via the Langfuse UI before being able to use ... - [Health and Readiness Check Endpoints](health-readiness-endpoints.md): Langfuse provides monitoring endpoints to check the health and readiness of your self-hosted deployment. These endpoi... - [Manage Organizations via API](instance-management-api.md): This is only available in the Enterprise Edition. Please add your [license key](/self-hosting/license-key) to activat... - [Instrument your application with the Langfuse SDKs](instrumentation.md): import { PropagationRestrictionsCallout } from "@/components/PropagationRestrictionsCallout"; - [Kubernetes (Helm) (self-hosted)](kubernetes-helm.md): This guide will walk you through the steps to deploy Langfuse on Kubernetes using the Helm package manager. - [Enterprise License Key (self-hosted)](license-key.md): All core Langfuse features and APIs are available in Langfuse OSS (MIT licensed) without any limits. - [Link to Traces](link-to-traces.md): Linking prompts to [traces](/docs/observability) enables tracking of metrics and evaluations per prompt version. It's... - [LLM API / Gateway (self-hosted)](llm-api.md): Optionally, you can configure Langfuse to use an external LLM API or gateway for add-on features. Langfuse tracing do... - [LLM-as-a-Judge Evaluation](llm-as-a-judge.md): Use an LLM to automatically score your application outputs. For this evaluation method, the LLM is presented with an ... - [LLM Connections](llm-connection.md): LLM connections are used to call models in the Langfuse Playground or for LLM-as-a-Judge evaluations. - [Log Levels](log-levels.md): Traces can have a lot of observations ([data model](/docs/tracing#introduction-to-traces-in-langfuse)). You can diffe... - [Masking of Sensitive LLM Data](masking.md): Masking is a feature that allows precise control over the [tracing](/docs/tracing/overview) data sent to the Langfuse... - [Langfuse MCP Server](mcp-server.md): Langfuse includes a native [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server that enables AI ass... - [MCP Tracing](mcp-tracing.md): [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) enables AI agents to interact with external tools an... - [Message Placeholders in Chat Prompts](message-placeholders.md): Message Placeholders allow you to insert a list of chat messages (`[{role: "...", content: "..."}]`) at specific posi... - [Metadata](metadata.md): import { PropagationRestrictionsCallout } from "@/components/PropagationRestrictionsCallout"; - [Metrics API](metrics-api.md): GET /api/public/metrics - [Multi-Modality & Attachments](multi-modality.md): Langfuse supports multi-modal traces including **text, images, audio, and other attachments**. - [Open Source Prompt Management for n8n](n8n-node.md): import { Callout } from "nextra/components"; - [Networking (self-hosted)](networking.md): Langfuse can be deployed in a VPC or on-premises in high-security environments. This guide covers the networking requ... - [Observability via OpenTelemetry](observability.md): Langfuse uses OpenTelemetry to provide observability into the application. - [Observation Types](observation-types.md): Langfuse supports different observation types to provide more context to your spans and allow efficient filtering. - [Observations API](observations-api.md): The Observations API allows you to retrieve observation data (spans, generations, events) from Langfuse for use in cu... - [Allowlist of organization creators (self-hosted)](organization-creators.md): This is only available in the Enterprise Edition. Please add your [license key](/self-hosting/license-key) to activat... - [Manage Organizations via API](organization-management-api.md): This is only available in the Enterprise Edition. Please add your [license key](/self-hosting/license-key) to activat... - [Open Source LLM API & Data Platform](overview.md): **Langfuse is designed to be open, extensible and flexible** (see [_why langfuse?_](/why)). People using Langfuse are... - [LLM Playground](playground.md): Test and iterate on your prompts directly in the Langfuse Prompt Playground. Tweak the prompt and model parameters to... - [Postgres Database (self-hosted)](postgres.md): Follow one of the [deployment guides](/self-hosting#deployment-options) to get started. - [Prompt Version Control](prompt-version-control.md): In Langfuse, version control & deployment of prompts is managed via`versions`and`labels`. - [Public API](public-api.md): Langfuse is open and meant to be extended via custom workflows and integrations. All Langfuse data and features are a... - [Query Data via SDKs](query-via-sdk.md): Langfuse is [open-source](/open-source) and data tracked with Langfuse is open. You can query data via: SDKs... - [Event queuing/batching](queuing-batching.md): Langfuse's client SDKs and integrations are all designed to queue and batch requests in the background to optimize AP... - [Deploy Langfuse v3 on Railway](railway.md): You can deploy Langfuse v3 on [Railway](https://railway.app/) via the prebuilt template. - [Role-Based Access Controls in Langfuse](rbac.md): The role-based access control (RBAC) in Langfuse is based on users, organizations, projects, and roles: - [Releases & Versioning](releases-and-versioning.md): import { PropagationRestrictionsCallout } from "@/components/PropagationRestrictionsCallout"; - [Langfuse Roadmap](roadmap.md): Langfuse is [open source](/open-source) and we want to be fully transparent what we're working on and what's next. Th... - [Sampling](sampling.md): Sampling can be used to control the volume of traces collected by Langfuse. Sampling is handled client-side. - [Scaling Langfuse Deployments](scaling.md): This guide covers how you can operate your Langfuse deployment at scale and includes best practices and tweaks to get... - [Organization-Key Scoped API Routes](scim-and-org-api.md): availability={{ - [Score Analytics](score-analytics.md): Score Analytics provides a lightweight, zero-configuration way to analyze your evaluation data out of the box. Whethe... - [Scores via API/SDK](scores-via-sdk.md): You can use the Langfuse SDKs or API to add scores to traces, observations, sessions and dataset runs. This is an eva... - [Add scores to traces via the UI](scores-via-ui.md): Adding scores via the UI is a manual [evaluation method](/docs/evaluation/core-concepts#evaluation-methods). It is us... - [LLM Security & Guardrails](security-and-guardrails.md): There are a host of potential safety risks involved with LLM-based applications. These include prompt injection, leak... - [Self-host Langfuse (Open Source LLM Observability)](self-hosting.md): import { Callout } from "nextra/components"; - [Sessions (Chats, Threads, etc.)](sessions.md): import { PropagationRestrictionsCallout } from "@/components/PropagationRestrictionsCallout"; - [Setup of the Langfuse Python SDK](setup.md): To get started with the Langfuse Python SDK, you need to install the SDK and initialize the client. - [Spend Alerts](spend-alerts.md): availability={{ - [Add tags to observations and traces in Langfuse](tags.md): import { PropagationRestrictionsCallout } from "@/components/PropagationRestrictionsCallout"; - [Model Usage & Cost Tracking for LLM applications (open source)](token-and-cost-tracking.md): UI](/images/docs/costs-and-usage-example.png) - [Trace IDs & Distributed Tracing](trace-ids-and-distributed-tracing.md): Langfuse allows you to bring your own trace IDs (e.g., messageId, traceId, correlationId) for - [Transactional Emails (self-hosted)](transactional-emails.md): Optionally, you can configure an SMTP server to send transactional emails. - [Troubleshooting and FAQ for Langfuse Administration](troubleshooting-and-faq.md): This page addresses frequently asked questions and common troubleshooting topics when administering Langfuse. - [UI Customization (self-hosted)](ui-customization.md): This is only available in the Enterprise Edition. Please add your [license key](/self-hosting/license-key) to activat... - [Langfuse SDK upgrade paths](upgrade-path.md): This page shows the migration guides to the latest versions of the Langfuse SDKs. Pick your SDK to follow the relevan... - [Migrate Langfuse v1 to v2 (self-hosted)](upgrade-v1-to-v2.md): Langfuse v2 ([released](https://github.com/langfuse/langfuse/releases/tag/v2.0.0) Jan 30, 2024) is a major release of... - [Migrate Langfuse v2 to v3 (self-hosted)](upgrade-v2-to-v3.md): This is a big upgrade and we tried to make it as seamless as possible. Please create a [GitHub Issue](/issues) or con... - [How to upgrade a self-hosted Langfuse deployment](upgrade.md): Langfuse evolves quickly ([changelog](/changelog)) and keeping your deployment up to date is key to benefit from secu... - [Trace URLs](url.md): Each trace has a unique URL that you can use to share it with others or to access it directly. - [Collect User Feedback in Langfuse](user-feedback.md): User feedback measures whether your AI actually helped users. Use it to find quality issues, build better evaluation ... - [User Tracking](users.md): import { PropagationRestrictionsCallout } from "@/components/PropagationRestrictionsCallout"; - [Self-host Langfuse v2](v2.md): This guide covers Langfuse v2. For Langfuse v3, see the [v3 - [Variables in Prompts](variables.md): import { FaqPreview } from "@/components/faq/FaqPreview"; - [Versioning](versioning.md): Versioning is key to ensure compatibility between Langfuse Server, SDKs, and custom integrations via the Public API. ... - [Webhooks & Slack Integration](webhooks-slack-integrations.md): Use webhooks to receive real‑time notifications whenever a prompt version is created, updated, or deleted in Langfuse...