With optional metadata
```bash curl -X POST http://localhost:8000/v1/users \ -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \ -H "Content-Type: application/json" \ -d ' { "user_id": "user-123", "alias": "Alice", "metadata": { "department": "Engineering", "team": "ML", "email": "alice@example.com" } }' ```Example Response
```json { "id": "abc-123", "key": "gw-...", "key_name": "mobile-app", "created_at": "2025-10-20T10:00:00", "expires_at": null, "is_active": true, "metadata": {} } ```Sample Response
```json { "budget_id": "abc-123", "max_budget": 10.0, "budget_duration_sec": 2592000, "created_at": "2025-10-22T10:00:00Z", "updated_at": "2025-10-22T10:00:00Z" } ```
One interface. Every LLM.
`any-llm` is a Python library providing a single interface to different llm providers. ```python from any_llm import completion # Using the messages format response = completion( model="gpt-4o-mini", messages=[{"role": "user", "content": "What is Python?"}], provider="openai" ) print(response) # Switch providers without changing your code response = completion( model="claude-sonnet-4-5-20250929", messages=[{"role": "user", "content": "What is Python?"}], provider="anthropic" ) print(response) ``` ### Why any-llm - Switch providers in one line - Unified exception handling across providers - Simple API, powerful features [View supported providers →](./providers.md) ### Getting Started **[Get started in 5 minutes →](./quickstart.md)** - Install the library and run your first API call. ### Demo Try `any-llm` in action with our interactive chat demo: **[📂 Run the Demo](https://github.com/mozilla-ai/any-llm/tree/main/demos/chat#readme)** Features: real-time streaming responses, multiple provider support, and collapsible "thinking" content display. ### API Documentation `any-llm` provides two main interfaces: **Direct API Functions** (recommended for simple use cases): - [completion](./api/completion.md) - Chat completions with any provider - [embedding](./api/embedding.md) - Text embeddings - [responses](./api/responses.md) - [OpenResponses](https://www.openresponses.org/) API for agentic AI systems **AnyLLM Class** (recommended for advanced use cases): - [Provider API](./api/any_llm.md) - Lower-level provider interface with metadata access and reusability ## For AI Systems This documentation is available in two AI-friendly formats: - **[llms.txt](https://mozilla-ai.github.io/any-llm/llms.txt)** - A structured overview with curated links to key documentation sections - **[llms-full.txt](https://mozilla-ai.github.io/any-llm/llms-full.txt)** - Complete documentation content concatenated into a single file --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/api/list_models.md ## Models ::: any_llm.api.list_models ::: any_llm.api.alist_models --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/api/types/model.md ## Model Types Data models and types for model operations. ::: any_llm.types.model --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/gateway/overview.md # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/platform/overview.md # Managed Platform Overview ## What is the any-llm Managed Platform? The any-llm managed platform is a cloud-hosted service that provides secure API key vaulting and usage tracking for all your LLM providers. Instead of managing multiple provider API keys across your codebase, you get a single virtual key that works with any supported provider while keeping your credentials encrypted and your usage tracked. The managed platform is available at [any-llm.ai](https://any-llm.ai). ## Why use the Managed Platform? Managing LLM API keys and tracking costs across multiple providers is challenging: - **Security risks**: API keys scattered across `.env` files, CI/CD pipelines, and developer machines - **No visibility**: Difficult to track spending across OpenAI, Anthropic, Google, and other providers - **Key rotation pain**: Updating keys means touching multiple systems and codebases - **No performance insights**: No easy way to measure latency, throughput, or reliability The managed platform solves these problems: - **Secure Key Vault**: Your provider API keys are encrypted client-side before storage—we never see your raw keys - **Single Virtual Key**: One `ANY_LLM_KEY` works across all providers - **Usage Analytics**: Track tokens, costs, and performance metrics without logging prompts or responses - **Zero Infrastructure**: No servers to deploy, no databases to manage ## How it works The managed platform acts as a secure credential manager and usage tracker. Here's the flow: 1. **You add provider keys** to the platform dashboard (keys are encrypted in your browser before upload) 2. **You get a virtual key** (`ANY_LLM_KEY`) that represents your project 3. **Your application** uses the `PlatformProvider` with your virtual key 4. **The SDK** authenticates with the platform, retrieves and decrypts your provider key client-side 5. **Your request** goes directly to the LLM provider (OpenAI, Anthropic, etc.) 6. **Usage metadata** (tokens, model, latency) is reported back—never your prompts or responses ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Your Application │ │ │ │ from any_llm import completion │ │ completion(provider="platform", model="openai:gpt-4", ...) │ └──────────────────────────────┬──────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────┐ │ any-llm SDK (PlatformProvider) │ │ │ │ 1. Authenticate with platform using ANY_LLM_KEY │ │ 2. Receive encrypted provider key │ │ 3. Decrypt provider key locally (client-side) │ │ 4. Make request directly to provider │ │ 5. Report usage metadata (tokens, latency) to platform │ └────────────────┬─────────────────────────────────────┬──────────────────┘ │ │ ▼ ▼ ┌─────────────────────────────┐ ┌────────────────────────────────────┐ │ any-llm Managed Platform │ │ LLM Provider │ │ │ │ (OpenAI, Anthropic, etc.) │ │ • Encrypted key storage │ │ │ │ • Usage tracking │ │ Your prompts/responses go │ │ • Cost analytics │ │ directly here—never through │ │ • Performance metrics │ │ our platform │ └─────────────────────────────┘ └────────────────────────────────────┘ ``` ## Key Features ### Client-Side Encryption Your provider API keys are encrypted in your browser using XChaCha20-Poly1305 before being sent to our servers. The encryption key is derived from your account credentials and never leaves your device. This means: - We cannot read your provider API keys - Even if our database were compromised, your keys remain encrypted - You maintain full control over your credentials ### Privacy-First Usage Tracking The platform tracks usage metadata to provide cost and performance insights: **What we track for you:** - Token counts (input and output) - Model name and provider - Request timestamps - Performance metrics (latency, throughput) **What we never track:** - Your prompts - Model responses - Any content from your conversations ### Project Organization Organize your usage by project, team, or environment: - Create separate projects for development, staging, and production - Track costs per project - Set up different provider keys per project ## Platform vs. Gateway any-llm offers two solutions for managing LLM access. Choose the one that fits your needs: | Feature | Managed Platform | Self-Hosted Gateway | |---------|-----------------|---------------------| | **Deployment** | Cloud-hosted (no infrastructure) | Self-hosted (Docker + Postgres) | | **Key Storage** | Client-side encrypted vault | Your own configuration | | **Budget Enforcement** | Coming soon | Built-in | | **User Management** | Per-project | Full user/key management | | **Request Routing** | Direct to provider, no proxy | Through your gateway | | **Best For** | Teams wanting zero-ops key management and usage tracking| Organizations needing full control | You can also use both together—store your provider keys in the managed platform and use them in a self-hosted gateway deployment. ## Current Status The any-llm managed platform is in **open beta**. During the beta: - **Free access** to all features - Core encryption and key management are **production-ready** - Dashboard UX and advanced features are being refined - Feedback is welcome at [any-llm.ai](https://any-llm.ai) ## Getting Started Ready to try the managed platform? 1. Create an account at [any-llm.ai](https://any-llm.ai) 2. Add your provider API keys 3. Get your virtual key 4. Make your first request --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/api/types/provider.md ## Provider Types Data models and types for provider operations. ::: any_llm.types.provider --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/providers.md --- schema: type: "TechArticle" name: "Supported Providers - any-llm" description: "Complete list of LLM providers supported by any-llm including OpenAI, Anthropic, Mistral, and more" datePublished: "2024-03-15" dateModified: "2024-11-18" --- # Supported Providers `any-llm` supports the below providers. In order to discover information about what models are supported by a provider as well as what features the provider supports for each model, refer to the provider documentation. Provider source code can be found in the [`src/any_llm/providers/`](https://github.com/mozilla-ai/any-llm/tree/main/src/any_llm/providers) directory of the repository. !!! note "Legend" - **Key**: Environment variable for the API key (e.g., `OPENAI_API_KEY`). - **Base**: Environment variable for a custom API base URL (e.g., `OPENAI_BASE_URL`). Useful for proxies or self-hosted endpoints. - **Reasoning (Completions)**: Provider can return reasoning traces alongside the assistant message via the completions and/or streaming endpoints. This does not indicate whether the provider offers separate "reasoning models". See [this](https://github.com/mozilla-ai/any-llm/issues/95) discussion for more information. - **Streaming (Completions)**: Provider can stream completion results back as an iterator. - **Image (Completions)**: Provider supports passing an `image_data` parameter for vision capabilities, as defined by the OpenAI spec [here](https://platform.openai.com/docs/api-reference/chat/create#chat_create-messages). - **OpenResponses API**: Provider supports the [OpenResponses](https://www.openresponses.org/) specification for agentic AI systems. See the [Responses API docs](api/responses.md) for usage details. - **List Models API**: Provider supports listing available models programmatically via the `list_models()` function. This allows you to discover what models are available from the provider at runtime, which can be useful for dynamic model selection or validation. --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/gateway/quickstart.md # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/quickstart.md --- schema: type: "HowTo" name: "How to Install and Use any-llm" description: "Step-by-step guide to installing any-llm and making your first API call with Python" totalTime: "PT5M" tool: - "Python 3.11 or newer" - "pip package manager" supply: - "API key from your chosen LLM provider" steps: - name: "Install any-llm" text: "Install any-llm with your chosen providers using pip. Use the all option to install support for all providers." url: "https://mozilla-ai.github.io/any-llm/quickstart/#installation" - name: "Set up API keys" text: "Configure your provider's API key as an environment variable. Make sure you have the appropriate environment variable set for your chosen provider." url: "https://mozilla-ai.github.io/any-llm/quickstart/#api-keys" - name: "Make your first completion call" text: "Import the completion function from any-llm and create your first API call with your chosen model and provider" url: "https://mozilla-ai.github.io/any-llm/quickstart/#your-first-api-call" --- ## Requirements - Python 3.11 or newer - API keys for your chosen LLM provider ## Installation ```bash pip install any-llm-sdk[all] # Install with all provider support ``` ### Installing Specific Providers If you want to install a specific provider from our [supported providers](./providers.md): ```bash pip install any-llm-sdk[mistral] # For Mistral provider pip install any-llm-sdk[ollama] # For Ollama provider # install multiple providers pip install any-llm-sdk[mistral,ollama] ``` ### Library Integration If you're building a library, install just the base package (`pip install any-llm-sdk`) and let your users install provider dependencies. > **API Keys:** Set your provider's API key as an environment variable (e.g., `export MISTRAL_API_KEY="your-key"`) or pass it directly using the `api_key` parameter. ## APIs ### Using the AnyLLM Class For applications making multiple requests with the same provider, use the `AnyLLM` class to avoid repeated provider instantiation: ```python import os from any_llm import AnyLLM # Make sure you have the appropriate API key set api_key = os.environ.get('MISTRAL_API_KEY') if not api_key: raise ValueError("Please set MISTRAL_API_KEY environment variable") llm = AnyLLM.create("mistral") response = llm.completion( model="mistral-small-latest", messages=[{"role": "user", "content": "Hello!"}] ) print(response.choices[0].message.content) metadata = llm.get_provider_metadata() print(f"Supports streaming: {metadata.streaming}") print(f"Supports tools: {metadata.completion}") ``` ### API Call ```python import os from any_llm import completion # Make sure you have the appropriate API key set api_key = os.environ.get('MISTRAL_API_KEY') if not api_key: raise ValueError("Please set MISTRAL_API_KEY environment variable") # Recommended: separate provider and model parameters response = completion( model="mistral-small-latest", provider="mistral", messages=[{"role": "user", "content": "Hello!"}] ) print(response.choices[0].message.content) ``` ### When to Choose Which Approach **Use Direct API Functions (`completion`, `acompletion`) when:** - Making simple, one-off requests - Prototyping or writing quick scripts - You want the simplest possible interface **Use Provider Class (`AnyLLM.create`) when:** - Building applications that make multiple requests with the same provider - You want to avoid repeated provider instantiation overhead **Finding model names:** Check the [providers page](./providers.md) for provider IDs, or use the [`list_models`](./api/list_models.md) API to see available models for your provider. ## Streaming For the [providers that support streaming](./providers.md), you can enable it by passing `stream=True`: ```python output = "" for chunk in completion( model="mistral-small-latest", provider="mistral", messages=[{"role": "user", "content": "Hello!"}], stream=True ): chunk_content = chunk.choices[0].delta.content or "" print(chunk_content) output += chunk_content ``` ## Embeddings [`embedding`][any_llm.embedding] and [`aembedding`][any_llm.aembedding] allow you to create vector embeddings from text using the same unified interface across providers. Not all providers support embeddings - check the [providers documentation](./providers.md) to see which ones do. ```python from any_llm import embedding result = embedding( model="text-embedding-3-small", provider="openai", inputs="Hello, world!" # can be either string or list of strings ) # Access the embedding vector embedding_vector = result.data[0].embedding print(f"Embedding vector length: {len(embedding_vector)}") print(f"Tokens used: {result.usage.total_tokens}") ``` ## Tools `any-llm` supports tool calling for providers that support it. You can pass a list of tools where each tool is either: 1. **Python callable** - Functions with proper docstrings and type annotations 2. **OpenAI Format tool dict** - Already in OpenAI tool format ```python from any_llm import completion def get_weather(location: str, unit: str = "F") -> str: """Get weather information for a location. Args: location: The city or location to get weather for unit: Temperature unit, either 'C' or 'F' Returns: Current weather description """ return f"Weather in {location} is sunny and 75{unit}!" response = completion( model="mistral-small-latest", provider="mistral", messages=[{"role": "user", "content": "What's the weather in Pittsburgh PA?"}], tools=[get_weather] ) ``` any-llm automatically converts your Python functions to OpenAI tools format. Functions must have: - A docstring describing what the function does - Type annotations for all parameters - A return type annotation ## Exception Handling The `any-llm` package provides a unified exception hierarchy that works consistently across all LLM providers. ### Enabling Unified Exceptions !!! info "Opt-in Feature" Unified exception handling is currently **opt-in**. Set the `ANY_LLM_UNIFIED_EXCEPTIONS` environment variable to enable it: ```bash export ANY_LLM_UNIFIED_EXCEPTIONS=1 ``` When enabled, provider-specific exceptions are automatically converted to `any-llm` exception types. When disabled (default), the original provider exceptions are raised with a deprecation warning. ### Basic Usage ```python from any_llm import completion from any_llm.exceptions import ( RateLimitError, AuthenticationError, ProviderError, AnyLLMError, ) try: response = completion( model="gpt-4", provider="openai", messages=[{"role": "user", "content": "Hello!"}] ) except RateLimitError as e: print(f"Rate limited: {e.message}") except AuthenticationError as e: print(f"Auth failed: {e.message}") except ProviderError as e: print(f"Provider error: {e.message}") except AnyLLMError as e: print(f"Error: {e.message}") ``` ### Accessing Original Exceptions All unified exceptions preserve the original provider exception for debugging: ```python from any_llm.exceptions import RateLimitError messages = [{"role": "user", "content": "Hello!"}] try: response = completion(model="gpt-4", provider="openai", messages=messages) except RateLimitError as e: print(f"Provider: {e.provider_name}") print(f"Original exception: {type(e.original_exception)}") ``` --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/api/types/responses.md # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/api/responses.md ## OpenResponses API The Responses API in any-llm implements the [OpenResponses](https://www.openresponses.org/) specification—an open-source standard for building multi-provider, interoperable LLM interfaces for agentic AI systems. !!! info "Learn More" - [OpenResponses Specification](https://www.openresponses.org/specification) - [OpenResponses Reference](https://www.openresponses.org/reference) - [HuggingFace Responses API Guide](https://huggingface.co/docs/inference-providers/guides/responses-api) ### Return Types The `responses()` and `aresponses()` functions return different types depending on the provider's level of OpenResponses compliance: | Return Type | When Returned | |-------------|---------------| | `openresponses_types.ResponseResource` | Providers fully compliant with the OpenResponses specification | | `openai.types.responses.Response` | Providers using OpenAI's native Responses API (not yet fully OpenResponses-compliant) | | `Iterator[dict]` / `AsyncIterator[dict]` | When `stream=True` is set | Both `ResponseResource` and `Response` share a similar structure, so in many cases you can access common fields like `output` without type checking. ::: any_llm.api.responses ::: any_llm.api.aresponses --- # Source: https://raw.githubusercontent.com/mozilla-ai/any-llm/refs/heads/main/docs/gateway/troubleshooting.md # Troubleshooting ## Database connection errors Make sure the database URL is correct and the database is accessible: ```bash python -c "from sqlalchemy import create_engine; engine = create_engine('postgresql://user:pass@host/db'); print('OK')" ``` ## Common Issues ### Authentication Errors - Ensure you're using the correct master key format: `Bearer your-secure-master-key` - Check that the `X-AnyLLM-Key` header is properly set - Verify that virtual API keys are active and not expired ### Configuration Issues - Verify your `config.yml` file is properly formatted - Check that environment variables are set correctly - Ensure provider API keys are valid and have proper permissions ### Budget Enforcement - Check that budgets are properly assigned to users - Verify budget limits are set correctly - Monitor user spending to ensure limits are being enforced ## Getting Help - Check the logs for detailed error messages - Verify your configuration matches the examples in the documentation - Ensure all required environment variables are set