# AI Providers

> Configure connections to Large Language Model providers

The AI Providers tab allows you to configure connections to different Large Language Models (LLMs). This page explains how to set up and manage AI provider integrations within Opik.

## Overview

Connecting AI providers enables you to:

* Send prompts and receive responses from different LLMs
* Set up a provider in one place and use it across projects
* Automatically record model metadata in the Playground
* Track and analyze traces using online evaluation rules

## Managing AI Providers

### Viewing Existing Providers

<Frame>
  <img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2026-01-30T17:33:07.444Z/img/configuration/ai_providers.png" alt="AI Providers Table" />
</Frame>

The AI Providers page displays a table of all configured connections with the following columns:

* **Name**: The name or identifier of the API key
* **Created**: The date and time when the provider was configured
* **Provider**: The type of AI provider (e.g., OpenAI)

### Adding a New Provider Configuration

To add a new AI provider:

1. Click the **Add configuration** button in the top-right corner

<Frame>
  <img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2026-01-30T17:33:07.444Z/img/configuration/new_ai_provider.png" alt="New AI Provider Modal" />
</Frame>

2. In the Provider Configuration dialog that appears:
   * Select a provider from the dropdown menu
   * Enter your API key for that provider
   * Click **Save** to store the configuration

### Supported Providers

Opik supports integration with various AI providers, including:

* OpenAI
* Anthropic
* OpenRouter
* Gemini
* VertexAI
* Azure OpenAI
* Amazon Bedrock
* Ollama (local or self-hosted, OpenAI-compatible)
* vLLM / any other OpenAI API-compliant provider

<Tip>
  If you would like us to support additional LLM providers, please let us know by opening an issue on
  [GitHub](https://github.com/comet-ml/opik/issues).
</Tip>

### Provider-Specific Setup

Below are instructions for obtaining API keys and other required information for each supported provider:

#### OpenAI

1. Create or log in to your [OpenAI account](https://platform.openai.com/)
2. Navigate to the [API keys page](https://platform.openai.com/api-keys)
3. Click "Create new secret key"
4. Copy your API key (it will only be shown once)
5. In Opik, select "OpenAI" as the provider and paste your key

#### Anthropic

1. Sign up for or log in to [Anthropic's platform](https://console.anthropic.com/)
2. Navigate to the [API Keys page](https://console.anthropic.com/settings/keys)
3. Click "Create Key" and select the appropriate access level
4. Copy your API key (it will only be shown once)
5. In Opik, select "Anthropic" as the provider and paste your key

#### OpenRouter

1. Create or log in to your [OpenRouter account](https://openrouter.ai/)
2. Navigate to the [API Keys page](https://openrouter.ai/settings/keys)
3. Create a new API key
4. Copy your API key
5. In Opik, select "OpenRouter" as the provider and paste your key

#### Gemini

1. Signup or login to [Google AI Studio](https://aistudio.google.com/)
2. Go to the [API keys page](https://aistudio.google.com/apikey)\\
3. Create a new API key for one your existing Google Cloud project
4. Copy your API key (it will only be shown once)
5. In Opik, select "Gemini" as the provider and paste your key

#### Azure OpenAI

Azure OpenAI provides enterprise-grade access to OpenAI models through Microsoft Azure. To use Azure OpenAI with Opik:

1. Get your Azure OpenAI endpoint URL

   * Go to [portal.azure.com](https://portal.azure.com)
   * Navigate to your Azure OpenAI resource
   * Copy your endpoint URL (it looks like `https://your-company.openai.azure.com`)

2. Construct the complete API URL

   * Add `/openai/v1` to the end of your endpoint URL
   * Your complete URL should look like: `https://your-company.openai.azure.com/openai/v1`

3. Configure in Opik
   * In Opik, go to **Configuration → AI Providers**
   * Click **"Add Configuration"**
   * Select **"vLLM / Custom provider"** from the dropdown
   * Enter your complete URL in the URL field: `https://your-company.openai.azure.com/openai/v1`
   * Add your Azure OpenAI API key in the API Key field
   * In the Models section, list the models you have deployed in Azure (e.g., `gpt-4o`)
   * Click **Save** to store the configuration

Once saved, you can use your Azure OpenAI models directly from Online Scores and the Playground.

#### Vertex AI

##### Option A: Setup via `gcloud` CLI

1. **Create a Custom IAM Role**

```bash
gcloud iam roles create opik \
  --project=<my-project> \
  --title="Opik" \
  --description="Custom IAM role for Opik" \
  --permissions=aiplatform.endpoints.predict,resourcemanager.projects.get \
  --stage=ALPHA
```

2. **Create a Service Account**

```bash
gcloud iam service-accounts create opik-sa \
  --description="Service account for Opik role" \
  --display-name="Opik Service Account"
```

3. **Assign the Role to the Service Account**

```bash
gcloud projects add-iam-policy-binding <my-project> \
  --member="serviceAccount:opik-sa@<my-project>.iam.gserviceaccount.com" \
  --role="projects/<my-project>/roles/opik"
```

4. **Generate the Service Account Key File**

```bash
gcloud iam service-accounts keys create opik-key.json \
  --iam-account=opik-sa@<my-project>.iam.gserviceaccount.com
```

> The file `opik-key.json` contains your credentials. **Open it in a text editor and copy the entire contents.**

***

##### Option B: Setup via Google Cloud Console (UI)

Step 1: Create the Custom Role

1. Go to [IAM → Roles](https://console.cloud.google.com/iam-admin/roles)
2. Click **Create Role**
3. Fill in the form:

* **Title**: `Opik`
* **ID**: `opik`
* **Description**: `Custom IAM role for Opik`
* **Stage**: `Alpha`

4. Click **Add Permissions**, then search for and add:

* `aiplatform.endpoints.predict`
* `resourcemanager.projects.get`

5. Click **Create**

Step 2: Create the Service Account

1. Go to [IAM → Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)
2. Click **Create Service Account**
3. Fill in:

* **Service account name**: `Opik Service Account`
* **ID**: `opik-sa`
* **Description**: `Service account for Opik role`

4. Click **Done**

Step 3: Assign the Role to the Service Account

1. Go to [IAM](https://console.cloud.google.com/iam-admin/iam)
2. Find the service account `opik-sa@<my-project>.iam.gserviceaccount.com`
3. Click the **edit icon**
4. Click **Add Another Role** → Select your custom role: **Opik**
5. Click **Save**

Step 4: Create and Download the Key

1. Go to [Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts)
2. Click on the `opik-sa` account
3. Open the **Keys** tab
4. Click **Add Key → Create new key**
5. Choose **JSON**, click **Create**, and download the file

> **Open the downloaded JSON file**, and **copy its entire content** to be used in the next step.

***

##### Final Step: Connect Opik to Vertex AI

1. In Opik, go to **Configuration → AI Providers**
2. Click **“Add Configuration”**
3. Set:

* **Provider**: `Vertex AI`
* **Location**: Your model region (e.g., `us-central1`)
* **Vertex AI API Key**: **Paste the full contents of the `opik-key.json` file here**

4. Click **Add configuration**

#### Amazon Bedrock

Amazon Bedrock provides access to foundation models from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through AWS. Opik connects to Bedrock using the [OpenAI Chat Completions API](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-chat-completions.html). Only models that support this API format will work with the Opik Playground. Check the [supported models documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-chat-completions.html#inference-chat-completions-supported-models) to verify compatibility before configuring.

##### Prerequisites

Before configuring Bedrock in Opik, ensure you have:

1. An active AWS account with Bedrock access
2. Model access enabled for the models you want to use (see [AWS documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html))
3. An API key or credentials configured for Bedrock access

<Tip>
  You can request access to models in the [AWS Bedrock console](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/providers). Not all models are available in all regions — check the [model availability documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) to verify availability in your chosen region.
</Tip>

##### Configuring Bedrock in Opik

1. In Opik, go to **Configuration → AI Providers**
2. Click **"Add Configuration"**
3. Select **"Bedrock"** from the provider dropdown
4. Fill in the configuration:
   * **Provider name**: A unique identifier for this provider instance (e.g., "Bedrock us-east-1")
   * **URL**: Your Bedrock endpoint URL (see format below)
   * **API Key**: Your AWS Bedrock API key (see [AWS documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started-api-keys.html) for setup instructions)
   * **Models list**: Comma-separated list of models you want to use (e.g., `us.anthropic.claude-3-5-sonnet-20241022-v2:0,us.meta.llama3-2-3b-instruct-v1:0`)
   * **Custom headers** (optional): Add any additional HTTP headers required by your configuration
5. Click **Add configuration** to save

##### Bedrock URL Format by Region

Bedrock endpoints follow this pattern: `https://bedrock-runtime.<region>.amazonaws.com/openai/v1`

**Examples by Region:**

* **US East 1**: `https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1`
* **US West 2**: `https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1`
* **Europe West 1 (Ireland)**: `https://bedrock-runtime.eu-west-1.amazonaws.com/openai/v1`
* **Europe Central 1 (Frankfurt)**: `https://bedrock-runtime.eu-central-1.amazonaws.com/openai/v1`
* **Asia Pacific (Tokyo)**: `https://bedrock-runtime.ap-northeast-1.amazonaws.com/openai/v1`
* **Asia Pacific (Singapore)**: `https://bedrock-runtime.ap-southeast-1.amazonaws.com/openai/v1`

##### Multiple Bedrock Instances

You can configure multiple Bedrock providers for different AWS regions or accounts. Each instance appears separately in the provider dropdown, making it easy to switch between configurations in the Playground and Online Evaluation.

#### Ollama

Opik connects to [Ollama](https://ollama.com/) using the OpenAI-compatible API, so you can use Ollama models for all LLM operations.

<Note>
  **URL must end with `/v1`.** The base URL you enter in Opik must end with `/v1` (e.g., `http://localhost:11434/v1`). Opik uses this to call the OpenAI-compatible chat completions endpoint on your Ollama instance.
</Note>

<Tip>
  **Self-hosted deployments:** The Ollama provider is enabled by default. To disable it, set the environment variable `TOGGLE_OLLAMA_PROVIDER_ENABLED=false` on the Opik backend service.
</Tip>

##### Configuring Ollama in Opik

1. In Opik, go to **Configuration → AI Providers**
2. Click **"Add configuration"**
3. Select **"Ollama"** from the provider dropdown
4. Fill in:
   * **Provider name**: A name for this instance (e.g., "Ollama local")
   * **URL**: Base URL of your Ollama instance, ending with `/v1` (e.g., `http://localhost:11434/v1`)
   * **API Key** (optional): Leave blank unless your Ollama instance requires authentication
   * Use **"Test connection"** to verify Opik can reach the instance, then **"Discover models"** to load the model list
5. Click **Save** to store the configuration

You can configure multiple Ollama instances with different provider names and URLs.

#### vLLM / Custom Provider

Use this option to add any other OpenAI API-compliant provider such as vLLM, etc. **You can configure multiple custom providers**, each with their own unique name, URL, and models.

<Frame>
  <img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2026-01-30T17:33:07.444Z/img/configuration/vllm_configuration.png" alt="New Custom AI Provider Modal" />
</Frame>

##### Configuration Steps

1. **Provider Name**: Enter a unique name to identify this custom provider (e.g., "vLLM Production", "Ollama Local", "Azure OpenAI Dev")
2. **URL**: Enter your server URL, for example: `http://host.docker.internal:8000/v1`
3. **API Key** (optional): If your model access requires authentication, enter the API key. Otherwise, leave this field blank.
4. **Models**: List all models available on your server. You'll be able to select one of them for use later.
5. **Custom Headers** (optional): Add any additional HTTP headers required by your custom endpoint as key-value pairs.

<Tip>
  If you're running Opik locally, you would need to use `http://host.docker.internal:<PORT>/v1` for Mac and Windows or `http://172.17.0.1:<PORT>/v1` for Linux, and not `http://localhost`.
</Tip>

##### Custom Headers

Some custom providers may require additional HTTP headers beyond the API key for authentication or routing purposes. You can configure these headers using the "Custom headers" section:

* Click **"+ Add header"** to add a new header
* Enter the header name (e.g., `X-Custom-Auth`, `X-Request-ID`)
* Enter the header value
* Add multiple headers as needed
* Use the trash icon to remove headers

**Common use cases for custom headers:**

* **Custom authentication**: Additional authentication tokens or headers required by your infrastructure
* **Request routing**: Headers for routing requests to specific model versions or deployments
* **Metadata tracking**: Custom headers for tracking or logging purposes
* **Enterprise features**: Headers required for enterprise proxy configurations

<Note>
  Custom headers are sent with every request to your custom provider endpoint. Ensure header values are kept secure and not exposed in logs or error messages.
</Note>

##### Managing Multiple Custom Providers

Once you've configured multiple custom providers, you can:

* **Edit** any custom provider by selecting it from the provider dropdown in the configuration dialog
* **Delete** custom providers that are no longer needed
* **Switch between** different custom providers in the Playground and Automation Rules

Each custom provider appears as a separate option in the provider dropdown, making it easy to work with multiple self-hosted or custom LLM deployments.

## API Key Security

API keys are securely stored and encrypted in the Opik system. Only the name and provider type are visible in the interface after configuration. The actual key values are not displayed after initial setup.

## Using AI Providers

Once configured, AI providers can be used in:

1. **Playground**: For interactive experimentation with different models
2. **Online Evaluation**: For systematic evaluation of model performance

## Best Practices

* Use descriptive names for your API keys to easily identify their purpose
* Regularly rotate API keys according to your organization's security policies
* Create separate keys for development and production environments
* Monitor usage to manage costs effectively

## Troubleshooting

### Common Issues

* **Authentication Errors**: Ensure your API key is valid and hasn't expired
* **Access Denied**: Check that your API key has the required permissions for the models you're trying to use
* **Rate Limiting**: If you're encountering rate limits, consider adjusting your request frequency or contacting your provider to increase your limits

## Additional Resources

* For programmatic configuration of AI providers, see the [API Reference](/reference/rest-api/overview)
* To learn about using different models in your application, see the [SDK Documentation](/reference/overview)