# In-memory storage (use a database in production)
alert_history = []

@app.post("/webhook")
async def receive_webhook(request: Request):
    data = await request.json()
    
    # Store alert
    alert_history.append({
        'timestamp': datetime.utcnow(),
        'event_type': data.get('eventType'),
        'alert_name': data['payload']['alertName'],
        'event_count': data['payload']['eventCount'],
        'data': data
    })
    
    # Keep only last 1000 alerts
    if len(alert_history) > 1000:
        alert_history.pop(0)
    
    return {"status": "success"}

@app.get("/dashboard")
async def get_dashboard():
    # Return aggregated statistics
    return {
        'total_alerts': len(alert_history),
        'by_type': group_by_type(alert_history),
        'recent_alerts': alert_history[-10:]
    }
```

## Supported event types

Opik supports ten types of alert events:

### Observability events

**Trace errors threshold exceeded**

* **Event type**: `trace:errors`
* **Triggered when**: Total trace error count exceeds the specified threshold within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires threshold value (error count) and time window (in seconds)
* **Payload**: Metrics alert payload with error count details
* **Use case**: Proactive error monitoring, detect error spikes, prevent system degradation

**Trace feedback score threshold exceeded**

* **Event type**: `trace:feedback_score`
* **Triggered when**: Average trace feedback score meets the specified threshold criteria within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires feedback score name, threshold value, operator (`>`, `<`), and time window
* **Payload**: Metrics alert payload with average feedback score details
* **Use case**: Track model performance, monitor user satisfaction, detect quality degradation

**Thread feedback score threshold exceeded**

* **Event type**: `trace_thread:feedback_score`
* **Triggered when**: Average thread feedback score meets the specified threshold criteria within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires feedback score name, threshold value, operator (`>`, `<`), and time window
* **Payload**: Metrics alert payload with average feedback score details
* **Use case**: Monitor conversation quality, track multi-turn interactions, detect thread satisfaction issues

**Guardrails triggered**

* **Event type**: `trace:guardrails_triggered`
* **Triggered when**: A guardrail check fails for a trace
* **Project scope**: Can be configured to specific projects
* **Payload**: Array of guardrail result objects
* **Use case**: Security monitoring, compliance tracking, PII detection

**Cost threshold exceeded**

* **Event type**: `trace:cost`
* **Triggered when**: Total trace cost exceeds the specified threshold within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires threshold value (in currency units) and time window (in seconds)
* **Payload**: Metrics alert payload with cost details
* **Use case**: Budget monitoring, cost control, prevent runaway spending

**Latency threshold exceeded**

* **Event type**: `trace:latency`
* **Triggered when**: Average trace latency exceeds the specified threshold within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires threshold value (in seconds) and time window (in seconds)
* **Payload**: Metrics alert payload with latency details
* **Use case**: Performance monitoring, SLA compliance, user experience tracking

### Prompt engineering events

**New prompt added**

* **Event type**: `prompt:created`
* **Triggered when**: A new prompt is created in the prompt library
* **Project scope**: Workspace-wide
* **Payload**: Prompt object with metadata
* **Use case**: Track prompt library changes, audit prompt creation

**New prompt version created**

* **Event type**: `prompt:committed`
* **Triggered when**: A new version (commit) is added to a prompt
* **Project scope**: Workspace-wide
* **Payload**: Prompt version object with template and metadata
* **Use case**: Monitor prompt iterations, track version history

**Prompt deleted**

* **Event type**: `prompt:deleted`
* **Triggered when**: A prompt is removed from the prompt library
* **Project scope**: Workspace-wide
* **Payload**: Array of deleted prompt objects
* **Use case**: Audit prompt deletions, maintain prompt governance

### Evaluation events

**Experiment finished**

* **Event type**: `experiment:finished`
* **Triggered when**: An experiment completes in the workspace
* **Project scope**: Workspace-wide
* **Payload**: Array of experiment objects with completion details
* **Use case**: Automate experiment notifications, track evaluation completions

### Want us to support more event types?

If you need additional event types for your use case, please [create an issue on GitHub](https://github.com/comet-ml/opik/issues/new?title=Alert%20Event%20Request%3A%20%3Cevent-name%3E\&labels=enhancement) and let us know what you'd like to monitor.

## Webhook payload structure

All webhook events follow a consistent payload structure:

```json
{
  "id": "webhook-event-id",
  "eventType": "trace:errors",
  "alertId": "alert-uuid",
  "alertName": "Production Errors Alert",
  "workspaceId": "workspace-uuid",
  "createdAt": "2025-01-15T10:30:00Z",
  "payload": {
    "alertId": "alert-uuid",
    "alertName": "Production Errors Alert",
    "eventType": "trace:errors",
    "eventIds": ["event-id-1", "event-id-2"],
    "userNames": ["user@example.com"],
    "eventCount": 2,
    "aggregationType": "consolidated",
    "message": "Alert 'Production Errors Alert': 2 trace:errors events aggregated",
    "metadata": [
      {
        "id": "trace-uuid",
        "name": "handle_query",
        "project_id": "project-uuid",
        "project_name": "Demo Project",
        "start_time": "2025-01-15T10:29:45Z",
        "end_time": "2025-01-15T10:29:50Z",
        "input": {
          "query": "User question"
        },
        "output": {
          "response": "LLM response"
        },
        "error_info": {
          "exception_type": "ValidationException",
          "message": "Validation failed",
          "traceback": "Full traceback..."
        },
        "metadata": {
          "customer_id": "customer_123"
        },
        "tags": ["production"]
      }
    ]
  }
}
```

### Payload fields

| Field                     | Type              | Description                                |
| ------------------------- | ----------------- | ------------------------------------------ |
| `id`                      | string            | Unique webhook event identifier            |
| `eventType`               | string            | Type of event (e.g., `trace:errors`)       |
| `alertId`                 | string (UUID)     | Alert configuration identifier             |
| `alertName`               | string            | Name of the alert                          |
| `workspaceId`             | string            | Workspace identifier                       |
| `createdAt`               | string (ISO 8601) | Timestamp when webhook was created         |
| `payload.eventIds`        | array             | List of aggregated event IDs               |
| `payload.userNames`       | array             | Users associated with the events           |
| `payload.eventCount`      | number            | Number of aggregated events                |
| `payload.aggregationType` | string            | Always "consolidated"                      |
| `payload.metadata`        | array             | Event-specific data (varies by event type) |

## Event-specific payloads

### Trace errors threshold exceeded payload

```json
{
  "metadata": {
    "event_type": "TRACE_ERRORS",
    "metric_name": "trace:errors",
    "metric_value": "15",
    "threshold": "10",
    "window_seconds": "900",
    "project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
    "project_names": "Demo Project,Default Project"
  }
}
```

### Trace feedback score threshold exceeded payload

```json
{
  "metadata": {
    "event_type": "TRACE_FEEDBACK_SCORE",
    "metric_name": "trace:feedback_score",
    "metric_value": "0.7500",
    "threshold": "0.8000",
    "window_seconds": "3600",
    "project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
    "project_names": "Demo Project,Default Project"
  }
}
```

### Thread feedback score threshold exceeded payload

```json
{
  "metadata": {
    "event_type": "TRACE_THREAD_FEEDBACK_SCORE",
    "metric_name": "trace_thread:feedback_score",
    "metric_value": "0.7500",
    "threshold": "0.8000",
    "window_seconds": "3600",
    "project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
    "project_names": "Demo Project,Default Project"
  }
}
```

### Prompt created payload

```json
{
  "metadata": {
    "id": "prompt-uuid",
    "name": "Prompt Name",
    "description": "Prompt description",
    "tags": ["system", "assistant"],
    "created_at": "2025-01-15T10:00:00Z",
    "created_by": "user@example.com",
    "last_updated_at": "2025-01-15T10:00:00Z",
    "last_updated_by": "user@example.com"
  }
}
```

### Prompt version created payload

```json
{
  "metadata": {
    "id": "version-uuid",
    "prompt_id": "prompt-uuid",
    "commit": "abc12345",
    "template": "You are a helpful assistant. {{question}}",
    "type": "mustache",
    "metadata": {
      "version": "1.0",
      "model": "gpt-4"
    },
    "created_at": "2025-01-15T10:00:00Z",
    "created_by": "user@example.com"
  }
}
```

### Prompt deleted payload

```json
{
  "metadata": [
    {
      "id": "prompt-uuid",
      "name": "Prompt Name",
      "description": "Prompt description",
      "tags": ["deprecated"],
      "created_at": "2025-01-10T10:00:00Z",
      "created_by": "user@example.com",
      "last_updated_at": "2025-01-15T10:00:00Z",
      "last_updated_by": "user@example.com",
      "latest_version": {
        "id": "version-uuid",
        "commit": "abc12345",
        "template": "Template content",
        "type": "mustache",
        "created_at": "2025-01-15T10:00:00Z",
        "created_by": "user@example.com"
      }
    }
  ]
}
```

### Guardrails triggered payload

```json
{
  "metadata": [
    {
      "id": "guardrail-check-uuid",
      "entity_id": "trace-uuid",
      "project_id": "project-uuid",
      "project_name": "Project Name",
      "name": "PII",
      "result": "failed",
      "details": {
        "detected_entities": ["EMAIL", "PHONE_NUMBER"],
        "message": "PII detected in response: email and phone number"
      }
    }
  ]
}
```

### Experiment finished payload

```json
{
  "metadata": [
    {
      "id": "experiment-uuid",
      "name": "Experiment Name",
      "dataset_id": "dataset-uuid",
      "created_at": "2025-01-15T10:00:00Z",
      "created_by": "user@example.com",
      "last_updated_at": "2025-01-15T10:05:00Z",
      "last_updated_by": "user@example.com",
      "feedback_scores": [
        {
          "name": "accuracy",
          "value": 0.92
        },
        {
          "name": "latency",
          "value": 1.5
        }
      ]
    }
  ]
}
```

### Cost threshold exceeded payload

```json
{
  "metadata": {
    "event_type": "TRACE_COST",
    "metric_name": "trace:cost",
    "metric_value": "150.75",
    "threshold": "100.00",
    "window_seconds": "3600",
    "project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
    "project_names": "Demo Project,Default Project"
  }
}
```

### Latency threshold exceeded payload

```json
{
  "metadata": {
    "event_type": "TRACE_LATENCY",
    "metric_name": "trace:latency",
    "metric_value": "5250.5000",
    "threshold": "5",
    "window_seconds": "1800",
    "project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
    "project_names": "Demo Project,Default Project"
  }
}
```

## Securing your webhooks

### Using secret tokens

Add a secret token to your webhook configuration to verify that incoming requests are from Opik:

1. Generate a secure random token (e.g., using `openssl rand -hex 32`)
2. Add it to your alert's "Secret token" field
3. Opik will send it in the `Authorization` header: `Authorization: Bearer your-secret-token`
4. Validate the token in your webhook handler before processing the request

### Example validation (Python/Flask)

```python
from flask import Flask, request, abort
import hmac

app = Flask(__name__)
SECRET_TOKEN = "your-secret-token-here"

@app.route('/webhook', methods=['POST'])
def handle_webhook():
    # Verify the secret token
    auth_header = request.headers.get('Authorization', '')
    if not auth_header.startswith('Bearer '):
        abort(401, 'Missing or invalid Authorization header')
    
    token = auth_header.split(' ', 1)[1]
    if not hmac.compare_digest(token, SECRET_TOKEN):
        abort(401, 'Invalid secret token')
    
    # Process the webhook
    data = request.json
    event_type = data.get('eventType')
    
    # Handle different event types
    if event_type == 'trace:errors':
        handle_trace_errors(data)
    elif event_type == 'trace:feedback_score':
        handle_feedback_score(data)
    elif event_type == 'experiment:finished':
        handle_experiment_finished(data)
    
    return {'status': 'success'}, 200
```

### Using custom headers

You can add custom headers for additional authentication or routing:

```python