# Count the total number of series by metric
count({__name__=~"temporal_cloud_v1_.*"}) by (__name__)
```

## API Limits

| Limit | Impact | Mitigation |
| ----- | ----- | ----- |
| 30k total datapoints per scrape | Response may be truncated | Use namespace/metric filtering |
| 180 requests per account per hour | HTTP 429 returned | Set appropriate scrape interval of 30-60s |

---

## Temporal Cloud OpenMetrics

:::tip SUPPORT, STABILITY, and DEPENDENCY INFO

Temporal Cloud OpenMetrics support is available in  [Public Preview](/evaluate/development-production-features/release-stages#public-preview).

:::

:::tip PRICING

Future pricing may apply to high-volume usage that exceeds standard [limits](/cloud/metrics/openmetrics/api-reference#api-limits).

:::

Temporal Cloud's [OpenMetrics](https://openmetrics.io/) endpoint provides operational metrics for your Temporal Cloud workloads in industry-standard Prometheus format, enabling comprehensive monitoring across Namespaces, Workflows, and Task Queues with your existing observability stack.

## Quick Links
* [Integrations](/cloud/metrics/openmetrics/metrics-integrations) - Get started exporting metrics with common integrations
* [API Documentation](/cloud/metrics/openmetrics/api-reference) - Endpoint specification and advanced configuration
* [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference) - Complete catalog of all metrics with descriptions and labels
* [Migration Guide](/cloud/metrics/openmetrics/migration-guide) - Guide on how to transition from the Prometheus query endpoint

## Overview
Temporal Cloud OpenMetrics exposes 30+ metrics covering workflow lifecycles, task queue operations, service performance, and system limits. All metrics are aggregated over one-minute windows and available for scraping within two minutes.

* [Set up authentication and scraping](/cloud/metrics/openmetrics/api-reference#authentication) with the API documentation. 
* Browse the [complete metrics catalog](/cloud/metrics/openmetrics/metrics-reference) for descriptions and labels. 
* Teams using the query endpoint should review the [migration guide](/cloud/metrics/openmetrics/migration-guide).

## API key authentication
Create a [service account](/cloud/metrics/openmetrics/migration-guide#create-an-api-key) with the "Metrics Read-Only" role, generate an API key, and start scraping immediately - no certificate rotation or distribution required.

## Global endpoint
This is a single endpoint at `metrics.temporal.io` which serves all metrics across your entire account with API key authentication and standard HTTPS.

## Namespace and metric filtering
You can use query parameters to enable selective scraping to manage data volume and costs, which support wildcards for flexible namespace selection and specific metric filtering.

## Dashboard templates
Production-ready [Grafana dashboards](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) provide immediate visibility with pre-built queries and visualizations.

---

## Metrics Integrations

Metrics can be exported from Temporal Cloud using the OpenMetrics endpoint. This document describes configuring integrations that have third party support or are based on open standards.
This document is for basic configuration only. For advanced concepts such as label management and high cardinality scenarios see the 
[general API reference](/cloud/metrics/openmetrics/api-reference).

:::tip SUPPORT, STABILITY, and DEPENDENCY INFO

Temporal Cloud OpenMetrics support is available in  [Public Preview](/evaluate/development-production-features/release-stages#public-preview).

:::

## Integrations

### Grafana Cloud

Grafana provides a serverless integration with the OpenMetrics endpoint for Grafana Cloud. This integration will scrape metrics, store them in Grafana Cloud, and provides a default dashboard
for visualizing the metrics in Grafana Cloud. See the [integration page](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-temporal/)
 for more details.

### ClickStack

ClickHouse provides an integration with the OpenMetrics endpoint for ClickStack. This integration uses an OpenTelemetry collector to read from the OpenMetrics endpoint, ingest data into ClickHouse, and
includes a default dashboard to visualize the data with HyperDX. See the [integration page](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics) for more details.

### New Relic

New Relic integrates with Temporal Cloud via the infrastructure agent using a flex integration that pull data from the OpenMetrics endpoint. See the [integration page](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/) for more details.

### Prometheus \+ Grafana

Self hosted Prometheus can be used to scrape the OpenMetrics endpoint.

1. Add a new scrape job for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys).

```yaml
scrape_configs:
  - job_name: 'temporal-cloud'
    scrape_interval: 60s
    scrape_timeout: 30s
    honor_timestamps: true
    scheme: https
    authorization:
      type: Bearer
      credentials: '<API_KEY>'
    static_configs:
      - targets: ['metrics.temporal.io']
    metrics_path: '/v1/metrics'
```

2. Import the [Grafana dashboard](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) and configure your Prometheus datasource.

### OpenTelemetry Collector Configuration

Collect metrics with a self-hosted OpenTelemetry Collector to ingest into the system of your choosing.

1. Add a new prometheus receiver for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys).

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: 'temporal-cloud'
        scrape_interval: 60s
        scrape_timeout: 30s
        honor_timestamps: true
        scheme: https
        authorization:
          type: Bearer
          credentials_file: <API_KEY_FILE>
        static_configs:
          - targets: ['metrics.temporal.io']
        metrics_path: '/v1/metrics'

processors:
  batch:

exporters:
  otlphttp:
    endpoint: <ENDPOINT>

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlphttp]
```

:::info

Examples for these integrations and more are [here](https://github.com/temporal-community/cloud-metrics-scrape-examples).

:::

---

## OpenMetrics Metrics Reference

This document describes all metrics available from the Temporal Cloud OpenMetrics endpoint.

:::tip SUPPORT, STABILITY, and DEPENDENCY INFO

Temporal Cloud OpenMetrics support is available in  [Public Preview](/evaluate/development-production-features/release-stages#public-preview).

:::

## Metric Conventions

### Metric Types

All metrics are exposed as OpenMetrics gauges, but represent different measurement types:

* *Rate Metrics*: per-second rate of the aggregated values  
* *Value Metrics*: The most recent aggregate value within a look-back window (e.g. backlogs, limits)  
* *Percentile Metrics*: Pre-calculated aggregated latency percentiles in seconds

:::note

All metrics are stored as 1 minute aggregates.

:::

### Common Labels

All metrics include these base labels:

| Label | Description |
| ----- | ----- |
| `temporal_namespace` | The Temporal namespace |
| `temporal_account` | The Temporal account identifier |
| `region` | Cloud region where the metric originated |

## Metrics Catalog

### Frontend Service Metrics

#### temporal\_cloud\_v1\_service\_request\_count

gRPC requests received per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_request\_throttled\_count

gRPC requests throttled per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_error\_count

gRPC errors per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_pending\_requests

The number of pollers that are waiting for a task. Use this to track against ``temporal_cloud_v1_poller_limit``

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Value

#### temporal\_cloud\_v1\_resource\_exhausted\_error\_count

Resource exhaustion errors per second. This metric does not include throttling due to Namespace limits.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Rate

#### temporal\_cloud\_v1\_service\_latency\_p50

:::caution

Avoid aggregating this metric across dimensions because the percentile won't be accurate.

:::

The 50th percentile latency of service requests in seconds

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Latency  

#### temporal\_cloud\_v1\_service\_latency\_p95

:::caution

Avoid aggregating this metric across dimensions because the percentile won't be accurate.

:::

The 95th percentile latency of service requests in seconds

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Latency  

#### temporal\_cloud\_v1\_service\_latency\_p99

:::caution

Avoid aggregating this metric across dimensions as the percentile won't be accurate.

:::

The 99th percentile latency of service requests in seconds

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |

**Type**: Latency  

### Workflow Completion Metrics

:::caution High Cardinality

These metrics could have high cardinality depending on number of workflow types and task queues.

:::

#### temporal\_cloud\_v1\_workflow\_success\_count

Successful workflow completions per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate  

#### temporal\_cloud\_v1\_workflow\_failed\_count

Workflow failures per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_timeout\_count

Workflow timeouts per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_cancel\_count

Workflow cancellations per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_terminate\_count

Workflow terminations per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

#### temporal\_cloud\_v1\_workflow\_continued\_as\_new\_count

Workflows continued as new per second.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |

**Type**: Rate

### Task Queue Metrics

:::caution High Cardinality

These metrics could have high cardinality depending on number of task queues present.

:::

#### temporal\_cloud\_v1\_approximate\_backlog\_count

The approximate number of tasks pending in a task queue. Started Activities are not included in the count as they have been dequeued from the task queue.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |

**Type**: Value

#### temporal\_cloud\_v1\_poll\_success\_count

Successfully matched tasks per second.

| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |

**Type**: Rate

#### temporal\_cloud\_v1\_poll\_success\_sync\_count

Tasks matched synchronously per second (no polling wait).

| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |

**Type**: Rate

#### temporal\_cloud\_v1\_poll\_timeout\_count

The rate of poll requests that timed out without receiving a task.

| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |

**Type**: Rate

#### temporal\_cloud\_v1\_no\_poller\_tasks\_count

The rate of tasks added to queues with no active pollers.

| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |

**Type**: Rate

### Namespace Metrics

#### temporal\_cloud\_v1\_namespace\_open\_workflows

The current number of open workflows in a namespace.

**Type**: Value

#### temporal\_cloud\_v1\_total\_action\_count

The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``.

| Label | Description |
| ----- | ----- |
| `is_background` | Whether the action was background: `true` or `false`. Background actions (e.g. History export) do not count toward the action rate limit |
| `namespace_mode` | Indicates if actions are produced by an `active` or a `standby` Namespace |

**Type**: Rate

#### temporal\_cloud\_v1\_total\_action\_throttled\_count

The total number of actions throttled per second.

**Type**: Rate

#### temporal\_cloud\_v1\_operations\_count

Operations performed per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
| `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit |
| `namespace_mode` | Indicates if operations are produced by an `active` or a `standby` Namespace |

**Type**: Rate

#### temporal\_cloud\_v1\_operations\_throttled\_count

Operations throttled due to rate limits per second.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
| `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit |
| `namespace_mode` | Indicates if actions are throttled in an `active` or a `standby` Namespace |

**Type**: Rate

### Schedule Metrics

#### temporal\_cloud\_v1\_schedule\_action\_success\_count

Successfully executed scheduled workflows per second.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_buffer\_overruns\_count

The rate of schedule buffer overruns when using `BUFFER_ALL` overlap policy.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_missed\_catchup\_window\_count

The rate of missed schedule executions outside the catchup window.

**Type**: Rate

#### temporal\_cloud\_v1\_schedule\_rate\_limited\_count

The rate of scheduled workflows delayed due to rate limiting.

**Type**: Rate

### Replication Metrics

#### temporal\_cloud\_v1\_replication\_lag\_p50

The 50th percentile cross-region replication lag in seconds.

**Type**: Latency

#### temporal\_cloud\_v1\_replication\_lag\_p95

The 95th percentile cross-region replication lag in seconds.

**Type**: Latency

#### temporal\_cloud\_v1\_replication\_lag\_p99

The 99th percentile cross-region replication lag in seconds.

**Type**: Latency

### Limit Metrics

#### temporal\_cloud\_v1\_operations\_limit

The current configured operations per second limit for a namespace.

**Type**: Value

#### temporal\_cloud\_v1\_action\_limit

The current configured actions per second limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_total_action_count`` and `is_background=false`.

**Type**: Value

#### temporal\_cloud\_v1\_service\_request\_limit

The current configured frontend service RPS limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_request_count``

**Type**: Value

#### temporal\_cloud\_v1\_poller\_limit

The current configured poller limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_pending_requests``.

**Type**: Value

---

## OpenMetrics Migration Guide

Temporal Cloud is transitioning from our Prometheus query endpoint to an industry-standard OpenMetrics
(Prometheus-compatible) endpoint for metrics collection. This migration represents a significant improvement in how you
can monitor your Temporal Cloud workloads, bringing enhanced capabilities, better integration with observability tools,
and access to high-cardinality metrics that were previously unavailable.

:::tip SUPPORT, STABILITY, and DEPENDENCY INFO

The OpenMetrics endpoint is available in
[Public Preview](/evaluate/development-production-features/release-stages#public-preview) for testing and validation.
The existing Prometheus query endpoint remains fully operational and supported.

:::

## Why We're Making This Change

1. **Industry-Standard Format**: Native compatibility with Prometheus and OpenTelemetry and all major observability
   platforms (Datadog, New Relic etc.) without custom integrations.

2. **High-Cardinality Metrics**: Access to previously unavailable dimensions including:
   - `temporal_task_queue` labels on multiple metrics
   - `temporal_workflow_type` labels for workflow-specific monitoring
   - New task queue backlog metrics for better operational visibility
3. **Accurate Percentiles**: Our new system provides accurate percentile calculations for latency metrics, even in the
   presence of substantial outliers, unlike Prometheus-style histograms.

4. **Simplified Integration**: Direct scraping from your observability tools without intermediate translation layers.

5. **Enhanced Performance**: Optimized for high-cardinality data with built-in safeguards for system stability. Data is
   available to scrape two minutes from the time it was emitted, in line with the freshest metrics
   [available from any major service provider](https://docs.datadoghq.com/integrations/guide/cloud-metric-delay/).

## What's Changing

| Aspect                 | Current Query Endpoint                             | New OpenMetrics Endpoint                    |
| ---------------------- | -------------------------------------------------- | ------------------------------------------- |
| **Protocol**           | Prometheus Query API (`/api/v1/query`)             | OpenMetrics scrape endpoint (`/v1/metrics`) |
| **Authentication**     | mTLS certificates with customer-specific endpoints | API keys with global endpoint               |
| **Metric Temporality** | Cumulative counters                                | Delta temporality (pre-computed rates)      |
| **Query Requirement**  | Direct queries supported                           | Requires observability platform             |
| **Cardinality**        | Limited labels                                     | High-cardinality labels available           |
| **Metric Naming**      | `*_v0_*` metrics                                   | `*_v1_*` metrics                            |

## Migration Timeline

Here is the current estimated timeline for migrating from the Prometheus query endpoint to the OpenMetrics endpoint.

:::caution

Timelines can shift so be sure to stay up to date on upcoming releases.

:::

**Public Preview (Current)**

- OpenMetrics endpoint available for onboarding.
- Both endpoints run in parallel with no changes required.

**General Availability [TBA]**:

- OpenMetrics endpoint becomes production-ready and the standard for metrics collection.

**Query Endpoint Deprecation (6 months after GA)**:

- Prometheus query endpoint deprecated and eventually removed.

:::important Action Required

Complete migration before the 6 month deprecation window ends.

:::

## Notable Differences

### 1\. No longer use `rate()` in Prometheus queries

Metrics are now pre-computed as per-second rates with delta temporality.

**Before (Prometheus query endpoint)**:

```
rate(temporal_cloud_v0_frontend_service_request_count[1m])
```

**After (OpenMetrics endpoint)**:

```
temporal_cloud_v1_service_request_count
```

### 2\. Functions that no longer apply

Metrics from OpenMetrics are already rates, therefore certain Prometheus functions no longer make sense. Below is a
non-exhaustive list of some of the functions:

- ❌ `rate()` \- Already computed
- ❌ `increase()` \- Increase of a rate is meaningless
- ❌ `irate()` \- Instant rate not applicable
- ❌ `histogram_quantile()` \- Not applicable (explicit percentiles provided instead)
- ✅ `sum()`, `avg()`, `max()`, `min()` \- Still work normally

### 3\. Percentile metrics

The new endpoint provides explicit percentile metrics (p50, p95, p99) rather than histogram buckets:

**Before (Prometheus query endpoint)**: Calculate percentiles using `histogram_quantile()`

```shell
histogram_quantile(0.95, rate(temporal_cloud_v0_service_latency_bucket[5m]))
```

**After (OpenMetrics endpoint)**: Use pre-calculated percentiles directly

```
temporal_cloud_v1_service_latency_p95
```

**Important Tradeoff**: While pre-calculated percentiles are more accurate for individual time series, they _cannot be
accurately aggregated_. For example:

- ❌ Cannot sum or average p95 values across Namespaces to get a global p95
- ❌ Cannot aggregate p95 values across regions or Task Queues
- ✅ Can still view individual namespace/task queue percentiles accurately
- ✅ More accurate percentile calculations for individual series, especially with outliers

### 4\. Authentication Setup

**Before**: mTLS certificates with customer-specific endpoint

```shell
curl --cert /path/to/client.pem \
     --key /path/to/client.key \
     --cacert /path/to/ca.pem \

"https://<customer-specific>.tmprl.cloud/api/v1/query?query=rate(temporal_cloud_v0_frontend_service_request_count[5m])&time=2025-01-15T10:00:00Z"
```

**After**: API key with global endpoint

```shell
curl -H "Authorization: Bearer <API_KEY>" https://metrics.temporal.io/v1/metrics
```

## Migration Steps

### Create an API Key

Create a service account within the Temporal Cloud UI settings with the “Metrics Read-Only” Account Level Role.

:::note

As this is an account-level role, scoping it to specific namespaces has no effect as it will have access to the full
account’s metrics.

:::

<CaptionedImage
  src="/img/cloud/metrics/service-account-with-metrics-role.png"
  title="Create Service Account with Metrics Read-Only Role"
/>

Once this is created, you can create an API key within this service account which will inherit the role. Save this API
key in a secure location and use it to access the metrics APIs.

To test that this works, curl the endpoint with your API Key.

The output should resemble the following example:

```shell
$ curl -H "Authorization: Bearer <API_KEY>" https://metrics.temporal.io/v1/metrics