# HELP temporal_cloud_v1_service_request_count The number of RPC requests received by the service..
```

Now you are ready to scrape your metrics\!

### Configuring Grafana \+ Prometheus

#### Update Prometheus Configuration

Add a new scrape job for the OpenMetrics endpoint with your API key.

```yaml
scrape_configs:
  - job_name: temporal-cloud
    static_configs:
      - targets:
          - 'metrics.temporal.io'
    scheme: https
    metrics_path: '/v1/metrics'
    honor_timestamps: true
    scrape_interval: 60s
    scrape_timeout: 30s
    authorization:
      type: Bearer
      credentials: 'API_KEY'
```

:::note

This replaces the direct Grafana datasource configuration you used with the query endpoint.

:::

#### Install New Dashboards

- Download the new Grafana dashboard:
  [temporal_cloud_openmetrics.json](https://github.com/temporalio/dashboards/blob/master/cloud/temporal_cloud_openmetrics.json)
- Import alongside existing dashboards during transition
- Update any custom alerts and queries to use new metrics and remove `rate()` functions

### Configuring Datadog

:::tip

Automated integration update coming soon.

:::

The Datadog team is working on updating the official Temporal Cloud integration to use the new endpoint. This transition
should be largely transparent for most users.

For users that want to get started immediately, Temporal Cloud metrics can be directly integrated into Datadog by
configuring the Datadog agent to scrape the OpenMetrics endpoint. An example for that lives
[here](https://github.com/temporal-community/cloud-metrics-scrape-examples/tree/main/datadog/openmetrics).

#### Other Observability Providers

Consult the documentation for your observability system for how to configure it to scrape this endpoint and retrieve
your metrics:

- [NewRelic](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-openmetrics/configure-prometheus-openmetrics-integrations/)
- [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/configuration/#receivers)

Examples for all these integrations live [here](https://github.com/temporal-community/cloud-metrics-scrape-examples).

### Metric Mapping Reference

Below is a template for mapping metrics from the old query endpoint to the new OpenMetrics endpoint. Note that all metrics follow the pattern of `v0` → `v1` version change, and the fundamental difference is the shift from cumulative counters to pre-computed rates for the majority of the metrics. Note that the labels below are only new labels added to the metrics. For the complete list of labels, see the /cloud/metrics/openmetrics/metrics-reference.

#### Frontend Service Metrics

| Old Metric (v0)                                    | New Metric (v1)                                    | New Labels |
| -------------------------------------------------- | -------------------------------------------------- | ---------- |
| `temporal_cloud_v0_frontend_service_error_count`   | `temporal_cloud_v1_service_error_count`            | `region`   |
| `temporal_cloud_v0_frontend_service_request_count` | `temporal_cloud_v1_service_request_count`          | `region`   |
| `temporal_cloud_v0_resource_exhausted_error_count` | `temporal_cloud_v1_resource_exhausted_error_count` | `region`   |
| `temporal_cloud_v0_total_action_count`             | `temporal_cloud_v1_total_action_count`             | `region`   |

#### Workflow Metrics

| Old Metric (v0)                                     | New Metric (v1)                                     | New Labels                                              |
| --------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------- |
| `temporal_cloud_v0_workflow_cancel_count`           | `temporal_cloud_v1_workflow_cancel_count`           | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_continued_as_new_count` | `temporal_cloud_v1_workflow_continued_as_new_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_failed_count`           | `temporal_cloud_v1_workflow_failed_count`           | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_success_count`          | `temporal_cloud_v1_workflow_success_count`          | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_terminate_count`        | `temporal_cloud_v1_workflow_terminate_count`        | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_timeout_count`          | `temporal_cloud_v1_workflow_timeout_count`          | `region` `temporal_workflow_type` `temporal_task_queue` |

#### Poll Metrics

| Old Metric (v0)                             | New Metric (v1)                             | New Labels                     |
| ------------------------------------------- | ------------------------------------------- | ------------------------------ |
| `temporal_cloud_v0_poll_success_count`      | `temporal_cloud_v1_poll_success_count`      | `region` `temporal_task_queue` |
| `temporal_cloud_v0_poll_success_sync_count` | `temporal_cloud_v1_poll_success_sync_count` | `region` `temporal_task_queue` |
| `temporal_cloud_v0_poll_timeout_count`      | `temporal_cloud_v1_poll_timeout_count`      | `region` `temporal_task_queue` |

#### Latency Metrics

| Old Metric (v0)                                                                                                          | New Metric (v1)                                                                                                     | New Labels |
| ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ---------- |
| `temporal_cloud_v0_service_latency_bucket temporal_cloud_v0_service_latency_count temporal_cloud_v0_service_latency_sum` | `temporal_cloud_v1_service_latency_p99 temporal_cloud_v1_service_latency_p95 temporal_cloud_v1_service_latency_p50` | `region`   |
| `temporal_cloud_v0_replication_lag_bucket temporal_cloud_v0_replication_lag_count temporal_cloud_v0_replication_lag_sum` | `temporal_cloud_v1_replication_lag_p99 temporal_cloud_v1_replication_lag_p95 temporal_cloud_v1_replication_lag_p50` | `region`   |

#### Schedule Metrics

| Old Metric (v0)                                          | New Metric (v1)                                          | New Labels |
| -------------------------------------------------------- | -------------------------------------------------------- | ---------- |
| `temporal_cloud_v0_schedule_action_success_count`        | `temporal_cloud_v1_schedule_action_success_count`        | `region`   |
| `temporal_cloud_v0_schedule_buffer_overruns_count`       | `temporal_cloud_v1_schedule_buffer_overruns_count`       | `region`   |
| `temporal_cloud_v0_schedule_missed_catchup_window_count` | `temporal_cloud_v1_schedule_missed_catchup_window_count` | `region`   |
| `temporal_cloud_v0_schedule_rate_limited_count`          | `temporal_cloud_v1_schedule_rate_limited_count`          | `region`   |

In addition to these metrics, there are a number of new metrics provided by our OpenMetrics endpoint.

:::info

See the [metrics reference](/cloud/metrics/openmetrics/metrics-reference) for an up-to-date list of all available metrics and their full descriptions.

:::

### Managing High-Cardinality

The new endpoint provides access to high-cardinality labels that can significantly increase your metric volume:

#### High-Cardinality Labels

- `temporal_task_queue`
- `temporal_workflow_type`

#### Best Practices

##### Namespace/Metric filtering

Namespace filtering can be used to ensure that metrics are scraped for relevant Namespaces, which reduces cardinality.

```
https://metrics.temporal.io/v1/metrics?namespaces=production-*
```

This can be taken further by only scraping relevant metrics for a given namespace which ensures that any new high
cardinality metrics won’t be an issue for your observability system.

```
https://metrics.temporal.io/v1/metrics?metrics=temporal_cloud_v1_workflow_success_count?namespaces=production-*
```

##### Relabeling

If the above doesn’t work, consider dropping problematic labels post-scrape but pre-ingestion into your observability
system.

For example, in Prometheus this can be done via
[relabeling rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).

```yaml
metric_relabel_configs:
- source_labels: [__name__]
  regex: 'temporal_cloud_v1_poll_success_count'
  action: labeldrop
  regex: 'temporal_task_queue'
```

Or you can even relabel certain label values in order to keep significant ones. For example, it’s possible to rename
less important task queues to “unknown” while retaining important ones.

```yaml
metric_relabel_configs:
  - source_labels: [temporal_task_queue]
    regex: '(critical-queue|payment-queue)'
    target_label: __tmp_keep_original
    replacement: 'true'
  # For anything without the keep flag, replace with "unknown"
  - source_labels: [__tmp_keep_original]
    regex: '' # empty/missing value
    target_label: temporal_task_queue
    replacement: 'unknown'
  # Clean up the temporary label
  - regex: '__tmp_keep_original'
    action: labeldrop
```

## Limits

See [API limits](/cloud/metrics/openmetrics/api-reference#api-limits) for details.

## FAQ

### Q: Will metrics match between promQL and OpenMetrics endpoints?

No. The metrics will be approximately the same but due to aggregation differences and windowing, values likely won't
match exactly between the two endpoints. Some metrics may be consistently different such as
`temporal_cloud_v1_total_action_count` which includes History Export actions in the OpenMetrics endpoint. In the case of
consistent differences the OpenMetrics endpoint is considered to be more accurate.

### Q: Can I still query metrics directly (e.g. with a Grafana dashboard)?

Currently, the OpenMetrics endpoint requires an observability platform to collect and query metrics. Direct querying via
API to return a time series of data is not supported. Supporting this type of query pattern is a future roadmap item.

### Q: What happens to my existing dashboards and alerts?

During the transition period, both endpoints remain active.

### Q: Will historical data be preserved?

Historical data from the query endpoint will remain in your observability platform. To maintain continuity:

- Combine old (`v0`) and new (`v1`) metrics in your queries during transition
- Consider using the PromQL `or` operator: `metric_v1 or metric_v0`

### Q: Are there limits to how frequently I can scrape or how much data will be returned?

The limits are documented [here](/cloud/metrics/openmetrics/api-reference#api-limits).

### Q: Why are some metrics missing from my scrapes? I don’t see all the metrics documented.

The OpenMetrics endpoint only returns metrics that were generated during the one-minute aggregation window. This is
different from the query endpoint which might return zeros.

**What this means:**

- If no workflows failed in the last minute, `temporal_cloud_v1_workflow_failed_count` won't appear in that scrape.
- If a specific task queue had no activity, its metrics will be absent.
- The set of metrics returned varies between scrapes based on system activity.

**This is normal behavior.** Unlike some metrics systems that populate zeros, the OpenMetrics endpoint follows a sparse
reporting pattern \- metrics only appear when there's actual data to report.

**How to handle this in queries:**

```
(temporal_cloud_v1_workflow_failed_count{namespace="production"} or vector(0))
```

This ensures your dashboards and alerts work correctly even when metrics are temporarily absent due to no activity.

---

## Prometheus Grafana setup

**How to set up Grafana with Temporal Cloud observability to view metrics.**

Temporal Cloud and SDKs generate metrics for monitoring performance and troubleshooting errors.

Temporal Cloud emits metrics through a [Prometheus HTTP API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/), which can be directly used as a Prometheus data source in Grafana or to query and export Cloud metrics to any observability platform.

The open-source SDKs require you to set up a Prometheus scrape endpoint for Prometheus to collect and aggregate the Worker and Client metrics.

This section describes how to set up your Temporal Cloud and SDK metrics and use them as data sources in Grafana.

The process for setting up observability includes the following steps:

1. Create or get your Prometheus endpoint for Temporal Cloud metrics and enable SDK metrics.
   - For Temporal Cloud, [generate a Prometheus HTTP API endpoint](/cloud/metrics/general-setup) on Temporal Cloud using valid certificates.
   - For SDKs, [expose a metrics endpoint](#sdk-metrics-setup) where Prometheus can scrape SDK metrics and [run Prometheus](#prometheus-configuration) on your host. The examples in this article describe running Prometheus on your local machine where you run your application code.
2. Run Grafana and [set up data sources for Temporal Cloud and SDK metrics](#grafana-data-sources-configuration) in Grafana. The examples in this article describe running Grafana on your local host where you run your application code.
3. [Create dashboards](#grafana-dashboards-setup) in Grafana to view Temporal Cloud metrics and SDK metrics. Temporal provides [sample community-driven Grafana dashboards](https://github.com/temporalio/dashboards) for Cloud and SDK metrics that you can use and customize according to your requirements.

If you're following through with the examples provided here, ensure that you have the following:

- Root CA certificates and end-entity certificates. See [Certificate requirements](/cloud/certificates#certificate-requirements) for details.
- Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud. See Connect to a Temporal Service for details.

  - [Go](/develop/go/temporal-client#connect-to-temporal-cloud)
  - [Java](/develop/java/temporal-client#connect-to-temporal-cloud)
  - [PHP](/develop/php/temporal-client#connect-to-a-dev-cluster)
  - [Python](/develop/python/temporal-client#connect-to-temporal-cloud)
  - [TypeScript](/develop/typescript/core-application#connect-to-temporal-cloud)
  - [.NET](/develop/dotnet/temporal-client#connect-to-temporal-cloud)

- Prometheus and Grafana installed.

## Temporal Cloud metrics setup

Before you set up your Temporal Cloud metrics, ensure that you have the following:

- Account Owner or Global Admin [role privileges](/cloud/users#account-level-roles) for the Temporal Cloud account.
- [CA certificate and key](/cloud/certificates) for the Observability integration.
  You will need the certificate to set up the Observability endpoint in Temporal Cloud.

The following steps describe how to set up Observability on Temporal Cloud to generate an endpoint:

1. Log in to Temporal Cloud UI with an Account Owner or Global Admin [role](/cloud/users#account-level-roles).
2. Go to **Settings** and select **Integrations**.
3. Select **Configure Observability** (if you're setting it up for the first time) or click **Edit** in the Observability section (if it was already configured before).
4. Add your root CA certificate (.pem) and save it.
   Note that if an observability endpoint is already set up, you can append your root CA certificate here to use the generated observability endpoint with your instance of Grafana.
5. To test your endpoint, run the following command on your host:
   ```
   curl -v --cert <path to your client-cert.pem> --key <path to your client-cert.key> "<your generated Temporal Cloud prometheus_endpoint>/api/v1/query?query=temporal_cloud_v0_state_transition_count"
   ```
   If you have Workflows running on a Namespace in your Temporal Cloud instance, you should see some data as a result of running this command.
6. Copy the HTTP API endpoint that is generated (it is shown in the UI).

This endpoint should be configured as a data source for Temporal Cloud metrics in Grafana.
See [Data sources configuration for Temporal Cloud and SDK metrics in Grafana](#grafana-data-sources-configuration) for details.

## SDK metrics setup

SDK metrics are emitted by SDK Clients used to start your Workers and to start, signal, or query your Workflow Executions.
You must configure a Prometheus scrape endpoint for Prometheus to collect and aggregate your SDK metrics.
Each language development guide has details on how to set this up.

- [Go SDK](/develop/go/observability#metrics)
- [Java SDK](/develop/java/observability#metrics)
- [TypeScript SDK](/develop/typescript/observability#metrics)
- [Python](/develop/python/observability#metrics)
- [.NET](/develop/dotnet/observability#metrics)

The following example uses the Java SDK to set the Prometheus registry and Micrometer stats reporter, set the scope, and expose an endpoint from which Prometheus can scrape the SDK metrics.

```java
//You need the following packages to set up metrics in Java.
//See the Developer's guide for packages required for other SDKs.

//…

//…
   {
     // See the Micrometer documentation for configuration details on other supported monitoring systems.
     // Set up the Prometheus registry.
     PrometheusMeterRegistry yourRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);

       public static Scope yourScope(){
     //Set up a scope, report every 10 seconds
       Scope yourScope = new RootScopeBuilder()
               .tags(ImmutableMap.of(
                       "customtag1",
                       "customvalue1",
                       "customtag2",
                       "customvalue2"))
               .reporter(new MicrometerClientStatsReporter(yourRegistry))
               .reportEvery(Duration.ofSeconds(10));

     //Start Prometheus scrape endpoint at port 8077 on your local host
     HttpServer scrapeEndpoint = startPrometheusScrapeEndpoint(yourRegistry, 8077);
     return yourScope;
   }

   /**
    * Starts HttpServer to expose a scrape endpoint. See
    * https://micrometer.io/docs/registry/prometheus for more info.
    */

   public static HttpServer startPrometheusScrapeEndpoint(
           PrometheusMeterRegistry yourRegistry, int port) {
       try {
           HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
           server.createContext(
                   "/metrics",
                   httpExchange -> {
                       String response = registry.scrape();
                       httpExchange.sendResponseHeaders(200, response.getBytes(UTF_8).length);
                       try (OutputStream os = httpExchange.getResponseBody()) {
                           os.write(response.getBytes(UTF_8));
                       }
                   });
           server.start();
           return server;
       } catch (IOException e) {
           throw new RuntimeException(e);
       }
   }
}

//…

// With your scrape endpoint configured, set the metrics scope in your Workflow service stub and
// use it to create a Client to start your Workers and Workflow Executions.

//…
{
    //Create Workflow service stubs to connect to the Frontend Service.
    WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(
               WorkflowServiceStubsOptions.newBuilder()
                      .setMetricsScope(yourScope()) //set the metrics scope for the WorkflowServiceStubs
                      .build());

   //Create a Workflow service client, which can be used to start, signal, and query Workflow Executions.
   WorkflowClient yourClient = WorkflowClient.newInstance(service,
          WorkflowClientOptions.newBuilder().build());
}

//…
```

To check whether your scrape endpoints are emitting metrics, run your code and go to [http://localhost:8077/metrics](http://localhost:8077/metrics) to verify that you see the SDK metrics.

You can set up separate scrape endpoints in your Clients that you use to start your Workers and Workflow Executions.

For more examples on setting metrics endpoints in other SDKs, see the metrics samples:

- [Java SDK Samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics)
- [Go SDK Samples](https://github.com/temporalio/samples-go/tree/main/metrics)

## SDK metrics Prometheus Configuration {#prometheus-configuration}

**How to configure Prometheus to ingest Temporal SDK metrics.**

For Temporal SDKs, you must have Prometheus running and configured to listen on the scrape endpoints exposed in your application code.

For this example, you can run Prometheus locally or as a Docker container.
In either case, ensure that you set the listen targets to the ports where you expose your scrape endpoints.
When you run Prometheus locally, set your target address to port 8077 in your Prometheus configuration YAML file. (We set the scrape endopint to port 8077 in the [SDK metrics setup](#sdk-metrics-setup) example.)

Example:

```yaml
global:
  scrape_interval: 10s # Set the scrape interval to every 10 seconds. Default is every 1 minute.
#...