```
For more information on Temporal CLI environment variables, see [Environment variables](/cli#environment-variables).
---
## Get started with Temporal Cloud
Getting started with Temporal Cloud involves a few key steps:
1. [Sign up for Temporal Cloud](#sign-up-for-temporal-cloud)
1. [Create a Namespace](#create-a-namespace)
1. [Set up your Clients and Workers](#set-up-your-clients-and-workers)
1. [Run your first Workflow](#run-your-first-workflow)
1. [Invite your team](#invite-your-team)
## Sign up for Temporal Cloud
To create a Temporal Cloud account, you can:
- Sign up [directly](https://temporal.io/get-cloud); or
- Subscribe at the AWS Marketplace for
[Temporal Cloud Pay-As-You-Go](https://aws.amazon.com/marketplace/pp/prodview-xx2x66m6fp2lo). Signing up through the
AWS Marketplace is similar to signing up directly on the Temporal Cloud site, but billing goes through your AWS
account.
- To purchase Temporal Cloud on the Google Cloud Marketplace, please contact our team at sales@temporal.io.
For information about Temporal Cloud Pricing, see our [Pricing Page](/cloud/pricing).
## Create a Namespace
See [Managing Namespaces](/cloud/namespaces#create-a-namespace) to create your first namespace.
Temporal Cloud supports either [API key](/cloud/api-keys) or [mTLS](/cloud/certificates) authentication for each
namespace. If you're not sure which to use, we recommend using [API keys](/cloud/api-keys) because they're easier to
manage and rotate for most teams. If your organization already has private key infrastructure (PKI) and is familiar with
cert management, then [mTLS](/cloud/certificates) is an excellent choice.
## Set up your Clients and Workers
See our guides for connecting each SDK to your Temporal Cloud Namespace:
- [Connect to Temporal Cloud in Go](/develop/go/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Java](/develop/java/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Python](/develop/python/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in TypeScript](/develop/typescript/core-application#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in .NET](/develop/dotnet/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in PHP](/develop/php/temporal-client#connect-to-temporal-cloud)
- [Connect to Temporal Cloud in Ruby](/develop/ruby/temporal-client#connect-to-temporal-cloud)
## Run your first Workflow
See our guides for starting a workflow using each SDK:
- [Start a workflow in Go](/develop/go/temporal-client#start-workflow-execution)
- [Start a workflow in Java](/develop/java/temporal-client#start-workflow-execution)
- [Start a workflow in Python](/develop/python/temporal-client#start-workflow-execution)
- [Start a workflow in TypeScript](/develop/typescript/core-application#start-workflow-execution)
- [Start a workflow in .NET](/develop/dotnet/temporal-client#start-workflow)
- [Start a workflow in PHP](/develop/php/temporal-client#start-workflow-execution)
- [Start a workflow in Ruby](/develop/ruby/temporal-client#start-workflow)
## Invite your team
See [Managing users](/cloud/users) to add other users and assign them roles.
You can also use [Service Accounts](/cloud/service-accounts) to represent machine identities.
Since you created the account when you signed up, your email address is the first
[Account Owner](/cloud/users#account-level-roles) for your account.
---
## Namespaces
:::info Temporal Cloud
This page covers namespace operations in **Temporal Cloud**.
For core namespace concepts, see [Temporal Namespace](/namespaces).
For open source Temporal, see [Managing Namespaces](/self-hosted-guide/namespaces).
:::
A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique
identifiers, and gRPC endpoints in Temporal Cloud.
- [Create a Namespace](#create-a-namespace)
- [Access a Namespace](#access-namespaces)
- [Manage Namespaces](#manage-namespaces)
- [Delete a Namespace](#delete-a-namespace)
- [Tag a Namespace](#tag-a-namespace)
## What is a Cloud Namespace Name? {#temporal-cloud-namespace-name}
A Cloud Namespace Name is a customer-supplied name for a [Namespace](/namespaces) in Temporal Cloud. Each Namespace
Name, such as `accounting-production`, is unique within the scope of a customer's account. It cannot be changed after
the Namespace is provisioned.
Each Namespace Name must conform to the following rules:
- A Namespace Name must contain at least 2 characters and no more than 39 characters.
- A Namespace Name must begin with a letter, end with a letter or number, and contain only letters, numbers, and the
hyphen (-) character.
- Each hyphen (-) character must be immediately preceded _and_ followed by a letter or number; consecutive hyphens are
not permitted.
- All letters in a Namespace Name must be lowercase.
## What is a Temporal Cloud Account ID? {#temporal-cloud-account-id}
A Temporal Cloud Account ID is a unique customer identifier assigned by Temporal Technologies. Each Id is a short string
of numbers and letters like `f45a2`, at least five characters long. This account identifier is retained throughout the
time each customer uses Temporal Cloud.
At times you may need to know your customer Account ID. Accessing the account's Namespaces provides an easy way to
capture this information. Each Temporal Namespace use an Account ID suffix. This is the alphanumeric character string
found after the period in any Temporal Cloud Namespace name.
You can retrieve an Account ID from the [Temporal Cloud](https://cloud.temporal.io) Web UI or by using the `tcld`
utility at a command line interface (CLI). Follow these steps.
Follow these steps to retrieve your Account ID:
1. Log into Temporal Cloud.
1. Select your account avatar at the top right of the page.
A profile dropdown menu appears.
1. Copy the Cloud Account ID from the menu.
In this example, the Account ID is `123de`.
1. Use the `tcld` utility to log into an account.
```
tcld login
```
The `tcld` output presents a URL with an activation code at the end. Take note of this code. The utility blocks until the login/activation process completes.
```
Login via this url: https://login.tmprl.cloud/activate?user_code=KTGC-ZPWQ
```
A Web page automatically opens for authentication in your default browser.
1. Visit the browser.
Ensure the user code shown by the CLI utility matches the code shown in the Web browser.
Then, click Confirm in the browser to continue.
After confirmation, Web feedback lets you know that the CLI "device" is now connected.
1. Return to the command line.
Issue the following command.
```
tcld namespace list
```
The CLI tool returns a short JSON packet with your namespace information.
This is the same list found in the Temporal Cloud Web UI Namespaces list.
Like the browser version, each Namespace uses an Account ID suffix.
```
{
"namespaces": [
"your-namespace.123de",
"another-namespace.123de"
],
"nextPageToken": ""
}
```
Each Namespace automatically appends an Account ID suffix to its customer-supplied identifier.
This five-character-or-longer string appears after the name, separated by a period.
In this Namespace listing sample, the Account ID is 123de.
## What is a Cloud Namespace Id? {#temporal-cloud-namespace-id}
A Cloud Namespace Id is a globally unique identifier for a [Namespace](/namespaces) in Temporal Cloud. A Namespace Id is
formed by concatenating the following:
1. A [Namespace Name](#temporal-cloud-namespace-name)
1. A period (.)
1. The [Account ID](#temporal-cloud-account-id) to which the Namespace belongs
For example, for the Account ID `123de` and Namespace Name `accounting-production`, the Namespace Id is
`accounting-production.123de`.
## What is a Cloud gRPC Endpoint? {#temporal-cloud-grpc-endpoint}
Temporal Clients communicate between application code and a Temporal Server by sending and receiving messages via the
gRPC protocol. gRPC is a Remote Procedure Call framework featuring low latency and high performance. gRPC provides
Temporal with an efficient, language-agnostic communication framework.
Every Temporal Namespace uses a gRPC endpoint for communication. When migrating to Temporal Cloud, you'll need to switch
the gRPC endpoint in your code from your current hosting, whether self-hosted or locally-hosted, to Temporal Cloud.
A gRPC endpoint appears on the detail page for each Cloud Namespace. Follow these steps to find it:
1. Log into your account on [cloud.temporal.io](https://cloud.temporal.io/namespaces).
2. Navigate to the Namespace list page from the left-side vertical navigation.
3. Tap or click on the Namespace Name to select and open the page for the Namespace whose endpoint you want to retrieve.
4. On the Namespace detail page, click on the "Connect" button in the top right corner of the page.
5. Click the copy icon next to the gRPC address to copy it to your clipboard.
See [How to access a Namespace in Temporal Cloud](/cloud/namespaces/#access-namespaces) for more information on
different gRPC endpoint types and how to access them.
## How to create a Namespace in Temporal Cloud {#create-a-namespace}
:::info
The user who creates a [Namespace](/namespaces) is automatically granted
[Namespace Admin](/cloud/users#namespace-level-permissions) permission for that Namespace.
To create a Namespace, a user must have the Developer, Account Owner, or Global Admin account-level
[Role](/cloud/users#account-level-roles).
:::
:::tip
By default, each account is allocated with a limit of ten Namespaces. As you start using Namespaces by scheduling
Workflows, Temporal Cloud automatically raises your allowance. This automatic adjustment happens whenever all your
Namespaces are in use, up to a maximum of 100 Namespaces. You can request further increases beyond the 100 Namespace
limit by opening a [support ticket](/cloud/support#support-ticket).
:::
### Information needed to create a Namespace
To create a Namespace in Temporal Cloud, gather the following information:
- [Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name), region, and Cloud Provider
- [Retention Period](/temporal-service/temporal-server#retention-period) for the
[Event History](/workflow-execution/event#event-history) of closed [Workflow Executions](/workflow-execution).
- [CA certificate](/cloud/certificates#certificate-requirements) for the Namespace, if you are using mTLS
authentication.
- [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) to
show decoded payloads to users in the Event History for Workflow Executions in the Namespace. For details, see
[Securing your data](/production-deployment/data-encryption).
- [Permissions](/cloud/users#namespace-level-permissions) for each user.
### Create a Namespace using Temporal Cloud UI
1. Gather the information listed earlier in
[Information needed to create a Namespace](#information-needed-to-create-a-namespace).
1. Go to the Temporal Cloud UI and log in.
1. On the left side of the window, click **Namespaces**.
1. On the **Namespaces** page, click **Create Namespace** in the upper-right portion of the window.
1. On the **Create Namespace** page in **Name**, enter the Namespace Name.
1. In **Cloud Provider**, select the cloud provider in which to host this Namespace.
1. In **Region**, select the region in which to host this Namespace.
1. In **Retention Period**, specify a value from 1 to 90 days. When choosing this value, consider your needs for Event
History versus the cost of maintaining that Event History. Typically, a development Namespace has a short retention
period and a production Namespace has a longer retention period. (If you need to change this value later, contact
[Temporal Support](/cloud/support#support-ticket).)
1. Select your authentication method: [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates).
1. If using mTLS authentication, paste the CA certificate for this Namespace.
1. Optional: In **Codec Server**, enter the HTTPS URL (including the port number) of your Codec Server endpoint. You may
also enable "Pass the user access token with your endpoint" and "Include cross-origin credentials." For details, see
[Hosting your Codec Server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli).
1. Click **Create Namespace**.
See the [`tcld` namespace create](/cloud/tcld/namespace/#create) command reference for details.
## What are some Namespace best practices? {#best-practices}
This section provides general guidance for organizing [Namespaces](/namespaces) across use cases, services,
applications, or domains. Temporal Cloud provides Namespace–as-a-service, so the Namespace is the endpoint. Customers
should consider not only a Namespace naming convention but also how to group or isolate workloads using the Namespace as
a boundary.
Each team can have their own Namespace for improved modularity, security, debugging, and fault isolation. Namespaces
contain the blast radius of misbehaving Workers that may exhaust rate limits. Sensitive Workflow state (PCI data) can be
secured with per-Namespace permissions and encrypted with a separate encryption key.
Temporal Applications in different Namespaces may be connected with [Nexus](/cloud/nexus) by exposing a clean service
contract for others to use with built-in [Nexus access controls](/cloud/nexus/security). Nexus supports cross-team,
cross-domain, multi-region, and multi-cloud use cases.
### Constraints and limitations
Before considering an appropriate Namespace configuration, you should be aware of the following constraints:
- By default, each account is allocated with a limit of ten Namespaces. As you create and use your Namespaces, for
example by scheduling Workflows, Temporal Cloud automatically raises your limit. Our service identifies your usage
patterns. It responds by slowly increasing your allowance, up to 100 Namespaces. You can request further increases
beyond the 100 Namespace limit by opening a [support ticket](/cloud/support#support-ticket).
- Each Namespace has a rate limit, which is measured in Actions per second (APS). A namespace may be throttled when its
throughput becomes too high. Throttling means limiting the rate at which actions are performed to prevent overloading
the system. A Namespace's default limit is set at 400 APS and automatically adjusts based on recent usage (over the
prior 7 days). Your APS limit will never fall below this default value.
- Each Namespace has a default service-level agreement (SLA) of 99.9% uptime.
- You can opt-in to using [High Availability features](https://docs.temporal.io/cloud/high-availability) with a 99.99%
contractual SLA.
- A Namespace is a security isolation boundary. Access to Temporal by [Worker Processes](/workers#worker-process) is
permitted at the Namespace level. Isolating applications or environments (development, test, staging, production)
should take this into consideration.
- A Namespace is provisioned with an endpoint for executing your Workflows. Accessing a Namespace from a Temporal Client
requires [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates) authentication.
- [Workflow Id](/workflow-execution/workflowid-runid#workflow-id)uniqueness is per Namespace.
- [Task Queue](/task-queue) names are unique per Namespace.
- Closed Workflow retention is per Namespace.
- RBAC [permissions](/cloud/users#namespace-level-permissions) are implemented at the Namespace level.
### General guidance
Namespace configuration requires some consideration. Following are some general guidelines to consider.
- Namespaces are usually defined per use case. A use case can encompass a broad range of Workflow types and a nearly
unlimited scale of concurrent [Workflow Executions](/workflow-execution).
- Namespaces can be split along additional boundaries such as service, application, domain or even sub-domain.
- Environments such as production and development usually have requirements for isolation. We recommend that each
environment has its own Namespace.
- Namespaces should be used to reduce the "blast radius" for mission-critical applications.
- Workflows that need to communicate with each other should (for now) be in the same Namespace.
- If you need to share Namespaces across team or domain boundaries, be sure to ensure the uniqueness of Workflow Ids.
### Examples
Following are some ideas about how to organize Namespaces.
#### Example 1: Namespace per use case and environment
We recommend using one Namespace for each use case and environment combination for simple configurations in which
multiple services and team or domain boundaries don't exist.
Sample naming convention:
<use-case>_<environment>
#### Example 2: Namespace per use case, service, and environment
We recommend using one Namespace for each use case, service, and environment combination when multiple services that are
part of same use case communicate externally to Temporal via API (HTTP/gRPC).
Sample naming convention:
<use-case>_<service>_<environment>
#### Example 3: Namespace per use case, domain, and environment
We recommend using one namespace per use case, domain, and environment combination when multiple services that are part
of the same use case need to communicate with each another via [Signals](/sending-messages#sending-signals) or by
starting [Child Workflows](/child-workflows). In this case, though, you must be mindful about Workflow Id uniqueness by
prefixing each Workflow Id with a service-specific string. The name of each Task Queue must also be unique. If multiple
teams are involved, the domain could also represent a team boundary.
Sample naming convention:
<use-case>_<domain>_<environment>
Sample workflowId convention:
<service-string>_<workflow-id>
## How to access a Namespace in Temporal Cloud {#access-namespaces}
{/* How to access a Namespace in Temporal Cloud */}
Temporal Cloud normally supports authentication to Namespaces using [API keys](/cloud/api-keys) _or_
[mTLS](/cloud/certificates). If you need to migrate from one authentication method to another, or you require both API
key and mTLS authentication to be enabled on your Namespace, please contact
[Support](https://docs.temporal.io/cloud/support#support-ticket).
:::info
Namespace authentication requiring both API key and mTLS is in
[pre-release](/evaluate/development-production-features/release-stages), and doesn't support
[High Availability features](/cloud/high-availability).
:::
See the documentation for [API keys](/cloud/api-keys) and [mTLS certificates](/cloud/certificates) for more information
on how to create and manage your credentials.
Programmatically accessing your Namespace requires specific endpoints based on your authentication method. There are two
types of gRPC endpoints for accessing a Namespace in Temporal Cloud:
- A namespace endpoint (`..tmprl.cloud:7233`)
- A regional endpoint (`..api.temporal.io:7233`).
Which one to use depends on your authentication method and whether your Namespace has
[High Availability features](/cloud/high-availability) enabled, as shown in the table below.
| | Not High Availability | High Availability |
| ---------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| mTLS Authentication | Namespace | Namespace |
| API Key Authentication | Regional | Both work, but we reommend using the Namespace endpoint because it reduces the unavailability window during a failover event |
:::info
When switching on or off High Availability features for a Namespace, you may need to update the gRPC endpoint used by
your clients and Workers, because the Namespace endpoint changes based on whether High Availability features are
enabled. See [Disable High Availability](/cloud/high-availability/enable#disable) for more information.
:::
For information on how to connect to Clients using a specific authentication method see the following documentation.
- To use API keys to connect with the [Temporal CLI](/cli), [Client SDK](/develop), [tcld](/cloud/tcld),
[Cloud Ops API](/ops), and [Terraform](/cloud/terraform-provider), see
[Use API keys to authenticate](/cloud/api-keys#using-apikeys).
- To use mTLS to connect with the [Temporal CLI](/cli) and [Client SDK](/develop), see
[Configure Clients to use Client certificates](/cloud/certificates#configure-clients-to-use-client-certificates).
For accessing the Temporal Web UI, use the HTTPS endpoint in the form:
`https://cloud.temporal.io/namespaces/.`. For example:
`https://cloud.temporal.io/namespaces/accounting-production.f45a2`.
To ensure the security of your data, all traffic to and from your Namespace is encrypted. However, for enhanced
protection, you have additional options:
- (Recommended) Set up private connectivity by [creating a ticket for Temporal Support](/cloud/support#support-ticket).
- Set up your allow list for outgoing network requests from your Clients and Workers with the IP address ranges of the
Cloud Provider region in which your Namespace is located:
- [AWS IP address ranges](https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html)
- [GCP IP address ranges](https://cloud.google.com/compute/docs/faq#find_ip_range)
## How to manage Namespaces in Temporal Cloud {#manage-namespaces}
{/* How to manage Namespaces in Temporal Cloud using Temporal Cloud UI */}
### Manage Namespaces in Temporal Cloud using Temporal Cloud UI
To list Namespaces:
- On the left side of the window, select **Namespaces**.
To edit a Namespace (including custom Search Attributes, certificates, certificate filters, Codec Server endpoint,
permissions, and users), find the Namespace and do either of the following:
- On the right end of the Namespace row, select the three vertical dots (⋮). Click **Edit**.
- Select the Namespace name. In the top-right portion of the page, select **Edit**.
On the **Edit** page, you can do the following:
- Add a [custom Search Attribute](/search-attribute#custom-search-attribute).
- [Manage CA certificates](/cloud/certificates).
- [Manage certificate filters](/cloud/certificates#manage-certificate-filters-using-temporal-cloud-ui).
- Set
[Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli)
for all users on the Namespace. Each user on the Namespace has the option to
[override this setting](/production-deployment/data-encryption#web-ui) in their browser.
- Manage [Namespace-level permissions](/cloud/users#namespace-level-permissions).
- Add users.
To add a user to a Namespace, scroll to the bottom of the page and select **Add User**.
After you make changes, select **Save** in the top-right or bottom-left portion of the page.
{/* How to manage Namespaces in Temporal Cloud using tcld */}
### Manage Namespaces in Temporal Cloud using tcld
To list Namespaces and get information about them, use the following [tcld](/cloud/tcld/) commands:
- [tcld namespace list](/cloud/tcld/namespace/#list)
- [tcld namespace get](/cloud/tcld/namespace/#get)
To manage certificates, use the [tcld namespace accepted-client-ca](/cloud/tcld/namespace/#accepted-client-ca) commands.
For more information, see [How to manage certificates in Temporal Cloud](/cloud/certificates).
To manage certificate filters, use the [tcld namespace certificate-filters](/cloud/tcld/namespace/#certificate-filters)
commands. For more information, see
[How to manage certificate filters in Temporal Cloud](/cloud/certificates#manage-certificate-filters).
## How to delete a Namespace in Temporal Cloud {#delete-a-namespace}
:::info
To delete a Namespace, a user must have Namespace Admin [permission](/cloud/users#namespace-level-permissions) for that
Namespace.
:::
### Delete a Namespace using Temporal Cloud UI
1. Go to the Temporal Cloud UI and log in.
1. On the left side of the window, select **Namespaces**.
1. On the **Namespaces** page, select a Namespace Name.
1. On the Namespace page, select **Edit** in the upper-right portion of the window.
1. On the **Edit** Namespace page, select **Delete Namespace** in the upper-right portion of the window.
1. In the **Delete Namespace** dialog, type `DELETE` to confirm the deletion of that Namespace.
1. Select **Delete**.
After deleting a Temporal Cloud Namespace, the Temporal Service immediately removes the Namespace's Workflow Executions
and Task Queues. Make sure all Workflows have been completed, canceled, or terminated before removing a Namespace. The
Namespace removal is permanent.
Closed Workflow Histories remain in Temporal storage until the user-defined retention period expires. This period
reflects the policy in effect when the Workflow Execution was closed.
For further questions or concerns, contact [Support](https://docs.temporal.io/cloud/support#support-ticket).
### Delete a Namespace using tcld
See the [tcld namespace delete](/cloud/tcld/namespace/#delete) command reference for details.
### Namespace deletion protection {#delete-protection}
To prevent accidental Namespace deletion, Temporal Cloud provides a protection feature. When you enable Deletion
Protection for your production environment Namespace, you ensure that critical data won't be deleted unintentionally.
Follow these steps:
- Visit the [Namespaces page](https://cloud.temporal.io/namespaces) on Temporal Cloud.
- Open your Namespace details page.
- Select the Edit button.
- Scroll down to Security and click the disclosure button (downward-facing caret).
- Enable **Deletion Protection**
To enable or disable this feature using [`tcld`](/cloud/tcld), use the following command. Set the value to `true` to
enable or `false` to disable:
```
tcld namespace lifecycle set \
--namespace \
--enable-delete-protection
```
## How to tag a Namespace in Temporal Cloud {#tag-a-namespace}
Tags are key-value metadata pairs that can be attached to namespaces in Temporal Cloud to help operators organize,
track, and manage namespaces more easily.
### Tag Structure and Limits
- Each namespace can have a maximum of 10 tags
- Each key must be unique for a given namespace (e.g., a namespace cannot have both `team:foo` and `team:bar` tags)
- Keys and values must be 1-63 characters in length
- Allowed characters: lowercase letters (`a-z`), numbers (`0-9`), periods (`.`), underscores (`_`), hyphens (`-`), and
at signs (`@`)
- Tags are not a secure storage mechanism and should not store PII or PHI
- Tags will not change the behavior of the tagged resource
- There is a soft limit of 1000 unique tag keys per account
### Permissions
- Only [**Account Admins** and **Account Owners**](/cloud/users#account-level-roles) can create and edit tags
- All users with access to a namespace can view its tags
### tcld
See the [tcld namespace tags](/cloud/tcld/namespace/#tags) command reference for details.
### Terraform
See the
[Terraform provider](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/docs/resources/namespace_tags.md)
for details.
### Web UI
Tags can be viewed and managed through the Temporal Cloud web interface. When viewing a namespace, you'll see tags
displayed and can add, edit, or remove them if you have the appropriate permissions.
---
## Manage service accounts
Temporal Cloud provides Account Owner and Global Admin [roles](/cloud/users#account-level-roles) with the option to create machine identities named Service Accounts.
Service Accounts are a type of identity in Temporal Cloud.
Temporal Cloud supports User identities as a representation of a human user who uses Temporal Cloud.
Service Accounts afford Temporal Cloud Account Owner and Global Admin [roles](/cloud/users#account-level-roles) the ability to create an identity for machine authentication, an identity not associated with a human user.
With the addition of Service Accounts, Temporal Cloud now supports 2 identity types:
- Users (tied to a human, identified by email address or ID)
- Service Accounts (not tied to a human, email address optional, identified by name or ID)
Service Accounts use API Keys as the authentication mechanism to connect to Temporal Cloud.
You should use Service Accounts to represent a non-human identity when authenticating to Temporal Cloud for operations automation or the Temporal SDKs and the Temporal CLI for Workflow Execution and management.
:::tip
Namespace Admins can now manage and create [Namespace-scoped Service Accounts](/cloud/service-accounts#scoped), regardless of their Account Role.
:::
## Manage Service Accounts
Account Owner and Global Admin [roles](/cloud/users#account-level-roles) can manage Service Accounts by creating, viewing, updating, deleting Service Accounts using the following tools:
- Temporal Cloud UI
- Temporal Cloud CLI (tcld)
- Use `tcld service-account --help` for a list of all service-account commands
Account Owner and Global Admin [roles](/cloud/users#account-level-roles) also have the ability to manage API Keys for Service Accounts.
### Prerequisites
- A Cloud user account with Account Owner or Global Admin [role](/cloud/users#account-level-roles) permissions
- Access to the Temporal Cloud UI or Temporal Cloud CLI (tcld)
- Enable access to API Keys for your Account
- To manage Service Accounts using the Temporal Cloud CLI (tcld), upgrade to the latest version of tcld (v0.18.0 or higher) using `brew upgrade tcld`.
- If using a version of tcld less than v0.31.0, enable Service Account commands with `tcld feature toggle-service-account`.
### Create a Service Account
Create a Service Account using the Temporal Cloud UI or tcld.
While User identities are invited to Temporal Cloud, Service Accounts are created in Temporal Cloud.
1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities)
2. Click the `Create Service Account` button located near the top of the `Identities` page
3. Provide the following information:
- **Name** (required)
- **Description** (optional)
- **Account Level Role** (required)
- **Namespace Permissions** (optional)
- Use this section of the Create Service Account page to grant the Service Account access to individual Namespaces
4. Click `Create Service Account` at the bottom of the page
- A status message is displayed at the bottom right corner of the screen and on the next screen
- You will be prompted to create an API Key for the Service Account (optional)
5. (Optional) Create API Key
- It is recommended to create an API Key for the Service Account right after you create the Service Account, though you can create/manage API Keys for Service Accounts at any time
- See the API Key [documentation](/cloud/api-keys) for more information on creating and managing API Keys
To create a Service Account using tcld, use the `tcld service-account create` command:
```
tcld service-account create -n "sa_test" -d "this is a test SA" --ar "Read"
```
This example creates a Service Account with the name `"sa_test"`, description `"this is a test SA"`, and a `Read` Account Role.
Creating a Service Account requires the following attributes: `name` and `account-role` (as above).
You can also provide the Namespace Permissions for the Service Account using the `—-np` flag.
Creating a Service Account returns the `ServiceAccountId` which is used to retrieve, update, or delete a Service Account.
### View Service Accounts
View a single or all Service Account(s) using the Temporal Cloud UI or tcld.
Service Accounts are listed on the `Identities` section of the `Settings` page, along with Users.
To locate a Service Account:
1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities)
2. Select the `Service Accounts` filter
To view all Service Accounts in your account using tcld, use the `tcld service-account list` command:
```
tcld service-account list
```
### Delete a Service Account
Delete a Service Account using the Temporal Cloud UI or tcld. When you delete a Service Account, all associated API keys are automatically deleted as well.
Therefore, you don't need to manually remove API keys after deleting a Service Account.
1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities)
2. Find the relevant Service Account
3. Select the vertical ellipsis menu in the Service Account row
4. Select `Delete`
5. Confirm the delete action when prompted
To delete a Service Account using tcld, use the `tcld service-account delete` command:
```
tcld service-account delete --service-account-id "e9d87418221548"
```
Use the tcld Service Account list command to validate the Service Account has been removed from the account.
The Service Account is deleted when it is no longer visible in the output of .
### Update a Service Account {#update}
Update a Service Account's description using the Temporal Cloud UI or tcld.
1. Go to [Settings → Identities](https://cloud.temporal.io/settings/identities)
2. Find the relevant Service Account
3. Select the vertical ellipsis menu in the Service Account row
4. Select `Edit`
5. Make changes to the Service Account
- You can change the Service Account's name, description, Account Level Role, and Namespace Permissions
6. Click the `Save` button located in the bottom left of the screen
- A status message is displayed at the bottom right corner of the screen
Three different commands exist to help users update a Service Account using tcld:
- `tcld service-account update`: to update a Service Account's name or description field
- `tcld service-account set-account-role`: to update a Service Account's Account Role
- `tcld service-account set-namespace-permissions`: to update a Service Account's Namespace Permissions
Example:
```
tcld service-account update --id "2f68507677904e09b9bcdbf93380bb95" -d "new description"
```
## Namespace-scoped Service Accounts {#scoped}
There is a special type of Service Account, called a Namespace-scoped Service Account, which shares the
same functionality as the Service Accounts above, but is limited (or scoped) to a single namespace.
In particular, a Namespace-scoped Service Account must _always_ have:
- A `Read` Account Role
- A single Namespace Permission
Note that a Namespace-scoped Service Account cannot be reassigned to a different Namespace after creation, but its Namespace permission can be modified (e.g. from `Read` to `Write`).
Namespace-scoped Service Accounts are useful in situations when you need to restrict a client's access to a single Namespace.
You can retrieve, update, and delete a Namespace-scoped Service Account using the same process and commands as above, but creation is slightly different.
### Permissions
Unlike regular Service Accounts, which require a Global Admin or Account Owner role, Namespace-scoped Service Accounts can be created and managed by Namespace Admins.
For example, an Account Developer with Namespace Admin for `test_ns` can create a Service Account scoped to `test_ns`.
Global Admins and Account Owners can also create Namespace-scoped Service Accounts, as they implicitly have Namespace Admin rights for all Namespaces.
### Create a Namespace-scoped Service Account
As with regular Service Accounts, Namespace-scoped Service Accounts can be created using Temporal Cloud UI or tcld.
#### Using the Cloud UI {#scoped-ui}
Currently, creating a Namespace-scoped Service Account from the Temporal Cloud UI happens on an individual [Namespace](/cloud/namespaces#manage-namespaces) page.
If the current Namespace has API key authentication enabled, then there will be a `Generate API Key` button as a banner on the top of the Namespace page or in the `Authentication` section.
By clicking on the `Generate API Key` button, a Namespace-scoped Service Account will be automatically created for the given Namespace (if one does not already exist) and an associated API key will be displayed. This key will have the maximum expiration time, which is 2 years.
The resulting Namespace-scoped Service Account will be named `-service-account` and will have an `Admin` Namespace permission by default.
#### Using tcld
To create a Namespace-scoped Service Account with tcld, use the `tcld service-accounted create-scoped` command:
```
tcld service-account created-scoped -n "test-scoped-sa" --np "test-ns=Admin"
```
This example creates a Namespace-scoped Service Account for the Namespace `test-ns`, named `test-scoped-sa`, with `Admin` Namespace Permission.
Note that the Account Role is omitted, since Namespace-scoped Service Accounts always have a `Read` Account Role.
### Lifecycle
When a Namespace is deleted, all associated Namespace-scoped Service Accounts and their associated API keys are automatically deleted as well.
Therefore, you do not need to manually remove Namespace-scoped Service Accounts and their API keys after deleting a Namespace.
---
## Manage user groups
## What are user groups?
User groups can be used to help manage sets of users that should have the same
access. Instead of separately assigning the same role to individual users, a user group can be
created, assigned the desired roles, and then users added to the user group. This
eases the toil of managing individual user permissions and can simplify access management. When
a new role is needed, it can be added to the group once and all users' access will reflect the
new role.
User groups can be assigned both [account-level roles](/cloud/users#account-level-roles) and [namespace-level permissions](/cloud/users#namespace-level-permissions).
One user can be assigned to many groups. In the event that a user's group memberships have multiple roles for the same resource, the user will have an effective role of the most permissive of the permissions. For example if `Group A` grants a read-only role to a namespace, but `Group B` grants a write role to a namespace then a user that belongs to both `Group A` and `Group B` would have the write role to the namespace.
[Service accounts](/cloud/service-accounts) cannot be assigned to user groups.
Only users with the Account Owner or Global Admin account-level [role](/cloud/users#account-level-roles) can manage user groups.
## How SCIM groups work with user groups {#scim-groups}
[SCIM groups](/cloud/scim) work similarly to user groups with respect to role assignment. Unlike a user group, the lifecycle of a SCIM group is fully managed by the SCIM integration which means:
1. SCIM groups cannot be created except through the SCIM integration
1. SCIM groups cannot be deleted except through the SCIM integration
1. SCIM group membership is managed through the SCIM integration
User groups and SCIM groups can be used simultaneously in a single Temporal Cloud account. One user may belong to multiple SCIM groups and to multiple user groups.
Using user group and SCIM groups together can be useful when the groups defined in the identity provider (IDP) don't map cleanly to the access you need to grant in Temporal Cloud. Instead of having to update the IDP (which is often sensitive and time-consuming), you can use Temporal Cloud user groups to manage access.
:::info
All user group administration requires an Account Owner or Global Admin account-level [role](/cloud/users#account-level-roles).
:::
## How to create a user group in your Temporal Cloud account {#create-group}
User group names must be 3-64 characters long and can only contain lowercase letters, numbers, hyphens, and underscores.
1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities)
1. Click the Create Group button
1. Name the group
1. Assign an account-level role to the group (you can assign namespace-level permissions after the group is created)
1. Click Save
See the [`tcld` user-group create](/cloud/tcld/user-group/#create) command reference for details.
See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.
## How to assign roles to a user group {#assign-group-roles}
To edit the account role of a group:
1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities)
1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
1. Click Edit Group
1. Click the Account Role dropdown
1. Select a new account role
1. Click Save
To add namespace permissions to a group:
1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities)
1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
1. Click Edit Group
1. Click Add Namespaces
1. Under Grant Access to a Namespace, search for the namespace you’d like to add permissions for
1. Select the namespace
1. Click the pencil to edit the permissions for the selected namespace
1. Click Save
To edit or remove namespace permissions from a group:
1. Click Edit Group
1. Click the pencil on a permission to edit it, or the trash can to delete it
1. Click Save
See the [`tcld` user-group set-access](/cloud/tcld/user-group/#set-access) command reference for details.
See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.
## How to manage users in a group {#assign-group-members}
To add users to the group:
1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities)
1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
1. Click Edit Group
1. Under Members, search for the user you’d like to add
1. Select the user
1. Click Save
To remove a user from the group:
1. Click Edit Group
1. Under Members, click the X next to the user you’d like to remove
1. Click Save
See the [`tcld` user-group add-users](/cloud/tcld/user-group/#add-users) and the [`tcld` user-group remove-users](/cloud/tcld/user-group/#remove-users) command reference for details.
See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.
## Delete a user group
1. Navigate to the [identities page](https://cloud.temporal.io/settings/identities)
1. Find the group to edit (You can filter the list of identities to only show groups to find the relevant group by clicking the Groups tab on the table)
1. Click the dropdown next to the edit button
1. Click Delete
1. Confirm by clicking Delete
See the [`tcld` user-group delete](/cloud/tcld/user-group/#delete) command reference for details.
See the [Terraform provider documentation](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/group) for details.
---
## User management
:::caution
Access to Temporal Cloud can be authorized through email and password, Google single sign-on, Microsoft single sign-on, or SAML, depending on your setup.
If you are using Google OAuth for single sign-on and an email address is not associated with a Google Account, the user must follow the instructions in the [Use an existing email address](https://support.google.com/accounts/answer/27441?hl=en#existingemail) section of [Create a Google Account](https://support.google.com/accounts/answer/27441).
**Important:** Do _not_ create a Gmail account when creating a Google Account.
If your organization uses Google Workspace or Microsoft Entra ID, and your IT administrator has enabled controls over single sign-on permissions, then you will need to work with your IT administrator to allow logins to Temporal Cloud.
:::
When a user is created in Temporal Cloud, they receive an invitation email with a link.
They must use this link to finalize their setup and access Temporal Cloud.
Accounts with SAML configurations can ignore this email.
However, those using Google/Microsoft SSO or email and password authentication need to accept the invitation link for their initial login to Temporal Cloud.
For future logins, they must use the same authentication method they originally signed up with.
:::info
To invite users, a user must have the Global Admin or Account Owner account-level [role](/cloud/users#account-level-roles).
:::
### Roles and permissions
Each user in Temporal Cloud is assigned a role.
Each user can be assigned permissions for individual Namespaces.
- [Account-level roles](/cloud/users#account-level-roles)
- [Namespace-level permissions](/cloud/users#namespace-level-permissions)
To invite users using the Temporal Cloud UI:
1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, select **Create Users** in the upper-right portion of the window.
1. On the **Create Users** page in the **Email Addresses** box, type or paste one or more email addresses.
1. In **Account-Level Role**, select a [Role](/cloud/users#account-level-roles).
The Role applies to all users whose email addresses appear in **Email Addresses**.
1. If the account has any Namespaces, they are listed under **Grant access to Namespaces**.
To add a permission, select the checkbox next to a Namespace, and then select a [permission](/cloud/users#namespace-level-permissions).
Repeat as needed.
1. When all permissions are assigned, select **Send Invite**.
Temporal sends an email message to each user.
To join Temporal Cloud, a user must select **Accept Invite** in the message.
To invite users using tcld, see the [tcld user invite](/cloud/tcld/user/#invite) command.
Temporal sends an email message to the specified user.
To join Temporal Cloud, the user must select **Accept Invite** in the message.
You can invite users pragmatically using the Cloud Ops API.
1. Create a connection to your Temporal Service using the Cloud Operations API.
2. Use the [CreateUser service](https://github.com/temporalio/api-cloud/blob/main/temporal/api/cloud/cloudservice/v1/service.proto) to create a user.
### Frequently Asked Questions
#### Can the same email be used across different Temporal Cloud accounts?
No — each email address can only be associated with a single Temporal Cloud account.
If you need access to multiple accounts, you’ll need a separate invite for each one using a different email address.
#### Can I use Google or Microsoft SSO after signing up with email and password?
If you originally signed up for Temporal Cloud using an email and password, you won’t be able to log in using Google or Microsoft single sign-on.
If you prefer SSO, ask your Account Owner to delete your current user and send you a new invitation.
During re-invitation, be sure to sign up using your preferred authentication method.
#### How do I complete the `Secure Your Account` step?
If you signed up to Temporal Cloud using an email and password, you're required to set up multi-factor authentication (MFA) for added security.
Currently, only authenticator apps are supported as an additional factor (such as Google Authenticator, Microsoft Authenticator, and Authy).
To proceed:
1. Download a supported authenticator app on your mobile device.
2. Scan the QR code shown on the **Secure Your Account** screen.
3. Enter the verification code from your app to complete MFA setup.
4. Securely store your recovery code.
This code allows you to access your account if you lose access to your authenticator app.
Once MFA is configured, you’ll be able to continue using Temporal Cloud.
#### What if I lose access to my authenticator app?
If you lose access to your authenticator app, you can still log in by clicking **Try another method** on the MFA screen.
From there, you can either:
- Enter your recovery code (provided when you first set up MFA)
- Receive a verification code through email
Once you're logged in, you can reset your authenticator app by navigating to **My Profile** > **Password and Authentication** and then clicking **Authenticator App** > **Remove method**.
#### How do I reset my password?
If you're currently logged in and would like to change your password, click your profile icon at the top right of the Temporal Cloud UI,
navigate to **My Profile** > **Password and Authentication**, and then click **Reset Password**.
If you're not currently logged in, navigate to the login page of the Temporal Cloud UI, enter your email address, click **Continue**, and then select **Forgot password**.
In both cases, you will receive an email with instructions on how to reset your password.
---
## Manage users
- [How to invite users to your Temporal Cloud account](#invite-users)
- [What are the account-level roles?](#account-level-roles)
- [What are the Namespace-level permissions?](#namespace-level-permissions)
- [How to update an account-level Role in Temporal Cloud](#update-roles)
- [How to update Namespace-level permissions in Temporal Cloud](#update-permissions)
- [How to delete a user from your Temporal Cloud account](#delete-users)
- [How to troubleshoot account access issues](#troubleshoot-access)
## How to invite users to your Temporal Cloud account {#invite-users}
## What are the account-level roles for users in Temporal Cloud? {#account-level-roles}
When an Account Owner or Global Admin invites a user to join an account, they select one of the following roles for that user:
- **Global Admin**
- Has full administrative permissions across the account, including users and usage
- Can create and manage [Namespaces](/namespaces) and [Nexus Endpoints](/nexus/endpoints)
- Has Namespace Admin [permissions](#namespace-level-permissions) on all Namespaces in the account.
This permission cannot be revoked
- **Developer**
- Can create Namespaces
- Is granted [Namespace Admin](/cloud/users#namespace-level-permissions) permission for each Namespace they create.
This permission can be revoked
- Can create and manage Nexus Endpoints where they are a [Namespace Admin](/cloud/users#namespace-level-permissions) on the Endpoint's target Namespace
- **Read-Only**
- Can read information
- Can be granted Namespace [permissions](#namespace-level-permissions), for example to read or write Workflow state in a given Namespace
- Can view all Nexus Endpoints in the account, which have separate [runtime access controls](/nexus/security#runtime-access-controls)
In addition, there are two roles that the Global Admin cannot assign:
- **Account Owner**
- Has full administrative permissions across the account, including users, usage and [billing](/cloud/billing-and-cost)
- Can create and manage Namespaces and Nexus Endpoints
- Has Namespace Admin [permissions](#namespace-level-permissions) on all [Namespaces](/namespaces) in the account.
This permission cannot be revoked
- **Finance Admin**
- Has permissions to view [billing](/cloud/billing-and-cost) information and update payment information
- Otherwise, has the same permissions as Account Read-only users
- Can be assigned to Service Accounts by a Global Admin, but otherwise can only be assigned by an Account Owner
:::note Default Role
When the account is created, the initial user who logs in is automatically assigned the Account Owner role.
If your account does not have an Account Owner, please reach out to [Support](https://temporalsupport.zendesk.com/) to assign the appropriate individual to this role.
:::
## Using the Account Owner role
The Account Owner role (i.e., users with the Account Owner system role) holds the highest level of access in the system.
This role configures account-level parameters and manages Temporal billing and payment information.
It allows users to perform all actions within the Temporal Cloud account.
:::tip Best Practices
Temporal strongly recommends the following precautions when assigning the Account Owner role to users:
- Assign the role to at least two users in your organization.
Otherwise, limit the number of users with this role.
- Associate a person’s direct email address to the Account Owner, rather than a shared or generic address, so Temporal Support can contact the right person in urgent situations.
This latter rule is useful for anyone on your team who may need to be contacted urgently, regardless of their Account role.
:::
## What are the Namespace-level permissions for users in Temporal Cloud? {#namespace-level-permissions}
An Account Owner or Global Admin can assign permissions for any [Namespace](/namespaces) in an account.
A Developer can assign permissions for a Namespace they create.
For a Namespace, a user can have one of the following permissions:
- **Namespace Admin:**
- Can [manage the Namespace](/cloud/namespaces#manage-namespaces) including identities and permissions
- Can create, rename, update, and delete [Workflows](/workflows) within the Namespace
- **Write:**
- Can create, rename, update, and delete [Workflows](/workflows) within the Namespace
- **Read-Only:**
- Can only read information from the Namespace
## How to update an account-level role in Temporal Cloud {#update-roles}
With Global Admin or Account Owner privileges, you can update any user's account-level [role](#account-level-roles) using either the Web UI or the tcld CLI utility.
The Account Owner role can only be granted by existing Account Owners.
For security reasons, changes to the Account Owner role must be made through Temporal Support.
To change or delete an Account Owner, you must submit a [support ticket](https://temporalsupport.zendesk.com/).
{/* How to update an account-level role in Temporal Cloud using Web UI */}
### How to update an account-level role using Web UI
1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, select the user.
1. On the user profile page, select **Edit User**.
1. On the **Edit User** page in **Account Level Role**, select the role.
1. Select **Save**.
{/* How to update an account-level role in Temporal Cloud using tcld */}
### How to update an account-level role using tcld
For details, see the [tcld user set-account-role](/cloud/tcld/user/#set-account-role) command.
## How to update Namespace-level permissions in Temporal Cloud {#update-permissions}
You can update Namespace-level [permissions](#namespace-level-permissions) by using either Web UI or tcld.
{/* How to update Namespace-level permissions for a Namespace in Temporal Cloud using Web UI */}
### How to use the Web UI to update a user's permissions across multiple Namespaces
1. In Temporal Web UI, select **Namespaces** in the left portion of the window.
1. On the **Namespaces** page, select the Namespace.
1. If necessary, scroll down to the list of permissions
1. On the user profile page in **Namespace permissions**, select the Namespace.
1. On the Namespace page in **Account Level Role**, select the role.
1. Select **Save**.
{/* How to update Namespace-level permissions for a user in Temporal Cloud using Web UI */}
### How to use the Web UI to update permissions for multiple users within a single Namespace
:::note
A user with the Account Owner or Global Admin account-level [role](#account-level-roles) has Namespace Admin permissions for all Namespaces.
:::
1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page in the **Users** tab, select the user.
1. On the user profile page, select **Edit User**.
1. On the **Edit User** page in **Namespace permissions**, change the permissions for one or more Namespaces.
1. Select **Save**.
{/* How to update an account-level role in Temporal Cloud using tcld */}
### How to use tcld to update Namespace-level permissions
For details, see the [tcld user set-namespace-permissions](/cloud/tcld/user/#set-namespace-permissions) command.
## How to delete a user from your Temporal Cloud account {#delete-users}
You can delete a user from your Temporal Cloud Account by using either Web UI or tcld.
:::info
To delete a user, a user must have the Account Owner or Global Admin account-level [role](#account-level-roles).
:::
{/* How to delete a user from your Temporal Cloud account using Web UI */}
### How to update an account-level role using Web UI
1. In Temporal Web UI, select **Settings** in the left portion of the window.
1. On the **Settings** page, find the user and, on the right end of the row, select **Delete**.
1. In the **Delete User** dialog, select **Delete**.
You can delete a user in two other ways in Web UI:
- User profile page: Select the down arrow next to **Edit User** and then select **Delete**.
- **Edit User** page: Select **Delete User**.
{/* How to delete a user from your Temporal Cloud account using tcld */}
### How to update an account-level role using tcld
For details, see the [tcld user delete](/cloud/tcld/user/#delete) command.
## Account-level roles and Namespace-level permissions {#account-level-roles-and-namespace-level-permissions}
Temporal account-level roles and Namespace-level permissions provide access to specific Temporal Workflow and Temporal Cloud operational APIs.
The following table provides the API details associated with each account-level role and Namespace-level permission.
:::note
Account Owners and Global Admins have Namespace Admin permissions on all Namespaces.
:::
#### Account-level role details
This table provides API-level details for the permissions granted to a user through account-level roles. These permissions are configured per user.
| Permission | Read-only | Developer | Finance Admin | Global Admin | Account Owner |
| --------------------------------- | --------- | --------- | ------------- | ------------ | ------------- |
| CountIdentities | ✔ | ✔ | ✔ | ✔ | ✔ |
| CreateAccountAuditLogSink | | | | ✔ | ✔ |
| CreateAPIKey | ✔ | ✔ | ✔ | ✔ | ✔ |
| CreateNamespace | | ✔ | | ✔ | ✔ |
| CreateNexusEndpoint | | ✔ | | ✔ | ✔ |
| CreateServiceAccount | | | | ✔ | ✔ |
| CreateServiceAccountAPIKey | | | | ✔ | ✔ |
| CreateStripeCustomerPortalSession | | | ✔ | | ✔ |
| CreateUser | | | | ✔ | ✔ |
| DeleteAccountAuditLogSink | | | | ✔ | ✔ |
| DeleteAPIKey | ✔ | ✔ | ✔ | ✔ | ✔ |
| DeleteNexusEndpoint | | ✔ | | ✔ | ✔ |
| DeleteServiceAccount | | | | ✔ | ✔ |
| DeleteUser | | | | ✔ | ✔ |
| GetAccount | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAccountAuditLogSink | | | | ✔ | ✔ |
| GetAccountAuditLogSinks | | | | ✔ | ✔ |
| GetAccountFeatureFlags | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAccountLimits | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAccountSettings | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAccountUsage | | | | ✔ | ✔ |
| GetAPIKey | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAPIKeys | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAsyncOperation | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetAuditLogs | | | | ✔ | ✔ |
| GetDecodedCertificate | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetIdentities | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetIdentity | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetNamespaces | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetNamespacesUsage | | | | ✔ | ✔ |
| GetNexusEndpoint | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetNexusEndpoints | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRegion | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRegions | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRequestStatus | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRequestStatuses | | | | ✔ | ✔ |
| GetRequestStatusesForNamespace | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRequestStatusesForUser | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRoles | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetRolesByPermissions | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetServiceAccount | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetServiceAccounts | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetStripeInvoice | | | ✔ | | ✔ |
| GetUser | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetUsers | ✔ | ✔ | ✔ | ✔ | ✔ |
| GetUsersWithAccountRoles | ✔ | ✔ | ✔ | ✔ | ✔ |
| InviteUsers | | | | ✔ | ✔ |
| ListCreditLedgerEntries | | | ✔ | | ✔ |
| ListGrants | | | ✔ | | ✔ |
| ListMetronomeInvoices | | | ✔ | | ✔ |
| ListMetronomeInvoicesForNamespace | | | ✔ | | ✔ |
| ListNamespaces | ✔ | ✔ | ✔ | ✔ | ✔ |
| ListPromotionGrantBalances | | | ✔ | | ✔ |
| ResendUserInvite | | | | ✔ | ✔ |
| SetAccountSettings | | | | ✔ | ✔ |
| SyncCurrentUserInvite | ✔ | ✔ | ✔ | ✔ | ✔ |
| UpdateAccount | | | | ✔ | ✔ |
| UpdateAccountAuditLogSink | | | | ✔ | ✔ |
| UpdateAPIKey | ✔ | ✔ | ✔ | ✔ | ✔ |
| UpdateNexusEndpoint | | ✔ | | ✔ | ✔ |
| UpdateServiceAccount | | | | ✔ | ✔ |
| UpdateUser | | | | ✔ | ✔ |
| ValidateAccountAuditLogSink | | | | ✔ | ✔ |
#### Namespace-level permissions details
This table provides API-level details for the permissions granted to a user through Namespace-level permissions.
These permissions are configured per Namespace per user.
| Permission | Read | Write | Namespace Admin |
| ---------------------------------- | ---- | ----- | --------------- |
| CountWorkflowExecutions | ✔ | ✔ | ✔ |
| CreateExportSink | | ✔ | ✔ |
| CreateSchedule | | ✔ | ✔ |
| DeleteExportSink | | ✔ | ✔ |
| DeleteNamespace | | ✔ | ✔ |
| DeleteSchedule | | ✔ | ✔ |
| DescribeBatchOperation | ✔ | ✔ | ✔ |
| DescribeNamespace | ✔ | ✔ | ✔ |
| DescribeSchedule | ✔ | ✔ | ✔ |
| DescribeTaskQueue | ✔ | ✔ | ✔ |
| DescribeWorkflowExecution | ✔ | ✔ | ✔ |
| FailoverNamespace | | | ✔ |
| GetExportSink | ✔ | ✔ | ✔ |
| GetExportSinks | ✔ | ✔ | ✔ |
| GetNamespace | ✔ | ✔ | ✔ |
| GetNamespaceUsage | ✔ | ✔ | ✔ |
| GetReplicationStatus | ✔ | ✔ | ✔ |
| GetSearchAttributes | ✔ | ✔ | ✔ |
| GetUsersForNamespace | ✔ | ✔ | ✔ |
| GetWorkerBuildIdCompatibility | ✔ | ✔ | ✔ |
| GetWorkerTaskReachability | ✔ | ✔ | ✔ |
| GetWorkflowExecutionHistory | ✔ | ✔ | ✔ |
| GetWorkflowExecutionHistoryReverse | ✔ | ✔ | ✔ |
| GlobalizeNamespace | | | ✔ |
| ListBatchOperations | ✔ | ✔ | ✔ |
| ListClosedWorkflowExecutions | ✔ | ✔ | ✔ |
| ListExportSinks | ✔ | ✔ | ✔ |
| ListFailoverHistoryByNamespace | ✔ | ✔ | ✔ |
| ListOpenWorkflowExecutions | ✔ | ✔ | ✔ |
| ListReplicaStatus | ✔ | ✔ | ✔ |
| ListScheduleMatchingTimes | ✔ | ✔ | ✔ |
| ListSchedules | ✔ | ✔ | ✔ |
| ListTaskQueuePartitions | ✔ | ✔ | ✔ |
| ListWorkflowExecutions | ✔ | ✔ | ✔ |
| PatchSchedule | | ✔ | ✔ |
| PollActivityTaskQueue | | ✔ | ✔ |
| PollWorkflowTaskQueue | | ✔ | ✔ |
| QueryWorkflow | ✔ | ✔ | ✔ |
| RecordActivityTaskHeartbeat | | ✔ | ✔ |
| RecordActivityTaskHeartbeatById | | ✔ | ✔ |
| RenameCustomSearchAttribute | | ✔ | ✔ |
| RequestCancelWorkflowExecution | | ✔ | ✔ |
| ResetStickyTaskQueue | | ✔ | ✔ |
| ResetWorkflowExecution | | ✔ | ✔ |
| RespondActivityTaskCanceled | | ✔ | ✔ |
| RespondActivityTaskCanceledById | | ✔ | ✔ |
| RespondActivityTaskCompleted | | ✔ | ✔ |
| RespondActivityTaskCompletedById | | ✔ | ✔ |
| RespondActivityTaskFailed | | ✔ | ✔ |
| RespondActivityTaskFailedById | | ✔ | ✔ |
| RespondQueryTaskCompleted | | ✔ | ✔ |
| RespondWorkflowTaskCompleted | | ✔ | ✔ |
| RespondWorkflowTaskFailed | | ✔ | ✔ |
| SetUserNamespaceAccess | | | ✔ |
| SignalWithStartWorkflowExecution | | ✔ | ✔ |
| SignalWorkflowExecution | | ✔ | ✔ |
| StartBatchOperation | | ✔ | ✔ |
| StartWorkflowExecution | | ✔ | ✔ |
| StopBatchOperation | | ✔ | ✔ |
| TerminateWorkflowExecution | | ✔ | ✔ |
| UpdateExportSink | | ✔ | ✔ |
| UpdateNamespace | | ✔ | ✔ |
| UpdateSchedule | | ✔ | ✔ |
| UpdateUserNamespacePermissions | | | ✔ |
| ValidateExportSink | | ✔ | ✔ |
| ValidateGlobalizeNamespace | | | ✔ |
Account Owners and Global Admins will have Namespace Admin permissions on Namespaces.
## How to troubleshoot account access issues {#troubleshoot-access}
### Why can't I sign in after my email domain changed? {#email-domain-change}
If your organization changed its email domain (for example, from `@oldcompany.com` to `@newcompany.com`), you may be unable to sign in to Temporal Cloud with your existing account.
**Why this happens:**
When you sign in using "Continue with Google" or "Continue with Microsoft", Temporal Cloud identifies your account by your email address.
If your email address changes, Temporal Cloud sees this as a different identity and cannot match it to your existing account.
**How to resolve this:**
[Create a support ticket](/cloud/support#support-ticket) with the following information:
- Your previous email address (the one originally used to access Temporal Cloud)
- Your new email address
- Your Temporal Cloud Account Id (if known)
Temporal Support can update your account to use your new email address.
:::tip Use SAML for enterprise identity management
If your organization frequently changes email domains or wants centralized control over user authentication, consider using [SAML authentication](/cloud/saml).
With SAML, your identity provider (IdP) manages user identities, and email domain changes can be handled within your IdP without affecting Temporal Cloud access.
:::
---
## Enable High Availability
:::tip Support, stability, and dependency info
Same-region Replication and Multi-cloud Replication are in
[Public Preview](/evaluate/development-production-features/release-stages#public-preview).
Multi-region Replication is in
[General Availability](/evaluate/development-production-features/release-stages#general-availability)
:::
You can enable High Availability features
([Single-region Replication](/cloud/high-availability#same-region-replication),
[Multi-region Replication](/cloud/high-availability#multi-region-replication), or
[Multi-cloud Replication](/cloud/high-availability#multi-cloud-replication)) for a new or existing Namespace by adding a
replica. When you add a replica, Temporal Cloud begins asynchronously replicating ongoing and existing Workflow
Executions.
Not all replication options are available in all regions. See the [region documentation](/cloud/regions) for the
replication options available in each region.
Using private network connectivity with a HA namespace requires extra setup. See
[Connectivity for HA](/cloud/high-availability/ha-connectivity).
There are charges associated with Replication and enabling High Availability features. For pricing details, visit
Temporal Cloud's [Pricing](/cloud/pricing) page.
## Create a Namespace with High Availability features {#create}
To create a new Namespace with High Availability features, you can use the Temporal Cloud UI or the tcld command line
utility.
1. Visit Temporal Cloud in your Web browser.
1. During Namespace creation, specify the primary [region](/cloud/regions) for the Namespace.
1. Select "Add a replica".
1. Choose the [region](/cloud/regions) for the replica.
The web interface will present an estimated time for replication to complete.
This time is based on your selection and the size and scale of the Workflows in your Namespace.
At the command line, enter:
```
tcld namespace create \
--namespace . \
--region \
--region
```
Specify the [region codes](/cloud/regions) as arguments to the two `--region` flags.
- Using the same region replicates to an isolation domain within that region.
- Using a different region replicates across regions.
If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before
`namespace create`.
Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use.
## Add High Availability to an existing Namespace {#upgrade}
A replica can be added after a namespace has already been created.
1. Visit Temporal Cloud Namespaces in your Web browser.
1. Navigate to the Namespace details page.
1. Select the “Add a replica” button.
1. Choose the [region](/cloud/regions) for the replica.
The web interface will present an estimated time for replication to complete. This time is based on your selection and
the size and scale of the Workflows in your Namespace.
Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use.
At the command line, enter:
```
tcld namespace add-region \
--namespace . \
--region
```
Specify the [region code](/cloud/regions) of the region where you want to create the replica as an argument to the
`--region` flag.
If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before
`namespace add-region`.
Temporal Cloud sends an email alert once your Namespace is ready for use.
## Change a replica location {#changing}
Temporal Cloud can't change replica locations directly. To change a replica's location, you need to remove the replica
and add a new one.
:::caution
We discourage changing the location of your replica for deployed applications, except under exceptional circumstances.
If you remove your replica, you lose the availability guarantees of the Namespace, and it can take time to add another
replica.
If you remove a replica from a region, you must wait seven days before you can re-enable High Availability (HA) in that
same location. During this period, you may add a replica to a different region, provided you have not had one active
there within the last seven days.
:::
Follow these steps to change the replica location:
1. [Remove your replica](#disable). This disables High Availability for your Namespace.
2. [Add a new replica](#upgrade) to your Namespace.
You will receive an email alert once your Namespace is ready for use.
## Disable High Availability (remove a replica) {#disable}
To disable High Availability features on a Namespace, remove the replica from that Namespace. Removing a replica
disables all High Availability features:
- Discontinues replication of the Workflows in the Namespace.
- Disables the Namespace's ability to trigger a failover to a different region or cloud.
- For Workers and Clients that use API keys, removing a replica requires connecting to the Namespace using the published
[regional endpoint](/cloud/regions) for the Namespace's region.
- Disables connecting to the Namespace with API keys and the Namespace's endpoint or the replica region's regional
endpoint.
- Ends High Availability charges.
:::caution
After removing a Namespace's replica, you cannot re-enable replication on that same Temporal Cloud Namespace in the same
location for seven days.
:::
Follow these steps to remove a replica from a Namespace:
1. If you are using API keys for authentication on this Namespace, configure your Workers and Clients that use API keys
to [connect with the regional Temporal Cloud endpoint](/cloud/api-keys#namespace-authentication) for the Namespace's
primary region.
1. Navigate to the Namespace details page in Temporal Cloud
1. Select the option to "Remove Replica" on the "Region" card.
First, if you are using API keys for authentication on this Namespace, configure your Workers and Clients that use API
keys to [connect with the regional Temporal Cloud endpoint](/cloud/api-keys#namespace-authentication) for the
Namespace's primary region.
Then, run the following command to remove the replica:
```
tcld namespace delete-region \
--api-key \
--namespace . \
--region
```
:::important
To remove a replica from a Namespace with API keys enabled, you need assistance from Temporal Support. Please
[contact support](/cloud/support#support-ticket) with the Namespace ID of the Namespace where you want to remove the
replica. You must confirm that Workers and Clients with API keys have been configured to connect to the Namespace using
the published [regional endpoint](/cloud/regions).
This safeguard ensures that Workers and Clients continue running uninterrupted once Temporal Support removes the
replica. After the replica is removed, if Workers and Clients with API keys attempt to use the Namespace endpoint or the
former replica's regional endpoint, their requests will fail.
:::
---
## Configure and Trigger Failovers
In case of an incident or an outage, Temporal will automatically your Namespace from the primary to the replica.
This lets Workflow Executions continue with minimal interruptions or data loss.
You can also [manually initiate failovers](/cloud/high-availability/failovers) based on your situational monitoring or for testing.
Returning control from the replica to the primary is called a .
The replica is active for a brief duration during an incident.
After the incident, Temporal fails back to the primary.
## Failovers
Occasionally, a Namespace may become temporarily unavailable due to an unexpected incident.
Temporal Cloud detects these issues using regular health checks.
### Health checks
Temporal Cloud monitors error rates, latencies, and infrastructure problems, such as request timeouts.
If it finds unhealthy conditions where indicators exceed the allowed thresholds, Temporal automatically switches the primary to the replica.
In most cases, the replica is unaffected by the issue.
This process is known as failover.
### Automatic failovers
Failovers prevent data loss and application interruptions.
Existing Workflows continue, and new Workflows start as the incident is addressed.
Once the incident is resolved, Temporal Cloud performs a "failback," shifting Workflow Execution processing back to the original Namespace.
Temporal Cloud handles failovers automatically, ensuring continuity without manual intervention.
For more control over the failover process, you can [disable automated failovers](/cloud/high-availability/failovers#disabling-temporal-initiated).
:::tip
You can test the failover of Namespace with High Availability features by manually triggering a failover using the UI or the 'tcld' CLI utility.
In most scenarios, we recommend you let Temporal handle failovers for you.
After failover, be aware of the following points:
- When working with Multi-region Namespaces, your CNAME may change.
For example, it may switch from aws-us-west-1.region.tmprl.cloud to aws-us-east-1.region.tmprl.cloud.
This change doesn't affect same-region Namespaces.
- Your Namespace endpoint _will not change_.
If it is `my_namespace.my_account.tmprl.cloud:7233` before failover, it will be `my_namespace.my_account.tmprl.cloud:7233` after failover.
:::
### The failover process {#failover-process}
Temporal's automated failover process works as follows:
- During normal operation, the primary asynchronously copies operations and metadata to its replica, keeping them in sync.
- If the primary becomes unavailable, Temporal detects the issue through health checks.
It automatically switches to the replica, using one of its available [failover scenarios](#scenarios).
- The replica takes over the active role and becomes the primary.
Operations continue with minimal disruption.
- When the original primary recovers, the roles can either switch back (failback, by default) or remain as they are, based on your Namespace settings.
Automatic role switching with failover and failback minimizes downtime for consistent availability.
:::info
A Namespace failover, which updates the "active region" field in the Namespace record, is a metadata update.
This update is replicated through the Namespace metadata mechanism.
:::
## Failover scenarios {#scenarios}
The Temporal Cloud failover mechanism supports several modes for executing Namespace failovers.
These modes include graceful failover ("handover"), forced failover, and a hybrid mode.
The hybrid mode is Temporal Cloud’s default Namespace behavior.
### Graceful failover (handover) {#graceful-failover}
In this mode, Temporal Cloud fully processes and drains replication Tasks.
Temporal Cloud pauses traffic to the Namespace before the failover.
Graceful failover prevents the loss of progress and avoids data conflicts.
During graceful failover, the Namespace may experience a brief period of unavailability.
This duration can be limited by the caller and defaults to 10 seconds.
If the system is unable to reach a consistent state within this timeout, the failover attempt is aborted and the Namespace reverts to its previous state.
During this unavailable period:
- Existing Workflows stop progress.
- Temporal Cloud returns a "Service unavailable error".
This error is retryable by the Temporal SDKs.
- State transitions will not happen and tasks are not dispatched.
- User requests like start/signal Workflow are rejected.
- Operations are paused during handover.
This mode favors _consistency_ over availability.
### Forced failover {#forced-failover}
In this mode, a Namespace immediately activates in the replica.
Events not replicated due to replication lag undergo conflict resolution upon reaching the new active Namespace.
This mode prioritizes _availability_ over consistency.
### Hybrid failover mode {#hybrid-failover}
While graceful failovers are preferred for consistency, they aren’t always practical.
Temporal Cloud’s hybrid failover mode (the default mode) limits the initial graceful failover attempt to 10 seconds or less.
During this period:
- Existing Workflows stop progress.
- Temporal Cloud returns a "Service unavailable error", which is retried by SDKs.
If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover.
This strategy balances _consistency_ and _availability_ requirements.
### Scenario summary
| Failover Scenario | Characteristics |
| ---------------------------- | ------------------------------------------------------- |
| Graceful failover (handover) | Favors _consistency_ over availability. |
| Forced failover | Prioritizes _availability_ over consistency. |
| Hybrid failover mode | Balances _consistency_ and _availability_ requirements. |
## Network partitions
At any time only the primary or the replica is active.
The only exception occurs in the event of a [network partition](https://en.wikipedia.org/wiki/Network_partition), when a Network splits into separate subnetworks.
Should this occur, you can promote a replica to active status.
**Caution:** This temporarily makes both regions active.
After the network partition is resolved and communication between the isolation domains/regions is restored, a conflict resolution algorithm determines whether the primary or replica remains active.
:::tip
In traditional active/active replication, multiple nodes serve requests and accept writes simultaneously, ensuring strong synchronous data consistency.
In contrast, with a Temporal Cloud Namespace with High Availability Features, only the primary accepts requests and writes at any given time.
Workflow History Events are written to the primary first and then asynchronously replicated to the replica, ensuring that the replica remains in sync.
:::
## Conflict resolution {#conflict-resolution}
Namespaces with replicas rely on asynchronous event replication.
Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers.
In the event of a non-graceful failover, replication lag may cause a temporary setback in Workflow progress.
Namespaces that aren't replicated can be configured to provide _at-most-once_ semantics for Activities execution when a retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0.
High Availability Namespaces provide _at-least-once_ semantics for execution of Activities.
Completed Activities _may_ be re-dispatched in a newly active Namespace, leading to repeated executions.
When a Workflow Execution is updated in a newly active replica following a failover, events from the previously active Namespace that arrive after the failover can't be directly applied.
At this point, Temporal Cloud has forked the Workflow History.
After failover, Temporal Cloud creates a new branch history for execution, and begins its process.
The Temporal Service ensures that Workflow Histories remain valid and are replayable by SDKs post-failover or after conflict resolution.
This capability is crucial for ensuring Workflow Executions continue forward without losing progress, and for maintaining consistency across replication, even during incidents that cause disruptions in replication.
## Perform a manual failover {#triggering-failovers}
For some users, Temporal's automated health checks and failovers don't provide sufficient nuance and control.
For this reason, you can manually trigger failovers based on your own custom alerts and for testing purposes.
This section explains how and what to expect afterward.
:::warning Check Your Replication Lag
Always check the before initiating a failover.
A forced failover when there is a significant replication lag has a higher likelihood of rolling back Workflow progress.
:::
### Trigger the failover {#manual-failovers}
You can trigger a failover manually using the Temporal CloudWeb UI or the tcld CLI, depending on your preference and setup.
The following instructions outline the steps for each method:
1. Visit the [Namespace page](https://cloud.temporal.io/namespaces) on the Temporal Cloud Web UI.
1. Navigate to your Namespace details page and select the **Trigger a failover** option from the menu.
1. Confirm your action.
After confirmation, Temporal initiates the failover.
To manually trigger a failover, run the following command in your terminal:
```
tcld namespace failover \
--namespace . \
--region
```
If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace failover`.
Temporal fails over the primary to the replica.
When you're ready to fail back, follow these failover instructions to move the primary back to the original.
### Post-failover event information {#info}
After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs.
The audit log entry for Failover uses the `"operation": "FailoverNamespace"` event.
After failover, the replica becomes active, taking over in the isolation domain or region.
You don't need to monitor Temporal Cloud's failover response in real time.
Whenever there is a failover event, Temporal Cloud [notifies you via email](/cloud/notifications#admin-notifications)
### Returning to the primary with failbacks
After Temporal-initiated failovers, Temporal Cloud shifts Workflow Execution processing back to the original region or isolation domain that was active before the incident once the incident is resolved.
This is called a "failback".
:::note
To failback a manually-initiated failover, follow the [Manual Failover](#manual-failovers) directions to failover back to the original primary.
:::
## Disabling Temporal-initiated failovers {#disabling-temporal-initiated}
When you add a replica to a Namespace, in the event of an incident or an outage Temporal Cloud automatically fails over the Namespace to its replica.
_This is the recommended and default option._
However if you prefer to disable Temporal-initiated failovers and handle your own failovers, you can do so by following these instructions:
1. Navigate to the Namespace detail page in Temporal Cloud.
1. Choose the "Disable Temporal-initiated failovers" option.
To disable Temporal-initiated failovers, run the following command in your terminal:
```
tcld namespace update-high-availability \
--namespace . \
--disable-auto-failover=true
```
If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace update-high-availability`
Temporal Cloud disables its health-check initiated failovers.
To restore the default behavior, unselect the option in the WebUI or change `true` to `false` in the CLI command.
## Best practices: Workers and failovers {#worker}
Enabling High Availability for Namespaces doesn't require specific Worker configuration.
The process is invisible to the Workers.
When a Namespace fails over to the replica, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption.
When a Namespace fails over to a replica in a different region, Workers will be communicating cross-region.
- If your application can’t tolerate this latency, deploy a second set of Workers in the replica's region or opt for a replica in the same region:
- In the case of a complete regional outage, Workers in the original region may fail alongside the original Namespace.
To keep Workflows moving during this level of outage, deploy a second set of Workers to the secondary region.
:::tip
Temporal Cloud enforces a maximum connection lifetime of 5 minutes.
This offers your Workers an opportunity to re-resolve the DNS.
:::
## Best practices: scheduled failover testing {#testing}
Microservices and external dependencies will fail at some point.
Testing failovers ensures your app can handle these failures effectively.
Temporal recommends regular and periodic failover testing for mission-critical applications in production.
By testing in non-emergency conditions, you verify that your app continues to function, even when parts of the infrastructure fail.
:::tip Safety First
If this is your first time performing a failover test, run it with a test-specific namespace and application.
This helps you gain operational experience before applying it to your production environment.
Practice runs help ensure the process runs smoothly during real incidents in production.
:::
Failover testing (also known as ")" can:
- **Validate replicated deployments**:
In multi-region setups, failover testing ensures your app can run from another region when the primary region experiences outages.
In standard setups, failover testing instead works with an isolation domain.
This maintains high availability in mission-critical deployments.
Manual testing confirms the failover mechanism works as expected, so your system handles incidents effectively.
- **Assess replication lag**:
In multi-region deployment, monitoring [replication lag](/cloud/high-availability/monitoring#replication-lag-metric) between regions is crucial.
Check the lag before initiating a failover to avoid rolling back Workflow progress.
This is less important when using isolation domains as failover is usually instantaneous.
Manual testing helps you practice this critical step and understand its impact.
- **Assess recovery time**:
Manual testing helps you measure actual recovery time.
You can check if it meets your expected Recovery Time Objective (RTO) of 20 minutes or less, as stated in the [High Availability Namespace SLA](/cloud/sla).
- **Identify potential issues**:
Failover testing uncovers problems not visible during normal operation.
This includes issues like [backlogs and capacity planning](https://temporal.io/blog/workers-in-production#testing-failure-paths-2438) and how external dependencies behave during a failover event.
- **Validate fault-oblivious programming**:
Temporal uses a "fault-oblivious programming" model, where your app doesn’t need to explicitly handle many types of failures.
Testing failovers ensures that this model works as expected in your app.
- **Operational readiness**:
Regular testing familiarizes your team with the failover process, improving their ability to handle real incidents when they arise.
Testing failovers regularly ensures your Temporal-based applications remain resilient and reliable, even when infrastructure fails.
---
## Connectivity for High Availability
:::tip Namespaces with High Availability features and AWS PrivateLink
Proper networking configuration is required for failover to be transparent to clients and workers when using PrivateLink.
This page describes how to configure routing for Namespaces with High Availability features on AWS PrivateLink.
:::
To use AWS PrivateLink with High Availability features, you may need to:
- Override the regional DNS zone.
- Ensure network connectivity between the two regions.
These instructions assume you already have the PrivateLink connections in place. If not, follow our [guide for creating AWS PrivateLink connections and configuring private DNS](/cloud/connectivity/aws-connectivity).
## Customer side solutions
When using PrivateLink, you connect to Temporal Cloud through a VPC Endpoint, which uses addresses local to your network.
Temporal treats each `region.` as a separate zone.
This setup allows you to override the default zone, ensuring that traffic is routed internally for the regions you’re using.
A Namespace's active region is reflected in the target of a CNAME record.
For example, if the active region of a Namespace is AWS us-west-2, the DNS configuration would look like this:
| ha-namespace.account-id.tmprl.cloud | CNAME | aws-us-west-2.region.tmprl.cloud |
| ----------------------------------- | ----- | -------------------------------- |
After a failover, the CNAME record will be updated to point to the failover region, for example:
| ha-namespace.account-id.tmprl.cloud | CNAME | aws-us-east-1.region.tmprl.cloud |
| ----------------------------------- | ----- | -------------------------------- |
The Temporal domain did not change, but the CNAME updated from us-west-2 to us-east-1.
## Setting up the DNS override
:::caution
Private connectivity is not yet offered for GCP Multi-region Namespaces.
:::
To set up the DNS override, configure specific regions to target the internal VPC Endpoint IP addresses.
For example, you might set aws-us-west-1.region.tmprl.cloud to target 192.168.1.2.
In AWS, this can be done using a Route 53 private hosted zone for `region.tmprl.cloud`.
Link that private zone to the VPCs you use for Workers.
When your Workers connect to the Namespace, they first resolve the `..` record.
This points to `.region.tmprl.cloud`, which then resolves to your internal IP addresses.
Consider how you’ll configure Workers for this setup.
You can either have Workers run in both regions continuously or establish connectivity between regions using Transit Gateway or VPC Peering.
This way, Workers can access the newly activated region once failover occurs.
## Available regions, PrivateLink endpoints, and DNS record overrides
:::caution
The `sa-east-1` region is not yet available for use with Multi-region Namespaces. Currently, it is the only region on the continent.
:::
The following table lists the available Temporal regions, PrivateLink endpoints, and DNS record overrides:
When using a Namespace with High Availability features, the Namespace's DNS record `..` points to a regional DNS record in the format `.region.`.
Here, `` is the currently active region for your Namespace.
During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another.
Namespace DNS records are configured with a 15 second TTL.
Any DNS cache should re-resolve the record within this time.
As a rule of thumb, receiving an updated DNS record takes about twice (2x) the TTL.
Clients should converge to the newly targeted region within, at most, a 30-second delay.
---
## High Availability
Temporal Cloud's High Availability features use asynchronous across multiple to provide enhanced resilience and a 99.99% [SLA](/cloud/sla).
When you enable High Availability features, Temporal deploys your primary and its in separate isolation domains, giving you control over the location of both. This redundancy, combined with capability, enhances availability during outages.
## Built-in reliability
Even without High Availability features, Temporal Cloud provides robust reliability and a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)) guarantee against service errors.
Each standard Temporal Namespace uses replication across three availability zones to ensure high availability.
An availability zone is a part of the system where tasks or operations are handled and executed.
This design helps manage workloads and ensure tasks are completed while improving resource use and reducing delays.
Replication makes sure that any changes to Workflow state or History are saved in all three zones _before_ the Temporal Service acknowledges a change back to the Client.
As a result, your standard Temporal Namespace stays operational even if one of its three zones becomes unavailable.
This provides the basis of our 99.9% service level.
## High Availability features {#high-availability-features}
:::tip Support, stability, and dependency info
Same-region Replication and Multi-cloud Replication are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).
Multi-region Replication is in [General Availability](/evaluate/development-production-features/release-stages#general-availability)
:::
High Availability features provide three types of replication:
| **Deployment** | **Description** |
| --------------------------------------- | ---------------------------------------------------------- |
| **Same‑region Replication** | Isolation domains are located within the same region. |
| **Multi‑region Replication** | Isolation domains are located in separate regions. |
| **Multi‑cloud Replication** | Isolation domains are located in separate cloud providers. |
### Same-region Replication
Temporal replicates Namespaces across isolation domains within one region.
This option is a good fit when your application is built for one region and you prefer to failover within that region.
This provides a reliable failover mechanism while maintaining deployment simplicity.
### Multi-region Replication
Temporal replicates Namespaces across regions, making sure Workflows and data are available even if a region fails.
Asynchronous replication means changes aren’t immediately reflected in other regions but will sync over time, ensuring data integrity.
This setup allows failovers between replicas without needing immediate consistency across regions.
Replication across different regions enhances resilience and reliability.
### Multi-cloud Replication
Temporal asynchronously replicates all Workflows (live and historical) and data to a Namespace in an entirely different cloud provider.
If a provider outage, regional outage, service disruption, or network issue occurs, traffic automatically shifts to the replica.
Replicated data is securely encrypted and transmitted across the public internet between cloud providers.
Internet connectivity allows workers in one cloud to fail over to a replica in a different cloud.
:::info
When you adopt Temporal's High Availability features, don't forget to consider the reliability of your own workers, infrastructure, and dependencies.
Issues like network outages, hardware failures, or misconfigurations in your own systems can affect your application performance.
For the highest level of reliability, distribute your dependencies across regions, and use our Multi-region or Multi-cloud replication features.
Using physically separated regions improves the fault tolerance of your application.
:::
## Service levels and recovery objectives
Namespaces using High Availability have a 99.99% [uptime SLA](/cloud/sla) with sub-1-minute [RPO](/cloud/rpo-rto) and 20-minute [RTO](/cloud/rpo-rto). For detailed information:
- [Service Level Agreement (SLA)](/cloud/sla)
- [Recovery Point Objective (RPO) and Recovery Time Objective (RTO)](/cloud/rpo-rto)
## Failover
High Availability Namespaces can automatically or manually [fail over](/cloud/high-availability/failovers) to the replica if the primary is unavailable or unhealthy.
## Target workloads
High Availability Namespaces are a great solution for Workloads where an outage would cause:
- Revenue loss
- Poor customer experience
- Problems stemming from policy/legal requirements that demand high availability
These are often major concerns for financial services, e-commerce, gaming, global SaaS platforms, bookings & reservations, delivery & shipping, and order management.
---
## Monitoring High Availability
Temporal Cloud offers several ways for you to track the health and performance of your
[High Availability](/cloud/high-availability) namespaces.
## Replication status
You can monitor your replica status with the Temporal Cloud UI. If the replica is unhealthy, Temporal Cloud disables the
“Trigger a failover” option to prevent failing over to an unhealthy replica. An unhealthy replica might be due to:
- **Data synchronization issues:** The replica fails to remain in sync with the primary due to network or performance
problems.
- **Replication lag:** The replica falls behind the primary, causing it to be out of sync.
- **Network issues:** Loss of communication between the replica and the primary causes problems.
- **Failed health checks:** If the replica fails health checks, it’s marked as unhealthy.
These issues prevent the replica from being used during a failover, ensuring system stability and consistency.
## Replication lag metric
Temporal Cloud’s High Availability features use asynchronous replication between the primary and the replica. Workflow
updates in the primary, along with associated History Events, are transmitted to the replica. Replication lag refers to
the transmission delay of Workflow updates and history events from the primary to the replica.
:::tip
Temporal Cloud strives to maintain a replication lag of less than 1 minute. In this context,
P95 means 95% of updates are processed faster than this limit.
:::
A forced failover, when there is significant replication lag, increases the likelihood of rolling back Workflow
progress. Always check the replication lag metrics before initiating a failover.
Temporal Cloud emits three replication lag-specific
[metrics](/cloud/metrics/reference#replication-lag). The following samples demonstrate how you can
use these metrics to monitor and explore replication lag:
**P99 replication lag histogram**:
```
histogram_quantile(0.99, sum(rate(temporal_cloud_v0_replication_lag_bucket[$__rate_interval])) by (temporal_namespace, le))
```
**Average replication lag**:
```
sum(rate(temporal_cloud_v0_replication_lag_sum[$__rate_interval])) by (temporal_namespace)
/
sum(rate(temporal_cloud_v0_replication_lag_count[$__rate_interval])) by (temporal_namespace)
```
When a Namespace is using a replica, you may notice that the Action count in `temporal_cloud_v0_total_action_count` is
2x what it was before adding a replica. This happens because Actions are replicated; they occur on both the primary and
the replica.
## Failover audit log
When Temporal triggers failovers, the [audit log](/cloud/audit-logs) will update with details.
Look for `"operation": "FailoverNamespace"` in the logs.
---
## Temporal Cloud guide
Welcome to the Temporal Cloud guide.
In this guide you will find information about Temporal Cloud, onboarding, features, and how to use them.
To create a Temporal Cloud account, sign up [here](https://temporal.io/get-cloud).
**[Get started with Temporal Cloud.](/cloud/get-started)**
## Become familiar with Temporal Cloud
- [Overview of Temporal Cloud](/cloud/overview)
- [Security model](/cloud/security)
- [Service availability](/cloud/service-availability) (availability, region support, throughput, latency, and limits)
- [Account, Namespace, and application level configurations](/cloud/limits)
- [Service Level Agreement (SLA)](/cloud/sla)
- [Pricing](/cloud/pricing)
- [Support](/cloud/support)
## Feature guides
- [Get started with Temporal Cloud](/cloud/get-started)
- [Manage certificates](/cloud/certificates)
- [Manage API keys](/cloud/api-keys)
- [Manage Namespaces](/cloud/namespaces)
- [Manage users](/cloud/users)
- [Manage user groups](/cloud/user-groups)
- [Manage billing](/cloud/billing-and-cost)
- [Manage Service Accounts](/cloud/service-accounts)
- [API key feature guide](/cloud/api-keys)
- [Metrics feature guide](/cloud/metrics)
- [Temporal Nexus](/cloud/nexus)
- [SAML authentication feature guide](/cloud/saml)
- [Cloud Ops API](/ops)
- [Audit logging feature guide](/cloud/audit-logs)
- [`tcld` (Temporal Cloud command-line interface) reference](/cloud/tcld)
---
## Account Access
Temporal Cloud offers several ways to manage access to your Temporal Cloud account.
- [**Users**](/cloud/users) - Manage individual user accounts and permissions
- [**User Groups**](/cloud/user-groups) - Organize users into groups for simplified access management
- [**Service Accounts**](/cloud/service-accounts) - Configure service accounts for automated access
- [**SAML**](/cloud/saml) - Configure SAML-based single sign-on integration
- [**SCIM**](/cloud/scim) - Use your IDP to manage Temporal Cloud users and access via SCIM integration
---
## Datadog metrics setup
Datadog in partnership with Temporal Cloud has created a native integration with Temporal Cloud metrics.
This integration is available to all Datadog customers.
Exporting cloud metrics to Datadog provides enhanced observability, allowing you to monitor, alert, and visualize key performance indicators of your applications and infrastructure.
Temporal's integration with Datadog extends the monitoring capabilities of your Temporal Cloud deployment. Benefits of using this integration include:
- Out-of-the-box Temporal Cloud metrics dashboard in Datadog
- No need to run a service, Datadog connects directly to Temporal Cloud
For detailed instructions on how to use the integration, see [the documentation on Datadog's site](https://docs.datadoghq.com/integrations/temporal_cloud/).
---
## General observability setup with metrics
You will learn how to do the following:
- [Configure an endpoint using the UI](#configure-via-ui)
- [Configure an endpoint using tcld](#configure-via-cli-tcld)
- [Query for metrics with a PromQL endpoint](#query-promql)
## Configure using the UI {#configure-via-ui}
**How to configure a metrics endpoint using Temporal Cloud UI**
:::note
To view and manage third-party integration settings, your user account must have the Account Owner or Global Admin [role](/cloud/users#account-level-roles).
:::
To assign a certificate and generate your metrics endpoint, follow these steps:
1. Log in to Temporal Cloud UI with an Account Owner or Global Admin [role](/cloud/users#account-level-roles).
2. Go to **Settings** and select **Observability**.
4. Add your root CA certificate (.pem) and save it.
Note that if an observability endpoint is already set up, you can append your root CA certificate here to use the generated observability endpoint in your observability tool.
5. To test your endpoint, run the following command on your host:
```
curl -v --cert --key "/api/v1/query?query=temporal_cloud_v0_state_transition_count"
```
If you have Workflows running on a Namespace in your Temporal Cloud instance, you should see some data as a result of running this command.
After the page refreshes, the new metrics endpoint appears below **Endpoint**, in the form `https://.tmprl.cloud/prometheus`.
Use the endpoint to configure your observability tool.
For example, if you use Grafana, see [Grafana data sources configuration](/cloud/metrics/prometheus-grafana#grafana-data-sources-configuration).
You can also query via the [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) at URLs like:
```
https://.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count
```
For example:
```
$ curl --cert client.pem --key client-key.pem "https://.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count" | jq .
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"__name__": "temporal_cloud_v0_state_transition_count",
"__rollup__": "true",
"operation": "WorkflowContext",
"temporal_account": "your-account",
"temporal_namespace": "your-namespace.your-account-is",
"temporal_service_type": "history"
},
"value": [
1672347471.2,
"0"
]
},
...
}
```
## Configure endpoint using tcld {#configure-via-cli-tcld}
**How to configure a metrics endpoint using the tcld CLI.**
To add a certificate to a metrics endpoint, use [`tcld account metrics accepted-client-ca add`](/cloud/tcld/account#add).
To enable a metrics endpoint, use [`tcld account metrics enable`](/cloud/tcld/account#enable).
To disable a metrics endpoint, use [`tcld account metrics disable`](/cloud/tcld/account#disable).
For more information, see [tcld account metrics command](/cloud/tcld/account#metrics).
## Query for metrics with a PromQL endpoint {#query-promql}
Temporal Cloud emits metrics in a Prometheus-supported format.
Prometheus is an open-source toolkit for alerting and monitoring.
The Temporal Service exposes Cloud metrics with a [Prometheus HTTP API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/).
Temporal Cloud metrics provide a compatible data source for visualizing, monitoring, and observability platforms like Grafana and Datadog.
You can use functions like [rate](https://prometheus.io/docs/prometheus/latest/querying/functions/#rate) or [increase](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase) to calculate the rate of increase for a Temporal Cloud metric:
```
rate(temporal_cloud_v0_frontend_service_request_count[$__rate_interval])
```
Or you might use Prometheus to calculate average latencies or histogram quartiles:
```
---
# Average latency
rate(temporal_cloud_v0_service_latency_sum[$__rate_interval])
/ rate(temporal_cloud_v0_service_latency_count[$__rate_interval])
---
# Approximate 99th percentile latency broken down by operation
histogram_quantile(0.99, sum(rate(temporal_cloud_v0_service_latency_bucket[$__rate_interval])) by (le, operation))
```
Metrics are scraped every 30 seconds and exposed to the metrics endpoint with a 1-minute lag.\
The endpoint returns data with a 15-second resolution, which results in displaying the same value twice.
Set up Grafana with Temporal Cloud observability to view metrics by creating or getting your Prometheus endpoint for Temporal Cloud metrics and enabling SDK metrics.
---
## Temporal Cloud Observability and Metrics
Temporal offers two distinct sources of metrics: [Cloud/Server Metrics](/cloud/metrics/reference) and [SDK Metrics](/references/sdk-metrics).
Each source provides options for levels of granularity and filtering, monitoring-tool integrations, and configuration.
Before implementing Temporal Cloud observability, decide what you need to measure for your use case. There are two primary use cases for metrics:
- To measure the health and performance of Temporal-backed applications and key business processes.
- To measure the health and performance of Temporal infrastructure and user provided infrastructure in the form of Temporal Workers and Temporal Clients.
When measuring the performance of Temporal-backed applications and key business processes, you should rely on Temporal SDK metrics as a source of truth.
This is because Temporal SDKs provide visibility from the perspective of your application, not from the perspective of the Temporal Service.
SDK metrics monitor individual workers and your code's behavior.
Cloud metrics monitor Temporal behavior.
When used together, Temporal Cloud and SDK metrics measure the health and performance of your full Temporal infrastructure, including the Temporal Cloud Service and user-supplied Temporal Workers.
Cloud Metrics for all Namespaces in your account are available from two sources:
- [OpenMetrics Endpoint](/cloud/metrics/openmetrics) - A Prometheus-compatible scrapable endpoint.
- [PromQL Endpoint](/cloud/metrics/promql) - A Prometheus query endpoint.
:::note
OpenMetrics is the recommended option for most users.
:::
---
## OpenMetrics API Reference
The Temporal Cloud OpenMetrics API provides actionable operational metrics about your Temporal Cloud deployment. This is a scrapable HTTP API that returns metrics in OpenMetrics format, suitable for ingestion by Prometheus-compatible monitoring systems.
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Cloud OpenMetrics support is available in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).
:::
## Available Metrics Reference
Metrics descriptions are also available programmatically via the `/v1/descriptors` endpoint. You can see the Metrics Reference for a list of available metrics.
## Authentication
Temporal uses API keys for integrating with the OpenMetrics endpoint. Applications must be authorized and authenticated before they can access metrics from Temporal Cloud.
An API key is owned by a Service Account and inherits the permissions granted to the owner.
### Creating API Keys
API keys can be created using the [Temporal Cloud UI](https://cloud.temporal.io):
1. Navigate to Settings → Service Accounts
2. Create a service account with **"Metrics Read-Only"** Account Level Role
3. Generate an API key within the service account
:::info
See the [docs](https://docs.temporal.io/cloud/api-keys#serviceaccount-api-keys) for more details on generating API keys.
:::
### Using API Keys
All API requests must be made over HTTPS. Calls made over plain HTTP will fail. API requests without authentication will also fail.
```shell
curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics
```
## Object Model
The object model for the Metrics API follows the [OpenMetrics](https://openmetrics.io/) standard.
### Metrics
A metric is a numeric attribute measured at a specific point in time, labeled with contextual metadata gathered at the point of instrumentation.
### Metric Types
All Temporal Cloud metrics are exposed as *gauges* in OpenMetrics format, but represent different measurement types:
* **Rate metrics**: Pre-computed per-second rates with delta temporality (e.g., `temporal_cloud_v1_workflow_success_count` \- workflows completed per second)
* **Value metrics**: Current or instantaneous values (e.g., `temporal_cloud_v1_approximate_backlog_count` \- current number of tasks in queue)
The list of metrics and their labels are available via the [List Descriptors](/cloud/metrics/openmetrics/api-reference#list-metric-descriptors) endpoint or in the [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference).
### Labels
A label is a key-value attribute associated with a metric data point. Labels can be used to filter or aggregate metrics.
Common labels include:
* `temporal_namespace`: The Temporal namespace
* `temporal_account`: The Temporal account
* `region`: The cloud region where the metric originated
* `temporal_workflow_type`: The workflow type (where applicable)
* `temporal_task_queue`: The task queue name (where applicable)
Each metric has its own set of applicable labels. See the Metrics Reference for complete details.
### Metric Family
A [Metric Family](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#metricfamily) may have zero or more metrics. The set of metrics returned will vary based on actual system activity. Metrics only appear in a Metric Family if they were reported during the aggregation window.
## Client Considerations
### Rate Limiting
To protect the stability of the API and keep it available to all users, Temporal employs multiple safeguards.
When a rate limit is breached, an HTTP `429 Too Many Requests` error is returned with the following headers:
| Header | Description |
| ----- | ----- |
| `Retry-After` | The time in seconds until the rate limit window resets |
#### Rate Limit Scopes
:::note
Rate limit scopes are subject to change.
:::
| Scope | Limit |
| ----- | ----- |
| Account | 180 requests per hour |
### Response Completeness
The `X-Completeness` header indicates whether the response contains all available data:
* `complete`: The response contains all metrics requested
* `limited`: Response truncated due to size limits (30k metric data points max). Use namespace or metric filtering to reduce the response size.
* `unknown`: Completeness cannot be determined (possibly due to regional issues or timeouts). Clients are encouraged to retry.
### Retry Logic
Implement retry logic in your client to gracefully handle transient API failures. Use exponential backoff with jitter to avoid retry storms with reasonable retry intervals to avoid reaching rate limits.
### Data Latency
Metric data points are available for query within 2 minutes of their origination. This is in line with the freshest metrics [available from any major service provider](https://docs.datadoghq.com/integrations/guide/cloud-metric-delay/). This latency should be accounted for when setting up monitoring alerts.
## Endpoints
:::info
All endpoints are served from: `metrics.temporal.io`
:::
### Get Metrics
`GET /v1/metrics`
Returns metrics in OpenMetrics format suitable for scraping by Prometheus-compatible systems.
#### Timestamp Offset
To account for metric data latency, this endpoint returns metrics from the current timestamp minus a fixed offset. The current offset is 2 minutes rounded down to the start of the minute. To accommodate this offset, the timestamps in the response should be honored when importing the metrics. For example, in Prometheus this can be controlled using the `honor\_timestamps` flag.
#### Query Parameters
| Parameter | Type | Description |
| ----- | ----- | ----- |
| `namespaces` | string array | Filter to specific Namespaces. Supports wildcards (e.g., `production-*`) |
| `metrics` | string array | Filter to specific metrics |
#### Response Headers
| Header | Description |
| ----- | ----- |
| `X-Completeness` | Indicates the response status: `complete`, `limited`, or `unknown` |
| `Content-Type` | `application/openmetrics-text` |
:::info Example
Request:
```shell
curl -H "Authorization: Bearer " \
"https://metrics.temporal.io/v1/metrics?namespaces=production-*"
```
Response:
```
---
# TYPE temporal_cloud_v1_workflow_success_count gauge
---
# HELP temporal_cloud_v1_workflow_success_count The number of successful workflows per second
temporal_cloud_v1_workflow_success_count{temporal_namespace="production",temporal_workflow_type="payment-processing",region="aws-us-west-2"} 42.0 1609459200000
temporal_cloud_v1_workflow_success_count{temporal_namespace="production",temporal_workflow_type="order-fulfillment",region="aws-us-west-2"} 128.0 1609459200000
---
# TYPE temporal_cloud_v1_approximate_backlog_count gauge
---
# HELP temporal_cloud_v1_approximate_backlog_count Approximate number of tasks in a task queue
temporal_cloud_v1_approximate_backlog_count{temporal_namespace="production",temporal_task_queue="critical-queue",task_type="workflow", region="aws-us-west-2"} 15.0 1609459200000
```
:::
#### Summary of Best Practices
* *Honor timestamps*: Set `honor_timestamps: true` in Prometheus
* *Scrape interval*: Use 30 or 60 second intervals
* *Timeout*: Set scrape timeout to 10 seconds for large responses
* *Filtering*: Use query parameters to reduce response size
### List Metric Descriptors
`GET /v1/descriptors`
Lists all metric descriptors including metadata, data types, and available dimensions (a.k.a. labels).
#### Query Parameters
| Parameter | Type | Description |
| ----- | ----- | ----- |
| `limit` | integer | Page size (1-100, default: 100\) |
| `offset` | integer | Page offset |
:::info Example
Request:
```shell
curl -H "Authorization: Bearer " \
"https://metrics.temporal.io/v1/descriptors"
```
Response:
```json
{
"meta": {
"pagination": {
"total": 35,
"limit": 100,
"offset": 0
}
},
"descriptors": [
{
"name": "temporal_cloud_v1_workflow_success_count",
"help": "The number of successful workflows per second",
"dimensions": [
"temporal_namespace",
"temporal_workflow_type",
"temporal_task_queue",
"region"
]
}
]
}
```
:::
## Managing High Cardinality
:::caution
High-cardinality labels like `temporal_task_queue` and `temporal_workflow_type` can significantly increase metric volume and impact performance of your monitoring system.
:::
### Cardinality Estimation
To estimate your metric cardinality and see if this is an issue:
```
Total series = Base metrics × Namespaces × Task queues × Workflow types
```
Example:
* 6 workflow metrics with both labels
* 10 namespaces
* 50 task queues
* 20 workflow types
* \= 6 × 10 × 50 × 20 \= 60,000 time series
:::note
60,000 time series in the above example results in exceeding the 30,000 data points per scrape limit.
:::
If the cardinality is too high or you are hitting API limits, consider the following strategies.
### Filtering at Scrape Time
You can isolate only the metrics/namespaces you need. For example, the following shows examples of filtering by modifying the `metrics_path.`
```shell
---
# Only specific namespaces matching the wildcard pattern
/v1/metrics?namespaces=production-*
---
# Only specific metrics
/v1/metrics?metrics=temporal_cloud_v1_workflow_success_count
---
# Combined filtering
/v1/metrics?namespaces=prod-*&metrics=temporal_cloud_v1_approximate_backlog_count
```
:::info
In Prometheus, the `params` config can be set to match the same behavior as above.
```yaml
scrape_configs:
- job_name: 'temporal-cloud'
...
static_configs:
- targets: ['metrics.temporal.io']
metrics_path: '/v1/metrics'
params:
namespaces: ['prod-*']
metrics: ['temporal_cloud_v1_approximate_backlog_count']
```
:::
### Label Management
#### Prometheus
If using Prometheus, you can configure it to drop metrics with a specific label or even rename specific label values to reduce the cardinality.
```yaml
metric_relabel_configs:
---
# Consolidate non-critical task queues
- source_labels: [temporal_task_queue]
regex: '(critical-queue|payment-queue)'
target_label: __tmp_keep_original
replacement: 'true'
- source_labels: [__tmp_keep_original]
regex: ''
target_label: temporal_task_queue
replacement: 'other'
- regex: '__tmp_keep_original'
action: labeldrop
```
#### OpenTelemetry Collector
To accomplish the same as Prometheus, a filter can be used in the collector along with any other processors.
```
processors:
filter:
metrics:
include:
match_type: regexp
expressions:
# Only keep metrics with critical-queue or payment-queue
- Label("temporal_task_queue") == nil or IsMatch(Label("temporal_task_queue"), "^(critical-queue|payment-queue)$")
```
### Monitoring Cardinality
Cardinality can be monitored using this PromQL query.
```shell
---
# Count the total number of series
count({__name__=~"temporal_cloud_v1_.*"})
---
# Count the total number of series by metric
count({__name__=~"temporal_cloud_v1_.*"}) by (__name__)
```
## API Limits
| Limit | Impact | Mitigation |
| ----- | ----- | ----- |
| 30k total datapoints per scrape | Response may be truncated | Use namespace/metric filtering |
| 180 requests per account per hour | HTTP 429 returned | Set appropriate scrape interval of 30-60s |
---
## Temporal Cloud OpenMetrics
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Cloud OpenMetrics support is available in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).
:::
:::tip PRICING
Future pricing may apply to high-volume usage that exceeds standard [limits](/cloud/metrics/openmetrics/api-reference#api-limits).
:::
Temporal Cloud's [OpenMetrics](https://openmetrics.io/) endpoint provides operational metrics for your Temporal Cloud workloads in industry-standard Prometheus format, enabling comprehensive monitoring across Namespaces, Workflows, and Task Queues with your existing observability stack.
## Quick Links
* [Integrations](/cloud/metrics/openmetrics/metrics-integrations) - Get started exporting metrics with common integrations
* [API Documentation](/cloud/metrics/openmetrics/api-reference) - Endpoint specification and advanced configuration
* [Metrics Reference](/cloud/metrics/openmetrics/metrics-reference) - Complete catalog of all metrics with descriptions and labels
* [Migration Guide](/cloud/metrics/openmetrics/migration-guide) - Guide on how to transition from the Prometheus query endpoint
## Overview
Temporal Cloud OpenMetrics exposes 30+ metrics covering workflow lifecycles, task queue operations, service performance, and system limits. All metrics are aggregated over one-minute windows and available for scraping within two minutes.
* [Set up authentication and scraping](/cloud/metrics/openmetrics/api-reference#authentication) with the API documentation.
* Browse the [complete metrics catalog](/cloud/metrics/openmetrics/metrics-reference) for descriptions and labels.
* Teams using the query endpoint should review the [migration guide](/cloud/metrics/openmetrics/migration-guide).
## API key authentication
Create a [service account](/cloud/metrics/openmetrics/migration-guide#create-an-api-key) with the "Metrics Read-Only" role, generate an API key, and start scraping immediately - no certificate rotation or distribution required.
## Global endpoint
This is a single endpoint at `metrics.temporal.io` which serves all metrics across your entire account with API key authentication and standard HTTPS.
## Namespace and metric filtering
You can use query parameters to enable selective scraping to manage data volume and costs, which support wildcards for flexible namespace selection and specific metric filtering.
## Dashboard templates
Production-ready [Grafana dashboards](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) provide immediate visibility with pre-built queries and visualizations.
---
## Metrics Integrations
Metrics can be exported from Temporal Cloud using the OpenMetrics endpoint. This document describes configuring integrations that have third party support or are based on open standards.
This document is for basic configuration only. For advanced concepts such as label management and high cardinality scenarios see the
[general API reference](/cloud/metrics/openmetrics/api-reference).
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Cloud OpenMetrics support is available in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).
:::
## Integrations
### Grafana Cloud
Grafana provides a serverless integration with the OpenMetrics endpoint for Grafana Cloud. This integration will scrape metrics, store them in Grafana Cloud, and provides a default dashboard
for visualizing the metrics in Grafana Cloud. See the [integration page](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/integrations/integration-reference/integration-temporal/)
for more details.
### ClickStack
ClickHouse provides an integration with the OpenMetrics endpoint for ClickStack. This integration uses an OpenTelemetry collector to read from the OpenMetrics endpoint, ingest data into ClickHouse, and
includes a default dashboard to visualize the data with HyperDX. See the [integration page](https://clickhouse.com/docs/use-cases/observability/clickstack/integrations/temporal-metrics) for more details.
### New Relic
New Relic integrates with Temporal Cloud via the infrastructure agent using a flex integration that pull data from the OpenMetrics endpoint. See the [integration page](https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/temporal-cloud-integration/) for more details.
### Prometheus \+ Grafana
Self hosted Prometheus can be used to scrape the OpenMetrics endpoint.
1. Add a new scrape job for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys).
```yaml
scrape_configs:
- job_name: 'temporal-cloud'
scrape_interval: 60s
scrape_timeout: 30s
honor_timestamps: true
scheme: https
authorization:
type: Bearer
credentials: ''
static_configs:
- targets: ['metrics.temporal.io']
metrics_path: '/v1/metrics'
```
2. Import the [Grafana dashboard](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) and configure your Prometheus datasource.
### OpenTelemetry Collector Configuration
Collect metrics with a self-hosted OpenTelemetry Collector to ingest into the system of your choosing.
1. Add a new prometheus receiver for the OpenMetrics endpoint with your [API key](/cloud/metrics/openmetrics/api-reference#creating-api-keys).
```yaml
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'temporal-cloud'
scrape_interval: 60s
scrape_timeout: 30s
honor_timestamps: true
scheme: https
authorization:
type: Bearer
credentials_file:
static_configs:
- targets: ['metrics.temporal.io']
metrics_path: '/v1/metrics'
processors:
batch:
exporters:
otlphttp:
endpoint:
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [otlphttp]
```
:::info
Examples for these integrations and more are [here](https://github.com/temporal-community/cloud-metrics-scrape-examples).
:::
---
## OpenMetrics Metrics Reference
This document describes all metrics available from the Temporal Cloud OpenMetrics endpoint.
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Cloud OpenMetrics support is available in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).
:::
## Metric Conventions
### Metric Types
All metrics are exposed as OpenMetrics gauges, but represent different measurement types:
* *Rate Metrics*: per-second rate of the aggregated values
* *Value Metrics*: The most recent aggregate value within a look-back window (e.g. backlogs, limits)
* *Percentile Metrics*: Pre-calculated aggregated latency percentiles in seconds
:::note
All metrics are stored as 1 minute aggregates.
:::
### Common Labels
All metrics include these base labels:
| Label | Description |
| ----- | ----- |
| `temporal_namespace` | The Temporal namespace |
| `temporal_account` | The Temporal account identifier |
| `region` | Cloud region where the metric originated |
## Metrics Catalog
### Frontend Service Metrics
#### temporal\_cloud\_v1\_service\_request\_count
gRPC requests received per second.
| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |
**Type**: Rate
#### temporal\_cloud\_v1\_service\_request\_throttled\_count
gRPC requests throttled per second.
| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |
**Type**: Rate
#### temporal\_cloud\_v1\_service\_error\_count
gRPC errors per second.
| Label | Description |
| ----- | ----- |
| `operation` | The name of the RPC operation |
**Type**: Rate
#### temporal\_cloud\_v1\_service\_pending\_requests
The number of pollers that are waiting for a task. Use this to track against ``temporal_cloud_v1_poller_limit``
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
**Type**: Value
#### temporal\_cloud\_v1\_resource\_exhausted\_error\_count
Resource exhaustion errors per second. This metric does not include throttling due to Namespace limits.
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
**Type**: Rate
#### temporal\_cloud\_v1\_service\_latency\_p50
:::caution
Avoid aggregating this metric across dimensions because the percentile won't be accurate.
:::
The 50th percentile latency of service requests in seconds
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
**Type**: Latency
#### temporal\_cloud\_v1\_service\_latency\_p95
:::caution
Avoid aggregating this metric across dimensions because the percentile won't be accurate.
:::
The 95th percentile latency of service requests in seconds
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
**Type**: Latency
#### temporal\_cloud\_v1\_service\_latency\_p99
:::caution
Avoid aggregating this metric across dimensions as the percentile won't be accurate.
:::
The 99th percentile latency of service requests in seconds
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
**Type**: Latency
### Workflow Completion Metrics
:::caution High Cardinality
These metrics could have high cardinality depending on number of workflow types and task queues.
:::
#### temporal\_cloud\_v1\_workflow\_success\_count
Successful workflow completions per second.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
**Type**: Rate
#### temporal\_cloud\_v1\_workflow\_failed\_count
Workflow failures per second.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
**Type**: Rate
#### temporal\_cloud\_v1\_workflow\_timeout\_count
Workflow timeouts per second.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
**Type**: Rate
#### temporal\_cloud\_v1\_workflow\_cancel\_count
Workflow cancellations per second.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
**Type**: Rate
#### temporal\_cloud\_v1\_workflow\_terminate\_count
Workflow terminations per second.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
**Type**: Rate
#### temporal\_cloud\_v1\_workflow\_continued\_as\_new\_count
Workflows continued as new per second.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `temporal_workflow_type` | The workflow type |
**Type**: Rate
### Task Queue Metrics
:::caution High Cardinality
These metrics could have high cardinality depending on number of task queues present.
:::
#### temporal\_cloud\_v1\_approximate\_backlog\_count
The approximate number of tasks pending in a task queue. Started Activities are not included in the count as they have been dequeued from the task queue.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |
**Type**: Value
#### temporal\_cloud\_v1\_poll\_success\_count
Successfully matched tasks per second.
| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |
**Type**: Rate
#### temporal\_cloud\_v1\_poll\_success\_sync\_count
Tasks matched synchronously per second (no polling wait).
| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |
**Type**: Rate
#### temporal\_cloud\_v1\_poll\_timeout\_count
The rate of poll requests that timed out without receiving a task.
| Label | Description |
| ----- | ----- |
| `operation` | The poll operation name |
| `task_type` | Type of task: `workflow` or `activity` |
| `temporal_task_queue` | The task queue name |
**Type**: Rate
#### temporal\_cloud\_v1\_no\_poller\_tasks\_count
The rate of tasks added to queues with no active pollers.
| Label | Description |
| ----- | ----- |
| `temporal_task_queue` | The task queue name |
| `task_type` | Type of task: `workflow` or `activity` |
**Type**: Rate
### Namespace Metrics
#### temporal\_cloud\_v1\_namespace\_open\_workflows
The current number of open workflows in a namespace.
**Type**: Value
#### temporal\_cloud\_v1\_total\_action\_count
The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``.
| Label | Description |
| ----- | ----- |
| `is_background` | Whether the action was background: `true` or `false`. Background actions (e.g. History export) do not count toward the action rate limit |
| `namespace_mode` | Indicates if actions are produced by an `active` or a `standby` Namespace |
**Type**: Rate
#### temporal\_cloud\_v1\_total\_action\_throttled\_count
The total number of actions throttled per second.
**Type**: Rate
#### temporal\_cloud\_v1\_operations\_count
Operations performed per second.
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
| `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit |
| `namespace_mode` | Indicates if operations are produced by an `active` or a `standby` Namespace |
**Type**: Rate
#### temporal\_cloud\_v1\_operations\_throttled\_count
Operations throttled due to rate limits per second.
| Label | Description |
| ----- | ----- |
| `operation` | The name of the operation |
| `is_background` | Whether the operation was background: `true` or `false`. Background operations do not count toward the operation rate limit |
| `namespace_mode` | Indicates if actions are throttled in an `active` or a `standby` Namespace |
**Type**: Rate
### Schedule Metrics
#### temporal\_cloud\_v1\_schedule\_action\_success\_count
Successfully executed scheduled workflows per second.
**Type**: Rate
#### temporal\_cloud\_v1\_schedule\_buffer\_overruns\_count
The rate of schedule buffer overruns when using `BUFFER_ALL` overlap policy.
**Type**: Rate
#### temporal\_cloud\_v1\_schedule\_missed\_catchup\_window\_count
The rate of missed schedule executions outside the catchup window.
**Type**: Rate
#### temporal\_cloud\_v1\_schedule\_rate\_limited\_count
The rate of scheduled workflows delayed due to rate limiting.
**Type**: Rate
### Replication Metrics
#### temporal\_cloud\_v1\_replication\_lag\_p50
The 50th percentile cross-region replication lag in seconds.
**Type**: Latency
#### temporal\_cloud\_v1\_replication\_lag\_p95
The 95th percentile cross-region replication lag in seconds.
**Type**: Latency
#### temporal\_cloud\_v1\_replication\_lag\_p99
The 99th percentile cross-region replication lag in seconds.
**Type**: Latency
### Limit Metrics
#### temporal\_cloud\_v1\_operations\_limit
The current configured operations per second limit for a namespace.
**Type**: Value
#### temporal\_cloud\_v1\_action\_limit
The current configured actions per second limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_total_action_count`` and `is_background=false`.
**Type**: Value
#### temporal\_cloud\_v1\_service\_request\_limit
The current configured frontend service RPS limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_request_count``
**Type**: Value
#### temporal\_cloud\_v1\_poller\_limit
The current configured poller limit for a namespace. Track utilization against this limit with ``temporal_cloud_v1_service_pending_requests``.
**Type**: Value
---
## OpenMetrics Migration Guide
Temporal Cloud is transitioning from our Prometheus query endpoint to an industry-standard OpenMetrics
(Prometheus-compatible) endpoint for metrics collection. This migration represents a significant improvement in how you
can monitor your Temporal Cloud workloads, bringing enhanced capabilities, better integration with observability tools,
and access to high-cardinality metrics that were previously unavailable.
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
The OpenMetrics endpoint is available in
[Public Preview](/evaluate/development-production-features/release-stages#public-preview) for testing and validation.
The existing Prometheus query endpoint remains fully operational and supported.
:::
## Why We're Making This Change
1. **Industry-Standard Format**: Native compatibility with Prometheus and OpenTelemetry and all major observability
platforms (Datadog, New Relic etc.) without custom integrations.
2. **High-Cardinality Metrics**: Access to previously unavailable dimensions including:
- `temporal_task_queue` labels on multiple metrics
- `temporal_workflow_type` labels for workflow-specific monitoring
- New task queue backlog metrics for better operational visibility
3. **Accurate Percentiles**: Our new system provides accurate percentile calculations for latency metrics, even in the
presence of substantial outliers, unlike Prometheus-style histograms.
4. **Simplified Integration**: Direct scraping from your observability tools without intermediate translation layers.
5. **Enhanced Performance**: Optimized for high-cardinality data with built-in safeguards for system stability. Data is
available to scrape two minutes from the time it was emitted, in line with the freshest metrics
[available from any major service provider](https://docs.datadoghq.com/integrations/guide/cloud-metric-delay/).
## What's Changing
| Aspect | Current Query Endpoint | New OpenMetrics Endpoint |
| ---------------------- | -------------------------------------------------- | ------------------------------------------- |
| **Protocol** | Prometheus Query API (`/api/v1/query`) | OpenMetrics scrape endpoint (`/v1/metrics`) |
| **Authentication** | mTLS certificates with customer-specific endpoints | API keys with global endpoint |
| **Metric Temporality** | Cumulative counters | Delta temporality (pre-computed rates) |
| **Query Requirement** | Direct queries supported | Requires observability platform |
| **Cardinality** | Limited labels | High-cardinality labels available |
| **Metric Naming** | `*_v0_*` metrics | `*_v1_*` metrics |
## Migration Timeline
Here is the current estimated timeline for migrating from the Prometheus query endpoint to the OpenMetrics endpoint.
:::caution
Timelines can shift so be sure to stay up to date on upcoming releases.
:::
**Public Preview (Current)**
- OpenMetrics endpoint available for onboarding.
- Both endpoints run in parallel with no changes required.
**General Availability [TBA]**:
- OpenMetrics endpoint becomes production-ready and the standard for metrics collection.
**Query Endpoint Deprecation (6 months after GA)**:
- Prometheus query endpoint deprecated and eventually removed.
:::important Action Required
Complete migration before the 6 month deprecation window ends.
:::
## Notable Differences
### 1\. No longer use `rate()` in Prometheus queries
Metrics are now pre-computed as per-second rates with delta temporality.
**Before (Prometheus query endpoint)**:
```
rate(temporal_cloud_v0_frontend_service_request_count[1m])
```
**After (OpenMetrics endpoint)**:
```
temporal_cloud_v1_service_request_count
```
### 2\. Functions that no longer apply
Metrics from OpenMetrics are already rates, therefore certain Prometheus functions no longer make sense. Below is a
non-exhaustive list of some of the functions:
- ❌ `rate()` \- Already computed
- ❌ `increase()` \- Increase of a rate is meaningless
- ❌ `irate()` \- Instant rate not applicable
- ❌ `histogram_quantile()` \- Not applicable (explicit percentiles provided instead)
- ✅ `sum()`, `avg()`, `max()`, `min()` \- Still work normally
### 3\. Percentile metrics
The new endpoint provides explicit percentile metrics (p50, p95, p99) rather than histogram buckets:
**Before (Prometheus query endpoint)**: Calculate percentiles using `histogram_quantile()`
```shell
histogram_quantile(0.95, rate(temporal_cloud_v0_service_latency_bucket[5m]))
```
**After (OpenMetrics endpoint)**: Use pre-calculated percentiles directly
```
temporal_cloud_v1_service_latency_p95
```
**Important Tradeoff**: While pre-calculated percentiles are more accurate for individual time series, they _cannot be
accurately aggregated_. For example:
- ❌ Cannot sum or average p95 values across Namespaces to get a global p95
- ❌ Cannot aggregate p95 values across regions or Task Queues
- ✅ Can still view individual namespace/task queue percentiles accurately
- ✅ More accurate percentile calculations for individual series, especially with outliers
### 4\. Authentication Setup
**Before**: mTLS certificates with customer-specific endpoint
```shell
curl --cert /path/to/client.pem \
--key /path/to/client.key \
--cacert /path/to/ca.pem \
"https://.tmprl.cloud/api/v1/query?query=rate(temporal_cloud_v0_frontend_service_request_count[5m])&time=2025-01-15T10:00:00Z"
```
**After**: API key with global endpoint
```shell
curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics
```
## Migration Steps
### Create an API Key
Create a service account within the Temporal Cloud UI settings with the “Metrics Read-Only” Account Level Role.
:::note
As this is an account-level role, scoping it to specific namespaces has no effect as it will have access to the full
account’s metrics.
:::
Once this is created, you can create an API key within this service account which will inherit the role. Save this API
key in a secure location and use it to access the metrics APIs.
To test that this works, curl the endpoint with your API Key.
The output should resemble the following example:
```shell
$ curl -H "Authorization: Bearer " https://metrics.temporal.io/v1/metrics
---
# TYPE temporal_cloud_v1_service_error_count gauge
---
# HELP temporal_cloud_v1_service_error_count The number of gRPC errors returned by frontend service
---
# TYPE temporal_cloud_v1_service_pending_requests gauge
---
# HELP temporal_cloud_v1_service_pending_requests The number of pollers that are waiting for a task
---
# TYPE temporal_cloud_v1_service_request_count gauge
---
# HELP temporal_cloud_v1_service_request_count The number of RPC requests received by the service..
```
Now you are ready to scrape your metrics\!
### Configuring Grafana \+ Prometheus
#### Update Prometheus Configuration
Add a new scrape job for the OpenMetrics endpoint with your API key.
```yaml
scrape_configs:
- job_name: temporal-cloud
static_configs:
- targets:
- 'metrics.temporal.io'
scheme: https
metrics_path: '/v1/metrics'
honor_timestamps: true
scrape_interval: 60s
scrape_timeout: 30s
authorization:
type: Bearer
credentials: 'API_KEY'
```
:::note
This replaces the direct Grafana datasource configuration you used with the query endpoint.
:::
#### Install New Dashboards
- Download the new Grafana dashboard:
[temporal_cloud_openmetrics.json](https://github.com/temporalio/dashboards/blob/master/cloud/temporal_cloud_openmetrics.json)
- Import alongside existing dashboards during transition
- Update any custom alerts and queries to use new metrics and remove `rate()` functions
### Configuring Datadog
:::tip
Automated integration update coming soon.
:::
The Datadog team is working on updating the official Temporal Cloud integration to use the new endpoint. This transition
should be largely transparent for most users.
For users that want to get started immediately, Temporal Cloud metrics can be directly integrated into Datadog by
configuring the Datadog agent to scrape the OpenMetrics endpoint. An example for that lives
[here](https://github.com/temporal-community/cloud-metrics-scrape-examples/tree/main/datadog/openmetrics).
#### Other Observability Providers
Consult the documentation for your observability system for how to configure it to scrape this endpoint and retrieve
your metrics:
- [NewRelic](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-openmetrics/configure-prometheus-openmetrics-integrations/)
- [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/configuration/#receivers)
Examples for all these integrations live [here](https://github.com/temporal-community/cloud-metrics-scrape-examples).
### Metric Mapping Reference
Below is a template for mapping metrics from the old query endpoint to the new OpenMetrics endpoint. Note that all metrics follow the pattern of `v0` → `v1` version change, and the fundamental difference is the shift from cumulative counters to pre-computed rates for the majority of the metrics. Note that the labels below are only new labels added to the metrics. For the complete list of labels, see the /cloud/metrics/openmetrics/metrics-reference.
#### Frontend Service Metrics
| Old Metric (v0) | New Metric (v1) | New Labels |
| -------------------------------------------------- | -------------------------------------------------- | ---------- |
| `temporal_cloud_v0_frontend_service_error_count` | `temporal_cloud_v1_service_error_count` | `region` |
| `temporal_cloud_v0_frontend_service_request_count` | `temporal_cloud_v1_service_request_count` | `region` |
| `temporal_cloud_v0_resource_exhausted_error_count` | `temporal_cloud_v1_resource_exhausted_error_count` | `region` |
| `temporal_cloud_v0_total_action_count` | `temporal_cloud_v1_total_action_count` | `region` |
#### Workflow Metrics
| Old Metric (v0) | New Metric (v1) | New Labels |
| --------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------- |
| `temporal_cloud_v0_workflow_cancel_count` | `temporal_cloud_v1_workflow_cancel_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_continued_as_new_count` | `temporal_cloud_v1_workflow_continued_as_new_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_failed_count` | `temporal_cloud_v1_workflow_failed_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_success_count` | `temporal_cloud_v1_workflow_success_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_terminate_count` | `temporal_cloud_v1_workflow_terminate_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
| `temporal_cloud_v0_workflow_timeout_count` | `temporal_cloud_v1_workflow_timeout_count` | `region` `temporal_workflow_type` `temporal_task_queue` |
#### Poll Metrics
| Old Metric (v0) | New Metric (v1) | New Labels |
| ------------------------------------------- | ------------------------------------------- | ------------------------------ |
| `temporal_cloud_v0_poll_success_count` | `temporal_cloud_v1_poll_success_count` | `region` `temporal_task_queue` |
| `temporal_cloud_v0_poll_success_sync_count` | `temporal_cloud_v1_poll_success_sync_count` | `region` `temporal_task_queue` |
| `temporal_cloud_v0_poll_timeout_count` | `temporal_cloud_v1_poll_timeout_count` | `region` `temporal_task_queue` |
#### Latency Metrics
| Old Metric (v0) | New Metric (v1) | New Labels |
| ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | ---------- |
| `temporal_cloud_v0_service_latency_bucket temporal_cloud_v0_service_latency_count temporal_cloud_v0_service_latency_sum` | `temporal_cloud_v1_service_latency_p99 temporal_cloud_v1_service_latency_p95 temporal_cloud_v1_service_latency_p50` | `region` |
| `temporal_cloud_v0_replication_lag_bucket temporal_cloud_v0_replication_lag_count temporal_cloud_v0_replication_lag_sum` | `temporal_cloud_v1_replication_lag_p99 temporal_cloud_v1_replication_lag_p95 temporal_cloud_v1_replication_lag_p50` | `region` |
#### Schedule Metrics
| Old Metric (v0) | New Metric (v1) | New Labels |
| -------------------------------------------------------- | -------------------------------------------------------- | ---------- |
| `temporal_cloud_v0_schedule_action_success_count` | `temporal_cloud_v1_schedule_action_success_count` | `region` |
| `temporal_cloud_v0_schedule_buffer_overruns_count` | `temporal_cloud_v1_schedule_buffer_overruns_count` | `region` |
| `temporal_cloud_v0_schedule_missed_catchup_window_count` | `temporal_cloud_v1_schedule_missed_catchup_window_count` | `region` |
| `temporal_cloud_v0_schedule_rate_limited_count` | `temporal_cloud_v1_schedule_rate_limited_count` | `region` |
In addition to these metrics, there are a number of new metrics provided by our OpenMetrics endpoint.
:::info
See the [metrics reference](/cloud/metrics/openmetrics/metrics-reference) for an up-to-date list of all available metrics and their full descriptions.
:::
### Managing High-Cardinality
The new endpoint provides access to high-cardinality labels that can significantly increase your metric volume:
#### High-Cardinality Labels
- `temporal_task_queue`
- `temporal_workflow_type`
#### Best Practices
##### Namespace/Metric filtering
Namespace filtering can be used to ensure that metrics are scraped for relevant Namespaces, which reduces cardinality.
```
https://metrics.temporal.io/v1/metrics?namespaces=production-*
```
This can be taken further by only scraping relevant metrics for a given namespace which ensures that any new high
cardinality metrics won’t be an issue for your observability system.
```
https://metrics.temporal.io/v1/metrics?metrics=temporal_cloud_v1_workflow_success_count?namespaces=production-*
```
##### Relabeling
If the above doesn’t work, consider dropping problematic labels post-scrape but pre-ingestion into your observability
system.
For example, in Prometheus this can be done via
[relabeling rules](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config).
```yaml
metric_relabel_configs:
- source_labels: [__name__]
regex: 'temporal_cloud_v1_poll_success_count'
action: labeldrop
regex: 'temporal_task_queue'
```
Or you can even relabel certain label values in order to keep significant ones. For example, it’s possible to rename
less important task queues to “unknown” while retaining important ones.
```yaml
metric_relabel_configs:
- source_labels: [temporal_task_queue]
regex: '(critical-queue|payment-queue)'
target_label: __tmp_keep_original
replacement: 'true'
# For anything without the keep flag, replace with "unknown"
- source_labels: [__tmp_keep_original]
regex: '' # empty/missing value
target_label: temporal_task_queue
replacement: 'unknown'
# Clean up the temporary label
- regex: '__tmp_keep_original'
action: labeldrop
```
## Limits
See [API limits](/cloud/metrics/openmetrics/api-reference#api-limits) for details.
## FAQ
### Q: Will metrics match between promQL and OpenMetrics endpoints?
No. The metrics will be approximately the same but due to aggregation differences and windowing, values likely won't
match exactly between the two endpoints. Some metrics may be consistently different such as
`temporal_cloud_v1_total_action_count` which includes History Export actions in the OpenMetrics endpoint. In the case of
consistent differences the OpenMetrics endpoint is considered to be more accurate.
### Q: Can I still query metrics directly (e.g. with a Grafana dashboard)?
Currently, the OpenMetrics endpoint requires an observability platform to collect and query metrics. Direct querying via
API to return a time series of data is not supported. Supporting this type of query pattern is a future roadmap item.
### Q: What happens to my existing dashboards and alerts?
During the transition period, both endpoints remain active.
### Q: Will historical data be preserved?
Historical data from the query endpoint will remain in your observability platform. To maintain continuity:
- Combine old (`v0`) and new (`v1`) metrics in your queries during transition
- Consider using the PromQL `or` operator: `metric_v1 or metric_v0`
### Q: Are there limits to how frequently I can scrape or how much data will be returned?
The limits are documented [here](/cloud/metrics/openmetrics/api-reference#api-limits).
### Q: Why are some metrics missing from my scrapes? I don’t see all the metrics documented.
The OpenMetrics endpoint only returns metrics that were generated during the one-minute aggregation window. This is
different from the query endpoint which might return zeros.
**What this means:**
- If no workflows failed in the last minute, `temporal_cloud_v1_workflow_failed_count` won't appear in that scrape.
- If a specific task queue had no activity, its metrics will be absent.
- The set of metrics returned varies between scrapes based on system activity.
**This is normal behavior.** Unlike some metrics systems that populate zeros, the OpenMetrics endpoint follows a sparse
reporting pattern \- metrics only appear when there's actual data to report.
**How to handle this in queries:**
```
(temporal_cloud_v1_workflow_failed_count{namespace="production"} or vector(0))
```
This ensures your dashboards and alerts work correctly even when metrics are temporarily absent due to no activity.
---
## Prometheus Grafana setup
**How to set up Grafana with Temporal Cloud observability to view metrics.**
Temporal Cloud and SDKs generate metrics for monitoring performance and troubleshooting errors.
Temporal Cloud emits metrics through a [Prometheus HTTP API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/), which can be directly used as a Prometheus data source in Grafana or to query and export Cloud metrics to any observability platform.
The open-source SDKs require you to set up a Prometheus scrape endpoint for Prometheus to collect and aggregate the Worker and Client metrics.
This section describes how to set up your Temporal Cloud and SDK metrics and use them as data sources in Grafana.
The process for setting up observability includes the following steps:
1. Create or get your Prometheus endpoint for Temporal Cloud metrics and enable SDK metrics.
- For Temporal Cloud, [generate a Prometheus HTTP API endpoint](/cloud/metrics/general-setup) on Temporal Cloud using valid certificates.
- For SDKs, [expose a metrics endpoint](#sdk-metrics-setup) where Prometheus can scrape SDK metrics and [run Prometheus](#prometheus-configuration) on your host. The examples in this article describe running Prometheus on your local machine where you run your application code.
2. Run Grafana and [set up data sources for Temporal Cloud and SDK metrics](#grafana-data-sources-configuration) in Grafana. The examples in this article describe running Grafana on your local host where you run your application code.
3. [Create dashboards](#grafana-dashboards-setup) in Grafana to view Temporal Cloud metrics and SDK metrics. Temporal provides [sample community-driven Grafana dashboards](https://github.com/temporalio/dashboards) for Cloud and SDK metrics that you can use and customize according to your requirements.
If you're following through with the examples provided here, ensure that you have the following:
- Root CA certificates and end-entity certificates. See [Certificate requirements](/cloud/certificates#certificate-requirements) for details.
- Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud. See Connect to a Temporal Service for details.
- [Go](/develop/go/temporal-client#connect-to-temporal-cloud)
- [Java](/develop/java/temporal-client#connect-to-temporal-cloud)
- [PHP](/develop/php/temporal-client#connect-to-a-dev-cluster)
- [Python](/develop/python/temporal-client#connect-to-temporal-cloud)
- [TypeScript](/develop/typescript/core-application#connect-to-temporal-cloud)
- [.NET](/develop/dotnet/temporal-client#connect-to-temporal-cloud)
- Prometheus and Grafana installed.
## Temporal Cloud metrics setup
Before you set up your Temporal Cloud metrics, ensure that you have the following:
- Account Owner or Global Admin [role privileges](/cloud/users#account-level-roles) for the Temporal Cloud account.
- [CA certificate and key](/cloud/certificates) for the Observability integration.
You will need the certificate to set up the Observability endpoint in Temporal Cloud.
The following steps describe how to set up Observability on Temporal Cloud to generate an endpoint:
1. Log in to Temporal Cloud UI with an Account Owner or Global Admin [role](/cloud/users#account-level-roles).
2. Go to **Settings** and select **Integrations**.
3. Select **Configure Observability** (if you're setting it up for the first time) or click **Edit** in the Observability section (if it was already configured before).
4. Add your root CA certificate (.pem) and save it.
Note that if an observability endpoint is already set up, you can append your root CA certificate here to use the generated observability endpoint with your instance of Grafana.
5. To test your endpoint, run the following command on your host:
```
curl -v --cert --key "/api/v1/query?query=temporal_cloud_v0_state_transition_count"
```
If you have Workflows running on a Namespace in your Temporal Cloud instance, you should see some data as a result of running this command.
6. Copy the HTTP API endpoint that is generated (it is shown in the UI).
This endpoint should be configured as a data source for Temporal Cloud metrics in Grafana.
See [Data sources configuration for Temporal Cloud and SDK metrics in Grafana](#grafana-data-sources-configuration) for details.
## SDK metrics setup
SDK metrics are emitted by SDK Clients used to start your Workers and to start, signal, or query your Workflow Executions.
You must configure a Prometheus scrape endpoint for Prometheus to collect and aggregate your SDK metrics.
Each language development guide has details on how to set this up.
- [Go SDK](/develop/go/observability#metrics)
- [Java SDK](/develop/java/observability#metrics)
- [TypeScript SDK](/develop/typescript/observability#metrics)
- [Python](/develop/python/observability#metrics)
- [.NET](/develop/dotnet/observability#metrics)
The following example uses the Java SDK to set the Prometheus registry and Micrometer stats reporter, set the scope, and expose an endpoint from which Prometheus can scrape the SDK metrics.
```java
//You need the following packages to set up metrics in Java.
//See the Developer's guide for packages required for other SDKs.
//…
//…
{
// See the Micrometer documentation for configuration details on other supported monitoring systems.
// Set up the Prometheus registry.
PrometheusMeterRegistry yourRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
public static Scope yourScope(){
//Set up a scope, report every 10 seconds
Scope yourScope = new RootScopeBuilder()
.tags(ImmutableMap.of(
"customtag1",
"customvalue1",
"customtag2",
"customvalue2"))
.reporter(new MicrometerClientStatsReporter(yourRegistry))
.reportEvery(Duration.ofSeconds(10));
//Start Prometheus scrape endpoint at port 8077 on your local host
HttpServer scrapeEndpoint = startPrometheusScrapeEndpoint(yourRegistry, 8077);
return yourScope;
}
/**
* Starts HttpServer to expose a scrape endpoint. See
* https://micrometer.io/docs/registry/prometheus for more info.
*/
public static HttpServer startPrometheusScrapeEndpoint(
PrometheusMeterRegistry yourRegistry, int port) {
try {
HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
server.createContext(
"/metrics",
httpExchange -> {
String response = registry.scrape();
httpExchange.sendResponseHeaders(200, response.getBytes(UTF_8).length);
try (OutputStream os = httpExchange.getResponseBody()) {
os.write(response.getBytes(UTF_8));
}
});
server.start();
return server;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
//…
// With your scrape endpoint configured, set the metrics scope in your Workflow service stub and
// use it to create a Client to start your Workers and Workflow Executions.
//…
{
//Create Workflow service stubs to connect to the Frontend Service.
WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(
WorkflowServiceStubsOptions.newBuilder()
.setMetricsScope(yourScope()) //set the metrics scope for the WorkflowServiceStubs
.build());
//Create a Workflow service client, which can be used to start, signal, and query Workflow Executions.
WorkflowClient yourClient = WorkflowClient.newInstance(service,
WorkflowClientOptions.newBuilder().build());
}
//…
```
To check whether your scrape endpoints are emitting metrics, run your code and go to [http://localhost:8077/metrics](http://localhost:8077/metrics) to verify that you see the SDK metrics.
You can set up separate scrape endpoints in your Clients that you use to start your Workers and Workflow Executions.
For more examples on setting metrics endpoints in other SDKs, see the metrics samples:
- [Java SDK Samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics)
- [Go SDK Samples](https://github.com/temporalio/samples-go/tree/main/metrics)
## SDK metrics Prometheus Configuration {#prometheus-configuration}
**How to configure Prometheus to ingest Temporal SDK metrics.**
For Temporal SDKs, you must have Prometheus running and configured to listen on the scrape endpoints exposed in your application code.
For this example, you can run Prometheus locally or as a Docker container.
In either case, ensure that you set the listen targets to the ports where you expose your scrape endpoints.
When you run Prometheus locally, set your target address to port 8077 in your Prometheus configuration YAML file. (We set the scrape endopint to port 8077 in the [SDK metrics setup](#sdk-metrics-setup) example.)
Example:
```yaml
global:
scrape_interval: 10s # Set the scrape interval to every 10 seconds. Default is every 1 minute.
#...
---
# Set your scrape configuration targets to the ports exposed on your endpoints in the SDK.
scrape_configs:
- job_name: 'temporalsdkmetrics'
metrics_path: /metrics
scheme: http
static_configs:
- targets:
# This is the scrape endpoint where Prometheus listens for SDK metrics.
- localhost:8077
# You can have multiple targets here, provided they are set up in your application code.
```
See the [Prometheus documentation](https://prometheus.io/docs/introduction/first_steps/) for more details on how you can run Prometheus locally or using Docker.
Note that Temporal Cloud exposes metrics through a [Prometheus HTTP API endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/) (not a scrape endpoint) that can be configured as a data source in Grafana.
The Prometheus configuration described here is for scraping metrics data on endpoints for SDK metrics only.
To check whether Prometheus is receiving metrics from your SDK target, go to [http://localhost:9090](http://localhost:9090) and navigate to **Status > Targets**.
The status of your target endpoint defined in your configuration appears here.
## Grafana data sources configuration {#grafana-data-sources-configuration}
**How to configure data sources for Temporal Cloud and SDK metrics in Grafana.**
Depending on how you use Grafana, you can either install and run it locally, run it as a Docker container, or log in to Grafana Cloud to set up your data sources.
If you have installed and are running Grafana locally, go to [http://localhost:3000](http://localhost:3000) and sign in.
You must configure your Temporal Cloud and SDK metrics data sources separately in Grafana.
To add the Temporal Cloud Prometheus HTTP API endpoint that we generated in the [Temporal Cloud metrics setup](/cloud/metrics/general-setup) section, do the following:
1. Go to **Configuration > Data sources**.
1. Select **Add data source > Prometheus**.
1. Enter a name for your Temporal Cloud metrics data source, such as _Temporal Cloud metrics_.
1. In the **Connection** section, paste the URL that was generated in the Observability section on the Temporal Cloud UI.
1. The **Authentication** section may be left as **No Authentication**.
1. In the **TLS Settings** section, select **TLS Client Authentication**:
- Leave **ServerName** blank. This is not required.
- Paste in your end-entity certificate and key.
- Note that the end-entity certificate used here must be part of the certificate chain with the root CA certificates used in your [Temporal Cloud observability setup](/cloud/metrics/general-setup).
1. Click **Save and test** to verify that the data source is working.
If you see issues in setting this data source, verify your CA certificate chain and ensure that you are setting the correct certificates in your Temporal Cloud observability setup and in the TLS authentication in Grafana.
To add the SDK metrics Prometheus endpoint that we configured in the [SDK metrics setup](#sdk-metrics-setup) and [Prometheus configuration for SDK metrics](#prometheus-configuration) sections, do the following:
1. Go to **Configuration > Data sources**.
2. Select **Add data source > Prometheus**.
3. Enter a name for your Temporal Cloud metrics data source, such as _Temporal SDK metrics_.
4. In the **HTTP** section, enter your Prometheus endpoint in the URL field.
If running Prometheus locally as described in the examples in this article, enter `http://localhost:9090`.
5. For this example, enable **Skip TLS Verify** in the **Auth** section.
6. Click **Save and test** to verify that the data source is working.
If you see issues in setting this data source, check whether the endpoints set in your SDKs are showing metrics.
If you don't see your SDK metrics at the scrape endpoints defined, check whether your Workers and Workflow Executions are running.
If you see metrics on the scrape endpoints, but Prometheus shows your targets are down, then there is an issue with connecting to the targets set in your SDKs.
Verify your Prometheus configuration and restart Prometheus.
If you're running Grafana as a container, you can set your SDK metrics Prometheus data source in your Grafana configuration.
See the example Grafana configuration described in the [Prometheus and Grafana setup for open-source Temporal Service](/self-hosted-guide/monitoring#grafana) article.
### Grafana dashboards setup
To set up dashboards in Grafana, you can use the UI or configure them directly in your Grafana deployment.
:::tip
Temporal provides community-driven example dashboards for [Temporal Cloud](https://github.com/temporalio/dashboards/tree/master/cloud) and [Temporal SDKs](https://github.com/temporalio/dashboards/tree/master/sdk) that you can customize to meet your needs.
:::
To import a dashboard in Grafana:
1. In the left-hand navigation bar, select **Dashboards** > **Import dashboard**.
2. You can either copy and paste the JSON from the [Temporal Cloud](https://github.com/temporalio/dashboards/tree/master/cloud) and [Temporal SDK](https://github.com/temporalio/dashboards/tree/master/sdk) sample dashboards, or import the JSON files into Grafana.
3. Save the dashboard and review the metrics data in the graphs.
To configure dashboards with the UI:
1. Go to **Create > Dashboard** and add an empty panel.
2. On the **Panel configuration** page, in the **Query** tab, select the "Temporal Cloud metrics" or "Temporal SDK metrics" data source that you configured earlier.
If you need to add multiple queries from both data sources, choose `–Mixed–`.
3. Add your metrics queries:
- For Temporal Cloud metrics, expand the **Metrics browser** and select the metrics you want.
You can also select associated labels and values to sort the query data.
The [Cloud metrics documentation](/cloud/metrics/reference) lists all metrics emitted from Temporal Cloud.
- For Temporal SDK metrics, expand the **Metrics browser** and select the metrics you want.
A list of Worker performance metrics is described in the [Developer's Guide - Worker performance](/develop/worker-performance).
All SDK-related metrics are listed in the [SDK metrics](/references/sdk-metrics) reference.
4. The graph should now display data based on your selected queries.
Note that SDK metrics will only show if you have Workflow Execution data and running Workers.
If you don't see SDK metrics, run your Worker and Workflow Executions, then monitor the dashboard.
---
## PromQL Metrics
:::tip
Need to scrape metrics into your observability stack? Try out the new [OpenMetrics endpoint](/cloud/metrics/openmetrics).
:::
Metrics for all Namespaces in your account are available from your metrics endpoint. Keep in mind that your Temporal Cloud metrics lag real-time performance by about one minute. Temporal Cloud also only retains raw metrics for seven days.
To ensure security of your metrics, a CA certificate dedicated to observability is required.
Only clients that use certificates signed by that CA, or that chain up to the CA, can query the metrics endpoint.
For more information about CA certificates in Temporal Cloud, see [Certificate requirements](/cloud/certificates#certificate-requirements).
- [General setup](/cloud/metrics/general-setup)
- [Available metrics](/cloud/metrics/reference)
- [Prometheus & Grafana setup](/cloud/metrics/prometheus-grafana)
- [Datadog setup](/cloud/metrics/datadog)
---
## Temporal Cloud metrics reference
A metric is a measurement or data point that provides insights into the performance and health of a system.
This document describes the metrics available on the Temporal Cloud platform.
Temporal Cloud metrics help you monitor performance and troubleshoot errors.
They provide insights into different aspects of the Service.
This document describes:
- **[Available Temporal Cloud metrics](#available-metrics)**:
The metrics emitted by Temporal Cloud include counts of gRPC errors, requests, successful task matches to a poller, and more.
- **[Metrics labels](#metrics-labels)**:
Temporal Cloud metrics labels can filter metrics and help categorize and differentiate results.
- **[Operations](#metrics-operations)**:
An operation is a special type of label that categorizes the type of operation being performed when the metric was collected.
:::info SDK METRICS
This document discusses metrics emitted by [Temporal Cloud](/cloud).
Temporal SDKs also emit metrics, sourced from Temporal Clients and Worker processes.
You can find information about Temporal SDK metrics on its [dedicated page](/references/sdk-metrics).
Please note:
- SDK metrics start with the phrase `temporal_`.
- Temporal Cloud metrics start with `temporal_cloud_`.
:::
## Available Temporal Cloud metrics {#available-metrics}
**What metrics are emitted from Temporal Cloud?**
The following metrics are emitted for your Namespaces:
### Frontend Service metrics {#frontend}
#### temporal_cloud_v0_frontend_service_error_count
This is a count of gRPC errors returned aggregated by operation.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_frontend_service_request_count
This is a count of gRPC requests received aggregated by operation.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_resource_exhausted_error_count
gRPC requests received that were rate-limited by Temporal Cloud, aggregated by cause.
Labels: temporal_account, temporal_namespace, resource_exhausted_cause
#### temporal_cloud_v0_state_transition_count
Count of state transitions for each Namespace.
#### temporal_cloud_v0_total_action_count
Approximate count of Temporal Cloud Actions.
Labels: temporal_account, temporal_namespace, is_background, namespace_mode
### Poll metrics {#poll}
#### temporal_cloud_v0_poll_success_count
Tasks that are successfully matched to a poller.
Labels: temporal_account, temporal_namespace, operation, task_type, temporal_service_type
#### temporal_cloud_v0_poll_success_sync_count
Tasks that are successfully sync matched to a poller.
Labels: temporal_account, temporal_namespace, operation, task_type, temporal_service_type
#### temporal_cloud_v0_poll_timeout_count
When no tasks are available for a poller before timing out.
Labels: temporal_account, temporal_namespace, operation, task_type, temporal_service_type
### Replication lag metrics {#replication-lag}
#### temporal_cloud_v0_replication_lag_bucket
A histogram of [replication lag](/cloud/high-availability/monitoring#replication-lag-metric) during a specific time interval for a Namespace with high availability.
Labels: temporal_account, temporal_namespace, le
#### temporal_cloud_v0_replication_lag_count
The [replication lag](/cloud/high-availability/monitoring#replication-lag-metric) count during a specific time interval for a Namespace with high availability.
Labels: temporal_account, temporal_namespace
#### temporal_cloud_v0_replication_lag_sum
The sum of [replication lag](/cloud/high-availability/monitoring#replication-lag-metric) during a specific time interval for a Namespace with high availability.
Labels: temporal_account, temporal_namespace
### Schedule metrics {#schedule}
#### temporal_cloud_v0_schedule_action_success_count
Successful execution of a Scheduled Workflow.
Labels: temporal_account, temporal_namespace
#### temporal_cloud_v0_schedule_buffer_overruns_count
When average schedule run length is greater than average schedule interval while a `buffer_all` overlap policy is configured.
Labels: temporal_account, temporal_namespace
#### temporal_cloud_v0_schedule_missed_catchup_window_count
Skipped Scheduled executions when Workflows were delayed longer than the catchup window.
Labels: temporal_account, temporal_namespace
#### temporal_cloud_v0_schedule_rate_limited_count
Workflows that were delayed due to exceeding a rate limit.
Labels: temporal_account, temporal_namespace
### Service latency metrics {#service-latency}
#### temporal_cloud_v0_service_latency_bucket
Latency for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations.
Labels: temporal_account, temporal_namespace, le, operation, temporal_service_type
#### temporal_cloud_v0_service_latency_count
Count of latency observations for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_service_latency_sum
Sum of latency observation time for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
### Workflow metrics {#workflow}
#### temporal_cloud_v0_workflow_cancel_count
Workflows canceled before completing execution.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_workflow_continued_as_new_count
Workflow Executions that were Continued-As-New from a past execution.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_workflow_failed_count
Workflows that failed before completion.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_workflow_success_count
Workflows that successfully completed.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_workflow_terminate_count
Workflows terminated before completing execution.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
#### temporal_cloud_v0_workflow_timeout_count
Workflows that timed out before completing execution.
Labels: temporal_account, temporal_namespace, operation, temporal_service_type
## Metrics labels {#metrics-labels}
**What labels can you use to filter metrics?**
Temporal Cloud metrics include key-value pairs called labels in their associated metadata.
Labels help you categorize and differentiate metrics for precise filtering, querying, and aggregation.
Use labels to specific attributes or compare values, such as numeric buckets in histograms.
This added context enhances the monitoring and analysis capabilities, providing deeper insights into your data.
Use the following labels to filter metrics:
| Label | Explanation |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `le` | Less than or equal to (`le`) is used in histograms to categorize observations into buckets based on their value being less than or equal to a predefined upper limit. |
| `operation` | This includes gRPC operations and general Cloud operations such as:SignalWorkflowExecutionStartBatchOperationStartWorkflowExecutionTaskQueueMgrTerminateWorkflowExecutionUpdateNamespaceUpdateSchedule See: [Metric Operations](#metrics-operations) and [Temporal Cloud Operation reference](/references/operation-list)|
| `resource_exhausted_cause` | Cause for resource exhaustion. |
| `task_type` | Activity, Workflow, or Nexus. |
| `temporal_account` | Temporal Account. |
| `temporal_namespace` | Temporal Namespace. |
| `temporal_service_type` | Frontend or Matching or History or Worker. |
| `is_background` | This label on `temporal_cloud_v0_total_action_count` indicates when actions are produced by a Temporal background job, for example: hourly Workflow Export. |
| `namespace_mode` | This label on `temporal_cloud_v0_total_action_count` indicates if actions are produced by an active vs a standby Namespace. For a regular Namespace, `namespace_mode` will always be “active”. |
The following is an example of how you can filter metrics using labels:
```text
temporal_cloud_v0_poll_success_count{__rollup__="true", operation="TaskQueueMgr", task_type="Activity", temporal_account="12345", temporal_namespace="your_namespace.12345", temporal_service_type="matching"}
```
## Operations {#metrics-operations}
**What operation labels are captured by Temporal Cloud?**
Operations are a special class of metrics label.
They describe the context during which a metric was captured.
Temporal Cloud includes the following operations labels:
- AdminDescribeMutableState
- AdminGetWorkflowExecutionRawHistory
- AdminGetWorkflowExecutionRawHistoryV2
- AdminReapplyEvents
- CountWorkflowExecutions
- CreateSchedule
- DeleteSchedule
- DeleteWorkflowExecution
- DescribeBatchOperation
- DescribeNamespace
- DescribeSchedule
- DescribeTaskQueue
- DescribeWorkflowExecution
- GetWorkerBuildIdCompatibility
- GetWorkerTaskReachability
- GetWorkflowExecutionHistory
- GetWorkflowExecutionHistoryReverse
- ListBatchOperations
- ListClosedWorkflowExecutions
- OperatorDeleteNamespace
- PatchSchedule
- PollActivityTaskQueue
- PollNexusTaskQueue
- PollWorkflowExecutionHistory
- PollWorkflowExecutionUpdate
- PollWorkflowTaskQueue
- QueryWorkflow
- RecordActivityTaskHeartbeat
- RecordActivityTaskHeartbeatById
- RegisterNamespace
- RequestCancelWorkflowExecution
- ResetStickyTaskQueue
- ResetWorkflowExecution
- RespondActivityTaskCanceled
- RespondActivityTaskCompleted
- RespondActivityTaskCompletedById
- RespondActivityTaskFailed
- RespondActivityTaskFailedById
- RespondNexusTaskCompleted
- RespondNexusTaskFailed
- RespondQueryTaskCompleted
- RespondWorkflowTaskCompleted
- RespondWorkflowTaskFailed
- SignalWithStartWorkflowExecution
- SignalWorkflowExecution
- StartBatchOperation
- StartWorkflowExecution
- StopBatchOperation
- TerminateWorkflowExecution
- UpdateNamespace
- UpdateSchedule
- UpdateWorkerBuildIdCompatibility
- UpdateWorkflowExecution
As the following table shows, certain [metrics groups](#available-metrics) support [operations](#metrics-operations) for aggregation and filtering:
| Metrics Group / Operations | All Operations | SignalWithStartWorkflowExecution / SignalWorkflowExecution / StartWorkflowExecution | TaskQueueMgr | CompletionStats |
| ----------------------------------------------- | -------------- | ----------------------------------------------------------------------------------- | ------------ | --------------- |
| **[Frontend Service Metrics](#frontend)** | X | | | |
| **[Service Latency Metrics](#service-latency)** | | X | | |
| **[Poll Metrics](#poll)** | | | X | |
| **[Workflow Metrics](#workflow)** | | | | X |
---
## Automated Migration
Automated migration is designed to provide a zero-downtime, secure means of migrating to Temporal Cloud. This guide outlines the current process
for transitioning workflows from a self-hosted setup to one hosted within Temporal Cloud.
:::tip Support, stability, and dependency info
Automated migration is currently in [Pre-release](/evaluate/development-production-features/release-stages#pre-release).
:::
### Solution overview
As illustrated below, there are 2 components that support automated migrations:
1. Migration proxy - The [s2s proxy](https://hub.docker.com/r/temporalio/s2s-proxy/tags) provides a security layer between the self-hosted server
and the migration server. The customer-side proxy is installed on your infrastructure, while the cloud-side proxy is managed by Temporal. Communications to the
proxies are secured via [mutual TLS](https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/) (mTLS).
2. Migration server - A Temporal service (server) enables secure connections between your self-hosted setup and Temporal Cloud. It provides the core
functionality of the service.
### Process overview
The migration process is separated into several phases, part of which involves coordinating with Temporal to create necessary cloud-side resources.
Migration involves the following phases:
1. Prepare - Preparing for a migration will require you to deploy a customer-side migration proxy, and will also involve coordinating with Temporal to
configure a migration server.
2. Initiate - Use the _[StartMigrationRequest](https://pkg.go.dev/github.com/temporalio/tcld#readme-start-a-migration)_ API to specify namespaces for migration along with endpoint
details.
A corresponding namespace is created in Temporal Cloud with a “non-active” status. You will configure permissions and access controls during this phase.
3. Monitor - The _[GetMigrationResponse](https://pkg.go.dev/github.com/temporalio/tcld#readme-get-a-migration)_ API allows you to track replication progress, including workflows
replicated, remaining workflows, and estimated
time for completion.
4. Handover - Once replication is complete, you may use the _[HandoverNamespaceRequest](https://pkg.go.dev/github.com/temporalio/tcld#readme-perform-handover-during-a-migration)_ API to switch traffic between your source namespace (self-hosted)
and target namespace (cloud). This is the opportunity to validate functionality within Temporal Cloud prior to finalizing the migration.
5. Finalize - Use _[ConfirmMigrationRequest](https://pkg.go.dev/github.com/temporalio/tcld#readme-confirm-a-migration)_ API to finalize the migration. In the event of issues, you may use
the _[AbortMigrationRequest](https://pkg.go.dev/github.com/temporalio/tcld#readme-abort-a-migration)_ to
roll-back changes without impacting your workflows. These APIs provide granular control over every step of the process, ensuring transparency and flexibility.
### Current limitations
The following are known limitations for the Pre-release phase of this service.
- OSS server versions 1.22 or newer are required. Refer to the [upgrade](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server) procedure as needed.
- History shard counts must be a multiple of two.
- Enabling payload encryption as part of migration is not yet supported. If payloads are already [encrypted](https://docs.temporal.io/payload-codec#encryption) in your self-hosted server
via data converter, then they will remain encrypted during and after migration.
- If you are using multi-cluster replication in your self-hosted setup and have previously failed over namespaces, then this may impact your eligibility for automated migration.
- If you have multiple self-hosted servers and they are all configured with the same cluster name (by default Temporal uses 'active' as cluster name), they cannot be connected to
a single migration server simultaneously due to cluster name collision. There are 2 available options:
1. Migrate one server at a time using a single migration server.
2. Create multiple migration servers (one for each self-hosted server) if you need to migrate all servers simultaneously.
- OSS supports cross-namespace commands (e.g., parent-child, SignalExternal, CancelExternal) through the `system.enableCrossNamespaceCommands` configuration. This
configuration is disabled on Temporal Cloud. If cross-namespace calls exist within workflow code, they must be updated or removed prior to migration.
### Getting started
To prepare for migration, you must first provide qualification details to Temporal via a support ticket. If eligible, the Temporal team will work with you to facilitate your migration.
Submit a support ticket with the following details:
- A list of your Temporal accounts
- Target Temporal Cloud service regions
- For each cluster, provide the server configuration by running `temporal operator cluster describe --address --output json` (see [notes](#alternative-commands-for-versions-1281-and-prior) for Temporal server version 1.28.1 and prior).
- Metrics for the namespaces to be migrated:
- number of open/closed workflows
- storage used
- retention policy
- any custom search attributes
- peak request per second (RPS) and action per second (APS) - refer to [this document](https://docs.google.com/document/d/151xjeI53SBfJ94X1toi5krPp4oeyzJ6wVUrOBhgK714) for instructions on fetching these metrics
- If you use a SQL-based datastore for visibility, and you use custom search attributes, provide _CustomSearchAttributeAliases_ of your namespace by running
temporal operator namespace describe using the [latest Temporal CLI](https://github.com/temporalio/cli).
:::warning
Proceed only when your request has been approved by Temporal.
:::
### Create cloud-side resources
Cloud-side resources must be in place prior to starting a migration. Complete the following procedure.
1. Create one or more empty namespaces in Temporal Cloud to serve as the migration targets. Since migration cannot proceed into a namespace that's already in use, these namespaces should
remain empty (no workflows).
2. Create a support ticket requesting these namespaces be configured with [system limits](https://docs.temporal.io/cloud/limits)
(including APS/RPS) matching your existing self-hosted workload.
3. Verify that a migration endpoint has been created in Temporal Cloud (e.g., `your-endpoint.{your-acct}.tmprl.cloud`). If you don't have one, request one via a support ticket.
4. Create any required custom search attributes used by your workflow.
5. If you need [private connectivity](https://docs.temporal.io/cloud/connectivity) for the namespace in Temporal Cloud, then prepare this setup in advance.
### Prepare your self-hosted service
1. Set the following [dynamic configurations](https://docs.temporal.io/references/dynamic-configuration) settings and then restart the Temporal frontend.
```yaml
frontend.keepAliveMaxConnectionAge:
- value: "2h"
```
2. If not already enabled, enable _GlobalNamspace_ by updating the _clusterMetadata_ and the _dcRedirectionPolicy_ in your [server config yaml file](https://github.com/temporalio/temporal/tree/main/config) to
the following and restart all Temporal services (frontend, history, matching, worker), starting with the frontend.
```yaml
dcRedirectionPolicy:
policy: "all-apis-forwarding"
clusterMetadata:
enableGlobalNamespace: true # add this
failoverVersionIncrement: 1000000 # to match failoverVersionIncrement in our migration server
masterClusterName: _NO_CHANGE_
currentClusterName: _NO_CHANGE_
clusterInformation:
_NO_CHANGE_:
enabled: true
initialFailoverVersion: [1,100] # pick a unique number between 1 and 100 for each server
rpcName: _NO_CHANGE_
rpcAddress: _NO_CHANGE_
```
3. Run `temporal operator cluster describe` to check the output. The following output is expected:
```yaml
"failoverVersionIncrement": "1000000",
"initialFailoverVersion": "the number you picked"
"isGlobalNamespaceEnabled": true
```
4. Create a CA certificate as a PEM file and share it in your support ticket. This is required for [mutual TLS](https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/) (mTLS)
between your s2s-proxy (as client) and the Temporal s2s-proxy (as server). Note that this differs from the CA used for [accessing namespace](https://docs.temporal.io/cloud/certificates)
in Temporal Cloud.
### Deploying the self-hosted s2s proxy
Complete the following steps in order to deploy and test the self-hosted s2s proxy. The proxy needs to be accessible from your server and requires a public internet connection to connect
to the Temporal s2s-proxy via migration endpoint.
1. Obtain the latest Docker image from the [temporalio/s2s-proxy repository](https://hub.docker.com/r/temporalio/s2s-proxy/tags).
2. Gather the CA certificate generated in the previous steps.
3. Deploy 1 replica of the s2s-proxy (minimum 4 CPU and 512mb memory) and configure s2s-proxy using our [helm chart example](https://github.com/temporalio/s2s-proxy/blob/main/charts/s2s-proxy/README.md) as a reference.
4. Test access by running `temporal operator cluster describe --address {the-outbound-external-address-of-your-proxy}`. It should display the information of the migration server.
5. Request the Temporal team to perform a similar test to verify that the migration server can access your server.
Proceed only when you have received confirmation from Temporal that connectivity tests have passed.
### Migration process
Once all prerequisites have been completed, migration may commence.
#### Migration start
A Temporal operator will initiate the following migration commands on your behalf. The migration proceeds through these phases:
1. Start migration - Your self-hosted namespace remains active while the cloud namespace becomes passive. Workflows are replicated from the self-hosted namespace to the cloud namespace.
Once cloud namespace is in-sync with self-hosted namespace, migration is ready for handover.
2. Handover to the Cloud - The cloud namespace becomes active and the self-hosted namespace becomes passive. Workflows are then replicated from the cloud namespace to the self-hosted server.
3. Confirm migration completion - Workflow replication from the cloud namespace to the self-hosted server stops.
Before the final step, it is possible to hand over the namespace back to the self-hosted server or abort the migration process completely. After this step, migrating namespaces from the cloud
back to a self-hosted server is an unsupported operation.
#### Transfer to cloud
There are two options for switching Temporal clients to the cloud:
- Option 1 (recommended) - Deploy two sets of Temporal clients: one pointing to your Temporal server and one to the Cloud namespace endpoint. This is the recommended option since your workflows will
continue to make progress during the handover, even if your cloud Temporal client is unable to access the cloud (due to misconfiguration, for example). The process is as follows:
1. Direct your cloud-based Temporal clients to the cloud namespace endpoint. Initially, these clients will connect and send Poll requests but will not receive any tasks.
2. Start migration. Your self-hosted namespace is active while your cloud namespace is passive (or standby). Your cloud Temporal clients can begin receiving tasks, but all requests
from cloud clients to the Cloud namespace will automatically forward from the cloud to your self-hosted server.
3. Hand over namespace to the cloud. Your cloud namespace becomes active and your self-hosted namespace becomes passive. All requests from your self-hosted Temporal clients will
automatically forward from your server to the cloud.
4. Complete migration. Your self-hosted Temporal clients will no longer receive any tasks from your server, allowing you to stop these clients.
- Option 2 - Deploy one set of Temporal clients and switch the namespace endpoint during migration. With this option, if your workers are misconfigured during the switch, then it is
possible that workflows can stop making progress. It is important to ensure that all workers maintain connectivity to cloud to avoid this scenario.
The process is as follows:
1. Start migration.
2. Switch your Temporal clients to point to the cloud namespace endpoint. Requests from your Temporal clients will automatically forward from the cloud to your server. Alternatively, you may
switch Temporal clients to the cloud namespace endpoint after handover.
3. Hand over namespace to the cloud. Requests from your Temporal clients will now be served by the cloud and will not be forwarded to your server.
4. Confirm migration completion.
#### Final Validation
Before failing over your namespace to Temporal Cloud, review the following production readiness checklist. These steps help ensure a smooth transition and minimize surprises in production.
- Understand how to access metrics for your namespace on Temporal Cloud.
- General workflow monitoring metrics (schedule to start latency, start v.s. completion rate, sync match rate, etc).
- Learn how capacity management works in Temporal Cloud, including how to request capacity increases.
- Plan for a worker tuning session - performance change between Temporal Cloud v.s. self-hosted cluster, which could lead to unexpected symptoms and optimizations.
- Know how to reach out to your Temporal Solutions Architect (SA) and Account Executive (AE) for assistance.
### Additional Notes
#### Special dynamic configuration for Version 1.22.x-1.23.x
Temporal versions 1.22.x and 1.23.x include support for stream-based replication, but it is disabled by default. Since those releases, stream-based replication has been validated as more reliable than the poll-based replication that remained the default in 1.22 and 1.23.
When preparing for an S2C migration on these versions, configure the following dynamic settings to enable stream-based replication:
```yaml
history.enableReplicationStream:
- value: true
```
Enabling this configuration will require a restart of your history pods.
#### Alternative Commands for Versions 1.28.1 and Prior
The `temporal operator cluster describe` command is missing some details needed for migration when run against Version 1.28.1 or older. If you are running that version of Temporal or older, then substitute the below
commands for the "temporal operator cluster describe" command cited in this document.
If you are using tctl, then substitute the following command: `tctl --address admin cluster describe`
If you are not a tctl user, then use the following alternative:
`grpcurl -v -plaintext temporal.server.api.adminservice.v1.AdminService.DescribeCluster`
---
## Migrate
Learn how to migrate your Temporal workflows with zero downtime:
- [Automated Migration](/cloud/migrate/automated) - This process enables seamless transitions from self-hosted Temporal instances to Temporal Cloud.
- [Manual Migration](/cloud/migrate/manual) - This process enables transitions from self-hosted Temporal instances to Temporal Cloud by updating clients
and workflows to utilize new resources within Temporal Cloud.
- [Migrate between regions](/cloud/migrate/migrate-within-cloud) - This process allows you to migrate a Temporal Cloud Namespace between regions or providers.
---
## Manual Migration
Migrating to Temporal Cloud from a self-hosted Temporal Service will have different requirements depending on your usage.
This guide provides some guidance based on our experience helping customers of all sizes successfully migrate.
### What to expect from a migration
Depending on your Workflows' requirements, the migration process may be as simple as changing a few parameters, or may require more extensive code changes.
There are two aspects to consider when migrating: your Temporal Client connection code and your Workflow Executions.
Here's a high-level overview of what you can expect:
- **Introduce another Temporal Client to your Starter and Worker Processes:** Configure and deploy a new Temporal Client so that Temporal Cloud becomes responsible for new Workflow Executions.
- **Migrate Workflow Executions:** There are different approaches for new, running, and completed Workflow Executions.
- **New Workflow Executions:** When you no longer need to send Signals or Queries to your self-hosted Temporal Service, you can deprecate your old Client code. Until then, your self-hosted Temporal Service can receive relevant traffic, while new Workflow Executions are sent to Temporal Cloud.
- **Running Workflow Executions:** Short-running Workflows can often be drained and then started again on Temporal Cloud. Long-running Workflows that cannot be drained might require you to implement more code changes to pass the state of the currently running Workflow to Temporal Cloud.
- **Completed Workflow Executions:** Completed Workflow Execution History cannot be automatically migrated to Temporal Cloud. Refer to [Multi-Cluster Replication](#multi-cluster-replication) for more information.
### Updating Client connection code in your Workers
Whether you're self-hosting Temporal or using Temporal Cloud, you manage runtime of your code.
To migrate your Workflows to Temporal Cloud, you need to change some parameters in the Client connection code, such as updating the namespace and gRPC endpoint.
The changes needed to direct your Workflow to your Temporal Cloud
Namespace are only a few lines of code, including:
- [Add your SSL certificate and private key](/cloud/saml) associated with your Namespace.
- [Copy the Cloud-hosted endpoint](/cloud/namespaces#temporal-cloud-grpc-endpoint) from the Namespace detail Web page.
The endpoint uses this format: `..tmprl.cloud:port`.
- [Connect to Temporal Cloud](/cloud/get-started) with your Client.
- [Configure tcld, the Cloud CLI](/cloud/tcld), with the same address, Namespace, and
certificate used to create a Client through code.
### Migrating your Workflow Executions
A Temporal Service stores the complete Event History for the entire lifecycle of a
Workflow Execution.
To migrate from a self-hosted Temporal Service to Temporal Cloud, take into account the current state, Event History, and any future expectations of your Workflow Executions.
**New Workflows are automatically executed on Temporal Cloud.**
Once you've made the code changes in Step 1, and your new code is deployed, new Workflow Executions will be sent to Temporal Cloud.
Existing Workflows must receive Signals to migrate and re-execute on Cloud.
If you maintain your self-hosted instance, you will still be able to use it to access any execution history from before your migration.
You can also export JSON from your previous execution history, that you can then import into your own analytics system.
**Running Workflows can either be drained or migrated.**
If your Workflow can be completed before any compelling event which drives a move to Temporal Cloud, those Workflows can be automatically restarted on Temporal Cloud.
If your Workflows need to run continuously, you must migrate Workflows while they are running.
To accomplish this migration, cancel your current Workflow and pass the current state to a new Workflow in Temporal Cloud.
Refer to [this repository](https://github.com/temporalio/temporal-migration) for an example of migrating running Workflows in Java.
When performing a live migration, make sure your Worker capacity can support the migration load.
Both a [Signal](/sending-messages#sending-signals) and a [Query](/sending-messages#sending-queries) will be executed during the course of the migration.
Also, the Query API loads the entire history of Workflows into Workers to compute the result (if they are not already cached).
That means that your self-hosted Temporal Service Worker capacity will need to support having those executions in memory to serve those requests.
The volume of these requests might be high to execute against all the matches to a `ListFilter`.
### Considerations when resuming Workflows on a new Temporal Service or Namespace
- **Skipping Steps:** If your Workflow steps cannot guarantee idempotency, determine whether you need to skip those steps when resuming the execution in the target Namespace.
- **Elapsed Time:** If your Workflow is “resuming sleep” when in the target Namespace, determine how you will calculate the delta for the sleep invocation in the new execution.
- **Child Relationships:** If your Workflow has Child Workflow relationships (other than Detached Parent Close Policy children), determine how you can pass the state of those children into the parent to execute the child in a resumed state.
- **Heartbeat state:** If you have long running activities relying on heartbeat state, determine how you can resume these activities in the target Namespace.
- Child Workflows with the same type as their Parent types are returned in List Filters used to gather relevant executions. Unless these are Detached `ParentClosePolicy` children, this is not what you want since the Parent/Child relationship will not be carried over to the target Namespace.
- Long running activities that use heartbeat details will not receive the latest details in the target Namespace.
- Duration between Awaitables inside a Workflow definition needs to be considered for elapsed time accuracy when resuming in the target Namespace.
- When Signaling directly from one Workflow to another, make sure to handle `NotFound` executions in the target Namespace. The Workflows may resume out of order.
### Other considerations when migrating
- Have you added an mTLS certificate to your Temporal Namespace? Review our [documentation for adding a certificate to your Temporal Cloud account](/cloud/certificates) for more information.
- There are differences in how metrics are generated in self-hosted Temporal and Temporal Cloud. Review the [documentation on Temporal Cloud metrics](/cloud/metrics/) for more information.
- Consider the implications for [security and access to your Temporal Service](/cloud/security).
- Review your current load (actions per second) and speak to your Account Executive and Solutions Architect so we can set appropriate [Namespace limits](/cloud/limits).
### Multi-Cluster Replication
[Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) is an experimental feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters for backup and state reconstruction.
Migrating Execution History from a self-hosted Temporal Service to Temporal Cloud is not currently supported.
However, a migration tool based on Multi-Cluster Replication, which will enable this, is currently in development for Temporal Cloud.
If you have used this feature locally or you are interested in using it to migrate to Temporal Cloud, [create a support ticket](https://docs.temporal.io/cloud/support) or watch this space for more information about public availability.
---
## Migrate between regions
Temporal Cloud's [High Availability features](/cloud/high-availability) allow you to migrate a Temporal Cloud Namespace from one region or cloud provider to another with zero downtime.
## Preparing to migrate
Namespaces using Export will need to stop Export and migrate the region configuration to the new region for Export jobs to continue after migration.
See [failover scenarios](/cloud/export#failover-scenarios) for details.
[Using High Availability features affects pricing](/cloud/pricing#high-availability-features).
## Steps to migrate
1. Add a Namespace replica in the region you want to migrate to. See [regions](/cloud/regions) for a list of available regions and supported multi-region and multi-cloud configurations.
2. Wait for the replica to become active. The Cloud UI will display a time estimate, and namespace admins will receive an email when the replica is active.
3. If your workers are using API key authentication: ensure your workers (and all other client code) are updated to [use the regional endpoint of the new replica](/cloud/namespaces#access-namespaces).
4. Trigger a failover to the new region.
5. Remove the Namespace replica in the region you are migrating from.
:::note
If using [API keys](/cloud/api-keys) for worker authentication, you must open a [support ticket](/cloud/support#support-ticket) to remove the replica.
:::
:::note
All replica changes are subject to a [cooldown period](/cloud/high-availability/enable#changing) before further replica changes can be made.
:::
---
## Nexus
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Nexus is now [Generally Available](/evaluate/development-production-features/release-stages#general-availability).
Learn why you should use Nexus in the [evaluation guide](/evaluate/nexus).
:::
[Temporal Nexus](/nexus) allows you to connect Temporal Applications across (and within) isolated Namespaces.
This provides all the benefits of Durable Execution across team and application boundaries with improved modularity, security, debugging, and fault isolation.
Nexus supports cross-team, cross-domain, cross-namespace, multi-region, and multi-cloud use cases.
Temporal Cloud support is built on top of the [core Nexus experience](/nexus) and adds a global Nexus Registry within an Account, enhanced security, and multi-region connectivity within and across AWS and GCP.
:::tip RELATED
- [Evaluate](/evaluate/nexus) why you should use Nexus and learn more about [Nexus use cases](/evaluate/nexus#use-cases).
- [Learn Nexus concepts](/nexus) in the Encyclopedia.
:::
## Global Nexus Registry
The Nexus Registry in Temporal Cloud is scoped to an Account.
Workers in any Namespace can host Nexus Services for others to use within an Account.
## Built-in access controls
Temporal Cloud has built-in Endpoint access controls to restrict which callers can use a Nexus Endpoint.
## Audit logging
Temporal Cloud supports audit log streaming for Nexus Registry actions to create, update, or delete Endpoints.
## Multi-region connectivity
Nexus requests in Temporal Cloud are routed across Namespaces, within and across AWS and GCP, using a global mTLS-secured Envoy mesh.
Built-in Nexus Machinery provides reliable at-least-once execution and Workflow policy can deduplicate requests for exactly-once execution, even across multi-region boundaries.
## Terraform support
The [Terraform provider for Temporal Cloud](/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform) supports managing Nexus Endpoints.
## Learn more
- [Evaluate](/evaluate/nexus) why you should use Nexus and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- [Learn key Nexus concepts](/nexus) and how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4&t=934s)
- Explore [additional resources](/evaluate/nexus#learn-more) to learn more about Nexus.
---
## Latency and Availability - Temporal Nexus
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Nexus is now [Generally Available](/evaluate/development-production-features/release-stages#general-availability).
Learn why you should use Nexus in the [evaluation guide](/evaluate/nexus).
:::
Nexus requests (commands, polling) have the same latency SLOs and error rate SLAs as other Worker requests in both the caller and handler Namespaces.
## Latency metrics
Nexus supports various [latency metrics](/nexus/metrics).
## Worker to Temporal Cloud interactions
Nexus interactions between a Worker and Temporal Cloud use the Worker's Namespace gRPC endpoint.
Nexus-related Worker interactions with Temporal Cloud have the same [latency SLOs](/cloud/service-availability#latency) and [availability SLAs](/cloud/sla) as other calls to a Namespaces's gRPC endpoint.
This applies to the following Nexus-related interactions between a Worker and Temporal Cloud:
- Caller Namespace
- RespondWorkflowTaskCompleted \- schedule a Nexus Operation.
- Handler Namespace
- PollNexusTaskQueue \- get a [Nexus Task](/tasks#nexus-task) to process, for example to start a Nexus Operation.
- RespondNexusTaskCompleted \- report the Nexus Task was successful.
- RespondNexusTaskFailed \- report the Nexus Task failed.
## Nexus connectivity across Namespaces
Nexus connectivity in Temporal Cloud is provided by a global mTLS secured Envoy mesh.
The cross-namespace latency between the caller's Nexus Machinery and the handler's Nexus Machinery varies based on the locality of the caller and handler Namespaces, which may be placed in different regions.
Communication between Namespaces in the same region will have lower latency.
Communication across different regions will have higher latency.
Consult the cross-region latency tables for your cloud provider(s) to estimate the latency for Nexus communication across Namespaces in Temporal Cloud.
---
## Limits - Temporal Nexus
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Nexus is now [Generally Available](/evaluate/development-production-features/release-stages#general-availability).
Learn why you should use Nexus in the [evaluation guide](/evaluate/nexus).
:::
Temporal Cloud has default limits for several aspects of Nexus.
Many of these defaults are configurable, so if you need them changed please open a support ticket.
## Rate Limiting
Nexus requests (commands, polling) are counted as part of the overall Namespace RPS limit in both the caller and handler Namespaces.
Default Namespace RPS limits are set at 1600 and automatically adjust based on recent usage (over prior 7 days).
## Operational Limits
Nexus has operational limits for thing like the maximum number of Nexus Endpoints and the maximum request handler timeout.
### Max Nexus Endpoints
By default, each account is provisioned with a max of 100 Nexus Endpoints.
You can request further increases beyond the initial 100 Endpoint limit by opening a support ticket.
### Workflow Max Nexus Operations
A single Workflow Execution can have a maximum of 30 in-flight Nexus Operations.
See the Nexus Encyclopedia entry for [additional details](/workflow-execution/limits#workflow-execution-nexus-operation-limits).
### Nexus Request Handler Timeout
Nexus Operation handlers have less than 10 seconds to process a single Nexus start or cancel request.
Handlers should observe the context deadline and ensure they do not exceed it.
This includes fully processing a synchronous Nexus operation and starting an asynchronous Nexus operation, for example one that starts a Workflow.
If a handler doesn’t respond within a context deadline, a context deadline exceeded error will be tracked in the caller Workflow’s pending Nexus operations, and the Nexus Machinery will retry the Nexus request with an exponential backoff policy.
### Nexus Operation Maximum Duration
Each Nexus Operation has a maximum ScheduleToClose duration of 60 days, which is most applicable to asynchronous Nexus Operations that are completed with an asynchronous callback using a separate Nexus request from the handler back to the caller Namespace.
The 60 day maximum is a limit we will look to increase at some point in the future.
While the caller of a Nexus Operation can configure the ScheduleToClose duration to be shorter than 60 days, the maximum duration can not be extended beyond 60 days and will be capped by the server to 60 days.
---
## Obserability - Temporal Nexus
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Nexus is now
[Generally Available](/evaluate/development-production-features/release-stages#general-availability). Learn why you
should use Nexus in the [evaluation guide](/evaluate/nexus).
:::
Nexus provides metrics and audit log streaming, in addition to integrated
[execution debugging](/nexus/execution-debugging).
## Metrics
Nexus provides the following metrics:
- [SDK metrics](/nexus/metrics#sdk-metrics) \- emitted by a Worker.
- [Cloud metrics](/nexus/metrics#cloud-metrics) \- emitted by Temporal Cloud.
## Audit Logging
The following Nexus control plane actions are sent to the [Audit Logging](/cloud/audit-logs) integrations:
- Create Nexus Endpoint: `CreateNexusEndpoint`
- Update Nexus Endpoint: `UpdateNexusEndpoint`
- Delete Nexus Endpoint: `DeleteNexusEndpoint`
---
## Pricing for Temporal Nexus
:::tip SUPPORT, STABILITY, and DEPENDENCY INFO
Temporal Nexus is now [Generally Available](/evaluate/development-production-features/release-stages#general-availability).
Learn why you should use Nexus in the [evaluation guide](/evaluate/nexus).
:::
The pricing for [Temporal Nexus](/evaluate/nexus) is:
- **One Action to start or cancel a Nexus Operation** in the caller Namespace.
The underlying Temporal primitives such as Workflows, Activities, Signals created by a Nexus Operation handler (directly or indirectly) result in the normal Actions for those primitives.
This includes retries for underlying Temporal primitives like Activities.
- **No Action results for handling or retrying the Nexus Operation itself**.
However, while the retry of the Nexus Operation incurs no charge, any billable action initiated by the handler (such as an Activity) will be charged if it fails and is subsequently retried.
See [Pricing](/cloud/pricing) for additional details.
## Learn more
- [Evaluate](/evaluate/nexus) why you should use Nexus and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082).
- Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](/nexus).
- [Additional resources](/evaluate/nexus#learn-more) to learn more about Nexus.
---
## Notifications
## Get notified about Temporal Cloud status {#cloud-status}
In the event of an incident, Temporal updates the [Temporal Cloud status page](https://status.temporal.io/) with important updates.
Users can subscribe to updates in their preferred mode (e.g. email, Slack, SMS, etc.) by visiting this page.
## Get notified about administrative events {#admin-notifications}
Temporal Cloud sends emails to notify users of important administrative events.
| Reason for email | Who receives email |
|------------------ | -------------------|
| Certificate Expiring in 15 days | Global Administrator, Namespace Administrator, Account Owner |
| Certificate Expiring in 10 days | Global Administrator, Namespace Administrator, Account Owner |
| Certificate Expiring in 5 days | Global Administrator, Namespace Administrator, Account Owner |
| API Key Expiring in 30 days | Global Administrator, Account Owner, individual user (if API Key has an owner) |
| API Key Expiring in 20 days | Global Administrator, Account Owner, individual user (if API Key has an owner) |
| API Key Expiring in 10 days | Global Administrator, Account Owner, individual user (if API Key has an owner) |
| Sign up credit expiring in 30 days | Account Owner, Finance Administrator |
| Sign up credit expiring in 14 days | Account Owner, Finance Administrator |
| Sign up credit expiring in 7 days | Account Owner, Finance Administrator |
| Sign up credit expiring in 1 days | Account Owner, Finance Administrator |
| Sign up credit is 50% consumed | Account Owner, Finance Administrator |
| Sign up credit is 90% consumed | Account Owner, Finance Administrator |
| Account plan type changed | Global Administrator, Account Owner, Finance Administrator |
| Namespace Failover Completed/Failed | Global Administrator, Namespace Administrator, Account Owner |
To ensure that you receive email notifications, configure your junk-email filters to permit email from
`noreply@temporal.io`.
To provide feedback on notifications or request changes, [create a support ticket](/cloud/support#support-ticket).
---
## Cloud Ops API
:::tip Support, stability, and dependency info
The Temporal Cloud Operations API is in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).
:::
The Temporal Cloud Operations API, or the Cloud Ops API, is an open source, public [HTTP API](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) and [gRPC API](https://github.com/temporalio/cloud-api/tree/main) for programmatically managing Temporal Cloud control plane resources, including [Namespaces](/cloud/namespaces), [Users](/cloud/users), [Service Accounts](/cloud/service-accounts), [API keys](/cloud/api-keys), and others. The Temporal Cloud [Terraform Provider](/cloud/terraform-provider), [tcld CLI](/cloud/tcld), and Web UI all use the Cloud Ops API.
## Develop applications with the Cloud Ops API
You can use the HTTP API or the gRPC API depending on how you need to integrate with your platform. The URL to access both the HTTP and gRPC Cloud Ops API is `saas-api.tmprl.cloud`.
### Prerequisites
These prerequisites are required for using either HTTP or gRPC.
- [Temporal Cloud user account](/cloud/get-started)
- [API Key](/cloud/tcld/apikey#create) for authentication
### Use cases
Some common reasons you might use the API are to:
- Provision Namespaces per environment or tenant via pipelines.
- Bootstrap new projects by creating users, assigning roles, and creating Namespaces via custom scripts.
- Rotate service account keys on a schedule with a job.
- Audit and report access across orgs with scheduled HTTP requests.
### Using HTTP
[The HTTP API](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) supports the same operations as the [gRPC API](#using-grpc), but it's usable via standard HTTP methods and authentication. This may be a more convenient option if you are writing automation scripts for CI/CD or you can't use gRPC due to network policies, proxies, tooling gaps, or language/runtime constraints. Since it's standard HTTP, it's language agnostic giving you the ability to run cloud operations consistently.
:::note
This *does not* allow interaction with individual Workflows or Activities via HTTP.
:::
### Using gRPC
*For Go developers:*
- Use the [Go SDK](https://github.com/temporalio/cloud-sdk-go) for the simplest setup experience
*For other programming languages:*
- Basic familiarity with gRPC and Protocol Buffers (protobuf)
- [Protocol Buffers](https://github.com/protocolbuffers/protobuf/releases)
- [gRPC](https://grpc.io/docs/languages/) in your preferred programming language
You can use the provided proto files to generate client libraries in your desired programming language, and then use that client to access the gRPC API. You can also find the [full gRPC docs on Buf](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CloudService).
#### Using the Go SDK
If you're developing in Go, we recommend using the [Go SDK](https://github.com/temporalio/cloud-sdk-go) which provides pre-compiled Go bindings and a more idiomatic interface. The Go SDK handles all the protobuf compilation and provides ready-to-use Go types and client interfaces. You can also use the [Go samples](https://github.com/temporalio/cloud-samples-go) to help you get started with the Cloud Ops API using the Go SDK.
To start using the Go SDK with the Cloud Ops API, follow these steps:
1. Install the Go SDK:
```go
go get github.com/temporalio/cloud-sdk-go
```
2. Import and use the SDK:
```go
"github.com/temporalio/cloud-sdk-go/client"
)
```
3. The Go SDK provides pre-built client interfaces that handle authentication and connection setup. Refer to the [Go samples](https://github.com/temporalio/cloud-samples-go) for detailed usage examples.
The Go SDK eliminates the need to work directly with generated protobuf files and provides a more idiomatic Go experience.
#### Compile the API and use the generated code (For other languages)
For programming languages other than Go, download the gRPC protobufs from the [Cloud Ops API repository](https://github.com/temporalio/cloud-api/tree/main/temporal/api/cloud) and compile them manually.
Use [gRPC](https://grpc.io/docs/) to compile and generate code in your preferred [programming language](https://grpc.io/docs/#official-support). The steps below use Python as an example and require [Python's gRPC tools](https://grpc.io/docs/languages/python/quickstart/#grpc-tools) to be installed, but the approach can be adapted for other supported programming languages.
1. Clone the Temporal Cloud API repository:
```command
git clone https://github.com/temporalio/cloud-api.git
cd cloud-api
```
2. Copy Protobuf files:
- Navigate to the `temporal` directory.
- Copy the protobuf files to your project directory.
3. Compile the Protobuf files:
```python
python -m grpc_tools.protoc -I./ --python_out=./ --grpc_python_out=./ *.proto
```
- `-I` specifies the directory of the `.proto` files.
- `--python_out=` sets the output directory for generated Python classes.
- `--grpc_python_out=` sets the output directory for generated gRPC service classes.
- `*.proto` processes all `.proto` files.
After compiling the Protobuf files, you will have generated code files in your project directory.
These files enable interaction with the Temporal Cloud API in your chosen programming language.
4. Import the Generated Files:
- Locate the Python files (.py) generated in your project directory.
- Import these files into your Python application where you intend to interact with the Temporal Cloud API.
2. Use the API:
- Use the classes and methods defined in the imported files to communicate with the Temporal Cloud services.
- Ensure to handle any required authentication or configuration as needed for Temporal Cloud.
This approach can be adapted for other programming languages by following their respective import and usage conventions for the generated code files.
## Usage guidelines
When interacting with the Temporal Cloud Ops API, follow these guidelines:
- API version header:
- Always include the `temporal-cloud-api-version` header in your requests, specifying the API version identifier.
- The current API version can be found [here](https://github.com/temporalio/cloud-api/blob/main/VERSION#L1C1-L1C14).
- Connection URL:
- Connect to the Temporal Cloud using the gRPC URL: `saas-api.tmprl.cloud:443`.
- Engagement steps:
- Generate API key:
- Obtain an [API Key for authentication](/cloud/api-keys#manage-api-keys). Note that many operations may require Admin privileges.
- Set up client:
- Establish a secure connection to the Temporal Cloud. Refer to the example [Client setup in Go](https://github.com/temporalio/cloud-samples-go/blob/main/client/temporal/client.go) for guidance.
- Execute operations:
- For operation specifics, refer to the `cloudservice/v1/request_response.proto` for gRPC messages and `cloudservice/v1/service.proto` for gRPC services.
These steps provide a structured approach to using the Temporal Cloud Ops API effectively, ensuring proper authentication and connection setup.
## Rate limits
The Temporal Cloud Operations API implements rate limiting to ensure system stability and fair usage across all users. Rate limits are applied based on identity type, with different limits for users and service accounts.
### Account-level rate limit
**Total rate limit: 160 requests per second (RPS)**
This limit applies to all requests made to the Temporal Cloud control plane by any client (tcld, UI, Cloud Ops API) or identity type (user, service account) within your account. The total account throughput cannot exceed the limit regardless of the number of users or service accounts making requests.
### Per-identity rate limits
**User rate limit: 40 RPS per user**
This limit applies to all requests made by each user through any client (tcld, UI, Cloud Ops API), regardless of the authentication method used (SSO or API keys).
**Service account rate limit: 80 RPS per service account**
This limit applies to all requests made by each service account through any client (tcld, Cloud Ops API).
### Important considerations
- Rate limits are enforced across all Temporal Cloud control plane operations
- Multiple clients used by the same identity (user or service account) share the same rate limit
- Authentication method (SSO, API keys) does not affect rate limiting
- These limits help ensure system stability and prevent any single account or identity from overwhelming the service
### Request limit increases
If your use case requires higher rate limits, you can request an increase by [submitting a support ticket](/cloud/support#support-ticket). When requesting a limit increase, please provide:
- Your current usage patterns and requirements
- The specific limits you need increased
- A description of your use case and why higher limits are necessary
### Provide feedback
Your input is valuable!
You can provide feedback through the following channels:
- Submit request or feedback through a [support ticket](/cloud/support#support-ticket)
- Open an issue in the [GitHub Repo](https://github.com/temporalio/cloud-api)
---
## Awsregions
### Asia Pacific - Tokyo (`ap-northeast-1`)
- **Cloud API Code**: `aws-ap-northeast-1`
- **Regional Endpoint**: `aws-ap-northeast-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ap-northeast-2`
- `aws-ap-south-1`
- `aws-ap-south-2`
- `aws-ap-southeast-1`
- `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
- `gcp-asia-south1`
### Asia Pacific - Seoul (`ap-northeast-2`)
- **Cloud API Code**: `aws-ap-northeast-2`
- **Regional Endpoint**: `aws-ap-northeast-2.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ap-northeast-1`
- `aws-ap-south-1`
- `aws-ap-south-2`
- `aws-ap-southeast-1`
- `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
- `gcp-asia-south1`
### Asia Pacific - Mumbai (`ap-south-1`)
- **Cloud API Code**: `aws-ap-south-1`
- **Regional Endpoint**: `aws-ap-south-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ap-northeast-1`
- `aws-ap-northeast-2`
- `aws-ap-south-2`
- `aws-ap-southeast-1`
- `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
- `gcp-asia-south1`
### Asia Pacific - Hyderabad (`ap-south-2`)
- **Cloud API Code**: `aws-ap-south-2`
- **Regional Endpoint**: `aws-ap-south-2.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ap-northeast-1`
- `aws-ap-northeast-2`
- `aws-ap-south-1`
- `aws-ap-southeast-1`
- `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
- `gcp-asia-south1`
### Asia Pacific - Singapore (`ap-southeast-1`)
- **Cloud API Code**: `aws-ap-southeast-1`
- **Regional Endpoint**: `aws-ap-southeast-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ap-northeast-1`
- `aws-ap-northeast-2`
- `aws-ap-south-1`
- `aws-ap-south-2`
- `aws-ap-southeast-2`
- **Multi-Cloud Replication**:
- `gcp-asia-south1`
### Asia Pacific - Sydney (`ap-southeast-2`)
- **Cloud API Code**: `aws-ap-southeast-2`
- **Regional Endpoint**: `aws-ap-southeast-2.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ap-northeast-1`
- `aws-ap-northeast-2`
- `aws-ap-south-1`
- `aws-ap-south-2`
- `aws-ap-southeast-1`
- **Multi-Cloud Replication**:
- `gcp-asia-south1`
### Europe - Frankfurt (`eu-central-1`)
- **Cloud API Code**: `aws-eu-central-1`
- **Regional Endpoint**: `aws-eu-central-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-eu-west-1`
- `aws-eu-west-2`
- **Multi-Cloud Replication**:
- `gcp-europe-west3`
### Europe - Ireland (`eu-west-1`)
- **Cloud API Code**: `aws-eu-west-1`
- **Regional Endpoint**: `aws-eu-west-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-eu-central-1`
- `aws-eu-west-2`
- **Multi-Cloud Replication**:
- `gcp-europe-west3`
### Europe - London (`eu-west-2`)
- **Cloud API Code**: `aws-eu-west-2`
- **Regional Endpoint**: `aws-eu-west-2.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-eu-central-1`
- `aws-eu-west-1`
- **Multi-Cloud Replication**:
- `gcp-europe-west3`
### North America - Central Canada (`ca-central-1`)
- **Cloud API Code**: `aws-ca-central-1`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d`
- **Regional Endpoint**: `aws-ca-central-1.region.tmprl.cloud`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-us-east-1`
- `aws-us-east-2`
- `aws-us-west-2`
- **Multi-Cloud Replication**:
- `gcp-us-central1`
- `gcp-us-west1`
- `gcp-us-east4`
### North America - Northern Virginia (`us-east-1`)
- **Cloud API Code**: `aws-us-east-1`
- **Regional Endpoint**: `aws-us-east-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f`
- **Same Region Replication**: Available
- **Multi-Region Replication**:
- `aws-ca-central-1`
- `aws-us-east-2`
- `aws-us-west-2`
- **Multi-Cloud Replication**:
- `gcp-us-central1`
- `gcp-us-west1`
- `gcp-us-east4`
### North America - Ohio (`us-east-2`)
- **Cloud API Code**: `aws-us-east-2`
- **Regional Endpoint**: `aws-us-east-2.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `aws-ca-central-1`
- `aws-us-east-1`
- `aws-us-west-2`
- **Multi-Cloud Replication**:
- `gcp-us-central1`
- `gcp-us-west1`
- `gcp-us-east4`
### North America - Oregon (`us-west-2`)
- **Cloud API Code**: `aws-us-west-2`
- **Regional Endpoint**: `aws-us-west-2.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94`
- **Same Region Replication**: Available
- **Multi-Region Replication**:
- `aws-ca-central-1`
- `aws-us-east-1`
- `aws-us-east-2`
- **Multi-Cloud Replication**:
- `gcp-us-central1`
- `gcp-us-west1`
- `gcp-us-east4`
### South America - São Paulo (`sa-east-1`)
- **Cloud API Code**: `aws-sa-east-1`
- **Regional Endpoint**: `aws-sa-east-1.region.tmprl.cloud`
- **PrivateLink Endpoint Service**: `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- None
- **Multi-Cloud Replication**:
- None
---
## Gcpregions
### North America - Iowa (`us-central1`)
- **Cloud API Code**: `gcp-us-central1`
- **Regional Endpoint**: `gcp-us-central1.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `gcp-us-west1`
- `gcp-us-east4`
- **Multi-Cloud Replication**:
- `aws-ca-central-1`
- `aws-us-east-1`
- `aws-us-east-2`
- `aws-us-west-2`
### North America - Oregon (`us-west1`)
- **Cloud API Code**: `gcp-us-west1`
- **Regional Endpoint**: `gcp-us-west1.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `gcp-us-central1`
- `gcp-us-east4`
- **Multi-Cloud Replication**:
- `aws-ca-central-1`
- `aws-us-east-1`
- `aws-us-east-2`
- `aws-us-west-2`
### North America - Northern Virginia (`us-east4`)
- **Cloud API Code**: `gcp-us-east4`
- **Regional Endpoint**: `gcp-us-east4.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- `gcp-us-central1`
- `gcp-us-west1`
- **Multi-Cloud Replication**:
- `aws-ca-central-1`
- `aws-us-east-1`
- `aws-us-east-2`
- `aws-us-west-2`
### Europe - Frankfurt (`europe-west3`)
- **Cloud API Code**: `gcp-europe-west3`
- **Regional Endpoint**: `gcp-europe-west3.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- None
- **Multi-Cloud Replication**:
- `aws-eu-central-1`
- `aws-eu-west-1`
- `aws-eu-west-2`
### Asia Pacific - Mumbai (`asia-south1`)
- **Cloud API Code**: `gcp-asia-south1`
- **Regional Endpoint**: `gcp-asia-south1.region.tmprl.cloud`
- **Private Service Connect Service Attachment URI**: `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw`
- **Same Region Replication**: Not Available
- **Multi-Region Replication**:
- None
- **Multi-Cloud Replication**:
- `aws-ap-northeast-1`
- `aws-ap-northeast-2`
- `aws-ap-south-1`
- `aws-ap-south-2`
- `aws-ap-southeast-1`
- `aws-ap-southeast-2`
---
## Private Service
| Region | Private Service Connect Service Name |
| ---------------------- | ----------------------------------------------------------------------------------------- |
| `asia-south1` | `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw` |
| `us-central1` | `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzn` |
| `us-west1 ` | `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x` |
---
## RPO and RTO
When a cloud outage disrupts a Namespace, Temporal Cloud takes measures to maintain the Namespace's availability and data durability. The time it takes to recover from the outage is called the "recovery time." The amount of data (event histories) lost is called the "recovery point." A durable system should have a low recovery time and recovery point.
To help users plan for keeping critical Workflows available during a cloud outage, Temporal Cloud publishes goals for the recovery time and recovery point for each kind of outage. These goals are called the Recovery Time Objective (RTO) and Recovery Point Objective (RPO). These objectives are complementary to Temporal Cloud's [Service Level Agreement (SLA)](/cloud/sla).
To achieve the lowest RPO and RTO, Temporal Cloud offers [High Availability](/cloud/high-availability) features that keep Workflows operational with minimal downtime. When High Availability is enabled on a Namespace, the user chooses a region to place a "replica" that will take over in the event of a failure. The location of the replica determines the type of replication used and the type of outages that can be handled. Multi-region Replication is when the active and replica are in different regions on the same cloud (e.g., AWS us-east-1 and AWS us-west-2). Multi-cloud Replication is when the active and replica are in different clouds (e.g., AWS and GCP). Same-region Replication is when the active and replica are in the same region. Temporal always places the active and replica in different [cells](/cloud/overview#cell-based-infrastructure).
As Workflows progress in the active region, history events are asynchronously replicated to the replica.
Because replication is asynchronous, High Availability does not impact the latency or throughput of Workflow Executions in the active region.
If an outage hits the active region or cell, Temporal Cloud will fail over to the replica so that existing Workflow Executions will continue to run and new Workflow Executions can be started.
The Recovery Point Objective and Recovery Time Objective for Temporal Cloud depend on the type of outage and which [High Availability](/cloud/high-availability) feature your Namespace has enabled. Temporal Cloud can only set an RPO and RTO for cases where it has the ability to mitigate the outage. Therefore, the below RPOs and RTOs apply to Namespaces that have the corresponding type of replication and have enabled Temporal-initiated failovers, which comes enabled by default.
1. **Availability zone outage**:
1. _Applicable Namespaces:_ All Namespaces
2. _Goals:_ Zero RPO and near-zero RTO
3. _More details:_ Historically, these have been the most common type of outage in the cloud. Temporal Cloud replicates every Namespace across three availability zones. The failure of a single availability zone is handled automatically by Temporal Cloud behind the scenes, with no potential for data loss, and little-to-no observable downtime to the end user.
2. **Cell outage**:
1. _Applicable Namespaces:_ Namespaces with Same-region Replication, Multi-region Replication, or Multi-cloud Replication
2. _Goals:_ 1-minute RPO and 20-minute RTO
3. _More details:_ Temporal Cloud runs on a [cell architecture](/cloud/sla). Each cell contains the software and services necessary to host a Namespace. While unlikely, it's possible for a cell to experience a disruption due to uncaught software bugs or sub-component failures (e.g., an outage in the underlying database).
3. **Regional outage**:
1. _Applicable Namespaces:_ Namespaces with Multi-region Replication or Multi-cloud Replication
2. _Goals:_ 1-minute RPO and 20-minute RTO
3. _More details:_ On [rare occasions](https://temporal.io/blog/how-devs-kept-running-during-the-aws-us-east-1-oct-20-2025), an entire region within a cloud provider will be degraded. Since Namespaces depend on the cloud provider's infrastructure, Temporal Cloud is not immune to these outages.
4. **Cloud-wide outage**:
1. _Applicable Namespaces:_ Namespaces with Multi-cloud Replication
2. _Goals:_ 1-minute RPO and 20-minute RTO
3. _More details:_ An entire cloud provider has an outage across most or all regions. Since cloud providers strive to keep cloud regions de-coupled, these are the rarest outages of all. Still, they [have happened](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW) in the past.
Notes:
- The above goals are only applicable to Namespaces that have enabled Temporal-initiated failovers, which comes enabled by default. Temporal-initiated failovers are initiated by Temporal's tooling and/or on-call engineers without user action. Users can always initiate a failover on their Namespace, even when Temporal-initiated failovers are enabled. In an outage, a user-initiated failover will not cancel out or accidentally reverse a Temporal-initiated failover.
:::note
Temporal highly recommends keeping Temporal-initiated failovers enabled. When Temporal-initiated failovers are _disabled,_ Temporal Cloud cannot set an RPO and RTO for that Namespace, because it cannot control when or if the user will trigger a failover.
:::
- The above goals are for unplanned cloud outages. They do not apply to user-initiated failovers during healthy periods (e.g., for DR drills). Read about [triggering a failover](/cloud/high-availability/failovers) to see how a Namespace failover should perform during healthy periods.
- As soon as a cloud outage resolves, Temporal's on-call engineers will work to restore service to Namespaces that were not protected by High Availability. A cloud outage can leave lingering effects in Temporal's systems and applications, even after the cloud provider restores the underlying service. Because of this, affected Namespaces may not be immediately available when the underlying service is restored. An affected Namespace's outage may last longer than the cloud provider's outage.
- All Namespaces are backed up every 4 hours. If an outage causes data loss on a Namespace that was not protected by High Availability, then Temporal will use the backup to restore as much data as feasible.
## Minimizing the Recovery Point
Temporal has put extensive work into tools and processes that minimize the recovery point and achieve its RPO for Temporal-initiated failovers, including:
- Best-in-class [data replication technology](https://youtu.be/mULBvv83dYM?si=RDeWb3gVsEtgGM4z&t=334) that keeps the replica up to date with the active.
- Monitoring, alerting, and internal SLOs on the replication lag for every Temporal Cloud Namespace.
However, user actions on a Namespace can affect the recovery point. For example, suddenly spiking into much higher throughput than a Namespace has seen before could create a period of replication lag where the replica falls behind the active.
Temporal provides a [replication lag](/cloud/high-availability/monitoring#replication-lag-metric) metric for each Namespace. This metric approximates the recovery point the Namespace would achieve in a worst case failure at that given moment.
:::note
Temporal recommends monitoring the replication lag and alerting should it rise too high, e.g., above 1 minute.
:::
## Minimizing the Recovery Time
Temporal has put extensive work into tools and processes that minimize the recovery time and achieve its RTO for Temporal-initiated failovers, including:
- History events are replicated _asynchronously_. This ensures that the Namespace can still run workflows in the active region even if there are networking blips or outages with the replica region.
- Outages are detected automatically. We have extensive internal alerting to detect disruptions to Namespaces, and are ever improving this system.
- Battle-tested Temporal Workflows that execute failovers of all Temporal Cloud Namespaces in a given region quickly.
- Regular drills where we fail over our internal Namespaces to test our tooling.
- Expert engineers on-call 24/7 monitoring Temporal Cloud Namespaces and ready to assist should an outage occur.
To achieve the lowest possible recovery times, Temporal recommends that you:
- Keep Temporal-initiated failovers enabled on your Namespace (the default)
- Invest in a process to detect outages and trigger a manual failover.
Users can trigger manual failovers on their Namespaces even if Temporal-initiated failovers are enabled. There are several benefits to combining a manual failover process with Temporal-initiated failovers:
- You can detect outages that Temporal doesn't. In the cloud, regional outages don't affect all services equally. It's possible that Temporal--and the services it depends on--are unaffected by the outage, even while your Workers or other cloud infrastructure are disrupted. If you [monitor services in your critical path](https://sre.google/sre-book/monitoring-distributed-systems/) and alert on unusual error rates, you may catch outages before Temporal Cloud does.
- You can sequence your failovers in a particular order. Your cloud infrastructure probably contains more pieces than just your Temporal Namespace: Temporal Workers, compute pools, data stores, and other cloud services. If you manually fail over, you can choose the order in which these pieces switch to the replica region. You can then test that ordering with failover drills and ensure it executes smoothly without data consistency issues or bottlenecks.
- You can proactively fail over more aggressively than Temporal. While the 20-minute RTO should be sufficient for most use cases, some may strive to hit an even lower RTO. For workloads like high frequency trading, auctions, or popular sporting events, an outage at the wrong time could cause tremendous lost revenue per minute. You can adopt a posture that fails over more eagerly than Temporal does. For example, you could trigger a manual failover at the first sign of a possible disruption, before knowing whether there's a true regional outage.
- Even if you have robust tooling to detect an outage and trigger a failover, leaving Temporal-initiated failovers enabled provides a "safety net" in case your automation misses an outage. It also gives Temporal leeway to preemptively fail over your Namespace if we detect that it may be disrupted soon, e.g., by a rolling failure that has impacted other Namespaces but not yours, yet.
## Understanding Temporal's RTO vs. SLA
Temporal has both a Recovery Time Objective (RTO) and a Service Level Agreement (SLA). They serve complementary purposes and apply in different situations.
| Aspect | RTO | SLA |
|-----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| What is it? | An objective, or high-priority goal, for the total time that an outage disrupts a Namespace. | A contractual agreement that sets an upper bound on the service error rate, with financial repercussions. |
| How is it measured? | The achieved recovery time is measured in terms of minutes per outage. | The achieved service error rate is measured in terms of error rate per month. |
| How is the calculation performed? | The achieved recovery time in a given outage is the total time between when a disruption to a Namespace began and when the Namespace was restored to full functionality, either after a failover to a healthy region or after the outage has been mitigated. | Temporal measures the percentage of requests to Temporal Cloud that fail, and applies a [formula](/cloud/sla) to get the final percentage for the month. |
| Do partial degradations count? | Most outages contain periods of __partial degradation__ where some percentage of Namespace operations fail while the rest complete as normal. When they disrupt a Namespace, periods of partial degradation count in the calculation of the recovery time. | Partial degradations only partially count for the service error rate calculation. A 5-minute window with a 10% error rate would count less than a 5-minute window with a 100% error rate. |
| What is excluded? | For partial degradations, what counts as a disruption to a Namespace is subject to Temporal's expert judgment, but a good rule of thumb is a service error rate >=10%. | We exclude outages that are out of Temporal's control to mitigate, e.g., a failure of the underlying cloud provider infrastructure that affects a Namespace without High Availability and Temporal-initiated failovers enabled. If a Namespace has the relevant High Availability feature and has Temporal-initiated failovers enabled, then Temporal can act to mitigate the outage and it does usually count against the SLA. Full exclusions on the [SLA page](/cloud/sla). |
The following examples illustrate the RTO and SLA calculations for different types of in a regional outage. These hypothetical Namespaces are based on actual Temporal Cloud performance in a [real-world outage](https://temporal.io/blog/how-devs-kept-running-during-the-aws-us-east-1-oct-20-2025).
Suppose that region `middle-earth-1` experienced a cascading failure starting at 10:00:00 UTC, causing various instances and machines to fail over time. Temporal's automatic failover triggered for all Namespaces and completed at 10:15:00 UTC.
- Namespace 0 was in the region but its cell was not affected by the outage. The only downtime it had was for a few seconds during the failover operation. It experienced a near-zero Recovery Time, and its service error rate was neglible. Graceful failover was successful, and this Namespace achieved a recovery point of 0.
- Namespace 1_A was in the region and its cell experienced a partial degradation that caused 10% of requests to fail in the first 5 minutes, 25% in the second five minutes, and 50% in the third five minutes. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month, then its service error rate for the month would be: ( (1 - 10%) + (1 - 25%) + (1 - 50%) + 8925 * 100% ) / 8928 = 99.990%. (Note: there are 8928 5-minute periods in a 31-day month.) Graceful failover was successful, and this Namespace achieved a recovery point of 0.
- Namespace 1_B was in the same cell as Namespace 2_A, so it also experienced a partial degradation that caused 10% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually fail over at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 * (1 - 10%) + 8927 * 100% ) / 8928 = 99.998%. Graceful failover was successful, and this Namespace achieved a recovery point of 0.
- Namespace 2_A was in the region and its cell was fully network partitioned at the start of the outage, causing 100% of requests to fail. Since it was significantly impacted from 10:00:00 to 10:15:00, its Recovery Time was 15 minutes. If it had no other service errors that month, then its service error rate for the month would be: ( 3 * (1 - 100%) + 8928 * 100% ) / 8640 5-minute periods per month = 99.97%. Because the Namespace was network partitioned, graceful failover did not succeed, and forced failover was used. The recovery point achieved was equal to the replication lag at the time of the network partition, which was a few seconds.
- Namespace 2_B was in the region and was fully network partitioned, causing 100% of requests to fail. However, its owner detected the outage via their own tooling and decided to manually fail over at 10:05:00. This Namespace achieved a recovery time of 5 minutes and a service error rate of ( 1 5-minute periods * (1 - 100%) + 8639 5-minute periods * 100% ) / 8640 5-minute periods per month = 99.99%. Because the Namespace was network partitioned, graceful failover did not succeed, and forced failover was used. The recovery point achieved was equal to the replication lag at the time of the network partition, which was a few seconds.
All of the above Namespaces were in the affected region and beat the 1-minute RPO. But they achieved varying recovery times and service error rates.
- Notice how Namespace 1_A and Namespace 2_A were both automatically failed over with **the same recovery time but different service error rates**. Notice how Namespace 2_B and Namespace 1_A happen to have the **same service error rate but different recovery times**. This illustrates how RTO and SLA can differ, even in the same outage. Both are valuable tools for Temporal Cloud users to measure the availability of their Namespaces.
- Notice how the Namespaces that were manually failed over (Namespace 1_B and Namespace 2_B) achieved lower recovery times than the Namespaces that were automatically failed over (Namespace 1_A and Namespace 2_A). This illustrates how **proactive, aggressive manual failover can achieve a better recovery time than automatic failover**.
---
## SAML authentication
To authenticate the users of your Temporal Cloud account, you can connect an identity provider (IdP) to your account by
using Security Assertion Markup Language (SAML) 2.0.
:::info
SAML is a paid feature. See the [pricing page](/cloud/pricing) for details.
:::
## Integrate SAML with your Temporal Cloud account
1. Locate your [Temporal Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id). Your Account Id can be viewed
and copied from the Temporal Cloud user profile dropdown menu in the top right corner. Alternatively, find your
[Namespace Id](/cloud/namespaces#temporal-cloud-namespace-id). The Account Id is the five or six characters following
the period (.), such as `f45a2`. You will need the Account Id to construct your callback URL and your entity
identifier.
1. Configure SAML with your IdP by following one of these sets of instructions:
- [Microsoft Entra ID](#configure-saml-with-azure-ad)
- [Okta](#configure-saml-with-okta)
1. [Share your connection information with us and test your connection.](#finish-saml-configuration)
## How to configure SAML with Microsoft Entra ID {#configure-saml-with-azure-ad}
If you want to use the general Microsoft login mechanism, you don't need to set up SAML with Entra ID. Just select
**Continue with Microsoft** on the Temporal Cloud sign-in page.
To use Entra ID as your SAML IdP, create a Microsoft Entra ID Enterprise application.
1. Sign in to the [Microsoft Entra ID](https://portal.azure.com/).
1. On the home page, under **Manage Microsoft Entra ID**, select **View**.
1. On the **Overview** page near the top, select **Add > Enterprise application**.
1. On the **Browse Microsoft Entra ID Gallery** page near the top, select **Create your own application**.
1. In the **Create your own application** pane, provide a name for your application (such as `temporal-cloud`) and
select **Integrate any other application you don't find in the gallery**.
1. Select **Save**.
1. In the **Getting Started** section, select **2. Set up single sign on**.
1. On the **Single sign-on** page, select **SAML**.
1. In the **Basic SAML Configuration** section of the **SAML-based Sign-on** page, select **Edit**.
1. In **Identifier (Entity ID)**, enter the following entity identifier, including your Account Id where indicated:
```bash
urn:auth0:prod-tmprl:ACCOUNT_ID-saml
```
A correctly formed entity identifier looks like this:
```bash
urn:auth0:prod-tmprl:f45a2-saml
```
1. In **Reply URL (Assertion Consumer Service URL)**, enter the following callback URL, including your Account Id where
indicated:
```bash
https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml
```
A correctly formed callback URL looks like this:
```bash
https://login.tmprl.cloud/login/callback?connection=f45a2-saml
```
1. In **Sign on URL**, enter the following login url, including your Account Id where indicated:
```bash
https://cloud.temporal.io/login/saml?connection=ACCOUNT_ID-saml
```
A correctly formed login URL looks like this:
```bash
https://cloud.temporal.io/login/saml?connection=f45a2-saml
```
1. You can leave the other fields blank. Near the top of the pane, select **Save**.
1. In the **Attributes & Claims** section, select **Edit**. Configure the following settings. Under **Required claim**:
- Set **Unique User Identifier (NameID)** to `user.userprincipalname`
- Set the **NameID format** to `emailAddress`
These are the default settings for Microsoft Entra ID. Then under **Additional claims**, ensure **Email** and
**Name** are present.
1. Collect information that you need to send to us:
- In the **SAML Certificates** section of the **SAML-based Sign-on** page, select the download link for **Certificate
(Base64)**.
- In the **Set up _APPLICATION_NAME_** section of the **SAML-based Sign-on** page, copy the value of **Login URL**.
To finish setting up Microsoft Entra ID as your SAML IdP, see [Finish SAML configuration](#finish-saml-configuration).
## How to configure SAML with Okta {#configure-saml-with-okta}
To use Okta as your SAML IdP, configure a new Okta application integration.
1. Sign in to the [Okta Admin Console](https://www.okta.com/login/).
1. In the left navigation pane, select **Applications > Applications**.
1. On the **Applications** page, select **Create App Integration**.
1. In the **Create a new app integration** dialog, select **SAML 2.0** and then select **Next**.
1. On the **Create SAML Integration** page in the **General Settings** section, provide a name for your application
(such as `temporal-cloud`) and then select **Next**.
1. In the **Configure SAML** section in **Single sign on URL**, enter the following callback URL, including your Account
Id where indicated:
```bash
https://login.tmprl.cloud/login/callback?connection=ACCOUNT_ID-saml
```
A correctly formed callback URL looks like this:
```bash
https://login.tmprl.cloud/login/callback?connection=f45a2-saml
```
1. In **Audience URI (SP Entity ID)**, enter the following entity identifier, including your Account Id where indicated:
```bash
urn:auth0:prod-tmprl:ACCOUNT_ID-saml
```
A correctly formed entity identifier looks like this:
```bash
urn:auth0:prod-tmprl:f45a2-saml
```
1. We require the user's full email address when connecting to Temporal.
- In **Name ID format**, select `EmailAddress`.
- In **Attribute Statements**, set **email** and **name**.
1. Select **Next**.
1. In the **Feedback** section, select **Finish**.
1. On the **Applications** page, select the name of the application integration you just created.
1. On the application integration page, select the **Sign On** tab.
1. Under **SAML Setup**, select **View SAML setup instructions**.
1. Collect information that you need to send to us:
- Copy the IdP settings.
- Download the active certificate.
To finish setting up Okta as your SAML IdP, see the next section,
[Finish SAML configuration](#finish-saml-configuration).
## How to finish your SAML configuration {#finish-saml-configuration}
After you configure SAML with your IdP, we can finish the configuration on our side.
[Create a support ticket](/cloud/support#support-ticket) that includes the following information:
- The sign-in URL from your application
- The X.509 SAML sign-in certificate in PEM format
- One or more IdP domains to map to the SAML connection
Generally, the provided IdP domain is the same as the domain for your email address. You can provide multiple IdP
domains.
When you receive confirmation from us that we have finished configuration, log in to Temporal Cloud. This time, though,
enter your email address in **Enterprise identity** and select **Continue**. Do not select **Continue with Google** or
**Continue with Microsoft**. You will be redirected to the authentication page of your IdP.
---
## SCIM user management
[SCIM](https://scim.cloud/) lets you integrate your identity provider (IdP) with Temporal Cloud to automate user provisioning and access. Once SCIM is configured, changes in your IdP are automatically reflected in Temporal Cloud, including:
- User creation / onboarding
- User deletion / offboarding
- User membership in groups
You can map SCIM groups to Temporal Cloud [roles and permissions](/cloud/users#account-level-roles-and-namespace-level-permissions), so users automatically get the Temporal Cloud access they need based on the groups they belong to.
:::info
SCIM is a paid feature. See the [pricing page](/cloud/pricing) for details.
:::
## Supported IdP Vendors
Supported upstream IdP vendors include:
* [Okta](#configure-scim-with-okta)
* Microsoft Entra ID (Azure AD)
* Google Workspace
* OneLogin
* CyberArk
* JumpCloud
* PingFederate
* Any SCIM 2.0-compliant provider
## Preparing for SCIM
Before starting your work with SCIM, you'll need to complete this checklist:
1. Configure [SAML](/cloud/saml) SSO.
1. Identify your organization's **IdP administrator**, who is responsible for configuring and managing your SCIM integration.
Specify their contact details when you reach out to support in the next stage of this process.
After completing these steps, you're ready to submit your [support ticket](/cloud/support#support-ticket) to enable SCIM.
:::tip Adding and removing users
When SCIM is enabled for user management, you can still add and remove users outside of SCIM using the Temporal Cloud interface, until you disable user lifecycle management.
You can always change a user's or group's Account Role from the Temporal Cloud interface.
:::
## Onboarding with SCIM and Okta {#configure-scim-with-okta}
1. Temporal Support enables the SCIM integration on your account.
Enabling integration automatically emails a configuration link to your Okta administrator.
This authorizes them to set up the integration.
1. Your Okta administrator opens the supplied link.
The link leads to step-by-step instructions for configuring the integration.
1. Once configured in Okta, Temporal Cloud will begin to receive SCIM messages and automatically onboard and offboard the users and groups configured in Okta.
Some points to note:
- User and group change events are applied within 10 minutes of them being made in Okta.
- User lifecycle management with SCIM also allows user roles to be derived from group membership.
- Once a group has been synced in Temporal Cloud, you can use `tcld` to assign roles to the group.
For instructions, see the [User Group Management](https://github.com/temporalio/tcld?tab=readme-ov-file#user-group-management) page.
---
## Monitor Temporal Cloud
Temporal Cloud metrics help monitor production deployments.
This documentation covers best practices for monitoring Temporal Cloud.
## Monitor availability issues
When you see a sudden drop in Worker resource utilization, verify whether Temporal Cloud's API is showing increased latency and error rates.
### Reference Metrics
- [temporal\_cloud\_v1\_service\_latency\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_latency_p99)
This metric measures latency for `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `StartWorkflowExecution` operations.
These operations are mission critical and never [throttled](/cloud/service-availability#throughput).
This metric is a good indicator of your lowest possible latency for the 99th percentile of requests.
## Monitor Temporal Service errors
Check for Temporal Service gRPC API errors.
Note that Service API errors are not equivalent to guarantees mentioned in the [Temporal Cloud SLA](/cloud/sla).
### Reference Metrics
- [temporal\_cloud\_v1\_frontend\_service\_error\_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_error_count)
- [temporal\_cloud\_v1\_frontend\_service\_request\_count](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_service_request_count)
### Prometheus Query for this Metric
Measure your daily average errors over 10-minute windows:
```
avg_over_time((
(
(
sum(increase(temporal_cloud_v1_service_request_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"}[10m]))
-
sum(increase(temporal_cloud_v1_service_error_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"}[10m]))
)
/
sum(increase(temporal_cloud_v1_service_request_count{temporal_namespace=~"$namespace", operation=~"StartWorkflowExecution|SignalWorkflowExecution|SignalWithStartWorkflowExecution|RequestCancelWorkflowExecution|TerminateWorkflowExecution"}[10m]))
)
or vector(1)
)[1d:10m])
```
## Detecting Activity and Workflow Failures
The metrics `temporal_activity_execution_failed` and `temporal_cloud_v1_workflow_failed_count` together provide failure detection for Temporal applications. These metrics work in tandem to give you both granular component-level visibility and high-level workflow health insights.
Note that `temporal_activity_execution_failed` is an SDK metric that must be collected from the Worker.
### Activity failure cascade
If not using infinite retry policies, Activity failures can lead to Workflow failures:
```
Activity Failure --> Retry Logic --> More Activity Failures --> Workflow Decision --> Potential Workflow Failure
```
Activity failures are often recoverable and expected. Workflow failures represent terminal states requiring immediate attention.
A spike in activity failures may precede workflow failures.
Generally Temporal recommends that Workflows should be designed to always succeed. If an Activity fails more than its retry policy allows, we suggest having the Workflow handle Activity failure and take action to notify a human to take corrective action or be aware of the error.
### Ratio-based monitoring
#### Failure conversion rate
Monitor the ratio of workflow failures to activity failures:
```
workflow_failure_rate = temporal_cloud_v1_workflow_failed_count / temporal_activity_execution_failed
```
What to watch for:
- High ratio (greater than 0.1): Poor error handling - activities failing are causing workflow failures
- Low ratio (less than 0.01): Good resilience - activities fail but workflows recover
- Sudden spikes: May indicate systematic issues
#### Activity success rate
```
activity_success_rate = (total_activities - temporal_activity_execution_failed) / total_activities
```
Target: >95% for most applications. Lower success rate can be a sign of system troubles.
See also:
- [Crafting an Error Handling Strategy](https://learn.temporal.io/courses/errstrat/)
- [Temporal Failures reference](/references/failures)
- [Detecting Workflow failures](/encyclopedia/detecting-workflow-failures)
## Monitor replication lag for Namespaces with High Availability features
Replication lag refers to the transmission delay of Workflow updates and history events from the primary Namespace to the replica.
Always check the [metric replication lag](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99) before initiating a failover.
A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress.
**Who owns the replication lag?**
Temporal owns replication lag.
**What guarantees are available?**
There is no SLA for replication lag.
Temporal recommends that customers do not trigger failovers except for testing or emergency situations.
High Availability feature's four-9 guarantee SLA means Temporal will handle failovers and ensure high availability.
Temporal also monitors replication lag.
Customer who decide to trigger failovers should look at this metric before moving forward.
**If the lag is high, what should you do?**
We don't expect users to failover.
Please contact Temporal support if you feel you have a pressing need.
**Where can you read more?**
See [operations and metrics](/cloud/high-availability) for Namespaces with High Availability features.
### Reference Metrics
- [temporal\_cloud\_v1\_replication\_lag\_p99](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p99)
- [temporal\_cloud\_v1\_replication\_lag\_p95](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p95)
- [temporal\_cloud\_v1\_replication\_lag\_p50](/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_replication_lag_p50)
## Detecting Resource Exhaustion
The Cloud metric `temporal_cloud_v1_resource_exhausted_error_count` is the primary indicator for Cloud-side throttling, signaling system limits
are exceeded and `ResourceExhausted` gRPC errors are occurring. This generally does not break workflow processing due to how resources are prioritized.
Persistent non-zero values of this metric are unexpected.
## Monitoring Trends Against Limits {#rps-aps-rate-limits}
The set of [limit metrics](/cloud/metrics/openmetrics/metrics-reference#limit-metrics) provide a time series of values for limits. Use these
metrics with their corresponding count metrics to monitor general trends against limits and set alerts when limits are exceeded. Use the corresponding throttle metrics
to determine the severity of any active rate limiting.
| Limit Metric | Count Metric | Throttle Metric |
| ------------ | ------------ | --------------- |
| `temporal_cloud_v1_action_limit` | `temporal_cloud_v1_total_action_count` | `temporal_cloud_v1_total_action_throttled_count` |
| `temporal_cloud_v1_service_request_limit` | `temporal_cloud_v1_service_request_count` | `temporal_cloud_v1_service_request_throttled_count` |
| `temporal_cloud_v1_operations_limit` | `temporal_cloud_v1_operations_count` | `temporal_cloud_v1_operations_throttled_count` |
The [Grafana dashboard example](https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json) includes a Usage & Quotas section
that creates demo charts for these limits and count metrics respectively.
The limit metrics, throttle metrics, and count metrics are already directly comparable as per second rates. Keep in mind that each `count` metric is represented as a per second rate averaged
over each minute. For example, to get the total count of Actions, you must multiply this metric by 60.
When setting alerts against limits, consider if your workload is spiky or sensitive to throttling (e.g. does latency matter?). If your workload is sensitive, consider alerting
for `temporal_cloud_v1_total_action_count` at a 50% threshold of the `temporal_cloud_v1_action_limit`. If your workload is not sensitive, consider an alert at 90% of this threshold
or directly when throttling is detected as a value greater than zero for `temporal_cloud_v1_total_action_throttled_count`. This logic can also be used to automatically scale [Temporal
Resource Units](/cloud/capacity-modes#provisioned-capacity) up or down as needed. Some workloads choose to exceed limits and accept throttling because they are not latency sensitive.
---
## tcld account command reference
The `tcld account` commands manage accounts in Temporal Cloud.
Alias: `a`
- [tcld account audit-log](#audit-log)
- [tcld account get](#get)
- [tcld account list-regions](#list-regions)
- [tcld account metrics](#metrics)
## audit-log
The `tcld account audit-log` command manage Audit Logs in Temporal Cloud.
Alias: `al`
- [tcld account audit-log kinesis](#kinesis)
- [tcld account audit-log pubsub](#pubsub)
### kinesis
The `tcld account audit-log kinesis` command manages Kinesis audit log sinks.
Alias: `k`
- [tcld account audit-log kinesis create](#create)
- [tcld account audit-log kinesis delete](#delete)
- [tcld account audit-log kinesis get](#account-audit-log-kinesis-get)
- [tcld account audit-log kinesis list](#list)
- [tcld account audit-log kinesis update](#update)
- [tcld account audit-log kinesis validate](#validate)
#### create
The `tcld account audit-log kinesis` command creates a Kinesis audit log sink.
Alias: `c`
##### --destination-uri
The destination URI of the audit log sink.
Alias: `du`
##### --region
The region to use for the request.
Alias: `re`
##### --role-name
The role name to use to write to the sink.
Alias: `rn`
##### --sink-name
Provide a name for the sink.
#### delete
The `tcld account audit-log kinesis delete` command deletes an audit log sink.
Alias: `d`
##### --resource-version
The resource-version (etag) to update from, if not set the cli will use the latest (optional).
Alias: `v`
##### --sink-name
Provide a name for the sink.
#### get {#account-audit-log-kinesis-get}
The `tcld account audit-log kinesis get` command gets an audit log sink.
Alias: `g`
##### --sink-name
Provide a name for the sink.
#### list
The `tcld account audit-log kinesis list` command lists audit log sinks on the account.
Alias: `l`
##### --page-size
The page size for list operations.
##### --page-token
The page token for list operations.
#### update
The `tcld account audit-log kinesis update` command updates an audit log sink.
Alias: `u`
##### --destination-uri
The destination URI of the audit log sink.
Alias: `du`
##### --enabled
Whether the sink is enabled.
##### --region
The region to use for the request.
Alias: `re`
##### --resource-version
The resource-version (etag) to update from, if not set the cli will use the latest (optional).
Alias: `v`
##### --role-name
The role name to use to write to the sink.
Alias: `rn`
##### --sink-name
Provide a name for the sink.
#### validate
The `tcld account audit-log kinesis validate` command verifies Temporal Cloud can write to a Kinesis sink.
Alias: `v`
##### --destination-uri
The destination URI of the audit log sink.
Alias: `du`
##### --region
The region to use for the request.
Alias: `re`
##### --role-name
The role name to use to write to the sink.
Alias: `rn`
##### --sink-name
Provide a name for the sink.
### pubsub
The `tcld account audit-log pubsub` command manages Pub/Sub audit log sinks.
Alias: `ps`
- [tcld account audit-log pubsub create](#create)
- [tcld account audit-log pubsub delete](#delete)
- [tcld account audit-log pubsub get](#account-audit-log-pubsub-get)
- [tcld account audit-log pubsub list](#list)
- [tcld account audit-log pubsub update](#update)
- [tcld account audit-log pubsub validate](#validate)
#### create
The `tcld account audit-log pubsub` command creates a Pub/Sub audit log sink.
Alias: `c`
##### --service-account-email
The service account email to impersonate to write to the sink.
Alias: `sae`
##### --sink-name
Provide a name for the sink.
##### --topic-name
The topic name to write to the sink.
Alias: `tn`
#### delete
The `tcld account audit-log pubsub delete` command deletes an audit log sink.
Alias: `d`
##### --resource-version
The resource-version (etag) to update from, if not set the cli will use the latest (optional).
Alias: `v`
##### --sink-name
Provide a name for the sink.
#### get {#account-audit-log-pubsub-get}
The `tcld account audit-log pubsub get` command gets an audit log sink.
Alias: `g`
##### --sink-name
Provide a name for the sink.
#### list
The `tcld account audit-log pubsub list` command lists audit log sinks on the account.
Alias: `l`
##### --page-size
The page size for list operations.
##### --page-token
The page token for list operations.
#### update
The `tcld account audit-log pubsub update` command updates an audit log sink.
Alias: `u`
##### --enabled
Whether the sink is enabled.
##### --resource-version
The resource-version (etag) to update from, if not set the cli will use the latest (optional).
Alias: `v`
##### --service-account-email
The service account email to impersonate to write to the sink.
Alias: `sae`
##### --sink-name
Provide a name for the sink.
##### --topic-name
The topic name to write to the sink.
Alias: `tn`
#### validate
The `tcld account audit-log pubsub validate` command verifies Temporal Cloud can write to a Pub/Sub sink.
Alias: `v`
##### --service-account-email
The service account email to impersonate to write to the sink.
Alias: `sae`
##### --sink-name
Provide a name for the sink.
##### --topic-name
The topic name to write to the sink.
Alias: `tn`
## get
The `tcld account get` command gets information about the Temporal Cloud account you are logged into.
Alias: `g`
`tcld account get`
The command has no modifiers.
## list-regions
The `tcld account list-regions` lists all regions where the account can provision namespaces.
Alias: `l`
## metrics
The `tcld account metrics` commands configure the metrics endpoint for the Temporal Cloud account that is currently logged in.
Alias: `m`
- [tcld account metrics enable](#enable)
- [tcld account metrics disable](#disable)
- [tcld account metrics accepted-client-ca](#accepted-client-ca)
### accepted-client-ca
The `tcld account metrics accepted-client-ca` commands manage the end-entity certificates for the metrics endpoint of the Temporal Cloud account that is currently logged in.
:::info
The end-entity certificates for the metrics endpoint must chain up to the CA certificate used for the account. For more information, see [Certificate requirements](/cloud/certificates#certificate-requirements).
:::
Alias: `ca`
- [tcld account metrics accepted-client-ca add](#add)
- [tcld account metrics accepted-client-ca list](#list)
- [tcld account metrics accepted-client-ca set](#set)
- [tcld account metrics accepted-client-ca remove](#remove)
#### add
The `tcld account metrics accepted-client-ca add` command adds end-entity certificates to the metrics endpoint of a Temporal Cloud account.
:::info
The end-entity certificates for the metrics endpoint must chain up to the CA certificate used for the account. For more information, see [Certificate requirements](/cloud/certificates#certificate-requirements).
:::
`tcld account metrics accepted-client-ca add --ca-certificate `
Alias: `a`
The following modifiers control the behavior of the command.
##### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request identifier.
Alias: `-r`
**Example**
```bash
tcld account metrics accepted-client-ca add --request-id --ca-certificate
```
##### --resource-version
Specify a resource version (ETag) to update from. If not specified, the latest version is used.
Alias: `-v`
**Example**
```bash
tcld account metrics accepted-client-ca add --resource-version --ca-certificate
```
##### --ca-certificate
_Required modifier unless `--ca-certificate-file` is specified_
Specify a base64-encoded string of a CA certificate PEM file.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-c`
**Example**
```bash
tcld account metrics accepted-client-ca add --ca-certificate
```
##### --ca-certificate-file
_Required modifier unless `--ca-certificate` is specified_
Specify a path to a CA certificate PEM file.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-f`
**Example**
```bash
tcld account metrics accepted-client-ca add --ca-certificate-file
```
#### list
The `tcld account metrics accepted-client-ca list` command lists the end-entity certificates that are currently configured for the metrics endpoint of a Temporal Cloud account.
`tcld account metrics accepted-client-ca list`
Alias: `l`
The command has no modifiers.
#### remove
The `tcld account metrics accepted-client-ca remove` command removes end-entity certificates from the metrics endpoint of a Temporal Cloud account.
`tcld account metrics accepted-client-ca remove --ca-certificate `
Alias: `r`
The following modifiers control the behavior of the command.
##### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request identifier.
Alias: `-r`
**Example**
```bash
tcld account metrics accepted-client-ca remove --request-id --ca-certificate
```
##### --resource-version
Specify a resource version (ETag) to update from. If not specified, the latest version is used.
Alias: `-v`
**Example**
```bash
tcld account metrics accepted-client-ca remove --resource-version --ca-certificate
```
##### --ca-certificate
_Required modifier unless `--ca-certificate-fingerprint` or `--ca-certificate-file` is specified_
Specify a base64-encoded string of a CA certificate PEM file.
If `--ca-certificate-fingerprint` is also specified, both `--ca-certificate` and `--ca-certificate-file` are ignored.
If `--ca-certificate-file` is also specified but `--ca-certificate-fingerprint` is not, only `--ca-certificate` is used.
Alias: `-c`
**Example**
```bash
tcld account metrics accepted-client-ca remove --ca-certificate
```
##### --ca-certificate-file
_Required modifier unless `--ca-certificate-fingerprint` or `--ca-certificate` is specified_
Specify a path to a CA certificate PEM file.
If `--ca-certificate-fingerprint` is also specified, both `--ca-certificate-file` and `--ca-certificate` are ignored.
If `--ca-certificate` is also specified but `--ca-certificate-fingerprint` is not, only `--ca-certificate` is used.
Alias: `-f`
**Example**
```bash
tcld account metrics accepted-client-ca remove --ca-certificate-file
```
##### --ca-certificate-fingerprint
_Required modifier unless `--ca-certificate` or `--ca-certificate-file` is specified_
Specify the fingerprint of a CA certificate.
If `--ca-certificate`, `--ca-certificate-file`, or both are also specified, they are ignored.
Alias: `--fp`
**Example**
```bash
tcld account metrics accepted-client-ca remove --ca-certificate-fingerprint
```
#### set
The `tcld account metrics accepted-client-ca set` command sets the end-entity certificates for the metrics endpoint of a Temporal Cloud account.
:::info
The end-entity certificates for the metrics endpoint must chain up to the CA certificate used for the account. For more information, see [Certificate requirements](/cloud/certificates#certificate-requirements).
:::
`tcld account metrics accepted-client-ca set --ca-certificate `
Alias: `s`
The following modifiers control the behavior of the command.
##### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request identifier.
Alias: `-r`
**Example**
```bash
tcld account metrics accepted-client-ca set --request-id --ca-certificate
```
##### --resource-version
Specify a resource version (ETag) to update from. If not specified, the latest version is used.
Alias: `-v`
**Example**
```bash
tcld account metrics accepted-client-ca set --resource-version --ca-certificate
```
##### --ca-certificate
_Required modifier unless `--ca-certificate-file` is specified_
Specify a base64-encoded string of a CA certificate PEM file.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-c`
**Example**
```bash
tcld account metrics accepted-client-ca set --ca-certificate
```
##### --ca-certificate-file
_Required modifier unless `--ca-certificate` is specified_
Specify a path to a CA certificate PEM file.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-f`
**Example**
```bash
tcld account metrics accepted-client-ca set --ca-certificate-file
```
### enable
The `tcld account metrics enable` command enables the metrics endpoint for the Temporal Cloud account that is currently logged in.
:::info
The end-entity for the metrics endpoint _must_ be configured before the endpoint can be enabled. See the [tcld account metrics accepted-client-ca](#accepted-client-ca) commands.
:::
`tcld account metrics enable`
The command has no modifiers.
### disable
The `tcld account metrics disable` command disables the metrics endpoint for the Temporal Cloud account that is currently logged in.
`tcld account metrics disable`
The command has no modifiers.
---
## tcld apikey command reference
The `tcld apikey` commands manage API Keys in Temporal Cloud.
Alias: `ak`
- [tcld apikey create](#create)
- [tcld apikey get](#get)
- [tcld apikey list](#list)
- [tcld apikey delete](#delete)
- [tcld apikey disable](#disable)
- [tcld apikey enable](#enable)
## create
The `tcld apikey create` command creates an API Key in Temporal Cloud.
`tcld apikey create --name --description --duration --expiry --request-id `
The following options control the behavior of the command.
#### --name
_Required modifier_
Specify the display name of the API Key.
Alias: `-n`
**Example**
```bash
tcld apikey create --name
```
#### --description
Specify a description for the API Key.
Alias: `-desc`
**Example**
```bash
tcld apikey create --name --description "Your API Key"
```
#### --duration
Specify the duration from now when the API Key will expire.
This will be ignored if the expiry flag is set.
Example format: `24h` (default: 0s).
Alias: `-d`
**Example**
```bash
tcld apikey create --name --duration 24h
```
#### --expiry
Specify the absolute timestamp (RFC3339) when the API Key will expire.
Example: `2023-11-28T09:23:24-08:00`.
Alias: `-e`
**Example**
```bash
tcld apikey create --name --expiry '2023-11-28T09:23:24-08:00'
```
#### --request-id
Specify a request-id for the asynchronous operation.
If not set, the server will assign one.
Alias: `-r`
**Example**
```bash
tcld apikey create --name --request-id
```
## get
The `tcld apikey get` command retrieves the details of a specified API Key in Temporal Cloud.
`tcld apikey get --id `
The following option controls the behavior of the command.
#### --id
_Required modifier_
Specify the ID of the API Key to retrieve.
Alias: `-i`
**Example**
```bash
tcld apikey get --id
```
## list
The `tcld apikey list` command lists all API Keys in Temporal Cloud.
`tcld apikey list`
This command does not require any specific options.
Alias: `l`
**Example**
```bash
tcld apikey list
```
## delete
The `tcld apikey delete` command deletes an API Key in Temporal Cloud.
`tcld apikey delete --id [--resource-version ] [--request-id ]`
The following options control the behavior of the command.
#### --id
_Required modifier_
Specify the ID of the API Key to delete.
Alias: `-i`
**Example**
```bash
tcld apikey delete --id
```
#### --resource-version
Specify the resource-version (etag) to update from.
If not set, the CLI will use the latest.
Alias: `-v`
**Example**
```bash
tcld apikey delete --id --resource-version
```
#### --request-id
Specify a request-id for the asynchronous operation.
If not set, the server will assign one.
Alias: `-r`
**Example**
```bash
tcld apikey delete --id --request-id
```
## disable
The `tcld apikey disable` command disables an API Key in Temporal Cloud.
`tcld apikey disable --id [--resource-version ] [--request-id ]`
The following options control the behavior of the command.
#### --id
_Required modifier_
Specify the ID of the API Key to disable.
Alias: `-i`
**Example**
```bash
tcld apikey disable --id
```
#### --resource-version
Specify the resource-version (etag) to update from. If not set, the CLI will use the latest.
Alias: `-v`
**Example**
```bash
tcld apikey disable --id --resource-version
```
#### --request-id
Specify a request-id for the asynchronous operation. If not set, the server will assign one.
Alias: `-r`
**Example**
```bash
tcld apikey disable --id --request-id
```
## enable
The `tcld apikey enable` command enables a disabled API Key in Temporal Cloud.
`tcld apikey enable --id [--resource-version ] [--request-id ]`
The following options control the behavior of the command.
#### --id
_Required modifier_
Specify the ID of the API Key to enable.
Alias: `-i`
**Example**
```bash
tcld apikey enable --id
```
#### --resource-version
Specify the resource-version (etag) to update from.
If not set, the CLI will use the latest.
Alias: `-v`
**Example**
```bash
tcld apikey enable --id --resource-version
```
#### --request-id
Specify a request-id for the asynchronous operation.
If not set, the server will assign one.
Alias: `-r`
**Example**
```bash
tcld apikey enable --id --request-id
```
---
## tcld connectivity-rule command reference
The `tcld connectivity-rule` commands manage [connectivity rules](/cloud/connectivity#connectivity-rules) in Temporal Cloud.
Alias: `cr`
- [tcld connectivity-rule create](#create)
- [tcld connectivity-rule delete](#delete)
- [tcld connectivity-rule get](#get)
- [tcld connectivity-rule list](#list)
## create
The `tcld connectivity-rule create` command creates a connectivity rule.
Alias: `c`
#### --connection-id
The connection ID of the private connection.
Alias: `ci`
#### --connectivity-type
The type of connectivity, currently only support 'private' and 'public'.
Alias: `ct`
#### --gcp-project-id
The GCP project ID of the connection, required if the cloud provider is 'gcp'.
Alias: `gpi`
#### --region
The region of the connection.
Alias: `r`
## delete
The `tcld connectivity-rule delete` command deletes a connectivity rule.
Alias: `d`
#### --connectivity-rule-id
The connectivity rule ID.
Alias: `id`
## get
The `tcld connectivity-rule get` command gets a connectivity rule.
Alias: `g`
#### --connectivity-rule-id
The connectivity rule ID.
Alias: `id`
## list
The `tcld connectivity-rule list` command lists connectivity rules.
Alias: `l`
#### --namespace
The namespace hosted on temporal cloud.
Alias: `n`
---
## tcld feature command reference
The `tcld feature` commands manage features in Temporal Cloud.
Alias: `f`
- [tcld feature get](#get)
- [tcld feature toggle](#toggle)
## get
The `tcld feature get` command gets information about the Temporal Cloud features you've enabled.
Alias: `g`
`tcld feature get`
The command has no modifiers.
**Example**
`tcld feature get`
The following is an example output:
```json
[
{
"Name": "enable-apikey",
"Value": true
}
]
```
## toggle
The `tcld feature toggle-*` command turns on or off the `*` feature in Temporal Cloud.
:::note
The `*` symbol represents the name of the feature.
Replace `*` with the name of the available feature to toggle.
:::
Alias: `tak`
`tcld feature toggle-*`
The command has no modifiers.
**Example**
`tcld feature toggle-apikey`
The following is an example output:
```json
Feature flag enable-apikey is now true
```
:::note
The feature `apikey` is an example.
Update the feature name to toggle a different feature.
:::
---
## tcld generate-certificates command reference
The `tcld generate-certificates` commands generate certificate authority (CA) and end-entity TLS certificates for Temporal Cloud.
Alias: `gen`
- [tcld generate-certificates certificate-authority-certificate](#certificate-authority-certificate)
- [tcld generate-certificates end-entity-certificate](#end-entity-certificate)
## tcld generate-certificates certificate-authority-certificate {#certificate-authority-certificate}
The `tcld generate-certificates certificate-authority-certificate` command generates certificate authority (CA) certificates for Temporal Cloud.
`tcld generate-certificates certificate-authority-certificate `
Alias: `ca`
The following modifiers control the behavior of the command.
#### --organization
Specify an organization name for certificate generation.
Alias: `--org`
**Example**
```bash
tcld generate-certificates certificate-authority-certificate --organization
```
#### --validity-period
Specify the duration for which the certificate is valid.
Format values as d/h (for example, `30d10h` for a certificate lasting 30 days and 10 hours).
Alias: `-d`
**Example**
```bash
tcld generate-certificates certificate-authority-certificate --validity-period
```
#### --ca-certificate-file
Specify a path to a `.pem` file where the generated X.509 certificate file will be stored.
Alias: `--ca-cert`
**Example**
```bash
tcld generate-certificates certificate-authority-certificate --ca-certificate-file
```
#### --ca-key-file
Specify a path to a `.key` file where the certificate's private key will be stored.
Alias: `--ca-key`
**Example**
```bash
tcld generate-certificates certificate-authority-certificate --ca-key-file
```
#### --rsa-algorithm
When enabled, a 4096-bit RSA key pair is generated for the certificate instead of an ECDSA P-384 key pair.
Because an ECDSA P-384 key pair is the recommended default, this option is disabled.
Alias: `--rsa`
**Example**
```bash
tcld generate-certificates certificate-authority-certificate --rsa-algorithm
```
## tcld generate-certificates end-entity-certificate {#end-entity-certificate}
The `tcld generate-certificates end-entity-certificate` command generates end-entity (leaf) certificates for Temporal Cloud.
`tcld generate-certificates end-entity-certificate `
Alias: `leaf`
The following modifiers control the behavior of the command.
#### --organization
Specify an organization name for certificate generation.
Alias: `--org`
**Example**
```bash
tcld generate-certificates end-entity-certificate --organization
```
#### --organization-unit
Optional: Specify the name of the organization unit.
**Example**
```bash
tcld generate-certificates end-entity-certificate --organization-unit
```
#### --validity-period
Specify the duration for which the certificate is valid.
Format values as d/h (for example, `30d10h` for a certificate lasting 30 days and 10 hours).
Alias: `-d`
**Example**
```bash
tcld generate-certificates end-entity-certificate --validity-period
```
#### --ca-certificate-file
Specify the path of the X.509 CA certificate in a `.pem` file for the certificate authority.
Alias: `--ca-cert`
**Example**
```bash
tcld generate-certificates end-entity-certificate --ca-certificate-file
```
#### --ca-key-file
Specify the path of the private key in a `.key` file for the certificate authority.
Alias: `--ca-key`
**Example**
```bash
tcld generate-certificates end-entity-certificate --ca-key-file
```
#### --certificate-file
Specify a path to a `.pem` file where the generated X.509 leaf certificate file will be stored.
Alias: `--cert`
**Example**
```bash
tcld generate-certificates end-entity-certificate --certificate-file
```
#### --key-file
Specify a path to a `.key` file where the leaf certificate's private key will be stored.
Alias: `--key`
**Example**
```bash
tcld generate-certificates end-entity-certificate --key-file
```
---
## tcld command reference
The Temporal Cloud CLI (tcld) is a command-line tool that you can use to interact with Temporal Cloud.
- [How to install tcld](#install-tcld)
### tcld commands
- [tcld account](/cloud/tcld/account)
- [tcld apikey](/cloud/tcld/apikey)
- [tcld connectivity-rule](/cloud/tcld/connectivity-rule)
- [tcld feature](/cloud/tcld/feature)
- [tcld generate-certificates](/cloud/tcld/generate-certificates)
- [tcld login](/cloud/tcld/login)
- [tcld logout](/cloud/tcld/logout/)
- [tcld namespace](/cloud/tcld/namespace)
- [tcld nexus](/cloud/tcld/nexus)
- [tcld request](/cloud/tcld/request)
- [tcld user](/cloud/tcld/user)
- [tcld version](/cloud/tcld/version/)
### Global modifiers
#### --auto_confirm
Automatically confirm all prompts.
You can specify the value for this modifier by setting the AUTO_CONFIRM environment variable.
The default value is `false`.
## How to install tcld {#install-tcld}
You can install [tcld](/cloud/tcld) in two ways.
### Install tcld by using Homebrew
```bash
brew install temporalio/brew/tcld
```
### Build tcld from source
1. Verify that you have Go 1.18 or later installed.
```bash
go version
```
If Go 1.18 or later is not installed, follow the [Download and install](https://go.dev/doc/install) instructions on the Go website.
1. Clone the tcld repository and run make.
```bash
git clone https://github.com/temporalio/tcld.git
cd tcld
make
```
1. Copy the tcld executable to any directory that appears in the PATH environment variable, such as `/usr/local/bin`.
```bash
cp tcld /usr/local/bin/tcld
```
1. Verify that tcld is installed.
```bash
tcld version
```
---
## tcld login command reference
The `tcld login` command logs in a user to Temporal Cloud.
Follow instructions in the browser to log in to your Temporal account.
Alias: `l`
`tcld login`
The command has no modifiers.
---
## tcld logout command reference
The `tcld logout` command logs a user out of Temporal Cloud.
Alias: `lo`
`tcld logout`
The following modifier controls the behavior of the command.
#### --disable-pop-up
Disables a browser pop-up if set to `true`. The default value is `false`.
---
## tcld namespace command reference
The `tcld namespace` commands enable [Namespace](/namespaces) operations in Temporal Cloud.
Alias: `n`
:::info Namespace ID Format
The `--namespace` flag accepts a **Namespace ID** in the format `.` (e.g., `your-namespace.a1b2c`). This is the full identifier shown in Temporal Cloud, not just the [Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name). You can find your account suffix in the Temporal Cloud UI.
:::
- [tcld namespace add-region](#add-region)
- [tcld namespace create](#create)
- [tcld namespace delete](#delete)
- [tcld namespace failover](#failover)
- [tcld namespace get](#get)
- [tcld namespace list](#list)
- [tcld namespace export](#export)
- [tcld namespace accepted-client-ca](#accepted-client-ca)
- [tcld namespace certificate-filters](#certificate-filters)
- [tcld namespace search-attributes](#search-attributes)
- [tcld namespace retention](#retention)
- [tcld namespace update-codec-server](#update-codec-server)
- [tcld namespace update-high-availability](#update-high-availability)
- [tcld namespace tags](#tags)
- [tcld namespace set-connectivity-rules](#set-connectivity-rules)
## add-region
Use `tcld namespace add-region` to add a region to an existing Temporal Cloud [Namespace](/namespaces), upgrading it to support [High Availability](/cloud/high-availability).
See [Regions](/cloud/regions) for available regions and their supported replication options.
The following modifiers control the behavior of the command.
#### --request-id
The request identifier to use for the asynchronous operation. If not set, the server assigns an identifier.
Alias: `-r`
#### --namespace
**Required.** Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable `$TEMPORAL_CLOUD_NAMESPACE` is used.
Alias: `-n`
#### --region
**Required.** The region to add to the existing Namespace. See [Regions](/cloud/regions) for a list of supported regions.
:::tip Choosing Replica Regions
See [Regions](/cloud/regions) for available regions and their supported replication options. See [High Availability](/cloud/high-availability) to learn how replication and failover work.
:::
Alias: `--re`
**Example**
```bash
tcld namespace add-region \
--namespace \
--region
```
Specify the region code of the region where you want to create the replica as an argument to the `--region` flag.
See [High Availability](/cloud/high-availability) for details on same-region, multi-region, and multi-cloud replication options.
Temporal Cloud sends an email alert once your Namespace is ready for use.
#### --cloud-provider
The cloud provider of the region. One of [`aws`, `gcp`].
Default: `aws`
## create
The `tcld namespace create` command creates a Temporal [Namespace](/namespaces) in Temporal Cloud.
Alias: `c`
The following modifiers control the behavior of the command.
#### --namespace
**Required.** The name for the new Namespace. This becomes part of the Namespace ID (`.`).
Alias: `-n`
#### --region
**Required.** The cloud provider region to create the Namespace in. Supply one `--region` for a standard Namespace, or two for a Namespace with [High Availability](/cloud/high-availability).
See [Regions](/cloud/regions) for available regions and their supported replication options.
Alias: `--re`
#### --auth-method
The authentication method for the Namespace. One of [`mtls`, `api_key`].
- `mtls` (default): Requires `--ca-certificate` or `--ca-certificate-file`
- `api_key`: No other modifiers
**Example**
```bash
tcld namespace create \
--namespace test-namespace.a1b2c \
--region us-east-1 \
--auth-method api_key
```
#### --ca-certificate
A base64-encoded [CA certificate](/cloud/certificates). If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-c`
#### --ca-certificate-file
A path to a [CA certificate](/cloud/certificates) PEM file. If both options are specified, only `--ca-certificate` is used.
Alias: `--cf`
#### --certificate-filter-file
Path to a JSON file that defines the [certificate filters](/cloud/certificates#manage-certificate-filters) to be applied to the Namespace.
Sample JSON: `{ "filters": [ { "commonName": "test1" } ] }`
If both `--certificate-filter-file` and `--certificate-filter-input` are specified, the command returns an error.
Alias: `--cff`
#### --certificate-filter-input
A JSON string that defines the [certificate filters](/cloud/certificates#manage-certificate-filters) to be applied to the Namespace.
Sample JSON: `{ "filters": [ { "commonName": "test1" } ] }`
If both `--certificate-filter-input` and `--certificate-filter-file` are specified, the command returns an error.
Alias: `--cfi`
#### --cloud-provider
The cloud provider of the region. One of [`aws`, `gcp`].
Default: `aws`
Alias: `--cp`
#### --connectivity-rule-ids
A list of [connectivity rule](/cloud/connectivity#connectivity-rules) IDs to apply to the Namespace. Can be specified more than once.
Alias: `--ids`
**Example**
```bash
tcld namespace create \
--namespace test-namespace.a1b2c \
--region us-east-1 \
--auth-method api_key \
--connectivity-rule-ids \
--connectivity-rule-ids
```
#### --enable-delete-protection
Enable [delete protection](/cloud/namespaces#delete-protection) on the Namespace.
Default: `false`
Alias: `--edp`
#### --endpoint
The [codec server](/production-deployment/data-encryption) endpoint to decode payloads for all users interacting with this Namespace. Must be HTTPS.
Alias: `-e`
#### --include-credentials
Include cross-origin credentials when calling the [codec server](/production-deployment/data-encryption).
Default: `false`
Alias: `--ic`
#### --pass-access-token
Pass the user access token to the [codec server](/production-deployment/data-encryption) endpoint.
Default: `false`
Alias: `--pat`
#### --request-id
The request identifier to use for the asynchronous operation. If not set, the server assigns an identifier.
Alias: `-r`
#### --retention-days
The [retention period](/temporal-service/temporal-server#retention-period) in days for closed Workflow Executions.
Default: `30`
Alias: `--rd`
#### --search-attribute
A custom [Search Attribute](/search-attribute) in the form '_name_=_type_'. Can be specified more than once.
Valid values for _type_: `Bool` | `Datetime` | `Double` | `Int` | `Keyword` | `Text`
Alias: `--sa`
**Example**
```bash
tcld namespace create \
--namespace test-namespace.a1b2c \
--region us-east-1 \
--auth-method api_key \
--search-attribute "customer_id=Int" \
--search-attribute "customer_name=Text"
```
#### --tag
A [tag](/cloud/namespaces#tag-a-namespace) in the form "_key_=_value_". Can be specified more than once.
See [Tag structure and limits](/cloud/namespaces#tag-structure-and-limits).
Alias: `--t`
**Example**
```bash
tcld namespace create \
--namespace test-namespace.a1b2c \
--region us-east-1 \
--auth-method api_key \
--tag "key=value" \
--tag "key2=value2"
```
#### --user-namespace-permission
A [Namespace-level permission](/cloud/users#namespace-level-permissions) for a user in the form '_email_=_permission_'. Can be specified more than once.
Valid values for _permission_: `Admin` | `Write` | `Read`
Alias: `-p`
**Example**
```bash
tcld namespace create \
--namespace test-namespace.a1b2c \
--region us-east-1 \
--auth-method api_key \
--user-namespace-permission "user@example.com=Admin" \
--user-namespace-permission "user2@example.com=Write"
```
## delete
The `tcld namespace delete` command deletes the specified [Namespace](/namespaces) in Temporal Cloud.
Alias: `d`
`tcld namespace delete`
The following modifiers control the behavior of the command.
#### --namespace
**Required.** Specify the Namespace hosted on Temporal Cloud to be deleted.
Alias: `-n`
#### --request-id
The request identifier to use for the asynchronous operation. If not set, the server assigns an identifier.
Alias: `-r`
#### --resource-version
A resource version (ETag) to update from. If not set, the CLI uses the latest.
Alias: `-v`
**Example**
```bash
tcld namespace delete \
--namespace
```
## delete-region
Use `tcld namespace delete-region` to remove a for an existing Temporal Cloud
[Namespace](/namespaces). Removing a replica disables [High Availability features](/cloud/high-availability) and results
in a mandatory 7-day waiting period before you can re-enable High Availability features in the same location. Refer to
[Enable High Availability](/cloud/high-availability/enable) for more information.
The following modifiers control the behavior of the command.
#### --request-id
The request identifier to use for the asynchronous operation. If not set, the server assigns an identifier.
Alias: `-r`
#### --namespace
**Required.** Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable `$TEMPORAL_CLOUD_NAMESPACE` is used.
Alias: `-n`
#### --region
**Required.** The region to remove from the Namespace. Upon removal, Temporal stops replication and the Namespace becomes a Standard Namespace. You cannot re-add a region or add a new region for seven days after removing a Namespace region.
Alias: `--re`
**Example**
```bash
tcld namespace delete-region \
--namespace \
--region
```
When using API key authentication, add your API credentials before pressing Enter:
```bash
tcld --api-key \
delete-region \
--namespace \
--region
```
#### --cloud-provider
The cloud provider of the region to failover to. One of [aws, gcp].
Default: aws (default: "aws")
## failover
Failover a Temporal Namespace with [High Availability features](/cloud/high-availability). A failover switches a
Namespace region from a primary Namespace to its replica.
**Example**
```bash
tcld namespace failover \
--namespace \
--region
```
When using API key authentication, add your API credentials before pressing Enter:
```bash
tcld --api-key \
namespace failover \
--namespace \
--region
```
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
#### --namespace
**Required.** Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable `$TEMPORAL_CLOUD_NAMESPACE` is used.
Alias: `-n`
#### --region
**Required.** The region to failover _to_.
See [Regions](/cloud/regions) for a list of supported regions.
Alias: `--re`
#### --ca-certificate
_Required modifier unless `--ca-certificate-file` is specified_.
A base64-encoded CA certificate.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-c`
#### --cloud-provider
The cloud provider of the region to failover to. One of [aws, gcp].
Default: aws (default: "aws")
## get
The `tcld namespace get` command gets information about the specified [Namespace](/namespaces) in Temporal Cloud.
Alias: `g`
`tcld namespace get`
The following modifier controls the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
**Example**
```bash
tcld namespace get \
--namespace
```
## list
The `tcld namespace list` command lists all [Namespaces](/namespaces) in Temporal Cloud.
Alias: `l`
`tcld namespace list`
The command has no modifiers.
## export
The `tcld namespace export s3` commands manage Workflow History Exports.
Valid options: `s3`
Alias: `es`
- [tcld namespace export s3 create](#create)
- [tcld namespace export s3 get](#get)
- [tcld namespace export s3 delete](#delete)
- [tcld namespace export s3 list](#list)
- [tcld namespace export s3 update](#update)
- [tcld namespace export s3 validate](#validate)
### create
The `tcld namespace export s3 create` command allows users to create an export sink for the Namespace of a Temporal
Cloud account.
**Example**
```bash
tcld namespace export s3 create \
--namespace \
--sink-name \
--s3-bucket-name \
--role-arn
```
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
#### --sink-name
Provide a name for the export sink.
_Required modifier_
#### --role-arn
Provide role arn for the IAM Role.
_Required modifier_
#### --s3-bucket-name
Provide the name of an AWS S3 bucket that Temporal will send closed workflow histories to.
_Required modifier_
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
#### --kms-arn
Provide the ARN of the KMS key to use for encryption. Note: If the KMS ARN needs to be added or updated, users should
create the IAM Role with KMS or modify the created IAM Role accordingly. Providing it as part of the input won't help.
### get
The `tcld namespace export s3 get` command allows users to retrieve details about an existing export sink from the
Namespace of a Temporal Cloud account.
**Example**
```bash
tcld namespace export s3 get \
--namespace \
--sink-name
```
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
#### --sink-name
Provide the name of the export sink you wish to retrieve details for.
_Required modifier_
### delete
The `tcld namespace export s3 delete` command allows users to delete an existing export sink from the Namespace of a
Temporal Cloud account.
**Example**
```bash
tcld namespace export s3 delete \
--namespace \
--sink-name
```
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
#### --sink-name
Provide the name of the export sink you wish to delete.
_Required modifier_
#### --resource-version
Specify a resource version (ETag) to delete from. If not specified, the CLI will use the latest version.
Alias: `-v`
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
### list
The `tcld namespace export s3 list` command allows users to list all existing export sinks within the Namespace of a
Temporal Cloud account.
**Example**
```bash
tcld namespace export s3 list \
--namespace
```
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
#### --page-size
Determine the number of results to return per page for list operations. If not specified, the default value is 100.
#### --page-token
Provide the page token to continue listing results from where the previous list operation left off.
### update
The `tcld namespace export s3 update` command allows users to modify the details of an existing export sink within the
Namespace of a Temporal Cloud account.
**Example**
```bash
tcld namespace export s3 update \
--namespace \
--sink-name \
--enabled true
```
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
#### --sink-name
Provide the name of the export sink you wish to update.
_Required modifier_
#### --enabled
Specify whether the export is enabled or not.
#### --role-arn
Update the role ARN for the IAM Role.
#### --s3-bucket-name
Update the name of the AWS S3 bucket that Temporal will send closed workflow histories to.
#### --resource-version
Specify a resource version (ETag) to update from. If not specified, the CLI will use the latest version.
Alias: `-v`
#### --kms-arn
Update the ARN of the KMS key used for encryption. Note: If the KMS ARN needs to be added or updated, users should
create the IAM Role with KMS or modify the created IAM Role accordingly. Providing it as part of the input won't help.
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
### validate
The `tcld namespace export s3 validate` command allows users to validate an export sink from the Namespace of a Temporal
Cloud account.
**Example**
```bash
tcld namespace export s3 validate \
--namespace \
--sink-name \
--s3-bucket-name \
--role-arn
```
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
#### --sink-name
Provide the name of the export sink you wish to retrieve details for.
_Required modifier_
#### --role-arn
Provide role arn for the IAM Role.
_Required modifier_
#### --s3-bucket-name
Update the name of the AWS S3 bucket that Temporal will send closed workflow histories to.
#### --kms-arn
Update the ARN of the KMS key used for encryption. Note: If the KMS ARN needs to be added or updated, users should
create the IAM Role with KMS or modify the created IAM Role accordingly. Providing it as part of the input won't help.
## accepted-client-ca
The `tcld namespace accepted-client-ca` commands manage the client CA certificates of the specified
[Namespace](/namespaces) in Temporal Cloud. The certificates are used to verify client connections.
:::note
Base64 versions of the CA certificate files are accepted by these commands.
:::
Alias: `ca`
- [tcld namespace accepted-client-ca add](#add)
- [tcld namespace accepted-client-ca list](#list)
- [tcld namespace accepted-client-ca set](#set)
- [tcld namespace accepted-client-ca remove](#remove)
:::important
Do not use a CA certificate that is signed with an insecure signature algorithm, such as SHA-1. Such signatures will be
rejected. Existing CA certificates that use SHA-1 can stop working without warning.
For more information about the vulnerabilities of SHA-1, see [SHAttered](https://shattered.io/).
:::
### add
The `tcld namespace accepted-client-ca add` command adds client CA certificates to a [Namespace](/namespaces) in
Temporal Cloud.
`tcld namespace accepted-client-ca add --ca-certificate `
Alias: `a`
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
**Example**
```bash
tcld namespace accepted-client-ca add \
--namespace \
--ca-certificate
```
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
**Example**
```bash
tcld namespace accepted-client-ca add \
--request-id \
--ca-certificate
```
#### --resource-version
Specify a resource version (ETag) to update from. If not specified, the latest version is used.
Alias: `-v`
**Example**
```bash
tcld namespace accepted-client-ca add \
--resource-version \
--ca-certificate
```
#### --ca-certificate
_Required modifier unless `--ca-certificate-file` is specified_
Specify a base64-encoded string of a CA certificate PEM file.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-c`
**Example**
```bash
tcld namespace accepted-client-ca add \
--ca-certificate
```
#### --ca-certificate-file
_Required modifier unless `--ca-certificate` is specified_
Specify a path to a CA certificate PEM file.
If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used.
Alias: `-f`
**Example**
```bash
tcld namespace accepted-client-ca add \
--ca-certificate-file
```
### list
The `tcld namespace accepted-client-ca list` command lists the client CA certificates that are currently configured for
a [Namespace](/namespaces) in Temporal Cloud.
`tcld namespace accepted-client-ca list`
Alias: `l`
The following modifier controls the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
**Example**
```bash
tcld namespace accepted-client-ca list \
--namespace
```
### remove
The `tcld namespace accepted-client-ca remove` command removes client CA certificates from a [Namespace](/namespaces) in
Temporal Cloud.
`tcld namespace accepted-client-ca remove --ca-certificate `
Alias: `r`
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
**Example**
```bash
tcld namespace accepted-client-ca remove \
--namespace \
--ca-certificate
```
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
**Example**
```bash
tcld namespace accepted-client-ca remove \
--request-id \
--ca-certificate
```
#### --resource-version
Specify a resource version (ETag) to update from. If not specified, the latest version is used.
Alias: `-v`
**Example**
```bash
tcld namespace accepted-client-ca remove \
--resource-version \
--ca-certificate
```
#### --ca-certificate
_Required modifier unless `--ca-certificate-fingerprint` or `--ca-certificate-file` is specified_
Specify the base64-encoded string of a CA certificate PEM file.
If `--ca-certificate-fingerprint` is also specified, both `--ca-certificate` and `--ca-certificate-file` are ignored.
If `--ca-certificate-file` is also specified but `--ca-certificate-fingerprint` is not, only `--ca-certificate` is used.
Alias: `-c`
**Example**
```bash
tcld namespace accepted-client-ca remove \
--ca-certificate
```
#### --ca-certificate-file
_Required modifier unless `--ca-certificate-fingerprint` or `--ca-certificate` is specified_
Specify a path to a CA certificate PEM file.
If `--ca-certificate-fingerprint` is also specified, both `--ca-certificate-file` and `--ca-certificate` are ignored.
If `--ca-certificate` is also specified but `--ca-certificate-fingerprint` is not, only `--ca-certificate` is used.
Alias: `-f`
**Example**
```bash
tcld namespace accepted-client-ca remove \
--ca-certificate-file
```
#### --ca-certificate-fingerprint
_Required modifier unless `--ca-certificate` or `--ca-certificate-file` is specified_
Specify the fingerprint of a CA certificate.
If `--ca-certificate`, `--ca-certificate-file`, or both are also specified, they are ignored.
Alias: `--fp`
**Example**
```bash
tcld namespace accepted-client-ca remove \
--ca-certificate-fingerprint
```
### set
The `tcld namespace accepted-client-ca set` command sets the client CA certificates for a [Namespace](/namespaces) in
Temporal Cloud.
`tcld namespace accepted-client-ca set --ca-certificate `
Alias: `s`
{/* How to rollover accepted client CA certificates in Temporal Cloud using tcld */}
When updating CA certificates, it's important to follow a rollover process. Doing so enables your Namespace to serve
both CA certificates for a period of time until traffic to your old CA certificate ceases.
1. Create a single file that contains both your old and new CA certificate PEM blocks. Just concatenate the PEM blocks
on adjacent lines.
```
-----BEGIN CERTIFICATE-----
... old CA cert ...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... new CA cert ...
-----END CERTIFICATE-----
```
1. Run the `tcld namespace accepted-client-ca set` command with the CA certificate bundle file.
```bash
tcld namespace accepted-client-ca set \
--ca-certificate-file
```
1. Monitor traffic to your old certificate until it ceases.
1. Create another file that contains only the new CA certificate.
1. Run the `tcld namespace accepted-client-ca set` command again with the updated CA certificate bundle file.
The following modifiers control the behavior of the command.
#### --namespace
Specify a Namespace hosted on Temporal Cloud. If not specified, the value of the environment variable
$TEMPORAL_CLOUD_NAMESPACE is used.
Alias: `-n`
**Example**
```bash
tcld namespace accepted-client-ca set \
--namespace
--ca-certificate
```
#### --request-id
Specify a request identifier to use for the asynchronous operation. If not specified, the server assigns a request
identifier.
Alias: `-r`
**Example**
```bash
tcld namespace accepted-client-ca set \
--request-id \
--ca-certificate