# replace <your-secret-key> with the "secretKey": output from tcld apikey create command
set TEMPORAL_CLOUD_API_KEY=<your-secret-key>
```

:::tip ENVIRONMENT VARIABLES

Do not confuse environment variables, set with your shell, with temporal env options.

:::

</TabItem>
</Tabs>

Or, pass it in manually in your .tf file using the provider code block

```yml
provider "temporalcloud" { api_key = "my-temporalcloud-api-key" }
```

## Manage Temporal Cloud Namespaces with Terraform

Terraform is a great way to automate the management of Temporal Namespaces. It doesn't matter whether you want
management to be centralized within a platform team or federated to different product teams. The provider allows you to
import, create, update, and delete Namespaces with Terraform.

You must use an Identity with Temporal Cloud Namespace management privileges. This includes the Account Owner, Global
Admin, or Developer Account Role.

For more detailed examples on how to manage Namespaces via Terraform, check the [Terraform Registry documention for Namspaces](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace).

**How do I create a Namespace with Terraform?**

1. Create a Terraform configuration file (`terraform.tf`) to define a Namespace.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 14
   }
   ```

   In this example, you create a Temporal Cloud Namespace named `terraform`, specifying the AWS region `aws-us-east-1`,
   and specifying the path to the CA certificate.

1. Initialize the Terraform provider.

   Run the following command to initialize the Terraform provider.

   ```bash
   terraform init
   ```

1. Apply the Terraform configuration.

   Once initialization occurs, apply the Terraform configuration to your Temporal Cloud account.

   ```bash
   terraform apply
   ```

Follow the onscreen prompts.

Upon completion, you'll see a success message indicating your Namespace is created.

```bash
temporalcloud_namespace.terraform: Creation complete after 2m17s [id=<your-namespace>]
```

You can find more examples of Namespace management in the Terraform Provider docs located on HashiCorp's
[Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/namespace).
The Terraform Provider docs show how to generate CA certs within Terraform configuration files and create a Namespace
with API Key based authentication.

**How do I validate the creation of the Namespace?**

You can validate the creation of the Namespace through the Temporal Web UI or through the `tcld namespace get` command.

**Using the Temporal Web UI**

1. Log into the Temporal Cloud Web UI.
1. Navigate to the Namespaces page.
1. Search for the Namespace you created.

**Using the tcld CLI utility**

Validate the creation of your Namespace through the Terraform provider. To validate see your Namespace in the Cloud UI
or through the `tcld namespace get` command. Run the `tcld namespace get` command and pass in your
[Cloud Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name) and
[Cloud Account Id](/cloud/namespaces#temporal-cloud-account-id):

```bash
tcld namespace get -n "<your-namespace>.<your-account-id>"
```

**How do I update a Temporal Cloud Namespace?**

Terraform automatically recognizes changes made within `.tf` files and applies those changes to Temporal.

For example, change the retention period setting in the Terraform file from the previous example and watch Terraform
apply the change without any additional steps required by you.

1. Set the retention period to 30 days.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source  = "temporalio/temporalcloud"
         version = ">= 0.0.6"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 30
   }
   ```

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

Upon completion, you will see a success message indicating your Namespace has been updated. It may take several minutes
to update a Namespace.

```text
temporalcloud_namespace.namespace: Modifications complete after 10s [id=terraform.a1bb2]
```

**How do I delete a Temporal Cloud Namespace?**

To delete a Namespace, remove the `temporalcloud_namespace` resource and all dependent resource configurations from your
Terraform files and run the `terraform apply` command.

Upon completion, you will see a success message indicating the resource has been destroyed:

```text
temporalcloud_namespace.my_namespace: Destruction complete after 3s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
```

:::note Preventing Deletion

You can prevent deletion of any Terraform resource by including the `prevent_destroy` argument in the Terraform
configuration file.

:::

**How do I import a Temporal Cloud Namespace?**

If you have an existing Namespace in Temporal Cloud, you can import it into Terraform to manage the Namespace from
Terraform using the `terraform import` command.

1. Provide a configuration placeholder in your Terraform configuration.

   ```yml
   resource "temporalcloud_namespace" "namespace" { }
   ```

1. Run the `terraform import` command from the command line and pass in the Namespace ID. Your Namespace ID is available
   at the top of the Namespace's page in the Temporal Cloud UI and is in the format `namespaceid.acctid`.

   ```bash
   terraform import temporalcloud_namespace.terraform namespaceid.acctid
   ```

The Namespace is now a part of the Terraform state and all changes to the Namespace should be managed by Terraform.

:::caution

Once a resource has been imported into Terraform, outside changes to the resource will create Terraform "drift" errors
on subsequent Terraform operations.

:::

## Manage Temporal Cloud Nexus Endpoints with Terraform

Terraform provides a great way to automate the management of [Nexus Endpoints](/nexus/endpoints). The provider allows
you to import, create, update, and delete Nexus Endpoints with Terraform.

You must use an Identity with [Developer role (or higher)](/cloud/users#account-level-roles) and
[Namespace Admin permission](/cloud/users#namespace-level-permissions) on the Endpoint's target Namespace.

**How do I create a Nexus Endpoint with Terraform?**

1. Create a Terraform configuration file (`terraform.tf`) to define a Nexus Endpoint.

   From the
   [example in the Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/nexus_endpoint):

   ```yml
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "target_namespace" {
     name           = "terraform-target-namespace"
     regions        = ["aws-us-west-2"]
     api_key_auth   = true
     retention_days = 14
     timeouts {
       create = "10m"
       delete = "10m"
     }
   }

   resource "temporalcloud_namespace" "caller_namespace" {
     name           = "terraform-caller-namespace"
     regions        = ["aws-us-east-1"]
     api_key_auth   = true
     retention_days = 14
     timeouts {
       create = "10m"
       delete = "10m"
     }
   }

   resource "temporalcloud_namespace" "caller_namespace_2" {
     name           = "terraform-caller-namespace-2"
     regions        = ["gcp-us-central1"]
     api_key_auth   = true
     retention_days = 14
     timeouts {
       create = "10m"
       delete = "10m"
     }
   }

   resource "temporalcloud_nexus_endpoint" "nexus_endpoint" {
     name        = "terraform-nexus-endpoint"
     description = <<-EOT
       Service Name:
         my-hello-service
       Operation Names:
         echo
         say-hello

       Input / Output arguments are in the following repository:
       https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go
     EOT
     worker_target = {
       namespace_id = temporalcloud_namespace.target_namespace.id
       task_queue   = "terraform-task-queue"
     }
     allowed_caller_namespaces = [
       temporalcloud_namespace.caller_namespace.id,
       temporalcloud_namespace.caller_namespace_2.id,
     ]
   }
   ```

   In this example, 3 Namespaces are created:

   - target Namespace for a Nexus Endpoint - Nexus requests will be routed to a Worker that polls the target Namespace.
   - caller Namespace(s) - Nexus Operations are invoked from caller Namespace, for example from a caller Workflow.

   These Namespaces are referenced in the [Nexus Endpoint](/nexus/endpoints) configuration:

   - `worker_target` (Namespace and Task Queue) - currently only a single worker_target is supported.
   - `allowed_caller_namespaces` - used to enforce Nexus Endpoint
     [runtime access controls](/nexus/security#runtime-access-controls).

1. Initialize the Terraform provider.

   Run the following command to initialize the Terraform provider.

   ```bash
   terraform init
   ```

1. Apply the Terraform configuration.

   Once initialization occurs, apply the Terraform configuration to your Temporal Cloud account.

   ```bash
   terraform apply
   ```

   Follow the onscreen prompts.

   Upon completion, you'll see a success message indicating 3 Namespaces and a Nexus Endpoint are created.

   ```bash
   temporalcloud_nexus_endpoint.nexus_endpoint: Creation complete after 2s [id=b158063be978471fa1d200569b03834d]
   ```

You can find more examples of Nexus Endpoint management in the Terraform Provider docs located on HashiCorp's
[Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/nexus_endpoint).
The Terraform Provider docs show how to generate CA certs within Terraform configuration files and create a Namespace
with API Key based authentication.

**How do I validate the creation of the Nexus Endpoint?**

You can validate the creation of the Nexus Endpoint through the Temporal Web UI or through the `tcld nexus endpoint get`
command.

**Using the Temporal Web UI**

1. Log into the Temporal Cloud Web UI.
1. Navigate to [the Nexus page](https://cloud.temporal.io/nexus).
1. Search for the Nexus Endpoint you created, using only the Nexus Endpoint Name (without an account suffix).

**Using the tcld CLI utility**

Validate the creation of your Nexus Endpoint through the Terraform provider. To validate see your Nexus Endpoint in the
Cloud UI or through the `tcld nexus endpoint get` command.

Run the below command using your Nexus Endpoint Name. Do not use the account ID suffix with this endpoint name:

```bash
tcld nexus endpoint get -n "<your-nexus-endpoint-name-without-account-suffix>"
```

**How do I update a Nexus Endpoint?**

Terraform automatically recognizes changes made within `.tf` files and applies those changes to Temporal.

For example, to change the allowed caller Namespaces on a Nexus Endpoint:

1. Add or remove allowed caller Namespaces by updating the Nexus Endpoint configuration, for example by removing
   `caller_namespace_2` from the configuration above:

   ```yml
   resource "temporalcloud_nexus_endpoint" "nexus_endpoint" {
     name        = "terraform-nexus-endpoint"
     description = <<-EOT
       Service Name:
         my-hello-service
       Operation Names:
         echo
         say-hello

       Input / Output arguments are in the following repository:
       https://github.com/temporalio/samples-go/blob/main/nexus/service/api.go
     EOT
     worker_target = {
       namespace_id = temporalcloud_namespace.target_namespace.id
       task_queue   = "terraform-task-queue"
     }
     allowed_caller_namespaces = [
       temporalcloud_namespace.caller_namespace.id
     ]
   }
   ```

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

   Upon completion, you will see a success message indicating your Nexus Endpoint has been updated. It may take several
   seconds to update a Nexus Endpoint in the control plane which is visibile from the Temporal UI or tcld CLI.
   Propagation of Nexus Endpoint changes to the data plane may take longer, but usually complete in less than one
   minute.

   ```text
   temporalcloud_nexus_endpoint.nexus_endpoint: Modifications complete after 1s [id=b158063be978471fa1d200569b03834d]
   ```

**How do I delete a Nexus Endpoint?**

To delete a Nexus Endpoint, remove the `temporalcloud_nexus_endpoint` resource configuration from your Terraform files
and run the `terraform apply` command.

Upon completion, you will see a success message indicating the resource has been destroyed:

```text
temporalcloud_nexus_endpoint.my_nexus_endpoint: Destruction complete after 3s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
```

**How do I import a Temporal Cloud Nexus Endpoint?**

If you have an existing Nexus Endpoint in Temporal Cloud, you can import it into Terraform to manage the Nexus Endpoint
from Terraform using the `terraform import` command.

1. Initialize the Terraform provider in a new directory.

   Run the following command to initialize the Terraform provider.

   ```bash
   terraform init
   ```

1. Provide a configuration placeholder in your Terraform configuration and ensure you've included your
   [API key](#setup).

   ```yml
   terraform { required_providers { temporalcloud = { source = "temporalio/temporalcloud" } } }

   provider "temporalcloud" { }

   resource "temporalcloud_nexus_endpoint" "nexus_endpoint" { }
   ```

1. Run the `terraform import` command from the command line and pass in the Nexus Endpoint ID.

   ```bash
   terraform import temporalcloud_nexus_endpoint <your-nexus-endpoint-ID>
   ```

   Your Nexus Endpoint ID is available at the top of the Nexus Endpoint's page in the
   [Temporal Cloud UI](https://cloud.temporal.io/nexus).

   <CaptionedImage src="/img/cloud/nexus/nexus-endpoint-id.png" title="Nexus Endpoint ID" width="100%" zoom="false" />

   Upon completion, you will see a success message indicating the Nexus Endpoint was imported.

   ```text
   temporalcloud_nexus_endpoint.nexus_endpoint: Refreshing state... [id=3c0c75ccfa8144b092c13ce632463761]

   Import successful!
   ```

The Nexus Endpoint is now a part of the Terraform state and all changes to the Nexus Endpoint should be managed by
Terraform.

:::caution

Once a resource has been imported into Terraform, outside changes to the resource will create Terraform "drift" errors
on subsequent Terraform operations.

:::

## Manage Temporal Cloud Users with Terraform

Manage Temporal Cloud Users with the same process you use to manage Namespaces with Terraform. The following examples
create, update, delete, and import Temporal Cloud Users with `terraform apply` commands on the Terraform configuration
file.

:::note User Management

Cautions about Temporal User management:

- Terraform can't manage the Temporal Account Owner role. While you can import an Account Owner to Terraform, you cannot
  create, update, or delete an Account Owner with Terraform.
- Right now, you can't manage a user's access to a Namespace from the Namespace resource. You must manage Namespace
  access from the User resource. This is also true for Service Accounts.
- Account Owners and Global Admins automatically gain access to all Namespaces in Temporal. Therefore, you cannot
  specify Namespace access for these roles. This is also true for Service Accounts.
- Follow Terraform best practices for resource management. Manage a specific user in one and only one .tf file. There's
  a risk that you may overwrite a user's permissions if you don't.
- To Import a user, you'll need the User's ID which is currently not available in the Temporal Cloud UI. You can fetch
  current User ID by running the `tcld user list` command.

:::

For more detailed examples on how to manage Namespaces via Terraform, check the Terraform Registry documention for [provisioning a Temporal Cloud user](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/user).

**How do I create a Temporal Cloud User with Terraform?**

1. Add a Terraform User resources configuration to your Terraform file.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 14
   }

   # Global admins automatically have access to all namespaces.
   resource "temporalcloud_user" "global_admin" {
     email          = "admin@example.com"
     account_access = "Admin"
   }

   # Developers can be granted explicit namespace permissions.
   resource "temporalcloud_user" "namespace_admin" {
     email          = "developer@example.com"
     account_access = "Developer"

     namespace_accesses = [{
       namespace_id = temporalcloud_namespace.namespace.id
       permission   = "Write"
     }]
   }
   ```

   Replace the email and domain values with your Temporal Cloud User email and domain.

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

Upon completion, you will see a success message indicating your User has been created.

```text
temporalcloud_user.namespace_admin: Creation complete after 1s [id=12a34bc5678910d38d9e8390636e7412]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
```

**How do I update a Temporal Cloud User with Terraform?**

To update a User with Terraform, follow the same steps used to create a User.

**How do I delete a Temporal Cloud User with Terraform?**

To delete a User with Terraform, remove the Terraform User resources configuration from your Terraform file and run the
`terraform apply` command.

1. Remove the Terraform User resources configuration from your Terraform file.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source  = "temporalio/temporalcloud"
         version = ">= 0.0.6"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_namespace" "namespace" {
     name               = "terraform"
     regions            = ["aws-us-east-1"]
     accepted_client_ca = base64encode(file("ca.pem"))
     retention_days     = 14
   }

   # This user will be deleted after running `terraform apply`
   resource "temporalcloud_user" "global_admin" {
     email          = "admin@example.com"
     account_access = "Admin"
   }
   # The following user resource has been removed (or commented out),
   # so Terraform will delete it.
   # resource "temporalcloud_user" "namespace_admin" {
   #   email          = "developer@example.com"
   #   account_access = "Developer"
   #
   #   namespace_accesses = [{
   #     namespace_id = temporalcloud_namespace.namespace.id
   #     permission   = "Write"
   #   }]
   # }
   ```

1. Run the `terraform apply` command. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

Upon completion, you will see a success message indicating your User has been deleted.

```text
temporalcloud_user.namespace_admin: Destruction complete after 2s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
```

**How do I import a Temporal User?** If you have an existing User in Temporal Cloud, you can import it into Terraform
using the `terraform import` command.

1. Provide a configuration placeholder in your Terraform configuration.

   ```yml
   resource "temporalcloud_user" "user" { }
   ```

1. Run the `terraform import` command and pass in the User ID. Your User ID is available using the Temporal Cloud CLI `tcld u l` command.

 ```bash
 terraform import temporalcloud_user.user 72360058153949edb2f1d47019c1e85f
 ```

The User is now a part of the Terraform state and all changes to the User should be managed by Terraform.

## Manage Temporal Cloud Service Accounts with Terraform

The process and steps to managing a Service Account with Terraform are very similar to managing a User with Terraform
with a few small differences:

- Service Accounts use the Service Account Terraform resource not the User resource.
- Service Accounts do not have email addresses, they have names instead. This means you should specify a name for a
  Service Account instead of an email.

Everything else about managing Services Accounts with Terraform follows the same process, guidance, and limitations of
managing Users with Terraform.

## Manage Temporal Cloud API Keys with Terraform

You can manage your own, personal API Keys and Service Account API Keys with Terraform. The process and steps to
managing an API Key with Terraform are very similar to managing other resources with Terraform. You can create, delete,
update and import API Keys with Terraform. One difference between working with API Keys as a Terraform resource compared
to other Temporal Cloud resources is the need to access an API Keys secure token output from Terraform. Walk through the
process of securely accessing the API Key Token in the Create section of this guide.

:::note Limits and Best Practices

- See the API Key [documentation](https://docs.temporal.io/cloud/api-keys) for information about the limits and best
  practices for managing API Keys.
- See Terraform's documentation on working with
  [sensitive data](https://www.terraform.io/docs/language/values/variables.html#sensitive-values) for more information
  on how to manage sensitive data in Terraform.

:::

**How do I create a Temporal Cloud API Key with Terraform?**

From the example in the [Terraform Registry](https://registry.terraform.io/providers/temporalio/temporalcloud/latest/docs/resources/apikey):
1. Add a Terraform API Key resources configuration to your Terraform file.

   ```hcl
   terraform {
     required_providers {
       temporalcloud = {
         source = "temporalio/temporalcloud"
       }
     }
   }

   provider "temporalcloud" {

   }

   resource "temporalcloud_service_account" "global_service_account" {
     name           = "admin"
     account_access = "Admin"
   }

   resource "temporalcloud_apikey" "global_apikey" {
     display_name = "admin"
     owner_type   = "service-account"
     owner_id     = temporalcloud_service_account.global_service_account.id
     expiry_time  = "2024-11-01T00:00:00Z"
     disabled     = false
   }
   ```

   Make sure to:

   - Replace the display_name, expiry_time, and disabled values with your Temporal Cloud API Key configuration.
   - Replace the owner_type and owner_id values with your Temporal Cloud Service Account or other Identity information.

1. Create an output.tf file and add the following code to output the API Key Token.

   ```hcl
   output "apikey_token" {
     value     = temporalcloud_apikey.global_apikey.token
     sensitive = true
   }
   ```

1. Apply your configuration. When prompted, answer yes to continue:

   ```command
   terraform apply
   ```

   Upon completion, you will see a success message indicating the API Key has been created.

   ```text
   temporalcloud_apikey.global_apikey: Creation complete after 1s [id=kayBf38JIWkMPmnfr59iEIaEk2L7uqR4]
   ```

1. Access the API Key Token securely. You'll notice that if you view the state for the API Key resource, the token value
   is not displayed.

   ```bash
   terraform state show temporalcloud_apikey.global_apikey

   # temporalcloud_apikey.global_apikey:
   resource "temporalcloud_apikey" "global_apikey" {
       disabled     = false
       display_name = "adminKey3"
       expiry_time  = "2024-12-01T00:00:00Z"
       id           = "kayBf38JIWkMPmnfr59iEIaEk2L7uqR4"
       owner_id     = "b81336a6097449cba75c2e5500df3d31"
       owner_type   = "service-account"
       state        = "active"
       token        = (sensitive value)
   }
   ```

To access the token, you can use the Terraform output command.

```bash
terraform output -json apikey_token
```

This will display the token value in the terminal.

:::info Security and API Keys

Remember, keep your Terraform state files secure if you're managing API Keys with Terraform. The state file contains
sensitive information, like the API Key Token, that should not be shared or exposed.

:::

**How do I update a Temporal Cloud API Key with Terraform?**

To update an API Key with Terraform, follow the same steps used to create an API Key.

:::note Editing Fields

You can only edit an API Key's name or description field. Updating an API Key does not generate a new secure token

:::

**How do I delete a Temporal Cloud API Key with Terraform?**

To delete an API Key with Terraform, remove the Terraform API Key resources configuration from your Terraform and
output.tf files and run the `terraform apply` command.

**How do I Import a Temporal API Key?**

You cannot import an API Key into Terraform. Once created, the API Key secret isn't stored and can't be retrieved, so
you can't access it using import.

Instead, Temporal recommends creating a new API Key using Terraform directly.

## Data Sources - Regions and Namespaces

The Terraform provider also supports 2 data sources that provide you access to the available Regions and Namespaces in
your Temporal Cloud account.

:::note Terraform Data Sources

See Terraform [documentation](https://developer.hashicorp.com/terraform/language/data-sources) to learn more about
Terraform Data Sources

:::

For example, to retrieve a list of regions available for your account, you can use the regions data_source

```hcl
data "temporalcloud_regions" "regions" {}

output "regions" {
  value = data.temporalcloud_regions.regions.regions
}
```

## Community Involvement

Do you have feedback about the provider? Want to report a bug or request a feature? We'd love to hear from you.

- Please reach out to us in the Temporal Community
  [Slack](https://join.slack.com/t/temporalio/shared_invite/zt-2u2ey8ilu-LRxnd3PSoAk9GZ94UuzoBA) in the #terraform
  channel
- Feel free to create issues and contribute PRs in the Temporal Terraform
  [GitHub repository](https://github.com/temporalio/terraform-provider-temporalcloud/tree/main)

---

## Monitor worker health

This page is a guide to monitoring a Temporal Worker fleet and covers the following scenarios:

- [Configuring minimal observations](#minimal-observations)
- [How to detect a backlog of Tasks](#detect-task-backlog)
- [How to detect greedy Worker resources](#detect-greedy-workers)
- [How to detect misconfigured Workers](#detect-misconfigured-workers)
- [How to configure Sticky cache](#configure-sticky-cache)

## Minimal Observations {#minimal-observations}

These alerts should be configured and understood first to gain intelligence into your application health and behaviors.

1. Create monitors and alerts for Schedule To Start latency SDK metrics (both [Workflow Executions](/references/sdk-metrics#workflow_task_schedule_to_start_latency) and [Activity Executions](/references/sdk-metrics#activity_schedule_to_start_latency)).
   See [Detect Task backlog section](#detect-task-backlog) to explore [sample queries](#prometheus-query-samples) and appropriate responses that accompany these values.

- Alert at >200ms for your p99 value
- Plot >100ms for your p95 value

2. Create a [Grafana](/cloud/metrics/prometheus-grafana) panel called Sync Match Rate.
   See the [Sync Match Rate section](#sync-match-rate) to explore example queries and appropriate responses that accompany these values.

- Alert at \<95% for your p99 value
- Plot \<99% for your p95 value

3. Create a [Grafana](/cloud/metrics/prometheus-grafana) panel called Poll Success Rate.
   See the [Detect greedy Workers section](#detect-greedy-workers) for example queries and appropriate responses that accompany these values.

- Alert at \<90% for your p99 value
- Plot \<95% for your p95 value

The following alerts build on the above to dive deeper into specific potential causes for Worker related issues you might be experiencing.

1. Create monitors and alerts for the [temporal_worker_task_slots_available](/references/sdk-metrics#worker_task_slots_available) SDK metric.
   See the [Detect misconfigured Workers section](#detect-misconfigured-workers) for appropriate responses based on the value.

- Alert at 0 for your p99 value

2. Create monitors for the [temporal_sticky_cache_size](/references/sdk-metrics#sticky_cache_size) SDK metric.
   See the [Configure Sticky Cache section](#configure-sticky-cache) for more details on this configuration.

- Plot at \{value\} > \{WorkflowCacheSize.Value\}

3. Create monitors for the [temporal_sticky_cache_total_forced_eviction](/references/sdk-metrics#sticky_cache_total_forced_eviction) SDK metric.
   This metric is available in the Go SDK, and the Java SDK only.
   See the [Configure Sticky Cache section](#configure-sticky-cache) for more details and appropriate responses.

- Alert at >\{predetermined_high_number\}

## Detect Task Backlog {#detect-task-backlog}

### Symptoms of high Task backlog

If the Task backlog is too high, you will find that tasks are waiting to find Workers to run on. This can cause a delay in
Workflow execution. Detecting a growing Task backlog is possible by watching the Schedule To Start latency and sync match rate.

Metrics to monitor:

- **SDK metric**: [workflow_task_schedule_to_start_latency](/references/sdk-metrics#workflow_task_schedule_to_start_latency)
- **SDK metric**: [activity_schedule_to_start_latency](/references/sdk-metrics#activity_schedule_to_start_latency)
- **Temporal Cloud metric**: [temporal_cloud_v0_poll_success_count](/cloud/metrics/reference#temporal_cloud_v0_poll_success_count)
- **Temporal Cloud metric**: [temporal_cloud_v0_poll_success_sync_count](/cloud/metrics/reference#temporal_cloud_v0_poll_success_sync_count)

### Schedule To Start latency

The Schedule To Start metric represents how long Tasks are staying unprocessed in the Task Queues.
It is the time between when a Task is enqueued and when it is started by a Worker.
This time being long (likely) means that your Workers can't keep up — either increase the number of Workers (if the host load is already high) or increase the number of pollers per Worker.

If your Schedule To Start latency alert triggers or is high, check the [Sync Match Rate](#sync-match-rate) to decide if you need to adjust your Worker or fleet, or contact Temporal Cloud support.
If your Sync Match Rate is low, contact [Temporal Cloud support](/cloud/support#support-ticket).
If your Sync Match Rate is low, you can contact Temporal Cloud support.

The schedule_to_start_latency SDK metric for both [Workflow Executions](/references/sdk-metrics#workflow_task_schedule_to_start_latency) and [Activity Executions](/references/sdk-metrics#activity_schedule_to_start_latency) should have alerts.

#### Prometheus query samples

**Workflow Task Latency, 99th percentile**

```
histogram_quantile(0.99, sum(rate(temporal_workflow_task_schedule_to_start_latency_seconds_bucket[5m])) by (le, namespace, task_queue))
```

**Workflow Task Latency, average**

```
sum(increase(temporal_workflow_task_schedule_to_start_latency_seconds_sum[5m])) by (namespace, task_queue)
/
sum(increase(temporal_workflow_task_schedule_to_start_latency_seconds_count[5m])) by (namespace, task_queue)
```

**Activity Task Latency, 99th percentile**

```
histogram_quantile(0.99, sum(rate(temporal_activity_schedule_to_start_latency_seconds_bucket[5m])) by (le, namespace, task_queue))
```

**Activity Task Latency, average**

```
sum(increase(temporal_activity_schedule_to_start_latency_seconds_sum[5m])) by (namespace, task_queue)
/
sum(increase(temporal_activity_schedule_to_start_latency_seconds_count[5m])) by (namespace, task_queue)
```

**Target**

This latency should be very low, close to zero.
Any higher value indicates a bottleneck.

### Sync Match Rate {#sync-match-rate}

The sync match rate measures the rate of Tasks that are delivered to workers without having to be persisted (workers are up and available to pick them up) to the rate of all delivered tasks.

A sync match is when a task is immediately matched to a Worker via the Sticky Queue.

An async match is when a Task cannot be matched to the Sticky Queue for a Worker. This can happen when no Worker has cached the Workflow, or if the Task times out during processing. In this case, the Task returns to the general Task Queue.

**Calculate Sync Match Rate**

```
temporal_cloud_v0_poll_success_sync_count ÷ temporal_cloud_v0_poll_success_count = N
```

#### Prometheus query samples

**sync_match_rate query**

```
sum by(temporal_namespace) (
    rate(
        temporal_cloud_v0_poll_success_sync_count{temporal_namespace=~"$namespace"}[5m]
    )
)
/
sum by(temporal_namespace) (
    rate(
        temporal_cloud_v0_poll_success_count{temporal_namespace=~"$namespace"}[5m]
    )
)
```

**Target**

The Sync Match Rate should be at least >95%, but preferably >99%.

### Handling Task backlog issues {#task-backlog-handling}

Once you have detected the condition of a high Task backlog, consider the scenarios below to take action.

#### High Schedule To Start latency and high sync match rate

There are three typical causes for this:

- There are not enough workers to perform work
- Each worker is either under resourced, or is misconfigured, to handle enough work
- There is congestion caused by the environment (eg., network) hosting the worker(s) and Temporal Cloud

Consider

- Increasing either the number of available workers
- Verifying that your worker hosts are appropriately resourced
- Increasing the worker configuration value for concurrent pollers for workers/task executions (if your worker resources can accommodate the increased load)
- Doing some combination of these

#### High Schedule To Start latency and low sync match rate

Verify that you have not set a value for `ScheduleToStartTimeout` in your Activity Options. This may skew your observations.

It may be acceptable for your use case to have low sync match rate.
For example, if you have known workloads or you intentionally throttle tasks.

In this case it's also important to understand the fill and drain rates of the async tasks are during these windows:

Successful async polls

```
temporal_cloud_v0_poll_success_count - temporal_cloud_v0_poll_success_sync_count = N
```

```
sum(rate(temporal_cloud_v0_poll_success_count{temporal_namespace=~"$temporal_namespace"}[5m])) by (temporal_namespace, task_type)
-
sum(rate(temporal_cloud_v0_poll_success_sync_count{temporal_namespace=~"$temporal_namespace"}[5m])) by (temporal_namespace, task_type)
```

[//]: # (add `temporal_cloud_v1_approximate_backlog_count` once the v2 metrics has been GA'd)

**Actions**

- Verify that your Worker setup is optimized for your instance:
  - Check the system CPU usage against `task_slots` and adjust `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` settings as necessary.
  - Check the system memory usage against `sticky_cache_size` and adjust sticky cache size as necessary.
  - For a detailed explanation of settings, see the [Worker Performance](/develop/worker-performance#task-queues-processing-tuning) section.
- Increase the Worker config for concurrent pollers for Workflow or Activity `task_slots`, if your Worker resources can accommodate the increased load.
  - Reference [Worker Performance > Poller Count](/develop/worker-performance#poller-count).
- Increase the number of available Workers.

:::warning

Setting the [Schedule To Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) in your Activity Options can skew your observations.
Avoid setting a Schedule To Start Timeout when load testing for latency.

:::

## Detect greedy Worker resources {#detect-greedy-workers}

**How to detect greedy Worker resources.**

You can have too many Workers.
If you see the Poll Success Rate showing low numbers, you might have too many resources polling Temporal Cloud.

Metrics to monitor:

- **Temporal Cloud metric**: [temporal_cloud_v0_poll_success_count](/cloud/metrics/reference#temporal_cloud_v0_poll_success_count)
- **Temporal Cloud metric**: [temporal_cloud_v0_poll_success_sync_count](/cloud/metrics/reference#temporal_cloud_v0_poll_success_sync_count)
- **Temporal Cloud metric**: [temporal_cloud_v0_poll_timeout_count](/cloud/metrics/reference#temporal_cloud_v0_poll_timeout_count)
- **SDK metric**: [temporal_workflow_task_schedule_to_start_latency](/references/sdk-metrics#workflow_task_schedule_to_start_latency)
- **SDK metric**: [temporal_activity_schedule_to_start_latency](/references/sdk-metrics#activity_schedule_to_start_latency)

**Calculate Poll Success Rate**

```
(temporal_cloud_v0_poll_success_count + temporal_cloud_v0_poll_success_sync_count)
/
(temporal_cloud_v0_poll_success_count + temporal_cloud_v0_poll_success_sync_count + temporal_cloud_v0_poll_timeout_count)
```

**Target**

Poll Success Rate should be >90% in most cases of systems with a steady load.
For high volume and low latency, try to target >95%.

**Interpretation**

There may be too many Pollers for the amount of Workers.

If you see all of the following at the same time then you might have too many Workers:

- Low poll success rate
- Low Schedule To Start latency
- Low worker host resource utilization

**Actions**

Consider sizing down your Workers by either:

- Reducing the number of Workers polling the impacted Task Queue, OR
- Reducing the concurrent pollers per Worker, OR
- Both of the above

#### Prometheus query samples

**poll_success_rate query**

```
(
    (
        sum by(temporal_namespace) (
          rate(
            temporal_cloud_v0_poll_success_count{temporal_namespace=~"$namespace"}[5m]
          )
        )
      +
        sum by(temporal_namespace) (
          rate(
            temporal_cloud_v0_poll_success_sync_count{temporal_namespace=~"$namespace"}[5m]
          )
        )
    )
  /
    (
        (
            sum by(temporal_namespace) (
              rate(
                temporal_cloud_v0_poll_success_count{temporal_namespace=~"$namespace"}[5m]
              )
            )
          +
            sum by(temporal_namespace) (
              rate(
                temporal_cloud_v0_poll_success_sync_count{temporal_namespace=~"$namespace"}[5m]
              )
            )
        )
      +
        sum by(temporal_namespace) (
          rate(
            temporal_cloud_v0_poll_timeout_count{temporal_namespace=~"$namespace"}[5m]
          )
        )
    )
)
```

## Detect misconfigured Workers {#detect-misconfigured-workers}

**How to detect misconfigured Workers.**

Worker configuration can negatively affect Task processing efficiency.

Metrics to monitor:

- **SDK metric**: [temporal_worker_task_slots_available](/references/sdk-metrics#worker_task_slots_available)
- **SDK metric**: [sticky_cache_size](/references/sdk-metrics#sticky_cache_size)
- **SDK metric**: [sticky_cache_total_forced_eviction](/references/sdk-metrics#sticky_cache_total_forced_eviction)

**Execution Size Configuration**

The `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` define the number of total available slots for the Worker.
If this is set too low, the Worker will not be able to keep up processing Tasks.

**Target**

The `temporal_worker_task_slots_available` metric should always be >0.

#### Prometheus query samples

**Over Time**

```
avg_over_time(temporal_worker_task_slots_available{namespace="$namespace",worker_type="WorkflowWorker"}[10m])
```

**Current Time**

```
temporal_worker_task_slots_available{namespace="default", worker_type="WorkflowWorker", task_queue="$task_queue_name"}
```

**Interpretation**

You are likely experiencing a Task backlog if you are seeing inadequate slot counts frequently.
The work is not getting processed as fast as it should/can.

**Action**

Increase the `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` values and keep an eye on your Worker resource metrics (CPU utilization, etc) to make sure you haven't created a new issue.

### Configure Sticky Execution Cache {#configure-sticky-cache}

Sticky Execution means that a Worker caches a Workflow Execution Event History and creates a dedicated Task Queue to listen on.
It significantly improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories.

**Target**

The `sticky_cache_size` should report less than or equal to your `WorkflowCacheSize` value.
Also, sticky_cache_total_forced_eviction should not be reporting high numbers (relative).

**Action**

If you see a high eviction count, verify there are no other inefficiencies in your Worker configuration or resource provisioning (backlog).
If you see the cache size metric exceed the `WorkflowCacheSize`, increase this value if your Worker resources can accommodate it or provision more Workers.
Finally, take time to review [the Worker performance guide](/develop/worker-performance) and see if it addresses other potential cache issues.

#### Prometheus query samples

**Sticky Cache Size**

```
max_over_time(temporal_sticky_cache_size{namespace="$namespace"}[10m])
```

**Sticky Cache Evictions**

```
rate(temporal_sticky_cache_total_forced_eviction_total{namespace="$namespace"}[5m]))
```

## Manage Worker Heartbeating {#manage-worker-heartbeating}

:::tip SUPPORT, STABILITY, and DEPENDENCY INFO

This feature is currently in [Public Preview](/evaluate/development-production-features/release-stages#public-preview).

:::

Workers send a heartbeat to Temporal Server every 60 seconds by default. This heartbeat serves to provide liveness and configuration data from the Worker to the Server.
Specific data sent can be found in the [API](https://github.com/temporalio/api/blob/master/temporal/api/worker/v1/message.proto). By providing a consistent heartbeat from
Workers, the Server can obtain an accurate count of Workers, understand Worker performance, and respond to Worker heartbeats with commands. Some examples of how this is useful:

- understanding the difference between a Worker that is down and a Worker that is processing tasks for a long time
- identifying a Worker with high CPU usage from the Server point of view

Use the Temporal CLI to view information about all Workers connected to Temporal Server. Use `temporal worker describe` to see details of a specific Worker. Use `temporal worker list` to get a complete list of all connected Workers.

If you wish to disable Worker heartbeating (features above will not work with this feature disabled) or set heartbeating to be more frequent than every 60 seconds (allowed range is 1s to 60s), set the configuration relevant to your SDK.
<SdkTabs hideUnsupportedLanguages>
<SdkTabs.Python>
Use `TelemetryConfig()` to adjust heartbeat settings. See the [Python SDK documentation](https://python.temporal.io/temporalio.bridge.runtime.RuntimeOptions.html#worker_heartbeat_interval_millis) for more details.
</SdkTabs.Python>
<SdkTabs.Ruby>
Add configurations to `Runtime()` to adjust heartbeat settings. See the [Ruby SDK documentation](https://ruby.temporal.io/Temporalio/Runtime.html) for more details.
</SdkTabs.Ruby>
</SdkTabs>

---

## How to visualize an Activity Retry Policy with timeouts

Use this tool to visualize total Activity Execution times and experiment with different Activity timeouts and Retry Policies.
For a list of Activity Task Execution times, use [this calculator](https://temporal-time.netlify.app/?initialInterval=1&maxInterval=100&maxReties=10&backoffCoeificent=2).

The simulator is based on a common Activity use-case, which is to call a third party HTTP API and return the results.
See the example code snippets below.

Use the Activity Retries settings to configure how long the API request takes to succeed or fail.
There is an option to generate scenarios.
The _Task Time in Queue_ simulates the time the Activity Task might be waiting in the Task Queue.

Use the Activity Timeouts and Retry Policy settings to see how they impact the success or failure of an Activity Execution.

<RetrySimulator />

---

## Asynchronous Activity completion - .NET SDK

This page describes how to asynchronously complete an Activity.

[Asynchronous Activity Completion](/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing.

There are three steps to follow:

1. The Activity provides the external system with identifying information needed to complete the Activity Execution.
   Identifying information can be a [Task Token](/activity-execution#task-token), or a combination of Namespace, Workflow Id, and Activity Id.
2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system.
3. The Temporal Client is used to Heartbeat and complete the Activity.

To mark an Activity as completing asynchronously, do the following inside the Activity.

```csharp
// Capture token for later completion
capturedToken = ActivityExecutionContext.Current.Info.TaskToken;

// Throw special exception that says an activity will be completed somewhere else
throw new CompleteAsyncException();
```

To update an Activity outside the Activity, use the [GetAsyncActivityHandle()](https://dotnet.temporal.io/api/Temporalio.Client.ITemporalClient.html#Temporalio_Client_ITemporalClient_GetAsyncActivityHandle_System_Byte___) method to get the handle of the Activity.

```csharp
var handle = myClient.GetAsyncActivityHandle(capturedToken);
```

Then, on that handle, you can call the results of the Activity, `HeartbeatAsync`, `CompleteAsync`, `FailAsync`, or `ReportCancellationAsync` method to update the Activity.

```csharp
await handle.CompleteAsync("Completion value.");
```

---

## Benign exceptions - .NET SDK

**How to mark an Activity error as benign using the Temporal .NET SDK**

When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues.
By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic.

To mark an error as benign, set the `category` parameter to `ApplicationErrorCategory.Benign` when throwing an [`ApplicationFailureException`](https://dotnet.temporal.io/api/Temporalio.Exceptions.ApplicationFailureException.html).

Benign errors:
- Have Activity failure logs downgraded to DEBUG level
- Do not emit Activity failure metrics
- Do not set the OpenTelemetry failure status to ERROR

```csharp
using Temporalio.Activities;
using Temporalio.Api.Enums.V1;
using Temporalio.Exceptions;

public class MyActivities
{
    [Activity]
    public async Task<string> MyActivityAsync()
    {
        try
        {
            return await CallExternalServiceAsync();
        }
        catch (Exception e)
        {
            // Mark this error as benign since it's expected
            throw new ApplicationFailureException(
                "Service is down",
                inner: e,
                category: ApplicationErrorCategory.Benign);
        }
    }
}
```

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried.

---

## Interrupt a Workflow - .NET SDK

This page shows how to interrupt a Workflow Execution.

You can interrupt a Workflow Execution in one of the following ways:

- [Cancel](#cancellation): Canceling a Workflow provides a graceful way to stop Workflow Execution.
- [Terminate](#termination): Terminating a Workflow forcefully stops Workflow Execution.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

- The system records a `WorkflowExecutionTerminated` event in the Workflow History.
- The termination forcefully and immediately stops the Workflow Execution.
- The Workflow code gets no chance to handle termination.
- A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the
Workflow is stuck and cannot be canceled normally.

## Cancellation {#cancellation}

To give a Workflow and its Activities the ability to be cancelled, do the following:

- Handle a Cancellation request within a Workflow.
- Set Activity Heartbeat Timeouts.
- Listen for and handle a Cancellation request within an Activity.
- Send a Cancellation request from a Temporal Client.

### Handle Cancellation in Workflow {#handle-cancellation-in-workflow}

**How to handle a Cancellation in a Workflow in .NET.**

Workflow Definitions can be written to respond to cancellation requests. It is common for an Activity to be run on
Cancellation to perform cleanup.

Cancellation Requests on Workflows cancel the `Workflow.CancellationToken`. This is the token that is implicitly used
for all calls within the workflow as well (e.g. Timers, Activities, etc) and therefore cancellation is propagated to
them to be handled and bubble out.

```csharp
[WorkflowRun]
public async Task RunAsync()
{
    try
    {
        // Whether this workflow waits on the activity to handle the cancellation or not is
        // dependent upon the CancellationType option. We leave the default here which sends the
        // cancellation but does not wait on it to be handled.
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyNormalActivity(),
            new() { ScheduleToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
    catch (Exception e) when (TemporalException.IsCanceledException(e))
    {
        // The "when" clause above is because we only want to apply the logic to cancellation, but
        // this kind of cleanup could be done on any/all exceptions too.
        Workflow.Logger.LogError(e, "Cancellation occurred, performing cleanup");

        // Call cleanup activity. If this throws, it will swallow the original exception which we
        // are ok with here. This could be changed to just log a failure and let the original
        // cancellation continue. We use a different cancellation token since the default one on
        // Workflow.CancellationToken is now marked cancelled.
        using var detachedCancelSource = new CancellationTokenSource();
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyCancellationCleanupActivity(),
            new()
            {
                ScheduleToCloseTimeout = TimeSpan.FromMinutes(5),
                CancellationToken = detachedCancelSource.Token;
            });

        // Rethrow the cancellation
        throw;
    }
}
```

### Handle Cancellation in an Activity {#handle-cancellation-in-an-activity}

**How to handle a Cancellation in an Activity using the Temporal .NET SDK**

Ensure that the Activity is [Heartbeating](/develop/dotnet/failure-detection#activity-heartbeats) to receive the
Cancellation request and stop execution. Also make sure that the
[Heartbeat Timeout](/develop/dotnet/failure-detection#heartbeat-timeout) is set on the Activity Options when calling
from the Workflow. An Activity Cancellation Request cancels the `CancellationToken` on the `ActivityExecutionContext`.

```csharp
[Activity]
public async Task MyActivityAsync()
{
    // This is a naive loop simulating work, but similar heartbeat/cancellation logic applies to
    // other scenarios as well
    while (true)
    {
        // Send heartbeat
        ActivityExecutionContext.Current.Heartbeat();

        // Do some work, passing the cancellation token
        await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken);
    }
}
```

### Request Cancellation {#request-cancellation}

**How to request Cancellation of a Workflow using the Temporal .NET SDK**

Use `CancelAsync` on the `WorkflowHandle` to cancel a Workflow Execution.

```csharp
// Get a workflow handle by its workflow ID. This could be made specific to a run by passing run ID.
// This could also just be a handle that is returned from StartWorkflowAsync instead.
var handle = myClient.GetWorkflowHandle("my-workflow-id");

// Send cancellation. This returns when cancellation is received by the server. Wait on the handle's
// result to wait for cancellation to be applied.
await handle.CancelAsync();
```

**How to request Cancellation of an Activity in .NET using the Temporal .NET SDK**

By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation token
is used by activities by default. To issue a cancellation explicitly, a new cancellation token can be created.

```csharp
[WorkflowRun]
public async Task RunAsync()
{
    // Create a source linked to workflow cancellation. A new source could be created instead if we
    // didn't want it associated with workflow cancellation.
    using var cancelActivitySource = CancellationTokenSource.CreateLinkedTokenSource(
        Workflow.CancellationToken);

    // Start the activity. Whether this workflow waits on the activity to handle the cancellation
    // or not is dependent upon the CancellationType option. We leave the default here which sends
    // the cancellation but does not wait on it to be handled.
    var activityTask = Workflow.ExecuteActivityAsync(
        (MyActivities a) => a.MyNormalActivity(),
        new()
        {
            ScheduleToCloseTimeout = TimeSpan.FromMinutes(5),
            CancellationToken = cancelActivitySource.Token;
        });
    activityTask.Start();

    // Wait 5 minutes, then cancel it
    await Workflow.DelayAsync(TimeSpan.FromMinutes(5));
    cancelActivitySource.Cancel();

    // Wait on the activity which will throw cancellation which will fail the workflow
    await activityTask;
}
```

## Termination {#termination}

**How to Terminate a Workflow Execution in .NET using the Temporal .NET SDK**

To Terminate a Workflow Execution in .NET, use the
[TerminateAsync()](https://dotnet.temporal.io/api/Temporalio.Client.WorkflowHandle.html#Temporalio_Client_WorkflowHandle_TerminateAsync_System_String_Temporalio_Client_WorkflowTerminateOptions_)
method on the Workflow handle.

```csharp
// Get a workflow handle by its workflow ID. This could be made specific to a run by passing run ID.
// This could also just be a handle that is returned from StartWorkflowAsync instead.
var handle = myClient.GetWorkflowHandle("my-workflow-id");

// Terminate
await handle.TerminateAsync();
```

Workflow Executions can also be Terminated directly from the WebUI. In this case, a custom note can be logged from the
UI when that happens.

## Reset a Workflow Execution {#reset}

Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a
point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other
issues that prevent it from completing.

When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the
Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as
removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when
resetting, as it will be recorded in the Event History.

<Tabs>

<TabItem value="web-ui" label="Web UI">

1. Navigate to the Workflow Execution details page,
2. Click the **Reset** button in the top right dropdown menu,
3. Select the Event ID to reset to,
4. Provide a reason for the reset,
5. Confirm the reset.

The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

</TabItem>

<TabItem value="cli" label="Temporal CLI">

Use the `temporal workflow reset` command to reset a Workflow Execution:

```bash
temporal workflow reset \
    --workflow-id <workflow-id> \
    --event-id <event-id> \
    --reason "Reason for reset"
```

For example:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code"
```

By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a
specific run. Use `--namespace` to specify a different Namespace:

```bash
temporal workflow reset \
    --workflow-id my-background-check \
    --event-id 4 \
    --reason "Fixed non-deterministic code" \
    --namespace my-namespace \
    --tls-cert-path /path/to/cert.pem \
    --tls-key-path /path/to/key.pem
```

Monitor the new Workflow Execution after resetting to ensure it completes successfully.

</TabItem>

</Tabs>

---

## Child Workflows - .NET SDK

This page shows how to do the following:

- [Start a Child Workflow Execution](#child-workflows)
- [Set a Parent Close Policy](#parent-close-policy)

## Start a Child Workflow Execution {#child-workflows}

**How to start a Child Workflow Execution using the Temporal .NET SDK**

A [Child Workflow Execution](/child-workflows) is a Workflow Execution that is scheduled from within another Workflow using a Child Workflow API.

When using a Child Workflow API, Child Workflow related Events ([StartChildWorkflowExecutionInitiated](/references/events#startchildworkflowexecutioninitiated), [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted), [ChildWorkflowExecutionCompleted](/references/events#childworkflowexecutioncompleted), etc...) are logged in the Workflow Execution Event History.

Always block progress until the [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted) Event is logged to the Event History to ensure the Child Workflow Execution has started.
After that, Child Workflow Executions may be abandoned using the _Abandon_ [Parent Close Policy](/parent-close-policy) set in the Child Workflow Options.

To spawn a Child Workflow Execution in .NET, use the `ExecuteChildWorkflowAsync()` method which starts the Child Workflow and waits for completion or
use the `StartChildWorkflowAsync()` method to start a Child Workflow and return its handle.
This is useful if you want to do something after it has only started, or to get the Workflow/Run ID, or to be able to signal it while running.

:::note

`ExecuteChildWorkflowAsync()` is a helper method for `StartChildWorkflowAsync()` plus `await handle.GetResultAsync()`.

:::

```csharp
await Workflow.ExecuteChildWorkflowAsync((MyChildWorkflow wf) => wf.RunAsync());
```

## Set a Parent Close Policy {#parent-close-policy}

**How to set a Parent Close Policy using the Temporal .NET SDK**

A [Parent Close Policy](/parent-close-policy) determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed Out).

The default Parent Close Policy option is set to terminate the Child Workflow Execution.

Set the `ParentClosePolicy` property inside the [`ChildWorkflowOptions`](https://dotnet.temporal.io/api/Temporalio.Workflows.ChildWorkflowOptions.html) for `ExecuteChildWorkflowAsync` or `StartChildWorkflowAsync` to specify the behavior of the Child Workflow when the Parent Workflow closes.

```csharp
await Workflow.ExecuteChildWorkflowAsync(
  (MyChildWorkflow wf) => wf.RunAsync(),
  new() { ParentClosePolicy = ParentClosePolicy.Abandon });
```

---

## Continue-As-New - .NET SDK

This page answers the following questions for .NET developers:

- [What is Continue-As-New?](#what)
- [How to Continue-As-New?](#how)
- [When is it right to Continue-as-New?](#when)
- [How to test Continue-as-New?](#how-to-test)

## What is Continue-As-New? {#what}

[Continue-As-New](/workflow-execution/continue-as-new) lets a Workflow Execution close successfully and creates a new Workflow Execution.
You can think of it as a checkpoint when your Workflow gets too long or approaches certain scaling limits.

The new Workflow Execution is in the same [chain](/workflow-execution#workflow-execution-chain); it keeps the same Workflow Id but gets a new Run Id and a fresh Event History.
It also receives your Workflow's usual parameters.

## How to Continue-As-New using the .NET SDK {#how}

First, design your Workflow parameters so that you can pass in the "current state" when you Continue-As-New into the next Workflow run.
This state is typically set to `None` for the original caller of the Workflow.

  
    View the source code
  {' '}
  in the context of the rest of the application code.

```csharp
public record Input
    {
        public State State { get; init; } = new();

        public bool TestContinueAsNew { get; init; }
    }

[WorkflowInit]
public ClusterManagerWorkflow(Input input)

````
The test hook in the above snippet is covered [below](#how-to-test).

Inside your Workflow, throw a [`CreateContinueAsNewException`](https://dotnet.temporal.io/api/Temporalio.Workflows.ContinueAsNewException.html) exception.
This stops the Workflow right away and starts a new one.

  
    View the source code
  {' '}
  in the context of the rest of the application code.

```csharp
throw Workflow.CreateContinueAsNewException((ClusterManagerWorkflow wf) => wf.RunAsync(new()
{
    State = CurrentState,
    TestContinueAsNew = input.TestContinueAsNew,
}));
````

### Considerations for Workflows with Message Handlers {#with-message-handlers}

If you use Updates or Signals, don't call Continue-as-New from the handlers.
Instead, wait for your handlers to finish in your main Workflow before you throw `CreateContinueAsNewException`.
See the [`AllHandlersFinished`](message-passing#wait-for-message-handlers) example for guidance.

## When is it right to Continue-as-New using the .NET SDK? {#when}

Use Continue-as-New when your Workflow might hit [Event History Limits](/workflow-execution/event#event-history).

Temporal tracks your Workflow's progress against these limits to let you know when you should Continue-as-New.
Call `Workflow.ContinueAsNewSuggested` to check if it's time.

## How to test Continue-as-New using the .NET SDK {#how-to-test}

Testing Workflows that naturally Continue-as-New may be time-consuming and resource-intensive.
Instead, add a test hook to check your Workflow's Continue-as-New behavior faster in automated tests.

For example, when `TestContinueAsNew == true`, this sample creates a test-only variable called `maxHistoryLength` and sets it to a small value.
A helper variable in the Workflow checks it each time it considers using Continue-as-New:

  
    View the source code
  {' '}
  in the context of the rest of the application code.

```csharp
private bool ShouldContinueAsNew =>
    // Don't continue as new while update running
    Workflow.AllHandlersFinished &&
    // Continue if suggested or, for ease of testing, max history reached
    (Workflow.ContinueAsNewSuggested || Workflow.CurrentHistoryLength > maxHistoryLength);
```

---

## Converters and encryption - .NET SDK

Temporal's security model is designed around client-side encryption of Payloads.
A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server.
This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data.
It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client.
Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads.
Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows.
When working with sensitive data, you should always implement Payload encryption.

## Custom Payload Codec {#custom-payload-codec}

**How to use a custom Payload Codec using the .NET SDK**

Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps.
If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`.

The `IPayloadCodec` needs to implement `EncodeAsync()` and `DecodeAsync()` methods.
These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field.
Do not mutate the existing payloads.
Here is an example of an encryption codec that just uses base64 in each direction:

```csharp
public class EncryptionCodec : IPayloadCodec
{
    public Task<IReadOnlyCollection<Payload>> EncodeAsync(IReadOnlyCollection<Payload> payloads) =>
        Task.FromResult<IReadOnlyCollection<Payload>>(payloads.Select(p =>
        {
            return new Payload()
            {
                // Set our specific encoding. We may also want to add a key ID in here for use by
                // the decode side
                Metadata = { ["encoding"] = "binary/my-payload-encoding" },
                Data = ByteString.CopyFrom(Encrypt(p.ToByteArray())),
            };
        }).ToList());

    public Task<IReadOnlyCollection<Payload>> DecodeAsync(IReadOnlyCollection<Payload> payloads) =>
        Task.FromResult<IReadOnlyCollection<Payload>>(payloads.Select(p =>
        {
            // Ignore if it doesn't have our expected encoding
            if (p.Metadata.GetValueOrDefault("encoding") != "binary/my-payload-encoding")
            {
                return p;
            }
            // Decrypt
            return Payload.Parser.ParseFrom(Decrypt(p.Data.ToByteArray()));
        }).ToList());

    private byte[] Encrypt(byte[] data) => Encoding.ASCII.GetBytes(Convert.ToBase64String(data));

    private byte[] Decrypt(byte[] data) => Convert.FromBase64String(Encoding.ASCII.GetString(data));
}
```

**Set Data Converter to use custom Payload Codec**

When creating a client, the default `DataConverter` can be updated with the payload codec like so:

```csharp
var myClient = await TemporalClient.ConnectAsync(new("localhost:7233")
{
    DataConverter = DataConverter.Default with { PayloadCodec = new EncryptionCodec() },
});
```

- Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
- Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](/cli/workflow#show) command or when browsing the Web UI to view output.

For reference, see the [Encryption](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption) sample.

### Using a Codec Server

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints.
You create, operate, and manage access to your Codec Server in your own environment.
The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
Refer to the [Codec Server](/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server.

## Payload conversion

Temporal SDKs provide a default [Payload Converter](/payload-converter) that can be customized to convert a custom data type to [Payload](/dataconversion#payload) and back.

### Conversion sequence {#conversion-sequence}

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter.
You can set multiple encoding Payload Converters to run your conversions.
When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec.
Temporal's Converter architecture looks like this:

<CaptionedImage
    src="/img/info/converter-architecture.png"
    title="Temporal converter architecture"
/>

### Custom Payload Converter {#custom-payload-converter}

**How to use a custom Payload Converter with the .NET SDK.**

Data converters are used to convert raw Temporal payloads to/from actual .NET types.
A custom data converter can be set via the `DataConverter` option when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters.
Payload converters convert .NET values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures.

Data converters are in the `Temporalio.Converters` namespace.
The default data converter uses a default payload converter, which supports the following types:

- `null`
- `byte[]`
- `Google.Protobuf.IMessage` instances
- Anything that `System.Text.Json` supports
- `IRawValue` as unconverted raw payloads

Custom converters can be created for all uses. For example, to create client with a data converter that converts all C#
property names to camel case, you would:

```csharp
using System.Text.Json;
using Temporalio.Client;
using Temporalio.Converters;

public class CamelCasePayloadConverter : DefaultPayloadConverter
{
    public CamelCasePayloadConverter()
      : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase })
    {
    }
}

var client = await TemporalClient.ConnectAsync(new()
{
    TargetHost = "localhost:7233",
    Namespace = "my-namespace",
    DataConverter = DataConverter.Default with { PayloadConverter = new CamelCasePayloadConverter() },
});
```

<!--  ### Custom Type Data Conversion TODO(cretz): Develop https://github.com/temporalio/samples-dotnet/issues/38 first -->

---

## Core application - .NET SDK

This page shows how to do the following:

- [Develop a basic Workflow Definition](#develop-workflow)
- [Develop a basic Activity Definition](#develop-activity)
- [Start an Activity from a Workflow](#activity-execution)
- [Run a Worker Process](#run-worker-process)
- [Set a Dynamic Workflow](#set-a-dynamic-workflow)
- [Set a Dynamic Activity](#set-a-dynamic-activity)

## Develop a Workflow {#develop-workflow}

**How to develop a basic Workflow using the Temporal .NET SDK**

Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](/workflow-definition).

In the Temporal .NET SDK programming model, Workflows are defined as classes.

Specify the `[Workflow]` attribute from the `Temporalio.Workflows` namespace on the Workflow class to identify a Workflow.

Use the `[WorkflowRun]` attribute to mark the entry point method to be invoked. This must be set on one asynchronous method defined on the same class as `[Workflow]`.

```csharp
using Temporalio.Workflows;

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        var param = MyActivityParams("Hello", name);
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivity(param),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}
```

Temporal Workflows may have any number of custom parameters.
However, we strongly recommend that objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow.
All Workflow Definition parameters must be serializable.

### Workflow logic requirements {#workflow-logic-requirements}

Workflow logic is constrained by [deterministic execution requirements](/workflow-definition#deterministic-constraints).
Therefore, each language is limited to the use of certain idiomatic techniques.
However, each Temporal SDK provides a set of APIs that can be used inside your Workflow to interact with external (to the Workflow) application code.

This means there are several things Workflows cannot do such as:

- Perform IO (network, disk, stdio, etc)
- Access/alter external mutable state
- Do any threading
- Do anything using the system clock (e.g. `DateTime.Now`)
  - This includes .NET timers (e.g. `Task.Delay` or `Thread.Sleep`)
- Make any random calls
- Make any not-guaranteed-deterministic calls (e.g. iterating over a dictionary)

#### .NET Task Determinism

Some calls in .NET do unsuspecting non-deterministic things and are easy to accidentally use.
This is especially true with `Task`s.
Temporal requires that the deterministic `TaskScheduler.Current` is used, but many .NET async calls will use `TaskScheduler.Default` implicitly (and some analyzers even encourage this).
Here are some known gotchas to avoid with .NET tasks inside of Workflows:

- Do not use `Task.Run` - this uses the default scheduler and puts work on the thread pool.
  - Use `Workflow.RunTaskAsync` instead.
  - Can also use `Task.Factory.StartNew` with current scheduler or instantiate the `Task` and run `Task.Start` on it.
- Do not use `Task.ConfigureAwait(false)` - this will not use the current context.
  - If you must use `Task.ConfigureAwait`, use `Task.ConfigureAwait(true)`.
  - There is no significant performance benefit to `Task.ConfigureAwait` in workflows anyways due to how the scheduler works.
- Do not use anything that defaults to the default task scheduler.
- Do not use `Task.Delay`, `Task.Wait`, timeout-based `CancellationTokenSource`, or anything that uses .NET built-in timers.
  - `Workflow.DelayAsync`, `Workflow.WaitConditionAsync`, or non-timeout-based cancellation token source is suggested.
- Do not use `Task.WhenAny`.
  - Use `Workflow.WhenAnyAsync` instead.
  - Technically this only applies to an enumerable set of tasks with results or more than 2 tasks with results. Other
    uses are safe. See [this issue](https://github.com/dotnet/runtime/issues/87481).
- Do not use `Task.WhenAll`
  - Use `Workflow.WhenAllAsync` instead.
  - Technically `Task.WhenAll` is currently deterministic in .NET and safe, but it is better to use the wrapper to be
    sure.
- Do not use `CancellationTokenSource.CancelAsync`.
  - Use `CancellationTokenSource.Cancel` instead.
- Do not use `System.Threading.Semaphore` or `System.Threading.SemaphoreSlim` or `System.Threading.Mutex`.
  - Use `Temporalio.Workflows.Semaphore` or `Temporalio.Workflows.Mutex` instead.
  - _Technically_ `SemaphoreSlim` does work if only the async form of `WaitAsync` is used without no timeouts and
    `Release` is used. But anything else can deadlock the workflow and its use is cumbersome since it must be disposed.
- Be wary of additional libraries' implicit use of the default scheduler.
  - For example, while there are articles for `Dataflow` about [using a specific scheduler](https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-specify-a-task-scheduler-in-a-dataflow-block), there are hidden implicit uses of `TaskScheduler.Default`. For example, see [this bug](https://github.com/dotnet/runtime/issues/83159).

In order to help catch wrong scheduler use, by default the Temporal .NET SDK adds an event source listener for info-level task events.
While this technically receives events from all uses of tasks in the process, we make sure to ignore anything that is not running in a Workflow in a high performant way (basically one thread local check).
For code that does run in a Workflow and accidentally starts a task in another scheduler, an `InvalidWorkflowOperationException` will be thrown which "pauses" the Workflow (fails the Workflow Rask which continually retries until the code is fixed).
This is unfortunately a runtime-only check, but can help catch mistakes early. If this needs to be turned off for any reason, set `DisableWorkflowTracingEventListener` to `true` in Worker options.

In the near future for modern .NET versions we hope to use the
[new `TimeProvider` API](https://github.com/dotnet/runtime/issues/36617) which will allow us to control current time and
timers.

#### Workflow .editorconfig

Since Workflow code follows some different logic rules than regular C# code, there are some common analyzer rules that developers may want to disable.
To ensure these are only disabled for Workflows, current recommendation is to use the `.workflow.cs` extension for files containing Workflows.

Here are the rules to disable:

- [CA1024](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1024) - This encourages properties instead of methods that look like getters. However for reflection reasons we cannot use property getters for queries, so it is very normal to have

  ```csharp
  [WorkflowQuery]
  public string GetSomeThing() => someThing;
  ```

- [CA1822](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1822) - This encourages static methods when methods don't access instance state. Workflows however use instance methods for run, Signals, Queries, or Updates even if they could be static.
- [CA2007](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2007) - This encourages users to use `ConfigureAwait` instead of directly waiting on a task. But in Workflows, there is no benefit to this and it just adds noise (and if used, needs to be `ConfigureAwait(true)` not `ConfigureAwait(false)`).
- [CA2008](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca2008) - This encourages users to always apply an explicit task scheduler because the default of `TaskScheduler.Current` is bad. But for Workflows, the default of `TaskScheduler.Current` is good/required.
- [CA5394](https://learn.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca5394) - This discourages use of non-crypto random. But deterministic Workflows, via `Workflow.Random` intentionally provide a deterministic non-crypto random instance.
- `CS1998` - This discourages use of `async` on async methods that don't `await`. But Workflows handlers like Signals are often easier to write in one-line form this way, e.g. `public async Task SignalSomethingAsync(string value) => this.value = value;`.
- [VSTHRD105](https://github.com/microsoft/vs-threading/blob/main/doc/analyzers/VSTHRD105.md) - This is similar to `CA2008` above in that use of implicit current scheduler is discouraged. That does not apply to Workflows where it is encouraged/required.

Here is the `.editorconfig` snippet for the above which may frequently change as more analyzers need to be adjusted:

```ini
##### Configuration specific for Temporal workflows #####
[*.workflow.cs]