# Run the worker
CMD ["python", "worker.py"]
```

Build the Docker image and target the `linux/amd64` architecture:

```bash
docker buildx build \
    --platform linux/amd64 \
    -t your-app .
```

## Publish the Worker Image to Amazon ECR

After building the Docker image, you’re ready to publish it to Amazon ECR.
Make sure that you’re authenticated with AWS, and that you’ve set your `AWS_REGION` and `AWS_ACCOUNT_ID` environment variables:

```bash
export AWS_ACCOUNT_ID=<your_aws_account_id>
export AWS_REGION=<your_aws_region>
```

Create an ECR repository and authenticate ECR with the Docker container client:

```bash
aws ecr create-repository \
    --repository-name your-app
aws ecr get-login-password --region $AWS_REGION | \
    docker login --username AWS --password-stdin \
            $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
```

After authenticating Docker with ECR, tag your container and publish it:

```bash
docker tag your-app $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest
```

## Deploy the Workers to EKS

With your Worker containerized, you’re ready to deploy it to EKS. Create a namespace in your EKS cluster. You’ll use the namespace to run your Temporal Workers:

```bash
kubectl create namespace your-namespace
```

Create a `ConfigMap` to hold non-sensitive values that Kubernetes will inject into your Worker deployment.
These enable dynamic routing for instances, Namespaces, and Task Queues.
To set these values, build a config-map.yaml file like the following example:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: temporal-worker-config
  namespace: temporal-system
data:
  TEMPORAL_HOST_URL: “<your-temporal-address>“
  TEMPORAL_NAMESPACE: “<your-temporal-cloud-namespace>”
  TEMPORAL_TASK_QUEUE: “<your-task-queue>”
```

Apply the `ConfigMap` to your namespace:

```bash
kubectl apply -f config-map.yaml \
    --namespace your-namespace
```

For sensitive values, use Kubernetes Secrets.
Create a secret to hold your Temporal API key:

```bash
kubectl create secret generic temporal-secret \
    --from-literal=TEMPORAL_API_KEY=$TEMPORAL_API_KEY \
    --namespace your-namespace
```

With your configuration in place, you can deploy the Worker.
Create a deployment.yaml file configuring your Worker image, resources, and secret values.
For common deployments, tune the resources you specify so they match your production workloads.
Note that the spun-up container reads your Temporal API key from the Kubernetes secret you just created:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
   name: your-app
   namespace: your-namespace
   labels:
      app: your-app
spec:
   selector:
      matchLabels:
         app: your-app
   replicas: 1
   template:
      metadata:
         labels:
            app: your-app
      spec:
         serviceAccountName: your-app
         containers:
            - name: your-app
              image: <your-ecr-image-name>
              env:
                - name: TEMPORAL_ADDRESS
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_ADDRESS
                - name: TEMPORAL_NAMESPACE
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_NAMESPACE
                - name: TEMPORAL_TASK_QUEUE
                  valueFrom:
                    configMapKeyRef:
                      name: temporal-worker-config
                      key: TEMPORAL_TASK_QUEUE
                - name: TEMPORAL_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: temporal-secret
                      key: TEMPORAL_API_KEY
              resources:
                limits:
                  cpu: "0.5"
                  memory: "512Mi"
                requests:
                  cpu: "0.2"
                  memory: "256Mi"
```

Apply the deployment.yaml file to the EKS cluster:

```bash
kubectl apply -f deployment.yaml \
    --namespace your-namespace
```

## Verify that the Workers are Connected

After deploying your Workers to EKS, confirm that they have connected to Temporal Cloud.
Retrieve the pod listing for the Kubernetes/EKS namespace that you created:

```
kubectl get pods -n temporal-system
```

After listing the pods, access the Worker logs to confirm you’re properly connected to Temporal Cloud:

```
kubectl logs <pod-name> -n temporal-system
```

You confirm connection when you see:

```
Initializing worker...
Starting worker... Waiting for tasks.
```

You have now successfully deployed your Temporal Worker to EKS.

---

## Temporal Worker Deployments

A core feature of Temporal is that you are able to deploy your Workers to any infrastructure where your Workflow and Activity code will actually run.
This way, you have total control over your runtime environment, and can be responsive to any security or scaling needs that may arise over time, whether you are using Temporal Cloud or self-hosting a Temporal Service.

If you are just getting started, you want more guidance, or a refresher on Temporal concepts, our [Tutorials and Courses](https://learn.temporal.io/) help by using only one or two Temporal Workers to demonstrate core functionality.
Once you have an understanding of the core concepts, the content in this section will provide clarity on real-world deployments that grow far beyond those examples.

Our Worker Deployments guide provides documentation of Temporal product features that make it easier to scale and revise your Workflows.

[Worker Versioning](/production-deployment/worker-deployments/worker-versioning) allows you to pin Workflows to individual versions of your workers, which are called Worker Deployment Versions.

You can optionally use the Temporal [Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to programmatically manage and scale your Worker deployments in Kubernetes pods.

This section also covers specific Worker Deployment examples:

- [**Deploy Workers to Amazon EKS**](/production-deployment/worker-deployments/deploy-workers-to-aws-eks)
  Containerize your Worker, publish it to Amazon Elastic Container Registry (ECR), and deploy it to Amazon Elastic Kubernetes Service (EKS) using the Temporal Python SDK.
  This guide covers the full deployment lifecycle and shows how to configure your Worker to connect to Temporal Cloud using Kubernetes-native tools like ConfigMaps and Secrets.
  Running Workers on EKS gives you fine-grained control over scaling, resource allocation, and availability—ideal for production systems that need reliability and flexibility in the cloud.

---

## Temporal Worker Controller

The [Temporal Worker Controller](https://github.com/temporalio/temporal-worker-controller) provides automation to enable rainbow deployments of your Workers by simplifying the tracking of which versions still have active Workflows, managing the lifecycle of versioned Worker deployments, and calling Temporal APIs to update the routing config of Temporal Worker Deployments.
The Temporal Worker Controller makes it simple and safe to deploy Temporal Workers on Kubernetes.

### Why adopt the Worker Controller?

The traditional approach to revising Temporal Workflows is to add branches using the [Versioning APIs](/workflow-definition#workflow-versioning).
Over time these checks can become a source of technical debt, as safely removing them from a codebase is a careful process that often involves querying all running Workflows.

[Worker Versioning](/production-deployment/worker-deployments/worker-versioning) is a Temporal feature that allows you to pin Workflows to individual versions of your Workers, which are called Worker Deployment Versions.
Using pinning, you will not need to add branching to your Workflows to avoid non-determinism errors.
This allows you to bypass the other Versioning APIs.

The Worker Controller gives you direct, programmatic control over your Worker deployments, and integrates with the [Temporal CLI](/production-deployment/worker-deployments/worker-versioning#rolling-out-changes-with-the-cli).
You do not need to use the Worker Controller to use Worker Versioning, but when used together, Worker Versioning and the Worker Controller can provide more graceful deployments and upgrades, and less need to manually tune your Workers.

Note that in Temporal, **Worker Deployment** is sometimes referred to as **Deployment**, but since the Worker Controller makes significant references to Kubernetes Deployment resource, within this page we will stick to these terms:

- [**Worker Deployment**](/worker-versioning#deployments): A Worker Deployment is a logical service that groups similar Workers together for unified management. Each Deployment has a name (such as your service name) and supports versioning through a series of Worker Deployment Versions.
- [**Worker Deployment Version**](/worker-versioning#deployment-versions): A Worker Deployment Version represents an iteration of a Worker Deployment. Each Deployment Version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Deployment Version to the Temporal Server.
- **Deployment**: A Kubernetes Deployment resource. A Deployment is "versioned" if it is running versioned Temporal workers/pollers.

### Features

- Registration of new Temporal Worker Deployment Versions
- Creation of versioned Deployment resources (that manage the Pods that run your Temporal pollers)
- Deletion of resources associated with drained Worker Deployment Versions
- `Manual`, `AllAtOnce`, and `Progressive` rollouts of new versions
- Ability to specify a "gate" Workflow that must succeed on the new version before routing real traffic to that version
- [Autoscaling](/develop/worker-performance#recommended-approach) of versioned Deployments

Refer to the [Temporal Worker Controller repo](https://github.com/temporalio/temporal-worker-controller/) for usage details.

## Configuring Worker Lifecycles

To use the Temporal Worker Controller, tag your Workers following the guidance for using [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).

Here is an example of a progressive rollout strategy gated on the success of the `HelloWorld` Workflow:

```
rollout:
  strategy: Progressive
  steps:
    - rampPercentage: 1
      pauseDuration: 30s
    - rampPercentage: 10
      pauseDuration: 1m
  gate:
    workflowType: "HelloWorld"
```

As you ship new deployment versions, the Worker Controller automatically detects them and gradually makes that version the new **Current Version** of the Worker deployment it is a part of.
As older pinned Workflows finish executing and deprecated deployment versions become drained, the Worker Controller also frees up resources by sunsetting the `Deployment` resources polling those versions.

## Running the Temporal Worker Controller

You can install the Temporal Worker Controller using our Helm chart:

```bash
RELEASE=temporal-worker-controller
NAMESPACE=temporal-system
VERSION=1.0.0

helm install $RELEASE oci://docker.io/temporalio/helm-charts/temporal-worker-controller \
  --version $VERSION \
  --namespace $NAMESPACE \
  --create-namespace
  
helm install temporal-worker-controller ./helm/temporal-worker-controller \
  --namespace $NAMESPACE \
  --create-namespace 
```

Refer to [GitHub](https://github.com/temporalio/temporal-worker-controller/tree/main/helm/temporal-worker-controller/templates) for other Worker Controller deployment templates.

---

## Worker Versioning(Worker-deployments)

Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on your Workers without breaking them.
Temporal enables this by helping you manage different builds or versions, formally called [Worker Deployment Versions](/worker-versioning#deployment-versions).

Worker Versioning unlocks important benefits for users of [blue-green or rainbow deployments](#deployment-systems).

- Ramping traffic gradually to a new Worker Deployment Version.
- Verifying a new Deployment Version with tests before sending production traffic to it.
- Instant rollback when you detect that a new Deployment Version is broken.
- Improved error rates when adopting it.

In addition, Worker Versioning introduces **Workflow Pinning**.
For pinned Workflow Types, each execution runs entirely on the Worker Deployment Version where it started.
You need not worry about making breaking code changes to running, pinned Workflows.

To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems).

:::tip
Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo.

    <iframe width="560" height="315"
        src="https://www.youtube.com/embed/rm4BlD9WXqc"
        title="YouTube video player"
        frameBorder="0"
        allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
        referrerPolicy="strict-origin-when-cross-origin" allowFullScreen></iframe>

:::

:::note
Worker Versioning is currently available in Public Preview.

Minimum versions:

- Go SDK version [v1.35.0](https://github.com/temporalio/sdk-go/releases/tag/v1.35.0)
- Python [v1.11](https://github.com/temporalio/sdk-python/releases/tag/1.11.0)
- Java [v1.29](https://github.com/temporalio/sdk-java/releases/tag/v1.29.0)
- Typescript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0)
- .NET [v1.7.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.7.0)
- Ruby [v0.5.0](https://github.com/temporalio/sdk-ruby/releases/tag/v0.5.0)
- Other SDKs: coming soon!

Self-hosted users:

- Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1)
- Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1)
- Minimum Temporal UI Version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0)
  :::

## Getting Started with Worker Versioning {#definition}

To get started with Worker Versioning, you should understand some concepts around versioning and deployments.

- A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two active Deployment Versions can run at once.
- A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on multiple Task Queues, but they all run the same build.
- A **Build ID**, in combination with a Worker Deployment name, identifies a single Worker Deployment Version.
- When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's Worker Deployment controls how the task queue matches Workflow Tasks with Workers.
- Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or Auto-Upgrade.
  - A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version.
  - An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single Deployment Version and need to be kept replay-safe manually, i.e. with [patching](/workflow-definition#workflow-versioning).
  - Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their Worker Deployment.
  - Pinned Workflows are designed for use with rainbow deployments. See [Deployment Systems](#deployment-systems).
  - Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete.
  - If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark your Workflow Type as Auto-Upgrade.
- Each Worker Deployment has a single [**Current Version**](/worker-versioning#versioning-definitions) which is where Workflows are routed to unless they were previously pinned on a different version.
- Each Worker Deployment can have a [**Ramping Version**](/worker-versioning#versioning-definitions) which is where a configurable percentage of Workflows are routed to unless they were previously pinned on a different version.
- For a given Workflow, its [**Target Worker Deployment Version**](/worker-versioning#versioning-definitions) is the version it will move to next.

## Setting up your deployment system {#deployment-systems}

If you haven't already, you'll want to pick a container deployment solution for your Workers.

You also need to pick among three common deployment strategies:

- A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower availability than the other strategies and are incompatible with Worker Versioning.
- A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can control how traffic is routed between them. This allows you to maximize your uptime with features like instant rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need.
- A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know when all the Workflows of a given version are drained so that you can sunset it.

:::note
You also have the option to use the [Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to automatically enable rainbow deployments of your Workers if you're using Kubernetes.
:::

## Configuring a Worker for Versioning

You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning.
There are three new parameters, with different names depending on the language:

- `UseVersioning`: This enables the Versioning functionality for this Worker.
- A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a deployment name and a build ID number.
- (Optional) The [Default Versioning Behavior](#definition). If unset, you'll be required to specify the behavior on each Workflow. Or you can default to Pinned or Auto-Upgrade.

Follow the example for your SDK below:

<SdkTabs>
<SdkTabs.Go>
```go
buildID:= mustGetEnv("MY_BUILD_ID")
w := worker.New(c, myTaskQueue, worker.Options{
  DeploymentOptions: worker.DeploymentOptions{
    UseVersioning: true,
    Version: worker.WorkerDeploymentVersion{
      DeploymentName: "llm_srv",
      BuildId:        buildID,
    },
    DefaultVersioningBehavior: workflow.VersioningBehaviorUnspecified,
  },
})
```
</SdkTabs.Go>
<SdkTabs.Java>
```java

  WorkerOptions.newBuilder()
  .setDeploymentOptions(
      WorkerDeploymentOptions.newBuilder()
      .setVersion(new WorkerDeploymentVersion("llm_srv", "1.0"))
      .setUseVersioning(true)
      .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE)
      .build())
  .build();

```
</SdkTabs.Java>
<SdkTabs.Python>
```python
from temporalio.common import WorkerDeploymentVersion, VersioningBehavior
from temporalio.worker import Worker, WorkerDeploymentConfig

Worker(
    client,
    task_queue="mytaskqueue",
    workflows=workflows,
    activities=activities,
    deployment_config=WorkerDeploymentConfig(
        version=WorkerDeploymentVersion(
            deployment_name="llm_srv",
            build_id=my_env.build_id),
        use_worker_versioning=True,
        default_versioning_behavior=VersioningBehavior.UNSPECIFIED
    ),
)
```

</SdkTabs.Python>
<SdkTabs.TypeScript>

```ts
const myWorker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue,
  workerDeploymentOptions: {
    useWorkerVersioning: true,
    version: { buildId: '1.0', deploymentName: 'llm_srv' },

    defaultVersioningBehavior: 'UNSPECIFIED',
  },
  connection: nativeConnection,
});
```

</SdkTabs.TypeScript>
<SdkTabs.DotNet>

```csharp
var myWorker = new TemporalWorker(
    Client,
    new TemporalWorkerOptions(taskQueue)
    {DeploymentOptions = new(new("llm_srv", "1.0"), true)
      { DefaultVersioningBehavior = VersioningBehavior.Unspecified },
    }.AddWorkflow<MyWorkflow>());
```

</SdkTabs.DotNet>
<SdkTabs.Ruby>

```ruby
worker = Temporalio::Worker.new(
  client: client,
  task_queue: task_queue,
  workflows: [MyWorkflow],
  deployment_options: Temporalio::Worker::DeploymentOptions.new(
      version: Temporalio::WorkerDeploymentVersion.new(
          deployment_name: 'llm_srv',
          build_id: '1.0'
      ),
      use_worker_versioning: true,
      default_versioning_behavior: Temporalio::VersioningBehavior::UNSPECIFIED
  )
)
```

</SdkTabs.Ruby>
</SdkTabs>

### Which Default Versioning Behavior should you choose?

If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`.

In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be short-running for the foreseeable future, you can default to Pinned.

Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to annotate their Workflows.
This default is the most similar to the legacy behavior.
Once each Workflow Type is annotated, you can remove the `DefaultVersioningBehavior`.

There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks or slowing down.

For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down, you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the Current Version are blocking the queue.

:::note
Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not Unspecified.
:::

You also want to make sure you understand how your Activities are going to work across different Worker Deployment Versions. Refer to the [Worker Versioning Activitiy behavior docs](/worker-versioning#actvity-behavior-across-versions) for more details.

## Rolling out changes with the CLI

Next, deploy your Worker with the additional configuration parameters.
Before making any Workflow revisions, you can use the `temporal` CLI to check which of your Worker versions are currently polling:

You can view the Versions that are part of a Deployment with `temporal worker deployment describe`:

```bash
temporal worker deployment describe --name="$MY_DEPLOYMENT"
```

To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name and a Build ID:

```bash
temporal worker deployment set-current-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID"
```

To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the same parameters and a ramping percentage:

```bash
temporal worker deployment set-ramping-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID" \
    --percentage=5
```

You can verify that Workflows are cutting over to that version with `describe -w YourWorkflowID`:

```bash
temporal workflow describe -w YourWorkflowID
```

That returns the new Version that the workflow is running on:

```
Versioning Info:

  Behavior               AutoUpgrade
  Version                llm_srv.2.0
  OverrideBehavior       Unspecified
```

## Marking a Workflow Type as Pinned

You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter.
This will cause it to remain on its original deployed version:

<SdkTabs>
<SdkTabs.Go>
```go
// w is the Worker configured as in the previous example
w.RegisterWorkflowWithOptions(HelloWorld, workflow.RegisterOptions{
	// or workflow.VersioningBehaviorAutoUpgrade
    VersioningBehavior: workflow.VersioningBehaviorPinned,
})
```
</SdkTabs.Go>
<SdkTabs.Java>
```java
@WorkflowInterface
public interface HelloWorld {
    @WorkflowMethod
    String hello();
}

public static class HelloWorldImpl implements HelloWorld {
    @Override
    @WorkflowVersioningBehavior(VersioningBehavior.PINNED)
    public String hello() {
        return "Hello, World!";
    }
}
```
</SdkTabs.Java>
<SdkTabs.Python>
```python
@workflow.defn(versioning_behavior=VersioningBehavior.PINNED)
class HelloWorld:
    @workflow.run
    async def run(self):
        return "hello world!"
```

</SdkTabs.Python>
<SdkTabs.TypeScript>

```ts
setWorkflowOptions({ versioningBehavior: 'PINNED' }, helloWorld);
export async function helloWorld(): Promise<string> {
  return 'hello world!';
}
```

</SdkTabs.TypeScript>
<SdkTabs.DotNet>

```csharp
[Workflow(VersioningBehavior = VersioningBehavior.Pinned)]
public class HelloWorld
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        return "hello world!";
    }
}
```

</SdkTabs.DotNet>
<SdkTabs.Ruby>

```ruby
class HelloWorld < Temporalio::Workflow::Definition
  workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED

  def execute
    'hello world!'
  end
end
```

</SdkTabs.Ruby>
</SdkTabs>

## Moving a pinned Workflow

Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the fix.

If you need to move a pinned Workflow to a new version, use `temporal workflow update-options`:

```bash
temporal workflow update-options \
    --workflow-id "$WORKFLOW_ID" \
    --versioning-override-behavior pinned \
    --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \
    --versioning-override-build-id "$TARGET_BUILD_ID"
```

You can move several Workflows at once matching a `--query` parameter:

```bash
temporal workflow update-options \
  --query="TemporalWorkerDeploymentVersion=$TARGET_DEPLOYMENT:$BAD_BUILD_ID" \
  --versioning-override-behavior pinned \
  --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \
  --versioning-override-build-id "$FIXED_BUILD_ID"
```

In this scenario, you may also need to use the other [Versioning APIs](/workflow-definition#workflow-versioning) to patch
your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly.
If you made a [version-incompatible change](/workflow-definition#deterministic-constraints) to your Workflow, and you
want to roll back to an earlier version, it's not possible to patch it. Considering using [Workflow Reset](/workflow-execution/event#reset)
along with your move.

"Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset Workflow,
so when it resumes execution, all new Workflow Tasks will be executed on your new Worker.

```bash
temporal workflow reset with-workflow-update-options \
    --workflow-id "$WORKFLOW_ID" \
    --event-id "$EVENT_ID" \
    --reason "$REASON" \
    --versioning-override-behavior pinned \
    --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \
    --versioning-override-build-id "$TARGET_BUILD_ID"
```

## Migrating a Workflow from Pinned to Auto-Upgrade

There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake.

Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a new build, but that would just push the problem to the next build.

In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade. You can use `temporal workflow update-options` for this:

```bash
temporal workflow update-options \
    --workflow-id "$WORKFLOW_ID" \
    --versioning-override-behavior auto_upgrade
```

If you want to move all your Workflows of a certain type to this new configuration, you can do it with this command:

```bash
temporal workflow update-options \
    --query="WorkflowType='$WORKFLOW_TYPE'" \
    --versioning-override-behavior auto_upgrade
```

You can also filter on a certain build ID to limit the number of Workflows you apply it to:

```bash
temporal workflow update-options \
    --query="WorkflowType='$WORKFLOW_TYPE' AND TemporalWorkerDeploymentVersion='$TARGET_DEPLOYMENT:$OLD_VERSION'" \
    --versioning-override-behavior auto_upgrade
```

:::note
When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](/patching#patching) the Workflow code.
:::

## Sunsetting an old Deployment Version

A Worker Deployment Version moves through the following states:

1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never becomes Active, it will never be Draining or Drained.
2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade Workflows.
3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated periodically.
4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed.

You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each `VersionSummary`, or by describing the version directly.
When a version is Draining or Drained, that is displayed in a value called `DrainageStatus`.
Periodically, the Temporal Service will refresh this status by counting any open pinned Workflows using that version.

On each refresh, `DrainageInfo.last_checked_time` is updated.
Eventually, `DrainageInfo` will report that the version is fully drained.
At this point, no Workflows are still running on that version and no more will be automatically routed to it, so you can consider shutting down the running Workers.

You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with `temporal worker deployment describe-version`:

```bash
temporal worker deployment describe-version \
    --deployment-name "YourDeploymentName" \
    --build-id "YourBuildID"
```

```
Worker Deployment Version:
  Version                  llm_srv.1.0
  CreateTime               5 hours ago
  RoutingChangedTime       32 seconds ago
  RampPercentage           0
  DrainageStatus           draining
  DrainageLastChangedTime  31 seconds ago
  DrainageLastCheckedTime  31 seconds ago

Task Queues:
     Name        Type
  hello-world  activity
  hello-world  workflow
```

If you have implemented [Queries](/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep some Workers running to handle them.

### Adding a pre-deployment test

Before deploying a new Workflow revision, you can test it with synthetic traffic.

To do this, use pinning in your tests, following the examples following

<SdkTabs>
<SdkTabs.Go>
```go
workflowOptions := client.StartWorkflowOptions{
	ID:        "MyWorkflowId",
	TaskQueue: "MyTaskQueue",
	VersioningOverride: &client.PinnedVersioningOverride{
        Version: worker.WorkerDeploymentVersion{
            DeploymentName: "DeployName",
            BuildId:        "1.0",
        },
    },
}
// c is an initialized Client
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, HelloWorld, "Hello")
```
</SdkTabs.Go>
<SdkTabs.Java>
```java
MyWorkflow handle = client.newWorkflowStub(
    MyWorkflow.class,
    WorkflowOptions.newBuilder()
        .setWorkflowId("MyWorkflowId")
        .setTaskQueue("MyTaskQueue")
        .setVersioningOverride(new VersioningOverride.PinnedVersioningOverride(
            new WorkerDeploymentVersion("DeployName", "1.0")))
        .build()
);
WorkflowExecution we = WorkflowClient.start(handle::execute, "Hello");
```
</SdkTabs.Java>
<SdkTabs.Python>
```python
handle = client.start_workflow(
    MyWorkflow.run,
    "Hello",
    id="MyWorkflowId",
    task_queue="MyTaskQueue",
    versioning_override=PinnedVersioningOverride(
        WorkerDeploymentVersion("DeployName", "1.0")
    ),
)
```
</SdkTabs.Python>
<SdkTabs.TypeScript>
```ts
const handle = await client.workflow.start('helloWorld', {
  taskQueue: 'MyTaskQueue',
  workflowId: 'MyWorkflowId',
  versioningOverride: {
    pinnedTo: { buildId: '1.0', deploymentName: 'deploy-name' },
  },
});
```
</SdkTabs.TypeScript>
<SdkTabs.DotNet>
```csharp
var workerV1 = new WorkerDeploymentVersion("deploy-name", "1.0");
var handle = await Client.StartWorkflowAsync(
    (HelloWorld wf) => wf.RunAsync(),
      	new(id: "MyWorkflowId", taskQueue: "MyTaskQueue")
      	{
           VersioningOverride = new VersioningOverride.Pinned(workerV1),
        }
);
```
</SdkTabs.DotNet>
<SdkTabs.Ruby>
```ruby
worker_v1 = Temporalio::WorkerDeploymentVersion.new(
  deployment_name: 'deploy-name',
  build_id: '1.0'
)
handle = env.client.start_workflow(
  HelloWorld,
  id: 'MyWorkflowId',
  task_queue: 'MyTaskQueue',
  versioning_override: Temporalio::VersioningOverride.pinned(worker_v1)
)
```
</SdkTabs.Ruby>
</SdkTabs>

## Garbage collection

Worker Deployments are never garbage collected, but *Worker Deployment Versions* (often referred to as Versions, Worker Versions, Deployment Versions) are.

Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to [`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/a3a53266c002ae33b630a41977274f8b5b587031/common/dynamicconfig/constants.go#L1317-L1321), which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed.

For example, when you deploy your 101st Worker version in a Worker Deployment, the server looks at the oldest drained version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version still has pollers, the server will try the next oldest version.
If none of the 100 versions are eligible for deletion (ie. none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version would fail.

At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment` or stop polling from one of the old drained versions to make it eligible for clean up.

If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and Deployment Name as the deleted version and the server will recreate it.

This covers the complete lifecycle of working with Worker Versioning.
We are continuing to improve this feature, and we welcome any feedback or feature requests using the sidebar link!

---

## Quickstarts


Choose your language to get started quickly.

<QuickstartCards
  items={[
    { href: "/develop/go/set-up-your-local-go", title: "Go", description: "Install the Go SDK and run a Hello World Workflow in Go." },
    { href: "/develop/java/set-up-your-local-java", title: "Java", description: "Install the Java SDK and run a Hello World Workflow in Java." },
    { href: "/develop/php/set-up-your-local-php", title: "PHP", description: "Install the PHP SDK and run a Hello World Workflow in PHP." },
    { href: "/develop/python/set-up-your-local-python", title: "Python", description: "Install the Python SDK and run a Hello World Workflow in Python." },
    { href: "/develop/ruby/set-up-local-ruby", title: "Ruby", description: "Install the Ruby SDK and run a Hello World Workflow in Ruby." },
    { href: "/develop/typescript/set-up-your-local-typescript", title: "TypeScript", description: "Install the TypeScript SDK and run a Hello World Workflow in TypeScript." },
    { href: "/develop/dotnet/set-up-your-local-dotnet", title: ".NET", description: "Install the .NET SDK and run a Hello World Workflow in C#." },
  ]}
/>

---

## Environment configuration(References)

The following table details all available settings, their corresponding environment variables, and their TOML file
paths. For more information on using environment variables and configuration files to set up your Temporal Client, refer
to the [Environment Configuration](/develop/environment-configuration).

| Setting                   | Environment Variable                     | TOML Path                                      | Description                                                                                                                                                                                                                 |
| :------------------------ | :--------------------------------------- | :--------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Configuration File Path   | `TEMPORAL_CONFIG_FILE`                   | **NA**                                         | Path to the TOML configuration file                                                                                                                                                                                         |
| Server Address            | `TEMPORAL_ADDRESS`                       | `profile.<name>.address`                       | The host and port of the Temporal Frontend service (e.g., "localhost:7233").                                                                                                                                                |
| Namespace                 | `TEMPORAL_NAMESPACE`                     | `profile.<name>.namespace`                     | The Temporal Namespace to connect to.                                                                                                                                                                                       |
| API Key                   | `TEMPORAL_API_KEY`                       | `profile.<name>.api_key`                       | An API key for authentication. If present, TLS is enabled by default.                                                                                                                                                       |
| Enable/Disable TLS        | `TEMPORAL_TLS`                           | `profile.<name>.tls.disabled`                  | Set to "true" to enable TLS, "false" to disable. In TOML, disabled = true turns TLS off.                                                                                                                                    |
| Client Certificate        | `TEMPORAL_TLS_CLIENT_CERT_DATA`          | `profile.<name>.tls.client_cert_data`          | The raw PEM data containing the client's public TLS certificate. Alternatively, you can use `TEMPORAL_TLS_CLIENT_CERT_PATH` to provide a path to the certificate or the TOML `profile.<name>.tls.client_cert_path`.         |
| Client Certificate Path   | `TEMPORAL_TLS_CLIENT_CERT_PATH`          | `profile.<name>.tls.client_cert_path`          | A filesystem path to the client's public TLS certificate. Alternatively, you can provide the raw PEM data using `TEMPORAL_TLS_CLIENT_CERT_DATA` or the TOML `profile.<name>.tls.client_cert_data`.                          |
| Client Key                | `TEMPORAL_TLS_CLIENT_KEY_DATA`           | `profile.<name>.tls.client_key_data`           | The raw PEM data containing the client's private TLS key. Alternatively, you can use `TEMPORAL_TLS_CLIENT_KEY_PATH` to provide a path to the key or the TOML `profile.<name>.tls.client_key_path`.                          |
| Client Key Path           | `TEMPORAL_TLS_CLIENT_KEY_PATH`           | `profile.<name>.tls.client_key_path`           | A filesystem path to the client's private TLS key. Alternatively, you can provide the raw PEM data using `TEMPORAL_TLS_CLIENT_KEY_DATA` or the TOML `profile.<name>.tls.client_key_data`.                                   |
| Server CA Cert            | `TEMPORAL_TLS_SERVER_CA_CERT_DATA`       | `profile.<name>.tls.server_ca_cert_data`       | The raw PEM data for the Certificate Authority certificate used to verify the server. Alternatively, you can use `TEMPORAL_TLS_SERVER_CA_CERT_PATH` to provide a path or the TOML `profile.<name>.tls.server_ca_cert_path`. |
| Server CA Cert Path       | `TEMPORAL_TLS_SERVER_CA_CERT_PATH`       | `profile.<name>.tls.server_ca_cert_path`       | A filesystem path to the Certificate Authority certificate. Alternatively, you can provide the raw PEM data using `TEMPORAL_TLS_SERVER_CA_CERT_DATA` or the TOML `profile.<name>.tls.server_ca_cert_data`.                  |
| TLS Server Name           | `TEMPORAL_TLS_SERVER_NAME`               | `profile.<name>.tls.server_name`               | Overrides the server name used for Server Name Indication (SNI) in the TLS handshake.                                                                                                                                       |
| Disable Host Verification | `TEMPORAL_TLS_DISABLE_HOST_VERIFICATION` | `profile.<name>.tls.disable_host_verification` | A boolean to disable server hostname verification. Use with caution. Not supported by all SDKs.                                                                                                                             |
| Codec Endpoint            | `TEMPORAL_CODEC_ENDPOINT`                | `profile.<name>.codec.endpoint`                | The endpoint for a remote Data Converter. This is not supported by all SDKs. SDKs that support this configuration don't apply it by default. Intended mostly for CLI use.                                                   |
| Codec Auth                | `TEMPORAL_CODEC_AUTH`                    | `profile.<name>.codec.auth`                    | The authorization header value for the remote data converter.                                                                                                                                                               |
| gRPC Metadata             | `TEMPORAL_GRPC_META_*`                   | `profile.<name>.grpc_meta`                     | Sets gRPC headers. The part after `_META_` becomes the header key (e.g., `_SOME_KEY` -> `some-key`).                                                                                                                        |

---

## OSS Temporal Service metrics reference

:::info OSS Temporal Service metrics

The information on this page is relevant to open source [Temporal Service deployments](/temporal-service).

See [Cloud metrics](/cloud/metrics/) for metrics emitted by [Temporal Cloud](/cloud/overview).

See [SDK metrics](/references/sdk-metrics) for metrics emitted by the [SDKs](/encyclopedia/temporal-sdks).

:::

A Temporal Service emits a range of metrics to help operators get visibility into the Temporal Service's performance and to set up alerts.

All metrics emitted by the Temporal Service are listed in [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go).

For details on setting up metrics in your Temporal Service configuration, see the [Temporal Service configuration reference](/references/configuration#global).

The [dashboards repository](https://github.com/temporalio/dashboards) contains community-driven Grafana dashboard templates that can be used as a starting point for monitoring the Temporal Service and SDK metrics.
You can use these templates as references to build your own dashboards.
For any metrics that are missing in the dashboards, use [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) as a reference.

Note that, apart from these metrics emitted by the Temporal Service, you should also monitor infrastructure-specific metrics like CPU, memory, and network for all hosts that are running Temporal Service services.

## Common metrics

Temporal emits metrics for each gRPC service request.
These metrics are emitted with `type`, `operation`, and `namespace` tags, which provide visibility into Service usage and show the request rates across Services, Namespaces, and Operations.

- Use the `operation` tag in your query to get request rates, error rates, or latencies per operation.
- Use the `service_name` tag with the [service role tag values](https://github.com/temporalio/temporal/blob/bba148cf1e1642fd39fa0174423b183d5fc62d95/common/metrics/defs.go#L108) to get details for the specific service.

All common tags that you can add in your query are defined in the [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) file.

For example, to see service requests by operation on the Frontend Service, use the following:

`sum by (operation) (rate(service_requests{service_name="frontend"}[2m]))`

Note: All metrics queries in this topic are [Prometheus queries](https://prometheus.io/docs/prometheus/latest/querying/basics/).

The following list describes some metrics you can get started with.

### `service_requests`

Shows service requests received per Task Queue.
Example: Service requests by operation
`sum(rate(service_requests{operation=\"AddWorkflowTask\"}[2m]))`

### `service_latency`

Shows latencies for all Client request operations.
Usually these are the starting point to investigate which operation is experiencing high-latency issues.
Example: P95 service latency by operation for the Frontend Service
`histogram_quantile(0.95, sum(rate(service_latency_bucket{service_name="frontend"}[5m])) by (operation, le))`

### `service_error_with_type`

(Available only in v1.17.0+) Identifies errors encountered by the service.
Example: Service errors by type for the Frontend Service
`sum(rate(service_error_with_type{service_name="frontend"}[5m])) by (error_type)`

### `client_errors`

An indicator for connection issues between different Server roles.
Example: Client errors
`sum(rate(client_errors{service_name="frontend",service_role="history"}[5m]))`

In addition to these, you can define some service-specific metrics to get performance details for each service.
Start with the following list, and use [metric_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) to define additional metrics as required.

## Matching Service metrics

### `poll_success`

Shows for Tasks that are successfully matched to a poller.
Example: `sum(rate(poll_success{}[5m]))`

### `poll_timeouts`

Shows when no Tasks are available for the poller within the poll timeout.
Example: `sum(rate(poll_timeouts{}[5m]))`

### `asyncmatch_latency`

Measures the time from creation to delivery for async matched Tasks.
The larger this latency, the longer Tasks are sitting in the queue waiting for your Workers to pick them up.
Example: `histogram_quantile(0.95, sum(rate(asyncmatch_latency_bucket{service_name="matching"}[5m])) by (operation, le))`

### `no_poller_tasks`

Emitted whenever a task is added to a task queue that has no poller, and is a counter metric.
This is usually an indicator that either the Worker or the starter programs are using the wrong Task Queue.

## History Service metrics

A History Task is an internal Task in Temporal that is created as part of a transaction to update Workflow state and is processed by the Temporal History service.
It is critical to ensure that the History Task processing system is healthy.
The following key metrics can be used to monitor the History Service health:

### `task_requests`

Emitted on every Task process request.
Example: `sum(rate(task_requests{operation=~"TransferActive.*"}[1m]))`

### `task_errors`

Emitted on every Task process error.
Example: `sum(rate(task_errors{operation=~"TransferActive.*"}[1m]))`

### `task_attempt`

Number of attempts on each Task Execution.
A Task is retried forever, and each retry increases the attempt count.
Example: `histogram_quantile(0.95, sum(rate(task_attempt_bucket{operation=~"TransferActive.*"}[1m])) by (operation, le))`

### `task_latency_processing`

Shows the processing latency per attempt.
Example: `histogram_quantile(0.95, sum(rate(task_latency_processing_bucket{operation=~"TransferActive.*",service_name="history"}[1m])) by (operation, le))`

### `task_latency`

Measures the in-memory latency across multiple attempts.

### `task_latency_queue`

Measures the duration, end-to-end, from when the Task should be executed (from the time it was fired) to when the Task is done.

### `task_latency_load`

(Available only in v1.18.0+) Measures the duration from Task generation to Task loading (Task schedule to start latency for persistence queue).

### `task_latency_schedule`

(Available only in v1.18.0+) Measures the duration from Task submission (to the Task scheduler) to processing (Task schedule to start latency for in-memory queue).

### `queue_latency_schedule`

(Available only in v1.18.0+) Measures the time to schedule 100 Tasks in one Task channel in the host-level Task scheduler.
If fewer than 100 Tasks are in the Task channel for 30 seconds, the latency is scaled to 100 Tasks upon emission.
Note: This is still an experimental metric and is subject to change.

### `service_latency_userlatency`

Shows the latency introduced because of Workflow logic.
For example, if you have one Workflow scheduling many Activities or Child Workflows at the same time, it can cause a per-Workflow lock contention.
The wait period for the per-Workflow lock is counted as `userlatency`.

The `operation` tag contains details about Task type and Active versus Standby statuses, and can be used to get request rates, error rates, or latencies per operation, which can help identify issues caused by database problems.

## Persistence metrics

Temporal Server emits metrics for every persistence database read and write.
Some of the most important ones are the following:

### `persistence_requests`

Emitted on every persistence request.
Examples:

- Prometheus query for getting the total number of persistence requests by operation for the History Service:
  `sum by (operation) (rate(persistence_requests{service_name="history"}[1m]))`
- Prometheus query for getting the total number of persistence requests by operation for the Matching Service:
  `sum by (operation) (rate(persistence_requests{service_name="matching"}[1m]))`

### `persistence_errors`

Shows all persistence errors.
This metric is a good indicator for connection issues between the Temporal Service and the persistence store.
Example:

- Prometheus query for getting all persistence errors by service (history)
  `sum (rate(persistence_errors{service_name="history"}[1m]))`

### `persistence_error_with_type`

Shows all errors related to the persistence store with type, and contain an `error_type` tag.

- Prometheus query for getting persistence errors with type by (history) and by error type:
  `sum(rate(persistence_error_with_type{service_name="history"}[1m])) by (error_type)`

### `persistence_latency`

Shows the latency on persistence operations.
Example:

- Prometheus query for getting latency by percentile:
  `histogram_quantile(0.95, sum(rate(persistence_latency_bucket{service_name="history"}[1m])) by (operation, le))`

## Schedule metrics

Temporal emits metrics that track the performance and outcomes of these Scheduled Executions.

Below are additional metrics that can help you monitor and optimize your Scheduled Workflow Executions.

### `schedule_buffer_overruns`

Indicates instances where the buffer for holding Scheduled Workflows exceeds its maximum capacity.
This scenario typically occurs when schedules with a `buffer_all` overlap policy have their average run length exceeding the average schedule interval.

Example: To monitor buffer overruns.

`sum(rate(schedule_buffer_overruns{namespace="$namespace"}[5m]))`

### `schedule_missed_catchup_window`

Tracks occurrences when the system fails to execute a Scheduled Action within the defined catchup window.
Missed catchup windows can result from extended outages beyond the configured catchup period.

Example: To identify missed catchup opportunities.

`sum(rate(schedule_missed_catchup_window{namespace="$namespace"}[5m]))`

### `schedule_rate_limited`

Reflects instances where the creation of Workflows by a Schedule is throttled due to rate limiting policies within a Namespace.
This metric is crucial for identifying scheduling patterns that frequently hit rate limits, potentially causing missed catchup windows.

Example: To assess the impact of rate limiting on Scheduled Executions.

`sum(rate(schedule_rate_limited{namespace="$namespace"}[5m]))`

### `schedule_action_success`

Measures the successful execution of Workflows as per their schedules or through manual triggers.
This metric is confirms that Workflows are running as expected without delays or errors.

Example: To track the success rate of Scheduled Workflow Executions.

`sum(rate(schedule_action_success{namespace="$namespace"}[5m]))`

## Workflow metrics

These metrics pertain to Workflow statistics.

### `workflow_cancel`

Number of Workflows canceled before completing execution.

### `workflow_continued_as_new`

Number of Workflow Executions that were Continued-As-New from a past execution.

### `workflow_failed`

Number of Workflows that failed before completion.

### `workflow_success`

Number of Workflows that successfully completed.

### `workflow_timeout`

Number of Workflows that timed out before completing execution.

## Nexus metrics

These metrics pertain to Nexus Operations.

### Nexus Machinery in the History Service

See [architecture document](https://github.com/temporalio/temporal/blob/5d55d6c707bd68d8f3274c57ae702331adf05e6e/docs/architecture/nexus.md#scheduler)
for more info.

#### In-Memory Buffer

`dynamic_worker_pool_scheduler_enqueued_tasks`: A counter that is incremented when a task is enqueued to the buffer.

`dynamic_worker_pool_scheduler_dequeued_tasks`: A counter that is incremented when a task is dequeued from the buffer.

`dynamic_worker_pool_scheduler_rejected_tasks`: A counter that is incremented when the buffer is full and adding the
task is rejected.

`dynamic_worker_pool_scheduler_buffer_size`: A gauge that periodically samples the size of the buffer.

### Concurrency Limiter

`dynamic_worker_pool_scheduler_active_workers`: A gauge that periodically samples the number of running goroutines.

#### Rate Limiter

`rate_limited_task_runnable_wait_time`: A histogram representing the time a task spends waiting for the rate limiter.

#### Circuit Breaker

`circuit_breaker_executable_blocked`: A counter that is incremented every time a task execution is blocked by the
circuit breaker.

#### Task Executors

`nexus_outbound_requests`: A counter representing the number of Nexus outbound requests made by the history service.

`nexus_outbound_latency`: A histogram representing the latency of outbound Nexus requests made by the history service.

`callback_outbound_requests`: A counter representing the number of callback outbound requests made by the history
service.

`callback_outbound_latency`: A histogram representing the latency histogram of outbound callback requests made by the
history service.

### Nexus Machinery on the Frontend Service

#### `nexus_requests`

The number of Nexus requests received by the service.

Type: Counter

#### `nexus_latency`

Latency of Nexus requests.

Type: Histogram

#### `nexus_request_preprocess_errors`

The number of Nexus requests for which pre-processing failed.

Type: Counter

#### `nexus_completion_requests`

The number of Nexus completion (callback) requests received by the service.

Type: Counter

#### `nexus_completion_latency`

Latency histogram of Nexus completion (callback) requests.

Type: Histogram

#### `nexus_completion_request_preprocess_errors`

The number of Nexus completion requests for which pre-processing failed.

Type: Counter

---

## Temporal Commands reference

A [Command](/workflow-execution#command) is a requested action issued by a [Worker](/workers#worker) to the [Temporal Service](/temporal-service) after a [Workflow Task Execution](/tasks#workflow-task-execution) completes.

The following is a complete list of possible Commands.

### CompleteWorkflowExecution

This Command is triggered when the Workflow Function Execution returns.
It indicates to the Temporal Service that the [Workflow Execution](/workflow-execution) is complete.
The corresponding [Event](/workflow-execution/event#event) for this Command is one of the few Events that will be the last in a Workflow Execution [Event History](/workflow-execution/event#event-history).

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [WorkflowExecutionCompleted](/references/events#workflowexecutioncompleted)

### ContinueAsNewWorkflowExecution

This Command is triggered when there is a call to [Continue-As-New](/workflow-execution/continue-as-new) from within the [Workflow](/workflows).
The corresponding Event for this Command is one of the few Events that will be the last in a Workflow Execution Event History.

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [WorkflowExecutionContinuedAsNew](/references/events#workflowexecutioncontinuedasnew)

### FailWorkflowExecution

This Command is triggered when the Workflow Execution returns an error or an exception is thrown.

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [WorkflowExecutionFailed](/references/events#workflowexecutionfailed)

### CancelWorkflowExecution

This Command is triggered when the Workflow has successfully cleaned up after receiving a Cancellation Request (which will be present as [WorkflowExecutionCancelRequestedEvent](/references/events#workflowexecutioncancelrequested) in the Event History).
The Corresponding Event for this Command is one of the few Events that will be the last in a Workflow Execution Event History.

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [WorkflowExecutionCanceled](/references/events#workflowexecutioncanceled)

### StartChildWorkflowExecution

This Command is triggered by a call to spawn a [Child Workflow Execution](/child-workflows).

- Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command.
- Corresponding Event: [ChildWorkflowExecutionStarted](/references/events#childworkflowexecutionstarted)

By default, you cannot have more than 2,000 pending Child Workflows.

### SignalExternalWorkflowExecution

This Command is triggered by a call to [Signal](/sending-messages#sending-signals) another Workflow Execution.

- Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command.
- Corresponding Event: [SignalExternalWorkflowExecutionInitiated](/references/events#signalexternalworkflowexecutioninitiated)

By default, you cannot have more than 2,000 pending Signals to other Workflows.

### RequestCancelExternalWorkflowExecution

This Command is triggered by a call to request cancellation of another Workflow Execution.

- Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command.
- Corresponding Event: [RequestCancelExternalWorkflowExecutionInitiated](/references/events#requestcancelexternalworkflowexecutioninitiated)

By default, you cannot have more than 2,000 pending Signals to other Workflows.

### ScheduleActivityTask

This Command is triggered by a call to execute an [Activity](/activities).

- Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command.
- Corresponding Event: [ActivityTaskScheduled](/references/events#activitytaskscheduled)

By default, you cannot schedule more than 2,000 Activities concurrently.

### RequestCancelActivityTask

This Command is triggered by a call to request the cancellation of an [Activity Task](/tasks#activity-task).

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [ActivityTaskCancelRequested](/references/events#activitytaskcancelrequested)

### StartTimer

This Command is triggered by a call to start a Timer.

- Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command.
- Corresponding Event: [TimerStarted](/references/events#timerstarted)

### CancelTimer

This Command is triggered by a call to cancel a Timer.

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [TimerCanceled](/references/events#timercanceled)

### RecordMarker

This Command is triggered by the SDK.

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [MarkerRecorded](/references/events#markerrecorded)

### UpsertWorkflowSearchAttributes

This Command is triggered by a call to "upsert" Workflow [Search Attributes](/search-attribute).

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [UpsertWorkflowSearchAttributes](/references/events#upsertworkflowsearchattributes)

### ProtocolMessageCommand

This Command helps guarantee ordering constraints for features such as Updates.

This Command points at the message from which the Event is created.
Therefore, just from the Command, you can't predict the resulting Event type.

### ScheduleNexusOperation

This Command is triggered by a call to execute an Nexus Operation in the caller Workflow.

- Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command.
- Corresponding Event: [NexusOperationScheduled](/references/events#nexusoperationscheduled)

By default, you can't schedule more than 30 Nexus Operations concurrently, see [Limits](/workflow-execution/limits#workflow-execution-nexus-operation-limits) for details.

### CancelNexusOperation

This Command is triggered by a call to request the cancellation of a Nexus Operation.

- Awaitable: No, a Workflow Execution can not await on the action resulting from this Command.
- Corresponding Event: [NexusOperationCancelRequested](/references/events#nexusoperationcancelrequested)

---

## Temporal Cluster configuration reference

Much of the behavior of a Temporal Cluster is configured using the `development.yaml` file and may contain the following top-level sections:

- [`global`](#global)
- [`persistence`](#persistence)
- [`log`](#log)
- [`clusterMetadata`](#clustermetadata)
- [`services`](#services)
- [`publicClient`](#publicclient)
- [`archival`](#archival)
- [`namespaceDefaults`](#namespacedefaults)
- [`dcRedirectionPolicy`](#dcredirectionpolicy)
- [`dynamicConfigClient`](#dynamicconfigclient)

Changing any properties in the `development.yaml` file requires a process restart for changes to take effect.
Configuration parsing code is available [here](https://github.com/temporalio/temporal/blob/main/common/config/config.go).

## global

The `global` section contains process-wide configuration. See below for a minimal configuration (optional parameters are commented out.)

```yaml
global:
  membership:
    broadcastAddress: '127.0.0.1'
  metrics:
    prometheus:
      framework: 'tally'
      listenAddress: '127.0.0.1:8000'
```

### membership

The `membership` section controls the following membership layer parameters.

#### maxJoinDuration

The amount of time the service will attempt to join the gossip layer before failing.

Default is 10s.

#### broadcastAddress

Used by gossip protocol to communicate with other hosts in the same Cluster for membership info.
Use IP address that is reachable by other hosts in the same Cluster.
If there is only one host in the Cluster, you can use 127.0.0.1.
Check `net.ParseIP` for supported syntax, only IPv4 is supported.

### metrics

Configures the Cluster's metric subsystem.
Specific provides are configured using provider names as the keys.

- [`statsd`](#statsd)
- `prometheus`
- `m3`

#### prefix

The prefix to be applied to all outgoing metrics.

#### tags

The set of key-value pairs to be reported as part of every metric.

#### excludeTags

A map from tag name string to tag values string list.
This is useful to exclude some tags that might have unbounded cardinality.
The value string list can be used to whitelist values of that excluded tag to continue to be included.
For example, if you want to exclude `task_queue` because it has unbounded cardinality, but you still want to see a whitelisted value for `task_queue`.

#### statsd

:::caution

`statsd` is not supported natively by Temporal.

:::

The `statsd` sections supports the following settings:

- `hostPort`: The host:port of the statsd server.
- `prefix`: Specific prefix in reporting to `statsd`.
- `flushInterval`: Maximum interval for sending packets. (_Default_ 300ms).
- `flushBytes`: Specifies the maximum UDP packet size you wish to send. (_Default_ 1432 bytes).

#### prometheus

The `prometheus` sections supports the following settings:

- `framework`: The framework to use, currently supports `opentelemetry` and `tally`, default is `tally`. We plan to switch default to `opentelemetry` once its API become stable.
- `listenAddress`: Address for Prometheus to scrape metrics from.
  The Temporal Server uses the Prometheus client API, and the `listenAddress` configuration is used to listen for metrics.
- `handlerPath`: Metrics handler path for scraper; default is `/metrics`.

#### m3

The `m3` sections supports the following settings:

- `hostPort`: The host:port of the M3 server.
- `service`: The service tag to that this client emits.
- `queue`: M3 reporter queue size, default is 4k.
- `packetSize`: M3 reporter max packet size, default is 32k.

### pprof

- `port`: If specified, this will initialize pprof upon process start on the listed port.

### tls

The `tls` section controls the SSL/TLS settings for network communication and contains two subsections, `internode` and `frontend`.
The `internode` section governs internal service communication among roles where the `frontend` governs SDK client communication to the Frontend Service role.

Each of these subsections contain a `server` section and a `client` section.
The `server` contains the following parameters:

- `certFile`: The path to the file containing the PEM-encoded public key of the certificate to use.
- `keyFile`: The path to the file containing the PEM-encoded private key of the certificate to use.
- `requireClientAuth`: _boolean_ - Requires clients to authenticate with a certificate when connecting, otherwise known as mutual TLS.
- `clientCaFiles`: A list of paths to files containing the PEM-encoded public key of the Certificate Authorities you wish to trust for client authentication. This value is ignored if `requireClientAuth` is not enabled.

:::tip

See the [server samples repo](https://github.com/temporalio/samples-server/tree/master/tls) for sample TLS configurations.

:::

Below is an example enabling Server TLS (https) between SDKs and the Frontend APIs:

```yaml
global:
  tls:
    frontend:
      server:
        certFile: /path/to/cert/file
        keyFile: /path/to/key/file
      client:
        serverName: dnsSanInFrontendCertificate
```

Note, the `client` section generally needs to be provided to specify an expected DNS SubjectName contained in the presented server certificate via the `serverName` field; this is needed as Temporal uses IP to IP communication.
You can avoid specifying this if your server certificates contain the appropriate IP Subject Alternative Names.

Additionally, the `rootCaFiles` field needs to be provided when the client's host does not trust the Root CA used by the server.
The example below extends the above example to manually specify the Root CA used by the Frontend Services:

```yaml
global:
  tls:
    frontend:
      server:
        certFile: /path/to/cert/file
        keyFile: /path/to/key/file
      client:
        serverName: dnsSanInFrontendCertificate
        rootCaFiles:
          - /path/to/frontend/server/CA/files
```

Below is an additional example of a fully secured cluster using mutual TLS for both frontend and internode communication with manually specified CAs:

```yaml
global:
  tls:
    internode:
      server:
        certFile: /path/to/internode/cert/file
        keyFile: /path/to/internode/key/file
        requireClientAuth: true
        clientCaFiles:
          - /path/to/internode/serverCa
      client:
        serverName: dnsSanInInternodeCertificate
        rootCaFiles:
          - /path/to/internode/serverCa
    frontend:
      server:
        certFile: /path/to/frontend/cert/file
        keyFile: /path/to/frontend/key/file
        requireClientAuth: true
        clientCaFiles:
          - /path/to/internode/serverCa
          - /path/to/sdkClientPool1/ca
          - /path/to/sdkClientPool2/ca
      client:
        serverName: dnsSanInFrontendCertificate
        rootCaFiles:
          - /path/to/frontend/serverCa
```

**Note:** In the case that client authentication is enabled, the `internode.server` certificate is used as the client certificate among services. This adds the following requirements:

- The `internode.server` certificate must be specified on all roles, even for a frontend-only configuration.
- Internode server certificates must be minted with either **no** Extended Key Usages or **both** ServerAuth and ClientAuth EKUs.
- If your Certificate Authorities are untrusted, such as in the previous example, the internode server Ca will need to be specified in the following places:

  - `internode.server.clientCaFiles`
  - `internode.client.rootCaFiles`
  - `frontend.server.clientCaFiles`

## persistence

The `persistence` section holds configuration for the data store/persistence layer.
The following example shows a minimal specification for a password-secured Cluster using Cassandra.

```yaml
persistence:
  defaultStore: default
  visibilityStore: cass-visibility # The primary Visibility store.
  secondaryVisibilityStore: es-visibility # A secondary Visibility store added to enable Dual Visibility.
  numHistoryShards: 512
  datastores:
    default:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal'
        user: 'username'
        password: 'password'
    cass-visibility:
      cassandra:
        hosts: '127.0.0.1'
        keyspace: 'temporal_visibility'
    es-visibility:
      elasticsearch:
        version: 'v7'
        logLevel: 'error'
        url:
          scheme: 'http'
          host: '127.0.0.1:9200'
        indices:
          visibility: temporal_visibility_v1_dev
        closeIdleConnectionsInterval: 15s
```

The following top level configuration items are required:

### numHistoryShards

_Required_ - The number of history shards to create when initializing the Cluster.

**Warning:** This value is immutable and will be ignored after the first run.
Please ensure you set this value appropriately high enough to scale with the worst case peak load for this Cluster.

### defaultStore

_Required_ - The name of the data store definition that should be used by the Temporal server.

### visibilityStore

_Required_ - The name of the primary data store definition that should be used to set up [Visibility](/temporal-service/visibility) on the Temporal Cluster.

### secondaryVisibilityStore

_Optional_ - The name of the secondary data store definition that should be used to set up [Dual Visibility](/dual-visibility) on the Temporal Cluster.

### datastores

_Required_ - contains named data store definitions to be referenced.

Each definition is defined with a heading declaring a name (ie: `default:` and `visibility:` above), which contains a data store definition.

Data store definitions must be either `cassandra` or `sql`.

#### cassandra

A `cassandra` data store definition can contain the following values:

- `hosts`: _Required_ - "," separated Cassandra endpoints, e.g. "192.168.1.2,192.168.1.3,192.168.1.4".
- `port`: Default: 9042 - Cassandra port used for connection by `gocql` client.
- `user`: Cassandra username used for authentication by `gocql` client.
- `password`: Cassandra password used for authentication by `gocql` client.
- `keyspace`: _Required_ - the Cassandra keyspace.
- `datacenter`: The data center filter arg for Cassandra.
- `maxConns`: The max number of connections to this data store for a single TLS configuration.
- `tls`: See TLS below.

#### sql

A `sql` data store definition can contain the following values:

- `user`: Username used for authentication.
- `password`: Password used for authentication.
- `pluginName`: _Required_ - SQL database type.
  - _Valid values_: `mysql` or `postgres`.
- `databaseName` - _required_ - the name of SQL database to connect to.
- `connectAddr` - _required_ - the remote address of the database, e.g. "192.168.1.2".
- `connectProtocol` - _required_ - the protocol that goes with the `connectAddr`
  - _Valid values_: `tcp` or `unix`
- `connectAttributes` - a map of key-value attributes to be sent as part of connect `data_source_name` url.
- `maxConns` - the max number of connections to this data store.
- `maxIdleConns` - the max number of idle connections to this data store
- `maxConnLifetime` - is the maximum time a connection can be alive.
- `tls` - See below.

#### tls

The `tls` and `mtls` sections can contain the following values:

- `enabled` - _boolean_.
- `serverName` - name of the server hosting the data store.
- `certFile` - path to the cert file.
- `keyFile` - path to the key file.
- `caFile` - path to the ca file.
- `enableHostVerification` - _boolean_ - `true` to verify the hostname and server cert (like a wildcard for Cassandra cluster). This option is basically the inverse of `InSecureSkipVerify`. See `InSecureSkipVerify` in http://golang.org/pkg/crypto/tls/ for more info.

Note: `certFile` and `keyFile` are optional depending on server config, but both fields must be omitted to avoid using a client certificate.

## log

The `log` section is optional and contains the following possible values:

- `stdout` - _boolean_ - `true` if the output needs to go to standard out.
- `level` - sets the logging level.
  - _Valid values_ - debug, info, warn, error or fatal, default to info.
- `outputFile` - path to output log file.

## clusterMetadata

`clusterMetadata` contains the local cluster information. The information is used in [Multi-Cluster Replication](/temporal-service/multi-cluster-replication).

An example `clusterMetadata` section:

```yaml
clusterMetadata:
  enableGlobalNamespace: true
  failoverVersionIncrement: 10
  masterClusterName: 'active'
  currentClusterName: 'active'
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 0
      rpcAddress: '127.0.0.1:7233'
  #replicationConsumer:
  #type: kafka
```

- `currentClusterName` - _required_ - the name of the current cluster. **Warning:** This value is immutable and will be ignored after the first run.
- `enableGlobalNamespace` - _Default:_ `false`.
- `replicationConsumer` - determines which method to use to consume replication tasks. The type may be either `kafka` or `rpc`.
- `failoverVersionIncrement` - the increment of each cluster version when failover happens.
- `masterClusterName` - the master cluster name, only the master cluster can register/update namespace. All clusters can do namespace failover.
- `clusterInformation` - contains the local cluster name to `ClusterInformation` definition. The local cluster name should be consistent with `currentClusterName`. `ClusterInformation` sections consist of:
  - `enabled` - _boolean_ - whether a remote cluster is enabled for replication.
  - `initialFailoverVersion`
  - `rpcAddress` - indicate the remote service address (host:port). Host can be DNS name. Use `dns:///` prefix to enable round-robin between IP address for DNS name.

## services

The `services` section contains configuration keyed by service role type.
There are four supported service roles:

- `frontend`
- `matching`
- `worker`
- `history`

Below is a minimal example of a `frontend` service definition under `services`:

```yaml
services:
  frontend:
    rpc:
      grpcPort: 8233
      membershipPort: 8933
      bindOnIP: '0.0.0.0'
```

There are two sections defined under each service heading:

### rpc

_Required_

`rpc` contains settings related to the way a service interacts with other services. The following values are supported:

- `grpcPort`: Port on which gRPC will listen.
- `membershipPort`: Port used to communicate with other hosts in the same Cluster for membership info.
  Each service should use different port.
  If there are multiple Temporal Clusters in your environment (Kubernetes for example), and they have network access to each other, each Cluster should use a different membership port.
- `bindOnLocalHost`: Determines whether uses `127.0.0.1` as the listener address.
- `bindOnIP`: Used to bind service on specific IP, or `0.0.0.0`.
  Check `net.ParseIP` for supported syntax, only IPv4 is supported, mutually exclusive with `BindOnLocalHost` option.

**Note:** Port values are currently expected to be consistent among role types across all hosts.

## publicClient

The `publicClient` a required section describing the configuration needed to for worker to connect to Temporal server for background server maintenance.

- `hostPort` IPv4 host port or DNS name to reach Temporal frontend, [reference](https://github.com/grpc/grpc/blob/master/doc/naming.md)

Example:

```yaml
publicClient:
  hostPort: 'localhost:8933'
```

Use `dns:///` prefix to enable round-robin between IP address for DNS name.

## archival

_Optional_

Archival is an optional configuration needed to set up the [Archival store](/temporal-service/archival).
It can be enabled on `history` and `visibility` data.

The following list describes supported values for each configuration on the `history` and `visibility` data.

- `state`: State for Archival setting. Supported values are `enabled`, `disabled`. This value must be `enabled` to use Archival with any Namespace in your Cluster.
  - `enabled`: Enables Archival in your Cluster setup. When set to `enabled`, `URI` and `namespaceDefaults` values must be provided.
  - `disabled`: Disables Archival in your Cluster setup. When set to `disabled`, the `enableRead` value must be set to `false`, and under `namespaceDefaults`, `state` must be set to `disabled`, with no values set for `provider` and `URI` fields.
- `enableRead`: Supported values are `true` or `false`. Set to `true` to allow read operations from the archived Event History data.
- `provider`: Location where data should be archived. Subprovider configs are `filestore`, `gstorage`, `s3`, or `your_custom_provider`. Default configuration specifies `filestore`.

Example:

- To enable Archival in your Cluster configuration:

  ```yaml
  # Cluster-level Archival config enabled
  archival:
    # Event History configuration
    history:
      # Archival is enabled for the History Service data.
      state: 'enabled'
      enableRead: true
      # Namespaces can use either the local filestore provider or the Google Cloud provider.
      provider:
        filestore:
          fileMode: '0666'
          dirMode: '0766'
        gstorage:
          credentialsPath: '/tmp/gcloud/keyfile.json'
    # Configuration for archiving Visibility data.
    visibility:
      # Archival is enabled for Visibility data.
      state: 'enabled'
      enableRead: true
      provider:
        filestore:
          fileMode: '0666'
          dirMode: '0766'
  ```

- To disable Archival in your Cluster configuration:

  ```yaml
  # Cluster-level Archival config disabled
  archival:
    history:
      state: 'disabled'
      enableRead: false
    visibility:
      state: 'disabled'
      enableRead: false

  namespaceDefaults:
    archival:
      history:
        state: 'disabled'
      visibility:
        state: 'disabled'
  ```

For more details on Archival setup, see [Set up Archival](/self-hosted-guide/archival#set-up-archival).

## namespaceDefaults

_Optional_

Sets default Archival configuration for each Namespace using `namespaceDefaults` for `history` and `visibility` data.

- `state`: Default state of the Archival for the Namespace. Supported values are `enabled` or `disabled`.
- `URI`: Default URI for the Namespace.

For more details on setting Namespace defaults on Archival, see [Namespace creation in Archival setup](/self-hosted-guide/archival#namespace-creation)

Example:

```yaml