# Default values for a Namespace if none are provided at creation
namespaceDefaults:
  # Archival defaults
  archival:
    # Event History defaults
    history:
      state: 'enabled'
      # New Namespaces will default to the local provider
      URI: 'file:///tmp/temporal_archival/development'
```

You can disable Archival by setting `archival.history.state` and `namespaceDefaults.archival.history.state` to `"disabled"`.

Example:

```yaml
archival:
  history:
    state: 'disabled'

namespaceDefaults:
  archival:
    history:
      state: 'disabled'
```

The following table showcases acceptable values for each configuration and what purpose they serve.

| Config                                         | Acceptable values                                                                  | Description                                                                                                                  |
| ---------------------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `archival.history.state`                       | `enabled`, `disabled`                                                              | Must be `enabled` to use the Archival feature with any Namespace in the Temporal Service.                                    |
| `archival.history.enableRead`                  | `true`, `false`                                                                    | Must be `true` to read from the archived Event History.                                                                      |
| `archival.history.provider`                    | Sub provider configs are `filestore`, `gstorage`, `s3`, or `your_custom_provider`. | Default config specifies `filestore`.                                                                                        |
| `archival.history.provider.filestore.fileMode` | File permission string                                                             | File permissions of the archived files. We recommend using the default value of `"0666"` to avoid read/write issues.         |
| `archival.history.provider.filestore.dirMode`  | File permission string                                                             | Directory permissions of the archive directory. We recommend using the default value of `"0766"` to avoid read/write issues. |
| `namespaceDefaults.archival.history.state`     | `enabled`, `disabled`                                                              | Default state of the Archival feature whenever a new Namespace is created without specifying the Archival state.             |
| `namespaceDefaults.archival.history.URI`       | Valid URI                                                                          | Must be a URI of the file store location and match a schema that correlates to a provider.                                   |

Additional resources: [Temporal Service configuration reference](/references/configuration).

#### Namespace creation

Although Archival is configured at the Temporal Service level, it operates independently within each Namespace.
If an Archival URI is not specified when a Namespace is created, the Namespace uses the value of `defaultNamespace.archival.history.URI` from the `config/development.yaml` file.
The Archival URI cannot be changed after the Namespace is created.
Each Namespace supports only a single Archival URI, but each Namespace can use a different URI.
A Namespace can safely switch Archival between `enabled` and `disabled` states as long as Archival is enabled at the Temporal Service level.

Archival is supported in [Global Namespaces](/global-namespace) (Namespaces that span multiple clusters).
When Archival is running in a Global Namespace, it first runs on the active cluster; later it runs on the standby cluster. Before archiving, a history check is done to see what has been previously archived.

#### Test setup

To test Archival locally, start by running a Temporal server:

```bash
./temporal-server start
```

Then register a new Namespace with Archival enabled.

{/* ./tctl --ns samples-namespace namespace register --gd false --history_archival_state enabled --retention 3 */}

```bash
./temporal operator namespace create --namespace="my-namespace" --global false --history-archival-state="enabled" --retention="4d"
```

:::note

If the retention period isn't set, it defaults to 72h.
The minimum retention period is one day.
The maximum retention period is 30 days.

Setting the retention period to 0 results in the error _A valid retention period is not set on request_.

:::

Next, run a sample Workflow such as the [helloworld temporal sample](https://github.com/temporalio/temporal-go-samples/tree/master/helloworld).

When execution is finished, Archival occurs.

#### Retrieve archives

You can retrieve archived Event Histories by copying the `workflowId` and `runId` of the completed Workflow from the log output and running the following command:

{/* ./tctl --ns samples-namespace wf show --wid <workflowId> --rid <runId> */}

```bash
./temporal workflow show --workflow-id="my-workflow-id" --run-id="my-run-id" --namespace="my-namespace"
```

### How to create a custom Archiver {#custom-archiver}

To archive data with a given provider, using the [Archival](/temporal-service/archival) feature, Temporal must have a corresponding Archiver component installed.
The platform does not limit you to the existing providers.
To use a provider that is not currently supported, you can create your own Archiver.

#### Create a new package

The first step is to create a new package for your implementation in [/common/archiver](https://github.com/temporalio/temporal/tree/main/common/archiver).
Create a directory in the archiver folder and arrange the structure to look like the following:

```
temporal/common/archiver
  - filestore/                      -- Filestore implementation
  - provider/
      - provider.go                 -- Provider of archiver instances
  - yourImplementation/
      - historyArchiver.go          -- HistoryArchiver implementation
      - historyArchiver_test.go     -- Unit tests for HistoryArchiver
      - visibilityArchiver.go       -- VisibilityArchiver implementations
      - visibilityArchiver_test.go  -- Unit tests for VisibilityArchiver
```

#### Archiver interfaces

Next, define objects that implement the [HistoryArchiver](https://github.com/temporalio/temporal/blob/main/common/archiver/interface.go#L80) and the [VisibilityArchiver](https://github.com/temporalio/temporal/blob/main/common/archiver/interface.go#L121) interfaces.

The objects should live in `historyArchiver.go` and `visibilityArchiver.go`, respectively.

#### Update provider

Update the `GetHistoryArchiver` and `GetVisibilityArchiver` methods of the `archiverProvider` object in the [/common/archiver/provider/provider.go](https://github.com/temporalio/temporal/blob/main/common/archiver/provider/provider.go) file so that it knows how to create an instance of your archiver.

#### Add configs

Add configs for your archiver to the `config/development.yaml` file and then modify the [HistoryArchiverProvider](https://github.com/temporalio/temporal/blob/main/common/config/config.go#L376) and [VisibilityArchiverProvider](https://github.com/temporalio/temporal/blob/main/common/config/config.go#L393) structs in `/common/common/config.go` accordingly.

#### Custom archiver FAQ

**If my custom Archive method can automatically be retried by the caller, how can I record and access progress between retries?**

Handle this situation by using `ArchiverOptions`.
Here is an example:

```go
func(a * Archiver) Archive(ctx context.Context, URI string, request * ArchiveRequest, opts...ArchiveOption) error {
    featureCatalog: = GetFeatureCatalog(opts...) // this function is defined in options.go
    var progress progress
    // Check if the feature for recording progress is enabled.
    if featureCatalog.ProgressManager != nil {
        if err: = featureCatalog.ProgressManager.LoadProgress(ctx, & prevProgress);
        err != nil {
            // log some error message and return error if needed.
        }
    }

    // Your archiver implementation...

    // Record current progress
    if featureCatalog.ProgressManager != nil {
        if err: = featureCatalog.ProgressManager.RecordProgress(ctx, progress);
        err != nil {
            // log some error message and return error if needed.
        }
    }
}
```

**If my `Archive` method encounters an error that is non-retryable, how do I indicate to the caller that it should not retry?**

```go
func(a * Archiver) Archive(ctx context.Context, URI string, request * ArchiveRequest, opts...ArchiveOption) error {
    featureCatalog: = GetFeatureCatalog(opts...) // this function is defined in options.go

    err: = youArchiverImpl()

    if nonRetryableErr(err) {
        if featureCatalog.NonRetryableError != nil {
            return featureCatalog.NonRetryableError() // when the caller gets this error type back it will not retry anymore.
        }
    }
}
```

**How does my history archiver implementation read history?**

The archiver package provides a utility called [HistoryIterator](https://github.com/temporalio/temporal/blob/main/common/archiver/historyIterator.go) which is a wrapper of [ExecutionManager](https://github.com/temporalio/temporal/blob/main/common/persistence/data_interfaces.go#L1014).
`HistoryIterator` is more simple than the `HistoryManager`, which is available in the BootstrapContainer, so archiver implementations can choose to use it when reading Workflow histories.
See the [historyIterator.go](https://github.com/temporalio/temporal/blob/main/common/archiver/history_iterator.go) file for more details.
Use the [filestore historyArchiver implementation](https://github.com/temporalio/temporal/tree/main/common/archiver/filestore) as an example.

**Should my archiver define its own error types?**

Each archiver is free to define and return its own errors.
However, many common errors that exist between archivers are already defined in [common/archiver/constants.go](https://github.com/temporalio/temporal/blob/main/common/archiver/constants.go).

**Is there a generic query syntax for the visibility archiver?**

Currently, no.
But this is something we plan to do in the future.
As for now, try to make your syntax similar to the one used by our advanced list Workflow API.

- [s3store](https://github.com/temporalio/temporal/tree/main/common/archiver/s3store#visibility-query-syntax)
- [gcloud](https://github.com/temporalio/temporal/tree/main/common/archiver/gcloud#visibility-query-syntax)

---

## Temporal Platform's production readiness checklist

This page describes common challenges customers face who self-host Temporal and it shares recommendations to mitigate those issues.

Temporal at its core is about durability and reliability.
To ensure this durability and reliability, a Temporal Service must be deployed according to best practices.

This guide provides a path to demonstrate that Temporal consumers can be confident in a Temporal Service and provides a list of key tests you as a user should perform against the service.

## Self-Hosting Challenge Areas

Significant engineering and ongoing effort is required to resolve several potential challenges:

- Scalability with spiky or growing workloads
- Global hosting
- Uptime, availability and reliability
- Management and control plane
- Latency, which must be kept low and consistent
- [Security](/self-hosted-guide/security)
- Maintenance and upgrades
- Expert support to users of the service
- Cost management

Each of these components is an essential part of building a mission critical Temporal Service.
Without demonstrated architectural durability, the value of Temporal's [Durable Execution](https://temporal.io/how-it-works) model is compromised.

## Scalability with Variable or Growing Workloads {#scaling-and-metrics}

Workloads can be highly variable, and you may experience sustained workload spikes.
Temporal recommends scaling your clusters to well above the average throughput.
See [Scaling Temporal: The Basics](https://temporal.io/blog/scaling-temporal-the-basics) for an introduction to the topic.

Temporal server throughput is often limited by the number of [Shards](/temporal-service/temporal-server#history-shard) configured for the Temporal Service.
A Shard is an unit within a Temporal Service by which concurrent Workflow Execution throughput can be scaled.
Shard capacity, and often overall cluster throughput, is set at build time for a cluster and that cluster setting cannot be adjusted later.
Adding more Shards if needed requires a cluster rebuild, and a migration to the new cluster.

The requirements of your Temporal Service will vary widely based on your intended production workload.
You will want to run your own proof of concept tests and watch for key metrics to understand the system health and scaling needs.

**Load testing.** You can use [the Omes benchmarking tool](https://github.com/temporalio/omes/), see how we ourselves [stress test Temporal](https://temporal.io/blog/temporal-deep-dive-stress-testing/), or write your own.

All metrics emitted by the server are [listed in Temporal's source](https://github.com/temporalio/temporal/blob/main/common/metrics/defs.go).
There are also equivalent metrics that you can configure from the client side.
At a high level, you will want to track these 3 categories of metrics:

- **Service metrics**: For each request made by the service handler we emit `service_requests`, `service_errors`, and `service_latency` metrics with `type`, `operation`, and `namespace` tags.
  This gives you basic visibility into service usage and allows you to look at request rates across services, namespaces and even operations.
- **Persistence metrics**: The Server emits `persistence_requests`, `persistence_errors` and `persistence_latency` metrics for each persistence operation.
  These metrics include the `operation` tag such that you can get the request rates, error rates or latencies per operation.
  These are super useful in identifying issues caused by the database.
- **Workflow Execution stats**: The Server also emits counters for when Workflow Executions are complete.
  These are useful in getting overall stats about Workflow Execution completions.
  Use `workflow_success`, `workflow_failed`, `workflow_timeout`, `workflow_terminate` and `workflow_cancel` counters for each type of Workflow Execution completion.
  These include the `namespace` tag.

## Availability

A high level of availability and reliability (99.99%) is a requirement for mission critical deployments.
Temporal recommends testing for this availability level while load testing.
We also recommend validating this level of reliability while doing server upgrades, to ensure no loss of service availability.

Temporal Clusters can be deployed in as many regions as needed to meet various requirements:

- Data Residency
- Latency
- Security / Isolation
  This can multiply the effort to implement and maintain clusters.

[Temporal Cloud is available in various cloud provider regions](/cloud/service-availability).

## Management and Control Plane

Temporal success leads to larger Temporal deployments.
Needs can increase, and can go from having one or two production use cases in a single region to many use cases in many regions.
Running multiple Temporal Services is complex work, as each needs its own setup, tuning, and configuration.

Needing to monitor and manage all your Temporal Services in a unified way leads to operational management pain.
Consider adding a layer on top of Temporal to manage multiple Temporal Services: a control plane.
A control plane manages and directs data flow, deciding where data packets should be sent.
A Temporal Service data plan can streamline operations and improve efficiency.
Since Temporal does not ship its own open source data plane, rolling your own can be complex and take effort to add.

Temporal Cloud provides exactly that support.
With Temporal Cloud, all Namespaces in all regions can be managed from a single view.
[Temporal Cloud](https://temporal.io/cloud) also has RBAC functionality that can delegate responsibilities for individual Namespaces.

Self-hosted Temporal does not support RBAC or audit logging out of the box.
Temporal Cloud provides RBAC and SSO support, audit logging, data encryption, third party penetration test validation, and SOC 2-Type II and HIPAA compliance.

## Maintenance and Upgrades

Temporal recommends keeping up-to-date and not falling behind on your server versions.

Temporal Server is proactively updated, and releases as often as every two weeks.
Temporal recommends [upgrading sequentially](/self-hosted-guide/upgrade-server), not skipping any minor versions, although you can skip patch versions.
No support is guaranteed for Temporal Server, but very old servers will be hard for even the community to support, so we encourage you to keep up to date.
You must create and maintain the infrastructure to host and run your self-hosted Temporal installation, such as Kubernetes, as well as data stores for persistence.

Server upgrades can negatively affect self-hosted Temporal Service availability.
Temporal recommends load and availability testing during the upgrade process to understand the performance implications.

Temporal Cloud updates are managed by the Temporal Cloud team; Cloud upgrades are seamless.

## Expert Support

Temporal recommends that customer platform teams who are building out a Temporal service gain deep experience across the lifecycle and breadth of a Temporal application.

Specific activities include:

- [Worker tuning](/develop/worker-performance)
- [Worker best practices](/workers)
- Code reviews
- Design guidance
- Training
- Code reviews
- Security reviews
- [Metrics](/references/sdk-metrics) and monitoring
- Technical onboarding

[Temporal support](/cloud/support) provides guidance on all of the above.

## Cost Management

Running a mission critical, global Temporal Service can be expensive.
Temporal Server is a complex system to run and scale.
Temporal recommends performance testing and planning scaling as your performance requirements evolve.
Following our guidance can oversize your self-hosted Temporal Server installs, but this is necessary to handle unpredictable spiky workloads.
Performance testing can help you right-size your environments.
Running mission critical Temporal as a Service requires multiple Temporal Clusters for high availability and global coverage.

It is a good practice to have trained, experienced administrators familiar with Temporal Service architecture to maintain your Temporal servers and provide a mission critical service.
Staffing, training and skill development can be significant costs to maintaining a Temporal Service.

[Temporal Cloud](https://temporal.io/cloud) can be significantly less expensive to set up and scale.

---

## Self-hosted Temporal Service defaults

:::info Looking for Temporal Cloud defaults?

See the [Temporal Cloud defaults and limits page](/cloud/limits)

:::

This page details many of the defaults coded into the Temporal Platform that can produce errors and warnings.
Errors are hard limits that fail when reached.
Warnings are soft limits that produce a warning log on the server side.

:::info

These limits might apply specifically to each Workflow Execution and do not pertain to the entire Temporal Platform or individual Namespaces.

:::

- **Identifiers:** By default, the maximum length for identifiers (such as Workflow Id, Workflow Type, and Task Queue name) is 1000 characters.
  - This is configurable with the `limit.maxIDLength` dynamic config variable, set to 255 in [this SQL example](https://github.com/temporalio/docker-compose/blob/93d382ef9133e4cde8ce311de5153cd0cc9fbd0c/dynamicconfig/development-sql.yaml#L1-L2).
  - The character format is UTF-8.
- **gRPC:** gRPC has a limit of 4 MB for [each message received](https://github.com/grpc/grpc/blob/v1.36.2/include/grpc/impl/codegen/grpc_types.h#L466).
- **Event batch size:** The `DefaultTransactionSizeLimit` limit is [4 MB](https://github.com/temporalio/temporal/pull/1363).
  This is the largest transaction size allowed for the persistence of Event Histories.
- **Blob size limit** for Payloads (including Workflow context and each Workflow and Activity argument and return value; _[source](https://github.com/temporalio/temporal/blob/v1.7.0/service/frontend/service.go#L133-L134)_):
  - Temporal warns at 256 KB: `Blob size exceeds limit.`
  - Temporal errors at 2 MB: `ErrBlobSizeExceedsLimit: Blob data size exceeds limit.`
  - Refer to [Troubleshoot blob size limit error](/troubleshooting/blob-size-limit-error).
- **Workflow Execution Update limits**:
  - A single Workflow Execution can have a maximum of 10 in-flight Updates and 2000 total Updates in History.
- **History total size limit** (leading to a terminated Workflow Execution):
  - Temporal warns at 10 MB: [history size exceeds warn limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1238).
  - Temporal errors at 50 MB: [history size exceeds error limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1204).
  - This is configurable with [HistorySizeLimitError and HistorySizeLimitWarn](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/configs/config.go#L380-L381).
- **History total count limit** (leading to a terminated Workflow Execution):
  - Temporal warns after 10,240 Events: [history size exceeds warn limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1238).
  - Temporal errors after 51,200 Events: [history size exceeds error limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1204).
  - This is configurable with [HistoryCountLimitError and HistoryCountLimitWarn](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/configs/config.go#L382-L383).
- **Concurrent limit**
  - The following Commands are limited:
    - `ScheduleActivityTask`
    - `SignalExternalWorkflowExecution`
    - `RequestCancelExternalWorkflowExecution`
    - `StartChildWorkflowExecution`
  - These will fail if the concurrent pending count exceeds 2,000.
    For optimal performance, limit concurrent operations to 500 or fewer.
    This reduces Workflow's Event History size and decreases the loading time in the Web UI.
  - As of v1.21, the open source Temporal Service has a default limit of 2,000 pending Activities, Child Workflows, Signals, or Workflow cancellation requests, but you can override the limits in the dynamic configuration using these variables:
    - `limit.numPendingActivities.error`
    - `limit.numPendingSignals.error`
    - `limit.numPendingCancelRequests.error`
    - `limit.numPendingChildExecutions.error`
  - By default, [Batch jobs](/cli/batch) are limited to one job at a time.
- [Custom Search Attributes limits](/search-attribute#custom-search-attribute-limits)

For details on dynamic configuration keys, see [Dynamic configuration reference](/references/dynamic-configuration).

---

## Deploying a Temporal Service

There are many ways to self-host a [Temporal Service](/temporal-service).
The right way for you depends entirely on your use case and where you plan to run it.

For step-by-step guides on deploying and configuring Temporal, refer to our [Infrastructure tutorials](https://learn.temporal.io/tutorials/infrastructure/).

### Minimum requirements

The Temporal Server depends on a database.

Supported databases include the following:

- [Apache Cassandra](/self-hosted-guide/visibility#cassandra)
- [MySQL](/self-hosted-guide/visibility#mysql)
- [PostgreSQL](/self-hosted-guide/visibility#postgresql)
- [SQLite](/self-hosted-guide/visibility#sqlite)

### Docker & Docker Compose

You can run a Temporal Service in [Docker](https://docs.docker.com/engine/install) containers using [Docker Compose](https://docs.docker.com/compose/install).

If you have Docker and Docker Compose installed, all you need to do is clone the [temporalio/docker-compose](https://github.com/temporalio/docker-compose) repo and run the `docker compose up` command from its root.

The `temporalio/docker-compose` repo comes loaded with a variety of configuration templates that enable you to try all three databases that the Temporal Platform supports (PostgreSQL, MySQL, Cassandra).
It also enables you to try [Advanced Visibility](/visibility#advanced-visibility) using [Search Attributes](/search-attribute), emit metrics, and even play with the [Archival](/temporal-service/archival) feature.
The Docker images in this repo are produced using the Temporal Server [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script.
This script defaults to creating images that run all the Temporal Server services in a single process.
You can use this script as a starting point for producing your own images.

The following commands start and run a Temporal Service in Docker using the default configuration:

```bash
git clone https://github.com/temporalio/docker-compose.git
cd docker-compose
docker compose up
```

Local [Temporal Clients](/encyclopedia/temporal-sdks#temporal-client) and [Workers](/workers) can connect to the Temporal Service running in Docker at 127.0.0.1:7233 (default connection for most SDKs) and the Temporal Web UI at 127.0.0.1:8080.

To try other configurations (different dependencies and databases), or to try a custom Docker image, follow the [temporalio/docker-compose README](https://github.com/temporalio/docker-compose/blob/main/README.md).

### Temporal Server binaries

You can run a complete Temporal Server by deploying just two Go binaries -- the [core Temporal Server](https://github.com/temporalio/temporal/releases/), and the [Temporal UI Server](https://github.com/temporalio/ui-server/releases).
Refer to our [tutorial site](https://learn.temporal.io/) for more details on how to deploy Temporal binaries behind an [Nginx reverse proxy](https://learn.temporal.io/tutorials/infrastructure/nginx-sqlite-binary/) or an [Envoy edge proxy](https://learn.temporal.io/tutorials/infrastructure/envoy-sqlite-binary/).

Each service can also be deployed separately.
For example, if you are using Kubernetes, you could have one service per pod, so they can scale independently in the future.

In Docker, you could run each service in its own container, using the `SERVICES` flag to specify the service:

```bash
docker run
    # persistence/schema setup flags omitted
    -e SERVICES=history \                      -- Spin up one or more: history, matching, worker, frontend
    -e LOG_LEVEL=debug,info \                           -- Logging level
    -e DYNAMIC_CONFIG_FILE_PATH=config/foo.yaml         -- Dynamic config file to be watched
    temporalio/server:<tag>
```

The environment variables supported by the Temporal Docker images are documented [on Docker Hub](https://hub.docker.com/r/temporalio/auto-setup).

Each Temporal Server release ships an [Auto Setup](https://temporal.io/blog/auto-setup) Docker image that includes an [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script.
We recommend using this script for initial schema setup of each supported database.

### Importing the Server package

The Temporal Server is a standalone Go application that can be [imported](/references/server-options) into another project.

You might want to do this to pass custom plugins or any other customizations through the [Server Options](/references/server-options).
Then you can build and run a binary that contains your customizations.

This requires Go v1.19 or later, as specified in the Temporal Server [Build prerequisites](https://github.com/temporalio/temporal/blob/main/CONTRIBUTING.md#build-prerequisites).

### Helm charts

[Temporal Helm charts](https://github.com/temporalio/helm-charts) enable you to get a Temporal Service running on [Kubernetes](https://kubernetes.io/) by deploying the Temporal Server services to individual pods and connecting them to your existing database and Elasticsearch instances.

The Temporal Helm charts repo contains [extensive documentation](https://github.com/temporalio/helm-charts/blob/main/README.md) about Kubernetes deployments.

---

## Self-hosted Temporal Service guide

Welcome to the self-hosted Temporal Service guide.
This guide shows you how to self-host open source infrastructure software that orchestrates your durable applications.

:::info Sign up for Temporal Cloud!

Instead of self-hosting, you can use [Temporal Cloud](/cloud).

:::

:::info Getting started with Temporal?

If you are just getting started with Temporal, we recommend our [introductory tutorials and courses](https://learn.temporal.io)

:::

:::info Building an app?

If you are building a new Temporal Application, you might only need a [development server](/cli#start-dev-server) available through the [Temporal CLI](/cli).
Check out the [dev guide](/develop) for application development best practices.

:::

- [Deployment](/self-hosted-guide/deployment)
- [Defaults](/self-hosted-guide/defaults)
- [Production checklist](/self-hosted-guide/production-checklist)
- [Namespaces](/self-hosted-guide/namespaces)
- [Security](/self-hosted-guide/security)
- [Monitoring](/self-hosted-guide/monitoring)
- [Visibility](/self-hosted-guide/visibility)
- [Data encryption](/production-deployment/data-encryption)
- [Upgrading server](/self-hosted-guide/upgrade-server#upgrade-server)
- [Archival](/self-hosted-guide/archival)
- [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication)
- [Temporal Nexus](/production-deployment/self-hosted-guide/nexus)

---

## Monitor Temporal Platform metrics

The Temporal Service and SDKs emit metrics that can be used to monitor performance and troubleshoot issues. 
You can relay these metrics to any monitoring and observability platform.

This guide will provide an example of configuring [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) to work with the observability metrics emitted from Temporal.
This solution can work on its own, or serve as a baseline for you to further customize and integrate with other observability tooling.
For example, it is also possible to use the [OpenTelemetry Collector](https://temporal.io/code-exchange/temporal-opentelemetry) in your stack instead of scraping metrics directly with Prometheus, or [Datadog](#datadog) as a frontend instead of Grafana.

This configuration assumes that you have [Docker](https://www.docker.com/) installed and are running a [Temporal dev server](https://temporal.io/setup/start-development-server) via the CLI. 

## Prometheus

This section discusses exporting metrics from Temporal SDKs, and setting up Prometheus to collect metrics on Temporal Service, Temporal Client, and Temporal Worker performance.

The Temporal Service and SDKs emit all metrics by default.
However, you must enable Prometheus in your application code (using the Temporal SDKs) and your Temporal Service configuration to collect the metrics emitted from your SDK and Temporal Service.

First, you'll need to create a `prometheus.yml` configuration file with some target ports to collect metrics from.
Here is a sample with one Temporal Service metrics target and two Temporal Worker (SDK) metrics targets:

```
global:
 scrape_interval: 10s
scrape_configs:
 - job_name: 'temporalmetrics'
   metrics_path: /metrics
   scheme: http
   static_configs:
     # Temporal Service metrics target
     - targets:
         - 'host.docker.internal:8000'
       labels:
         group: 'server-metrics'

     # Local app targets (set in SDK code)
     - targets:
         - 'host.docker.internal:8077'
         - 'host.docker.internal:8078'
       labels:
         group: 'sdk-metrics'
```

In this example, Prometheus is configured to scrape at 10-second intervals and to listen for Temporal Service metrics on `host.docker.internal:8000` and SDK metrics on two targets, `host.docker.internal:8077` and `host.docker.internal:8078`.
The `8077` and `8078` ports must be set on `WorkflowServiceStubs` in your application code with your preferred SDK -- there is an example of this in the next section.
You can set up as many targets as required.

:::info

For further Prometheus configuration options, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/).

:::

You can use Docker to run the official Prometheus image with this configuration:

```bash
docker run -p 9090:9090 -v /path/to/prometheus.yml /etc/prometheus/prometheus.yml prom/prometheus
```

Next, launch your Temporal dev server from the CLI with an additional `--metrics-port 8000` parameter:

```bash
temporal server start-dev --metrics-port 8000
```

:::info

Refer to the [Temporal Cluster configuration reference](/references/configuration#global) to expose metrics from a production service.

:::

You should now have both Prometheus and a Temporal Service running locally, with Temporal providing Service metrics to Prometheus.
Next, you'll want to configure SDK metrics as well.

### SDK metrics setup

SDK metrics are emitted by Temporal Workers and other Clients, and must be configured in your application code.
The Metrics section in the Observability guide details how to create hooks for all supported SDKs:

- [Go](/develop/go/observability#metrics)
- [Java](/develop/java/observability#metrics)
- [PHP](/develop/php/observability)
- [Python](/develop/python/observability#metrics)
- [TypeScript](/develop/typescript/observability#metrics)
- [.NET](/develop/dotnet/observability#metrics)
- [Ruby](/develop/ruby/observability#metrics)

For end-to-end examples of how to expose metrics from each SDK, see the metrics samples:

- [Go SDK Sample](https://github.com/temporalio/samples-go/tree/main/metrics)
- [Java SDK Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics)
- [Python SDK Sample](https://github.com/temporalio/samples-python/tree/main/prometheus)
- [TypeScript SDK Sample](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry)
- [.NET SDK Sample](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry)

Some of these may require you to set different metrics port numbers based on the Prometheus example here, which is configured to scrape port `8077` and `8078` by default.
Follow the instructions from each of the samples to run Workflows and begin emitting metrics.
This will allow you to populate a dashboard in the next section and understand how to further customize Temporal observability for your needs.

### Verifying Prometheus configuration

Once your Workflows are running and emitting metrics, you can visit [http://localhost:9090/targets](http://localhost:9090/targets) on your local Prometheus instance to verify that it is able to scrape the provided endpoints.

![Prometheus scrape targets](/img/observability/prometheus-targets.png)

This example shows a response from the server metrics endpoint, provided by the Temporal dev server, and two SDK metrics endpoints, as defined in the Prometheus configuration.
To create this example, we used the Go and Python metrics samples, running on port 8077 and 8088 respectively.
If you are not pushing data to exactly 3 metrics endpoints, your environment may be different.

Next, you can visit the [local Prometheus query endpoint](http://localhost:9090/query) to manually run [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) queries on your exported metrics, or proceed to the next section to configure Grafana to generate dashboards from those metrics.

## Grafana

With [Prometheus](#prometheus) configured, deploy Grafana as a metrics frontend, and configure it to use Prometheus as a data source.

As before, you can use Docker to run the official Grafana image:

```bash
docker run -d -p 3000:3000 grafana/grafana-enterprise
```

This will deploy a Grafana instance with a default username and password of `admin`/`admin`.
In production, you would want to [configure authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/) and control port access to Grafana.

:::info

For more information on how to customize your Grafana setup, see the [Grafana documentation](https://grafana.com/docs/grafana/latest/setup-grafana/).

:::

Next, configure Grafana to use Prometheus as the data source.
To do this, click on "Add new data source" from the "Connections" menu in the Grafana sidebar, and add Prometheus from the list.

You will be prompted to add additional configuration parameters.
If you are following this guide using Docker, use `http://host.docker.internal:9090` as the Prometheus address.
This is a [DNS name provided by Docker Desktop](https://docs.docker.com/desktop/features/networking/#use-cases-and-workarounds) which resolves to the internal IP address used by the host machine, and allows you to connect applications across Docker containers without additional configuration rules.
This is the only parameter you will need to set for your Prometheus configuration.
After providing it, scroll down to the "Save and Test" button, and you can validate Prometheus as a data source for this Grafana instance.

![Grafana data sources](/img/observability/grafana-data-sources.png)

In this example, Grafana is set to pull metrics from Prometheus at the port 9090, as defined in the Prometheus configuration.

Now, you'll just need to add some of our provided dashboards for visualizing Temporal metrics.

### Dashboard setup

We provide community-driven Grafana dashboards that can be used for monitoring Temporal Server and SDK metrics in a [dashboards](https://github.com/temporalio/dashboards/) repo.
Follow the instructions in that repo's README to import the dashboards to Grafana.

This way, you can create at least one dashboard for monitoring server metrics:

![Grafana server metrics](/img/observability/grafana-server-metrics.png)

And at least one other dashboard for monitoring SDK metrics:

![Grafana SDK metrics](/img/observability/grafana-sdk-metrics.png)

:::info

You can provide additional queries in your dashboard to report other data as needed.
For more details on configuring Grafana dashboards, see the [Grafana Dashboards documentation](https://grafana.com/docs/grafana/latest/dashboards/).

:::

From here, you can configure Grafana [Alerts](https://grafana.com/docs/grafana/latest/alerting/) for any monitored parameters, add custom metrics to your Temporal SDK code, and use these observability features to help scale your Temporal deployment.
Refer to the [Cluster metrics](/references/cluster-metrics) and [SDK metrics](/references/sdk-metrics) reference for more.

## Configuring Temporal Service health checks {#health-checks}

The [Frontend Service](/temporal-service/temporal-server#frontend-service) supports TCP or [gRPC](https://github.com/grpc/grpc/blob/875066b61e3b57af4bb1d6e36aabe95a4f6ba4f7/src/proto/grpc/health/v1/health.proto#L45) health checks on port 7233.

If you use [Nomad](https://www.nomadproject.io/) to manage your containers, the [check stanza](https://developer.hashicorp.com/nomad/docs/job-specification/check) would look like this for TCP:

```
service {
  check {
    type     = "tcp"
    port     = 7233
    interval = "10s"
    timeout  = "2s"
  }
```

or like this for gRPC (requires Consul ≥ `1.0.5`):

```
service {
  check {
    type         = "grpc"
    port         = 7233
    interval     = "10s"
    timeout      = "2s"
  }
```

## Installing via Helm Chart

If you are installing and running Temporal via [Helm chart](https://github.com/temporalio/helm-charts), you can also [provide additional parameters](https://github.com/temporalio/helm-charts?tab=readme-ov-file#exploring-metrics-via-grafana) to populate and explore a Grafana dashboard out of the box.

## Datadog {#datadog}

Datadog has a Temporal integration for collecting Temporal Service metrics.
Once you've [configured Prometheus](#prometheus), you can configure the [Datadog Agent](https://docs.datadoghq.com/integrations/temporal/).

If you are using [Temporal Cloud](/cloud/overview), you can also [integrate Datadog directly](https://docs.datadoghq.com/integrations/temporal-cloud/), without needing to use Prometheus.

---

## Self-hosted Multi-Cluster Replication

Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction.
When necessary, for higher availability, Cluster operators can failover to any of the backup Clusters.

Temporal's Multi-Cluster Replication feature is considered **experimental** and not subject to normal [versioning and support policy](/temporal-service/temporal-server#versions-and-support).

Temporal automatically forwards Start, Signal, and Query requests to the active Cluster.
This feature must be enabled through a Dynamic Config flag per [Global Namespace](/global-namespace).

When the feature is enabled, Tasks are sent to the Parent Task Queue partition that matches that Namespace, if it exists.

All Visibility APIs can be used against active and standby Clusters.
This enables [Temporal UI](https://docs.temporal.io/web-ui) to work seamlessly for Global Namespaces.
Applications making API calls directly to the Temporal Visibility API continue to work even if a Global Namespace is in standby mode.
However, they might see a lag due to replication delay when querying the Workflow Execution state from a standby Cluster.

#### Namespace Versions

A _version_ is a concept in Multi-Cluster Replication that describes the chronological order of events per Namespace.

With Multi-Cluster Replication, all Namespace change events and Workflow Execution History events are replicated asynchronously for high throughput.
This means that data across clusters is **not** strongly consistent.
To guarantee that Namespace data and Workflow Execution data will achieve eventual consistency (especially when there is a data conflict during a failover), a **version** is introduced and attached to Namespaces.
All Workflow Execution History entries generated in a Namespace will also come with the version attached to that Namespace.

All participating Clusters are pre-configured with a unique initial version and a shared version increment:

- `initial version < shared version increment`

When performing failover for a Namespace from one Cluster to another Cluster, the version attached to the Namespace will be changed by the following rule:

- for all versions which follow `version % (shared version increment) == (active cluster's initial version)`, find the smallest version which has `version >= old version in namespace`

When there is a data conflict, a comparison will be made and Workflow Execution History entries with the highest version will be considered the source of truth.

When a cluster is trying to mutate a Workflow Execution History, the version will be checked.
A cluster can mutate a Workflow Execution History only if the following is true:

- The version in the Namespace belongs to this cluster, i.e.
  `(version in namespace) % (shared version increment) == (this cluster's initial version)`
- The version of this Workflow Execution History's last entry (event) is equal or less than the version in the Namespace, i.e.
  `(last event's version) <= (version in namespace)`

<details>
    <summary>
    Namespace version change example
    </summary>

Assuming the following scenario:

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Shared version increment: 10

T = 0: Namespace α is registered, with active Cluster set to Cluster A

```
namespace α's version is 1
all workflows events generated within this namespace, will come with version 1
```

T = 1: namespace β is registered, with active Cluster set to Cluster B

```
namespace β's version is 2
all workflows events generated within this namespace, will come with version 2
```

T = 2: Namespace α is updated to with active Cluster set to Cluster B

```
namespace α's version is 2
all workflows events generated within this namespace, will come with version 2
```

T = 3: Namespace β is updated to with active Cluster set to Cluster A

```
namespace β's version is 11
all workflows events generated within this namespace, will come with version 11
```

</details>

#### Version history

Version history is a concept which provides a high level summary of version information in regards to Workflow Execution History.

Whenever there is a new Workflow Execution History entry generated, the version from Namespace will be attached.
The Workflow Executions's mutable state will keep track of all history entries (events) and the corresponding version.

<details>
    <summary>
        Version history example (without data conflict)
    </summary>

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Shared version increment: 10

T = 0: adding event with event ID == 1 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 1               | 1       |
| -------- | -------------   | --------------- | ------- |
```

T = 1: adding event with event ID == 2 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 2: adding event with event ID == 3 & version == 1

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               |                 |         |
| 3        | 1               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 3: Namespace failover triggered, Namespace version is now 2
adding event with event ID == 4 & version == 2

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               | 4               | 2       |
| 3        | 1               |                 |         |
| 4        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 4: adding event with event ID == 5 & version == 2

View in both Cluster A & B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 3               | 1       |
| 2        | 1               | 5               | 2       |
| 3        | 1               |                 |         |
| 4        | 2               |                 |         |
| 5        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

</details>

Since Temporal is AP, during failover (change of active Temporal Cluster Namespace), there can exist cases where more than one Cluster can modify a Workflow Execution, causing divergence of Workflow Execution History. Below shows how the version history will look like under such conditions.

<details>
    <summary>
    Version history example (with data conflict)
    </summary>

Below, shows version history of the same Workflow Execution in 2 different Clusters.

- Cluster A comes with initial version: 1
- Cluster B comes with initial version: 2
- Cluster C comes with initial version: 3
- Shared version increment: 10

T = 0:

View in both Cluster B & C

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 3               | 2       |
| 3        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 1: adding event with event ID == 4 & version == 2 in Cluster B

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 4               | 2       |
| 3        | 2               |                 |         |
| 4        | 2               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 1: namespace failover to Cluster C, adding event with event ID == 4 & version == 3 in Cluster C

```
| -------- | --------------- | --------------- | ------- |
| Events   | Version History |                 |         |
| -------- | --------------- | --------------- | ------- |
| Event ID | Event Version   | Event ID        | Version |
| -------- | -------------   | --------------- | ------- |
| 1        | 1               | 2               | 1       |
| 2        | 1               | 3               | 2       |
| 3        | 2               | 4               | 3       |
| 4        | 3               |                 |         |
| -------- | -------------   | --------------- | ------- |
```

T = 2: replication task from Cluster C arrives in Cluster B

Note: below is a tree structures

```
| ------------- | ------------- |
| Events         |               |
| -------------- | ------------- |
| Event ID       | Event Version |
| -------------  | ------------- |
| 1              | 1             |
| 2              | 1             |
| 3              | 2             |
| -------------  | ------------- |
|                |               |
| -------------  | ------------- |
|                |               |
| -------------- | ------------- |  | -------- | ------------- |
| Event ID       | Event Version |  | Event ID | Event Version |
| -------------  | ------------- |  | -------- | ------------- |
| 4              | 2             |  | 4        | 3             |
| -------------- | ------------- |  | -------- | ------------- |

| --------------- | ----------- |
| Version History |              |
| --------------- | ------------ |
| Event ID        | Version      |
| --------------- | ------------ |
| 2               | 1            |
| 3               | 2            |
| --------------- | ------------ |

| --------------- | ----------- |  | --------------- | ------- |
| Event ID | Version |  | Event ID | Version |
| -------- | ------- || --------------- | ------- |
| 4   | 2   |  | 4 | 3 |
| --- | --- || --------------- | ------- |
```

T = 2: replication task from Cluster B arrives in Cluster C, same as above

</details>

#### Conflict resolution

When a Workflow Execution History diverges, proper conflict resolution is applied.

In Multi-cluster Replication, Workflow Execution History Events are modeled as a tree, as shown in the second example in [Version History](#version-history).

Workflow Execution Histories that diverge will have more than one history branch.
Among all history branches, the history branch with the highest version is considered the `current branch` and the Workflow Execution's mutable state is a summary of the current branch.
Whenever there is a switch between Workflow Execution History branches, a complete rebuild of the Workflow Execution's mutable state will occur.

Temporal Multi-Cluster Replication relies on asynchronous replication of Events across Clusters, so in the case of a failover it is possible to have an Activity Task dispatched again to the newly active Cluster due to a replication task lag.
This also means that whenever a Workflow Execution is updated after a failover by the new Cluster, any previous replication tasks for that Execution cannot be applied.
This results in loss of some progress made by the Workflow Execution in the previous active Cluster.
During such conflict resolution, Temporal re-injects any external Events like Signals in the new Event History before discarding replication tasks.
Even though some progress could roll back during failovers, Temporal provides the guarantee that Workflow Executions won't get stuck and will continue to make forward progress.

Activity Execution completions are not forwarded across Clusters.
Any outstanding Activities will eventually time out based on the configuration.
Your application should have retry logic in place so that the Activity gets retried and dispatched again to a Worker after the failover to the new Cluster.
Handling this is similar to handling an Activity Task timeout caused by a Worker restarting.

#### Zombie Workflows

There is an existing contract that for any Namespace and Workflow Id combination, there can be at most one run (Namespace + Workflow Id + Run Id) open / executing.

Multi-cluster Replication aims to keep the Workflow Execution History as up-to-date as possible among all participating Clusters.

Due to the nature of Multi-cluster Replication (for example, Workflow Execution History events are replicated asynchronously) different Runs (same Namespace and Workflow Id) can arrive at the target Cluster at different times, sometimes out of order, as shown below:

```
| ------------- |          | ------------- |          | ------------- |
| Cluster A |  | Network Layer |  | Cluster B |
| --------- || ------------- |          | ------------- |
        |                          |                          |
        | Run 1 Replication Events |                          |
        | -----------------------> |                          |
        |                          |                          |
        | Run 2 Replication Events |                          |
        | -----------------------> |                          |
        |                          |                          |
        |                          |                          |
        |                          |                          |
        |                          | Run 2 Replication Events |
        |                          | -----------------------> |
        |                          |                          |
        |                          | Run 1 Replication Events |
        |                          | -----------------------> |
        |     |  |
        | --- || ------------- |          | ------------- |
| Cluster A |  | Network Layer |  | Cluster B |
| --------- || ------------- |          | ------------- |
```

Because Run 2 appears in Cluster B first, Run 1 cannot be replicated as "runnable" due to the rule `at most one Run open` (see above), thus the "zombie" Workflow Execution state is introduced.
A "zombie" state is one in which a Workflow Execution which cannot be actively mutated by a Cluster (assuming the corresponding Namespace is active in this Cluster). A zombie Workflow Execution can only be changed by a replication Task.

Run 1 will be replicated similar to Run 2, except when Run 1's execution will become a "zombie" before Run 1 reaches completion.

#### Workflow Task processing

In the context of Multi-cluster Replication, a Workflow Execution's mutable state is an entity which tracks all pending tasks.
Prior to the introduction of Multi-cluster Replication, Workflow Execution History entries (events) are from a single branch, and the Temporal Server will only append new entries (events) to the Workflow Execution History.

After the introduction of Multi-cluster Replication, it is possible that a Workflow Execution can have multiple Workflow Execution History branches.
Tasks generated according to one history branch may become invalidated by switching history branches during conflict resolution.

Example:

T = 0: task A is generated according to Event Id: 4, version: 2

```
| -------- | ------------- |
| Events   |
| -------- | ------------- |
| Event ID | Event Version |
| -------- | ------------- |
| 1        | 1             |
| 2        | 1             |
| 3        | 2             |
| -------- | ------------- |
|          |
|          |
| -------- | ------------- |
| Event ID | Event Version |
| -------- | ------------- |
| 4        | 2             | <-- task A belongs to this event |
| -------- | ------------- |
```

T = 1: conflict resolution happens, Workflow Execution's mutable state is rebuilt and history Event Id: 4, version: 3 is written down to persistence

```
| ------------- | -------------- |
| Events        |
| ------------- | -------------- |
| Event ID      | Event Version  |
| ------------- | -------------- |
| 1             | 1              |
| 2             | 1              |
| 3             | 2              |
| ------------- | -------------- |

| --------------| -------------- |                                  |----------| ----------------- |
| Event ID | Event Version |  | Event ID | Event Version |
| -------- | ------------- || -------- | ----------------- |
| 4   | 2   | <-- task A belongs to this event | 4 | 3 | <-- current branch / mutable state |
| --- | --- || -------- | ----------------- |
```

T = 2: task A is loaded.

At this time, due to the rebuild of a Workflow Execution's mutable state (conflict resolution), Task A is no longer relevant (Task A's corresponding Event belongs to non-current branch).
Task processing logic will verify both the Event Id and version of the Task against a corresponding Workflow Execution's mutable state, then discard task A.

## How to set up Multi-Cluster Replication {#set-up-multi-cluster-replication}

The [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) feature asynchronously replicates Workflow Execution Event Histories from active Clusters to other passive Clusters, and can be enabled by setting the appropriate values in the `clusterMetadata` section of your configuration file.

1. `enableGlobalNamespace` must be set to `true`.
2. `failoverVersionIncrement` has to be equal across connected Clusters.
3. `initialFailoverVersion` in each Cluster has to assign a different value.
   No equal value is allowed across connected Clusters.

After the above conditions are satisfied, you can start to configure a multi-cluster setup.

#### Set up Multi-Cluster Replication prior to v1.14

You can set this up with [`clusterMetadata` configuration](/references/configuration#clustermetadata); however, this is meant to be only a conceptual guide rather than a detailed tutorial.

:::tip

If you need help when setting up, please reach out to our [community Slack](https://temporalio.slack.com).
Good places to start include the **\#support-community** channel, searching through previous conversations, and asking our unusually excellent Temporal-trained large language model (visit **\#ask-ai**).

Need a Slack invitation?
Here's an [invitation link](https://temporal.io/slack).

:::

For example:

```yaml