# Connect to the first ClickHouse replica
kubectl exec -it chi-opik-clickhouse-cluster-0-0-0 -- clickhouse-client
```

<Callout intent="warning">
  **Important**: The Opik schema name is typically `opik` but may vary depending on your installation. Before proceeding, verify your schema name by running `SHOW DATABASES;` in ClickHouse and identifying the Opik database. Use that database name in all subsequent commands.
</Callout>

Run the `SYSTEM RESTORE REPLICA` command for each table:

```sql
-- Restore system tables
SYSTEM RESTORE REPLICA default.DATABASECHANGELOG;
SYSTEM RESTORE REPLICA default.DATABASECHANGELOGLOCK;

-- Verify your Opik database name
SHOW DATABASES;

-- List all Opik tables (replace 'opik' with your actual schema name if different)
USE opik;
SHOW TABLES;

-- Restore each Opik table
SYSTEM RESTORE REPLICA opik.attachments;
SYSTEM RESTORE REPLICA opik.automation_rule_evaluator_logs;
SYSTEM RESTORE REPLICA opik.comments;
SYSTEM RESTORE REPLICA opik.dataset_items;
SYSTEM RESTORE REPLICA opik.experiment_items;
SYSTEM RESTORE REPLICA opik.experiments;
SYSTEM RESTORE REPLICA opik.feedback_scores;
SYSTEM RESTORE REPLICA opik.guardrails;
SYSTEM RESTORE REPLICA opik.optimizations;
SYSTEM RESTORE REPLICA opik.project_configurations;
SYSTEM RESTORE REPLICA opik.spans;
SYSTEM RESTORE REPLICA opik.traces;
SYSTEM RESTORE REPLICA opik.trace_threads;
SYSTEM RESTORE REPLICA opik.workspace_configurations;
```

<Callout intent="info">
  **Note**: The exact list of tables may vary depending on your Opik version. Use the `SHOW DATABASES;` or `\d` command to list all tables in your database and restore each one.
</Callout>

##### 4. Restart ClickHouse Again

Restart ClickHouse again to ensure it:

* Re-establishes connections to Zookeeper
* Verifies and synchronizes the newly restored metadata
* Automatically resumes normal replication operations

```bash
kubectl rollout restart statefulset/chi-opik-clickhouse-cluster-0-0
```

##### 5. Validate the Recovery

After the restart completes, verify that the replica status is healthy:

```sql
-- Check table creation
SHOW CREATE TABLE opik.attachments;

-- Verify replica status
SELECT table, is_readonly, replica_is_active, zookeeper_exception
FROM system.replicas;
```

**Expected Results:**

* `is_readonly = 0` (table is writable)
* `replica_is_active = 1` (replica is active)
* `zookeeper_exception = ''` (no exceptions)

You can also verify from the Zookeeper side:

```bash