Last update: 2025-12-11T11:38:13.000Z
Content:
---
title: "Kafka partitioning strategies · Tinybird Docs"
theme-color: "#171612"
description: "Best practices for Kafka topic partitioning, including partition key design, impact on ingestion performance, and relationship to ClickHouse® table partitions."
inkeep:version: "forward"
---




# Kafka partitioning strategies [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#kafka-partitioning-strategies)

Copy as MD This guide covers best practices for Kafka topic partitioning and how it affects Tinybird's Kafka connector performance.

## Understanding Kafka partitions [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#understanding-kafka-partitions)

Kafka topics are divided into partitions, which allow:

- **  Parallel processing**   - Multiple consumers can process different partitions simultaneously
- **  Scalability**   - Distribute load across partitions
- **  Ordering**   - Messages with the same key go to the same partition, maintaining order

Tinybird's Kafka connector automatically assigns partitions to consumer instances and scales consumers based on lag and load.

**Partition assignment is automatic** : Tinybird's Kafka connector automatically handles partition assignment. There is no configuration option to explicitly target specific partitions or turn off automatic assignment. The connector manages partition distribution across consumers automatically.

## Kafka partitions vs ClickHouse® table partitions [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#kafka-partitions-vs-clickhouse-table-partitions)

**Kafka partitions** and **ClickHouse® table partitions** are different concepts:

- **  Kafka partitions**   : Logical divisions of a topic for parallel processing and ordering. Each Kafka partition can be consumed independently.
- **  ClickHouse® table partitions**   : Physical divisions of data in ClickHouse® tables, typically organized by time (for example, monthly partitions).

**How they relate:**

- Each Kafka partition can write data to multiple ClickHouse® parts within a ClickHouse® partition
- Multiple Kafka partitions can write to the same ClickHouse® partition
- ClickHouse® typically partitions by time (for example, `ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"`   )

**TOO_MANY_PARTS error:**

If you have too many Kafka partitions writing to the same ClickHouse® partition, you may encounter the `TOO_MANY_PARTS` error. This happens when ClickHouse® creates too many parts in a single partition, which can impact query performance.

**Best practice:** For ClickHouse®, partitioning by time (for example, monthly or daily) is usually fine for time-series data. This is independent of your Kafka partition key strategy. Kafka partition keys should focus on distributing load evenly across Kafka partitions, while ClickHouse® partitioning organizes data by time for efficient querying.

## Partition design best practices [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#partition-design-best-practices)

### Number of partitions [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#number-of-partitions)

**Guidelines:**

- Start with**  3-6 partitions**   for most use cases
- Scale up if you see uneven lag distribution
- More partitions = more parallelism, but also more overhead
- Too many partitions can cause frequent rebalancing and ClickHouse® TOO_MANY_PARTS errors

**Factors to consider:**

- Expected message throughput
- Consumer capacity
- ClickHouse® partition strategy

### Kafka partition key strategy [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#kafka-partition-key-strategy)

The partition key determines which Kafka partition a message goes to. This is about **Kafka partitions** , not ClickHouse partitions.

**Goal: Even distribution across Kafka partitions**

- Use keys that distribute messages evenly across Kafka partitions
- Avoid keys that create "hot partitions" (one Kafka partition receiving most messages)
- Consider hash-based keys for even distribution

**Example of good partition key:**

# Even distribution using hash - distributes messages across Kafka partitions
partition_key = hash(user_id) % num_partitions **Example of problematic partition key:**

# Creates hot partition - all messages go to one Kafka partition
partition_key = "all-users" **Note:** Partition keys are for distributing load across Kafka partitions. For ClickHouse® table partitioning, you typically partition by time (for example, `ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"` ), which is usually fine for time-series data. ClickHouse® partitioning is independent of Kafka partition keys.

## Monitoring partition performance [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#monitoring-partition-performance)

### Monitor partition lag [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#monitor-partition-lag)

Partition lag indicates how far behind your consumer is:

SELECT
    datasource_id,
    topic,
    partition,
    max(lag) as current_lag,
    avg(lag) as avg_lag
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY current_lag DESC
### Detect hot partitions [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#detect-hot-partitions)

Check if messages are evenly distributed across partitions:

SELECT
    __partition,
    count(*) as message_count,
    count(*) * 100.0 / sum(count(*)) OVER () as percentage
FROM your_datasource
WHERE timestamp > now() - INTERVAL 1 hour
GROUP BY __partition
ORDER BY message_count DESC Uneven distribution (for example, one partition has >50% of messages) indicates a hot partition.

### Monitor partition assignment [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#monitor-partition-assignment)

Track how partitions are assigned to consumers:

SELECT
    datasource_id,
    topic,
    partition,
    count(*) as assignment_count
FROM tinybird.kafka_ops_log
WHERE timestamp > now() - INTERVAL 1 hour
  AND partition >= 0
GROUP BY datasource_id, topic, partition
ORDER BY assignment_count DESC
## How it works in practice [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#how-it-works-in-practice)

1. **  Producer sends messages**   to Kafka topics with optional partition keys
2. **  Kafka assigns messages**   to partitions based on the key (or round-robin if no key)
3. **  Tinybird connector**   consumes from all partitions in parallel
4. **  Data is ingested**   into ClickHouse®, where it's organized into ClickHouse® partitions (typically by time)
5. **  Multiple Kafka partitions**   can write to the same ClickHouse® partition
6. **  Each Kafka partition**   may create multiple ClickHouse® parts within a partition

**Example flow:**

Kafka Topic: events (6 partitions)
  ├─ Partition 0 → ClickHouse® partition 202401 (multiple parts)
  ├─ Partition 1 → ClickHouse® partition 202401 (multiple parts)
  ├─ Partition 2 → ClickHouse® partition 202402 (multiple parts)
  └─ ... **Rebalancing:**

When partitions are added, removed, or consumer instances change, Kafka rebalances partition assignments. This causes a brief pause in processing. Monitor for rebalancing by tracking partition assignment changes in `kafka_ops_log`.

## Troubleshooting [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#troubleshooting)

### Issue: Uneven partition lag [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#issue-uneven-partition-lag)

**Symptoms:** Some partitions have high lag while others are idle

**Solutions:**

1. Review partition key strategy
2. Check message distribution across partitions
3. Consider increasing partitions if needed
4. Adjust partition key to distribute more evenly

### Issue: Hot partitions [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#issue-hot-partitions)

**Symptoms:** One partition receives most messages, causing high lag

**Solutions:**

1. Change partition key strategy
2. Use hash-based keys for even distribution
3. Split high-volume keys across multiple partitions

### Issue: TOO_MANY_PARTS error [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#issue-too-many-parts-error)

**Symptoms:** ClickHouse® error indicating too many parts in a partition

**Solutions:**

1. Reduce number of Kafka partitions writing to the same ClickHouse® partition
2. Adjust ClickHouse® merge settings if needed
3. Consider increasing ClickHouse® partition granularity (for example, daily instead of monthly)
4. Note: Kafka partition key strategy doesn't directly affect this - ClickHouse® partitioning by time is usually fine

## Best practices [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#best-practices)

1. **  Start with 3-6 Kafka partitions**   and scale as needed
2. **  Use Kafka partition keys**   that distribute messages evenly across Kafka partitions
3. **  For ClickHouse®**   , partition by time (for example, monthly) - this is usually fine for time-series data
4. **  Monitor partition lag**   regularly
5. **  Avoid hot Kafka partitions**   by using hash-based keys
6. **  Watch for TOO_MANY_PARTS**   errors and adjust ClickHouse® partition granularity if needed
7. **  Remember**   : Kafka partition keys (for load distribution) and ClickHouse® partitioning (for data organization) serve different purposes

## Related documentation [¶](https://www.tinybird.co/docs/forward/get-data-in/connectors/kafka/guides/partitioning-strategies#related-documentation)

- [  Performance optimization guide](performance-optimization)   - Overall performance tuning
- [  Monitor Kafka connectors](/docs/forward/monitoring/kafka-clickhouse-monitoring)   - Monitoring partition performance
- [  Troubleshooting guide](../troubleshooting)   - Resolve partition issues


