v.25.6Experimental Feature

Support two types of partition locks: permanent

Official CH changelog text
Implement Kafka rebalance like logic for Kafka2 using ClickHouse Keeper For each replica we support two types of partition locks: permanent locks and temporary locks. The replica tries to hold permanent locks as long as possible, at any given time there are no more than all_topic_partitions / active_replicas_count (here all_topic_partitions is the number of all partitions, active_replicas_count is the number of active replicas) permanent locks on the replica, if there are more, then the replica releases some partitions. Some partitions are temporarily held by the replica. The maximum number of temporary locks on a replica changes dynamically to give other replicas a chance to take some partitions into permanent locks. When updating temporary locks, the replica releases them all and tries to take some others again. #78726 (Daria Fomina).
This feature implements Kafka-style rebalance logic for the Kafka2 engine in ClickHouse, using ClickHouse Keeper to coordinate partition locks across replicas.

Why it matters

The feature addresses the problem of evenly distributing Kafka topic partitions among active replicas to ensure balanced load and avoid overloading any single replica. It manages two types of locks on partitions—permanent and temporary—allowing replicas to hold partitions fairly and dynamically rebalance them as replicas join or leave, improving fault tolerance and resource allocation.

How to use it

ClickHouse manages partition locks automatically when using the Kafka2 engine with ClickHouse Keeper enabled. Users do not need to manually configure locks; the system maintains no more than all_topic_partitions / active_replicas_count permanent locks per replica and dynamically adjusts temporary locks to facilitate rebalancing.

Related resources

PreviousNext