v.24.1Improvement

Create Dynamic Kafka Consumers with TTL to Improve Statistics and Prevent Memory Leaks

Official CH changelog text
Create consumers for Kafka tables on the fly (but keep them for some period - kafka_consumers_pool_ttl_ms, since last used), this should fix problem with statistics for system.kafka_consumers (that does not consumed when nobody reads from Kafka table, which leads to live memory leak and slow table detach) and also this PR enables stats for system.kafka_consumers by default again. #58310 (Azat Khuzhin).
ClickHouse now creates Kafka table consumers on the fly and retains them for a configurable period (kafka_consumers_pool_ttl_ms) since they were last used. This change improves system.kafka_consumers statistics accuracy by ensuring consumers are active even when nobody reads from the Kafka table.

Why it matters

Previously, system.kafka_consumers did not consume data if no queries read from the Kafka table, causing live memory leaks and slow table detach operations. By creating and keeping consumers alive temporarily, this feature prevents memory leaks, improves resource management, and enables reliable Kafka consumer statistics reporting.

How to use it

This feature is enabled by default and users can configure the retention period of unused Kafka consumers by adjusting the kafka_consumers_pool_ttl_ms setting. No manual action is required to enable the updated statistics for system.kafka_consumers as they are enabled by default again.

Related resources

PreviousNext