v.20.5Improvement

Improve Kafka engine input formats and resolve performance issues

Official CH changelog text
Make more input formats to work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when kafka_num_consumers is greater than number of partitions in topic. #11599 (filimonov).
Improved compatibility of input formats with the Kafka engine, fixed premature flush issues, and enhanced performance when kafka_num_consumers exceeds the number of topic partitions.

Why it matters

This feature addresses stability and efficiency problems in the Kafka engine integration by ensuring more input formats work correctly, preventing data flushing too early, and optimizing consumer resource usage when the number of consumers is greater than the topic partitions. It helps users achieve better data ingestion reliability and performance.

How to use it

The improvements are applied automatically in the updated ClickHouse Kafka engine. Users should configure the kafka_num_consumers setting according to their topic partitions to optimize performance. No additional actions are needed to benefit from enhanced input format support and premature flush fixes.

Related resources

PreviousNext