v.19.8Improvements

Add New Setting Kafka_commit_every_batch to Regulate Kafka Committing Policy

Official CH changelog text
Add new setting kafka_commit_every_batch to regulate Kafka committing policy. It allows to set commit mode: after every batch of messages is handled, or after the whole block is written to the storage. It's a trade-off between losing some messages or reading them twice in some extreme situations. #5308 (Ivan)
Introduces a new setting kafka_commit_every_batch in ClickHouse to control the Kafka committing policy, allowing commits either after processing each batch of messages or after writing a complete block to storage.

Why it matters

This feature addresses the trade-off between potential message loss and reading duplicate messages when consuming from Kafka. By controlling the commit behavior, users can balance reliability and performance according to their application's needs.

How to use it

Users can enable or configure this feature by setting the kafka_commit_every_batch option in the Kafka engine settings. Setting it to true commits after every batch of messages is processed, whereas setting it to false commits only after the entire block is written to storage.

Related resources

PreviousNext