v.24.8Experimental Feature
Add Experimental Kafka Storage Engine to Store Offsets in Keeper
Add new experimental Kafka storage engine to store offsets in Keeper instead of relying on committing them to Kafka. It makes the commit to ClickHouse tables atomic with regard to consumption from the queue. #57625 (János Benjamin Antal).Why it matters
This feature solves the problem of non-atomic offset commits by making the commit to ClickHouse tables atomic in relation to consumption from the Kafka queue. It ensures more reliable and consistent data ingestion by coordinating offset storage and table updates.How to use it
Users can enable this feature by configuring the new experimentalKafka storage engine to store offsets in Keeper rather than Kafka. This typically involves setting up the table engine accordingly to utilize Keeper for offset management.