Added support for using multiple consumers simultaneously with the Kafka engine in ClickHouse, along with extended configuration options.
Why it matters
This feature was introduced to allow multiple independent consumers to read from the same Kafka topic concurrently through the ClickHouse Kafka engine. It solves problems related to scalability, parallel data processing, and improved throughput by enabling multiple consumers rather than a single consumer instance. This provides users with greater flexibility and efficiency in how they consume and process Kafka data streams within ClickHouse.How to use it
To use multiple consumers with the Kafka engine, configure multiple Kafka tables or table engines with distinct consumer group IDs and/or other related settings in the Kafka engine configuration. Adjust the extended Kafka engine configuration options to specify parameters such as consumer groups, partition assignments, and read options to enable concurrent processing. The configuration allows better control over how multiple consumers operate together.