v.18.16Improvement

Added Subscription Checks and kafka_max_block_size for Kafka Table Engine

Added for the Kafka table engine: checks for subscriptions before beginning to read from Kafka; the kafka_max_block_size setting for the table. Marek Vavruša
Added subscription checks before reading from Kafka and introduced the kafka_max_block_size setting for the Kafka table engine.

Why it matters

This feature ensures that the Kafka table engine verifies active subscriptions before starting to read data from Kafka topics. It also allows users to control the maximum block size read from Kafka using the kafka_max_block_size setting, improving stability and configurability when processing Kafka streams.

How to use it

To use this feature, ensure your Kafka table setup has valid subscriptions before initiating reads. Configure the kafka_max_block_size setting in the table definition or via settings to limit the maximum size of data blocks read from Kafka. Example setting in the table engine configuration:

SET kafka_max_block_size = <desired_value>;