v.21.6Improvement

Add Kafka Storage Support for Arrow Formats

Official CH changelog text
Add support for Kafka storage with Arrow and ArrowStream format messages. #23415 (Chao Ma).
Added support for Kafka storage to consume messages in Arrow and ArrowStream formats.

Why it matters

This feature enables ClickHouse users to ingest data from Kafka topics where messages are serialized in Apache Arrow formats. It solves the problem of integrating high-performance columnar data interchange formats directly into Kafka storage, improving data processing efficiency and compatibility.

How to use it

To use this feature, configure the Kafka engine table with the format setting set to either Arrow or ArrowStream. For example, create a Kafka table with FORMAT = 'Arrow' or FORMAT = 'ArrowStream' to consume messages in these formats.

Related resources

PreviousNext