Kafka Table Engine Integration

Why it matters

This feature was created to enable seamless and efficient ingestion of real-time streaming data from Kafka into ClickHouse. It solves the problem of integrating Kafka with ClickHouse without the need for external connectors or complex ETL pipelines, providing users with a native, performant, and scalable solution for handling Kafka data streams.

How to use it

To use the Kafka table engine, create a table in ClickHouse with the engine set to Kafka. Specify Kafka connection parameters such as kafka_broker_list, kafka_topic_list, kafka_group_name, kafka_format, and others in the table engine definition. After creating the table, you can consume the Kafka messages as if they were rows in a ClickHouse table. Usually, it is combined with a Materialized View to select and insert parsed data into a regular ClickHouse table for querying. Example:

CREATE TABLE kafka_table (
    key String,
    value String
) ENGINE = Kafka
SETTINGS
    kafka_broker_list = 'localhost:9092',
    kafka_topic_list = 'my_topic',
    kafka_group_name = 'my_group',
    kafka_format = 'JSONEachRow';

Then create a materialized view to materialize the data:

CREATE MATERIALIZED VIEW kafka_mv TO target_table AS
SELECT * FROM kafka_table;