v.25.11New Feature
Add kafka_schema_registry_skip_bytes
Added kafka_schema_registry_skip_bytes setting to Kafka table engine to skip envelope header bytes (e.g., AWS Glue Schema Registry's 19-byte prefix) before parsing message payload. This enables ClickHouse to consume messages from schema registries that add metadata headers. #89621 (Taras Polishchuk).Why it matters
This feature addresses the issue of consuming Kafka messages from schema registries that prepend metadata headers (such as AWS Glue Schema Registry's 19-byte prefix). By skipping these bytes, ClickHouse can correctly parse and process the actual message payload, improving compatibility with such schema registries.How to use it
To use this feature, set thekafka_schema_registry_skip_bytes parameter with the desired number of bytes to skip in the Kafka table engine settings. For example, to skip 19 bytes corresponding to AWS Glue Schema Registry's prefix, define the table with:CREATE TABLE example_kafka_table (
...
) ENGINE = Kafka SETTINGS kafka_schema_registry_skip_bytes = 19, ...;