Allow Kafka Engine to Ignore Some Number of Parsing Errors per Block

Why it matters

This feature addresses the issue of ingestion failures caused by occasional malformed messages. By allowing a set number of parsing errors to be skipped, it improves the robustness and reliability of data import from Kafka topics without requiring manual intervention or stopping the ingestion process.

How to use it

Users can enable this feature by setting the max_errors_per_block setting in the Kafka engine table definition to specify how many parsing errors are allowed to be ignored per block. When this threshold is reached within a block, errors will cause ingestion to fail as usual.