v.23.10New Feature

Allow Saving of Unparsed Records and Errors in RabbitMQ, NATS, and FileLog Engines

Official CH changelog text
Allow to save unparsed records and errors in RabbitMQ, NATS and FileLog engines. Add virtual columns _error and _raw_message(for NATS and RabbitMQ), _raw_record (for FileLog) that are filled when ClickHouse fails to parse new record. The behaviour is controlled under storage settings nats_handle_error_mode for NATS, rabbitmq_handle_error_mode for RabbitMQ, handle_error_mode for FileLog similar to kafka_handle_error_mode. If it's set to default, en exception will be thrown when ClickHouse fails to parse a record, if it's set to stream, erorr and raw record will be saved into virtual columns. Closes #36035. #55477 (Kruglov Pavel).
Introduces the ability to save unparsed records and errors in RabbitMQ, NATS, and FileLog table engines by adding virtual columns _error and _raw_message (for NATS and RabbitMQ) and _raw_record (for FileLog) that capture parsing failures.

Why it matters

This feature addresses the problem of losing data or stopping processing when ClickHouse fails to parse incoming messages from RabbitMQ, NATS, or FileLog engines. By enabling error handling modes that store problematic records along with error details, users gain better visibility into data issues and can continue processing streams without interruption.

How to use it

Configure the storage setting rabbitmq_handle_error_mode, nats_handle_error_mode, or handle_error_mode (for FileLog) in the corresponding table engine. Set the mode to stream to enable saving errors and raw records into the new virtual columns instead of throwing exceptions. The default mode (default) throws exceptions on parsing errors.

Related resources

PreviousNext