v.21.7Improvement

Improvement for Split Distributed Batch on Failures in Distributed Tables

Official CH changelog text
Improvement for Distributed tables. Add ability to split distributed batch on failures (i.e. due to memory limits, corruptions), under distributed_directory_monitor_split_batch_on_failure (OFF by default). #23864 (Azat Khuzhin).
Add support for splitting batches on failure in Distributed tables via the distributed_directory_monitor_split_batch_on_failure setting, which is OFF by default.

Why it matters

This feature addresses issues when batches processed by Distributed tables fail due to memory limits or data corruptions. By splitting the batch on failure, it improves the robustness and reliability of distributed batch processing, allowing partial success instead of complete batch failure.

How to use it

Enable the feature by setting distributed_directory_monitor_split_batch_on_failure = 1 in your ClickHouse configuration or session settings to allow retry with smaller batch splits upon failure.

Related resources

PreviousNext