v.23.5New Feature

Add Memory Tracking for Background Tasks in ClickHouse

Official CH changelog text
Add MemoryTracker for the background tasks (merges and mutation). Introduces merges_mutations_memory_usage_soft_limit and merges_mutations_memory_usage_to_ram_ratio settings that represent the soft memory limit for merges and mutations. If this limit is reached ClickHouse won't schedule new merge or mutation tasks. Also MergesMutationsMemoryTracking metric is introduced to allow observing current memory usage of background tasks. Resubmit #46089. Closes #48774. #48787 (Dmitry Novik).
Introduces memory tracking for ClickHouse background tasks, including merges and mutations, with new soft memory limit settings and a metric for monitoring memory usage.

Why it matters

To prevent excessive memory consumption by background merge and mutation tasks, enabling better resource control and system stability by stopping task scheduling when memory usage exceeds configured soft limits.

How to use it

Configure the new settings merges_mutations_memory_usage_soft_limit and merges_mutations_memory_usage_to_ram_ratio to set soft memory limits for merges and mutations. Monitor the MergesMutationsMemoryTracking metric to observe current memory usage of these background tasks.

Related resources

PreviousNext