v.22.2Experimental Feature

Add Memory Overcommit Feature to MemoryTracker with Guaranteed Settings

Official CH changelog text
Add memory overcommit to MemoryTracker. Added guaranteed settings for memory limits which represent soft memory limits. In case when hard memory limit is reached, MemoryTracker tries to cancel the most overcommited query. New setting memory_usage_overcommit_max_wait_microseconds specifies how long queries may wait another query to stop. Closes #28375. #31182 (Dmitry Novik).
Added memory overcommit support to MemoryTracker with soft memory limits via guaranteed settings and a mechanism to cancel the most overcommitted query when the hard memory limit is reached.

Why it matters

This feature solves the problem of rigid and immediate query cancellation upon reaching memory limits by introducing a soft limit that allows temporary memory overcommitment. It improves query scheduling and stability by enabling queries to wait for others to release memory, thus optimizing resource usage and reducing unnecessary query failures.

How to use it

Users can configure the new guaranteed memory limit in their MemoryTracker settings to establish soft memory limits and set the memory_usage_overcommit_max_wait_microseconds parameter to control how long queries wait for memory to be freed before cancellation happens.

Related resources

PreviousNext