v.21.3Improvement

Add Throttle for INSERT in Distributed Engine Based on Pending Bytes

Official CH changelog text
Add ability to throttle INSERT into Distributed based on amount of pending bytes for async send (bytes_to_delay_insert/max_delay_to_insert and bytes_to_throw_insert settings for Distributed engine has been added). #19673 (Azat Khuzhin).
Added capability to throttle INSERT operations into Distributed tables based on the amount of pending bytes waiting for asynchronous send, using new settings to control delay and rejection thresholds.

Why it matters

This feature addresses the issue of uncontrolled accumulation of data pending to be sent asynchronously to remote shards in Distributed tables. It helps prevent overloads and resource exhaustion by allowing users to limit the buffering of INSERT operations, improving stability and resource management.

How to use it

Use the new settings in the Distributed engine configuration: bytes_to_delay_insert and max_delay_to_insert to specify thresholds at which INSERT operations are delayed, and bytes_to_throw_insert to define when INSERT requests should be rejected (thrown). These settings enable fine control over throttling behavior for asynchronous sends.

Related resources

PreviousNext