v.23.5Improvement

Data Parts Loading and Deletion Jobs Moved to Shared Server-Wide Pools

Official CH changelog text
Data parts loading and deletion jobs were moved to shared server-wide pools instead of per-table pools. Pools sizes are controlled via settings max_active_parts_loading_thread_pool_size, max_outdated_parts_loading_thread_pool_size and max_parts_cleaning_thread_pool_size in top-level config. Table-level settings max_part_loading_threads and max_part_removal_threads became obsolete. #49474 (Nikita Mikhaylov).
Data parts loading and deletion tasks have been centralized into shared server-wide thread pools instead of being managed by individual per-table pools.

Why it matters

This change improves resource management by consolidating thread pools for parts loading and deletion across all tables, simplifying configuration and potentially enhancing performance and scalability. It eliminates the need for per-table thread pool settings, reducing complexity and avoiding inefficient resource allocation.

How to use it

Configure the sizes of the shared thread pools using the top-level settings max_active_parts_loading_thread_pool_size, max_outdated_parts_loading_thread_pool_size, and max_parts_cleaning_thread_pool_size in the main ClickHouse configuration file. Remove or ignore the obsolete per-table settings max_part_loading_threads and max_part_removal_threads.

Related resources

PreviousNext