v.24.8New Feature

Added deduplicate_merge_projection_mode setting for MergeTree engine projections

Official CH changelog text
Added a new MergeTree setting deduplicate_merge_projection_mode to control the projections during merges (for specific engines) and OPTIMIZE DEDUPLICATE query. Supported options: throw (throw an exception in case the projection is not fully supported for *MergeTree engine), drop (remove projection during merge if it can't be merged itself consistently) and rebuild (rebuild projection from scratch, which is a heavy operation). #66672 (jsc0218).
Introduced a new MergeTree setting deduplicate_merge_projection_mode to control how projections are handled during merges and the OPTIMIZE DEDUPLICATE query.

Why it matters

This feature allows users to manage projections on MergeTree tables more effectively during merges, addressing issues when projections cannot be merged consistently. It provides flexible handling options to either throw an exception, drop problematic projections, or rebuild them, thus improving data consistency and control during optimization.

How to use it

Set the deduplicate_merge_projection_mode in your MergeTree table settings with one of the following values: throw, drop, or rebuild. Use the OPTIMIZE DEDUPLICATE query to deduplicate data and apply the chosen projection merge behavior during optimization.

Related resources

PreviousNext