v.24.7Improvement

Change in Materialized Views Deduplication Implementation

Official CH changelog text
Change how deduplication for Materialized Views works. Fixed a lot of cases like: - on destination table: data is split for 2 or more blocks and that blocks is considered as duplicate when that block is inserted in parallel. - on MV destination table: the equal blocks are deduplicated, that happens when MV often produces equal data as a result for different input data due to performing aggregation. - on MV destination table: the equal blocks which comes from different MV are deduplicated. #61601 (Sema Checherinda).
Improved deduplication mechanism for Materialized Views and destination tables to handle parallel and aggregated insert scenarios more effectively.

Why it matters

This feature addresses issues where data blocks inserted in parallel or resulting from aggregations in Materialized Views were incorrectly considered duplicates or not properly deduplicated. It ensures that equal blocks from different sources or parallel inserts are correctly deduplicated, preventing data duplication and improving data consistency.

How to use it

The improved deduplication is applied automatically to destination tables and Materialized View destination tables without additional user configuration. Users benefit from enhanced deduplication behavior during parallel inserts and aggregation processing in Materialized Views.

Related resources

PreviousNext