v.25.8Improvement

Iceberg's current implementation of positional delete files

Official CH changelog text
Iceberg's current implementation of positional delete files keeps all data in RAM. This can be quite expensive if the positional delete files are large, which is often the case. My implementation keeps only the last row-group of Parquet delete files in RAM, which is significantly cheaper. #85329 (Konstantin Vedernikov).
Optimized handling of Iceberg positional delete files by keeping only the last row-group of Parquet delete files in RAM instead of the entire file.

Why it matters

This feature addresses the high memory consumption caused by loading large Iceberg positional delete files entirely into RAM. By limiting in-memory data to the last row-group only, it significantly reduces memory usage and improves performance when processing large delete files.

How to use it

The optimization is applied internally during the processing of Iceberg positional delete files. Users do not need to enable any settings explicitly; the improved memory handling is integrated within Iceberg table operations involving delete files.

Related resources

PreviousNext