v.25.7Improvement
Support partition pruning
An improvement forDeltaLaketable engine: delta-kernel-rs hasExpressionVisitorAPI which is implemented in this PR and is applied to partition column expressions transform (it will replace an old deprecated within the delta-kernel-rs way, which was used before in our code). In the future thisExpressionVisitorwill also allow to implement statistics based pruning and some delta-lake proprietary features. Additionally the purpose of this change is to support partition pruning inDeltaLakeClustertable engine (the result of a parsed expression - ActionsDAG - will be serialized and sent from the initiator along with the data path, because this kind of information, which is needed for pruning, is only available as meta information on data files listing, which is done by initiator only, but it has to be applied to data on each reading server). #81136 (Kseniia Sumarokova).
Why it matters
This feature modernizes and improves partition pruning support within theDeltaLake and DeltaLakeCluster table engines, enabling more efficient query performance by leveraging expression transformation. It also paves the way for future enhancements such as statistics-based pruning and proprietary delta-lake features by utilizing the centralized parsing of expressions on the initiator node and serializing ActionsDAG for distributed pruning across reading servers.How to use it
Users benefit from this improvement automatically when using theDeltaLake or DeltaLakeCluster table engines with partitioned data. No explicit user action is required to enable the updated partition pruning mechanism as it happens internally by serializing and transmitting the expression's ActionsDAG from the initiator to reading servers.