v.24.12Improvement

Azure's Iceberg Metadata Violations: Mislabeling v1 as v2 and Compatibility Issues

Official CH changelog text
Azure violated the Iceberg specification, mistakenly labeling Iceberg v1 as Iceberg v2. The problem is described here. Azure Iceberg Writer creates Iceberg metadata files (as well as manifest files) that violate specs. Now we attempt to read v1 Iceberg format metadata with the v2 reader (cause they write it in a this way), and added error when they didn't create corresponding fields in a manifest file. #72277 (Daniil Ivanik).
Fix reading Iceberg v1 metadata files mislabeled as v2 by Azure Iceberg Writer, ensuring correct handling and error reporting for manifests violating the specification.

Why it matters

Azure's Iceberg Writer incorrectly labels Iceberg v1 metadata files as v2 and creates manifest files that do not conform to the Iceberg specification. This causes failures when ClickHouse attempts to read these files with a v2 reader. This feature resolves this issue by properly detecting such mislabeling and adding explicit error handling for missing manifest fields, improving compatibility and user experience.

How to use it

No special configuration is required. ClickHouse now automatically detects Iceberg metadata version mismatches caused by Azure's incorrect labeling and will provide clear errors if manifest files are incomplete or invalid.

Related resources

PreviousNext