v.22.7Improvement

S3 Object Removal Enhancements and GCP Compatibility Improvements

Official CH changelog text
S3 single objects are now removed with RemoveObjectRequest. Implement compatibility with GCP which did not allow to use removeFileIfExists effectively breaking approximately half of remove functionality. Automatic detection for DeleteObjects S3 API, that is not supported by GCS. This will allow to use GCS without explicit support_batch_delete=0 in configuration. #37882 (Vladimir Chebotarev).
Improved compatibility with Google Cloud Storage (GCS) by switching to single object removal using RemoveObjectRequest instead of batch deletions, which are unsupported by GCS.

Why it matters

Previously, the removeFileIfExists method and batch deletions via DeleteObjects API broke about half of the remove functionality on GCS due to lack of support. This update enables automatic detection of the unsupported batch delete operation and uses single object delete requests instead, allowing seamless use of GCS without needing to disable batch deletes explicitly via support_batch_delete=0 in the configuration.

How to use it

No explicit user action is required. The system now automatically detects unsupported batch deletion on GCS and uses single object removal with RemoveObjectRequest. To ensure compatibility, users do not need to set support_batch_delete=0 when using GCS.

Related resources

PreviousNext