v.22.2Improvement
Allow Export of Large Files to S3 with New Settings
Allow to export arbitrary large files tos3. Add two new settings:s3_upload_part_size_multiply_factorands3_upload_part_size_multiply_parts_count_threshold. Now each times3_upload_part_size_multiply_parts_count_thresholduploaded to S3 from a single querys3_min_upload_part_sizemultiplied bys3_upload_part_size_multiply_factor. Fixes #34244. #34422 (alesapin).
Why it matters
This feature addresses the limitation on the size of files that can be exported tos3 by allowing ClickHouse to increase the upload part size based on the number of parts already uploaded. This prevents excessive fragmentation and optimizes upload performance for large files.How to use it
Configure the new settingss3_upload_part_size_multiply_factor and s3_upload_part_size_multiply_parts_count_threshold in the server configuration. When the number of parts uploaded from a single query reaches s3_upload_part_size_multiply_parts_count_threshold, the part size (s3_min_upload_part_size) will be multiplied by s3_upload_part_size_multiply_factor for subsequent parts, allowing efficient handling of large file uploads to s3.