v.24.7Improvement
StorageS3Queue Enhancements: Default Values, Exception Handling, and Commit Controls
StorageS3Queuerelated fixes and improvements. Deduce a default value ofs3queue_processing_threads_numaccording to the number of physical cpu cores on the server (instead of the previous default value as 1). Set default value ofs3queue_loading_retriesto 10. Fix possible vague "Uncaught exception" in exception column ofsystem.s3queue. Do not increment retry count onMEMORY_LIMIT_EXCEEDEDexception. Move files commit to a stage after insertion into table fully finished to avoid files being commited while not inserted. Add settingss3queue_max_processed_files_before_commit,s3queue_max_processed_rows_before_commit,s3queue_max_processed_bytes_before_commit,s3queue_max_processing_time_sec_before_commit, to better control commit and flush time. #65046 (Kseniia Sumarokova).
Why it matters
These enhancements address performance and reliability issues by automatically adjusting the number of processing threads based on physical CPU cores, increasing the default retry attempts, preventing retries on memory limit exceptions, and improving commit timing to avoid partial data commits. They also introduce new settings to give users finer control over commit and flush operations, leading to more efficient and stable S3 queue data processing.How to use it
Users can benefit from the new defaults automatically without changes, or customize behavior by settings3queue_processing_threads_num, s3queue_loading_retries, and the new commit control settings: s3queue_max_processed_files_before_commit, s3queue_max_processed_rows_before_commit, s3queue_max_processed_bytes_before_commit, and s3queue_max_processing_time_sec_before_commit to optimize processing according to their workload.