v.23.7Experimental Feature
Writing Parquet Files Now 10x Faster with Multi-threading
Writing parquet files is 10x faster, it's multi-threaded now. Almost the same speed as reading. #49367 (Michael Kolupaev). This is controlled by the setting output_format_parquet_use_custom_encoder which is disabled by default, because the feature is non-ideal.Why it matters
This feature addresses the slow write performance of Parquet files by introducing a multi-threaded encoding process. It greatly reduces the time required to export data to Parquet format, benefiting users who rely on efficient data serialization and export workflows.How to use it
To enable the enhanced Parquet write speed, users must set the settingoutput_format_parquet_use_custom_encoder to true. This setting is disabled by default due to the feature being experimental and potentially non-ideal under certain conditions.