v.23.8New Feature

Allow Variable Number of Columns in TSV/CustomSeparated/JSON Input Formats

Official CH changelog text
Allow variable number of columns in TSV/CustomSeparated/JSONCompactEachRow, make schema inference work with variable number of columns. Add settings input_format_tsv_allow_variable_number_of_columns, input_format_custom_allow_variable_number_of_columns, input_format_json_compact_allow_variable_number_of_columns. #52692 (Kruglov Pavel).
Enable support for a variable number of columns in TSV, CustomSeparated, and JSONCompactEachRow input formats, including schema inference that adapts to varying column counts.

Why it matters

This feature addresses the limitation where input formats required a fixed number of columns, causing failures or errors when encountering rows with differing column counts. By allowing a variable number of columns, it improves flexibility and robustness when importing semi-structured or inconsistent data, reducing manual preprocessing and errors.

How to use it

To enable this feature, set the corresponding settings to true:

SET input_format_tsv_allow_variable_number_of_columns = 1;
-- or
SET input_format_custom_allow_variable_number_of_columns = 1;
-- or
SET input_format_json_compact_allow_variable_number_of_columns = 1;


These settings allow the respective input formats to process rows with a variable number of columns dynamically during ingestion.

Related resources

PreviousNext