v.21.11
Backward Incompatible Changes
4Bug Fixes
82- Fix case-insensitive search functions for UTF-8 strings in rare mismatch cases
- Fix reading from empty file on encrypted disk
- Fix disjunction chain transformation to IN in distributed queries with specific settings
- Allow Materialized Columns as Sharding Key in Distributed Tables
- Fix ORDER BY ... WITH FILL with TO and FROM when result set is empty
- Fix set index in AND/OR expressions for multiple operands
- Fix Crash in Projection with Hashing Function Materialization
- Fix ZooKeeper Path Handling in ReplicatedMergeTree to Resolve Ambiguity and Deprecate Invalid Formats
- Clean Temporary Directory on Local Backup Failure
- Fixed race condition in MergeTree affecting REPLACE/MOVE PARTITION operations
- Fix PREWHERE Behavior with Always True Conditions
- Limit Push Down Optimization Error Fix
- Add Missing Parentheses for isNotNull/isNull Rewrites in SQL Queries
- Fix deadlock on ALTER with scalar subquery to the same table
- Fixed segfault during REPLACE PARTITION on session expiration
- Fix for Incorrect Results in Queries with IN (Subquery) and Aggregate Projections
- Fix JOIN Query Column Alias Resolution Issue
- Fix deficiency in replaceRegexpAll function
- Fix Parsing of Preallocate Option in ComplexKeyHashedDictionary and ComplexKeySparseHashedDictionary
- Fix [I]LIKE Function, Closes #28661
- Fix crash with shortcircuit and lowcardinality in multiIf
- FlatDictionary and HashedDictionary Fix for Nullable Attributes Bytes Calculation
- Allow Numeric Starting Identifiers in Multiple Joins
- Fix MergeTree Reading with max_read_buffer_size = 0 to Prevent Exceptions and Data Loss
- Fix pread_fake_async/pread_threadpool with min_bytes_to_use_direct_io
- Fix INSERT SELECT for MATERIALIZED column from Nullable column
- Support Nullable Arguments in initializeAggregation Function
- Fix "Port is already connected" error for GLOBAL IN and WITH TOTALS queries in versions 21.9 and 21.10
- Fix Race Condition in MergeTree During MOVE PARTITION Operations
- Memory Database Issue Resolved and New Drop Setting Introduced
- Fix crash of sample caused by tuple()
- Try to Close Issue #29965
- Fix data-race issue in FileChecker and StorageLog/StorageStripeLog
- Fix data race in LogSink::writeMarks and LogSource in StorageLog
- Fix Resource Leak in Concurrent Query Limit of Merge Tree Tables
- Fix system tables recreation check for enum value changes
- MaterializedMySQL: Fix partial transaction processing on MySQL connection loss
- Avoid Timeout Exceeded Error Due to Kernel Bug Fixes
- Fix bad cast in ATTACH TABLE query to prevent uninitialized memory access
- Fix Concurrent Access Issues with LowCardinality in GROUP BY Operations
- Fix Group By Issue in Distributed Queries with Mixed Version Shards
- Fixed Server Restart Issue with materialized_postgresql_tables_list
- Condition Loss in Filter Predicate After Push-Down Optimization
- Fix JIT Expression Compilation and Short-Circuit Evaluation Issues
- Fix segfault in ALTER MODIFY query with incorrect table identifier in DEFAULT expression
- Fix nullptr dereference in GROUP BY WITH TOTALS HAVING
- Avoid Deadlocks in Join Table Engine Operations
- Fix bug in pathStartsWith due to std::mismatch usage error
- ODBC Bridge: Add Retries for Invalid Cursor State Error
- Fixed Table Name Parsing for Lazy Database Loading
- Fix Block Structure Mismatch for Subqueries with Pushed-Down HAVING Predicate
- Fix Logical Error in Greatest/Least Functions
- RocksDB Table Engine: Fix Race Condition During Multiple DB Opens
- Fix clean shutdown of misconfigured replicated access storage
- Remove Memory-Unsafe nth_value Window Function
- Fix Vertical Merges of Projection Parts and Related Issues
- Fix Hanging DDL Queries on Replicated Database During New Replica Addition
- Fix connection timeouts (send_timeout/receive_timeout)
- Fix Table Column Structure Exception in ReplicatedMergeTree with Case-Insensitive Default Expressions
- Send Database Doesn't Exist Error to Client Instead of Attempt to Read After EOF
- Fix segfault in LowCardinality(Nullable) column with Avro input format
- Do Not Allow Reuse of Previous Credentials for Inter-Server Secrets
- Handle any_join_distinct_right_table_keys in Dictionary Joins
- Fix "Not found column ... in block" Error with Alias Columns
- Fix thread usage in GLOBAL IN subquery after bug #19414
- Fix ORDER BY Optimizations with WITH FILL
- Fix SIGSEGV and ILLEGAL_COLUMN issues in higher-order array functions
- Fix waiting for mutation with mutations_sync=2
- Fix Multi-Column Queries to External Databases in ClickHouse
- Fix LowCardinality Bug in Short-Circuit Function Evaluation
- Fix Subcolumn Reading from Compact Parts
- Fixed Race Condition Between DROP PART and REPLACE/MOVE PARTITION in ClickHouse
- Fix short circuit evaluation in expressions compilation
- Fix rare ReplicatedMergeTree replica divergence issue after hard reboot
- Improved RabbitMQ Connection Usability Checks and Exception Handling
- Fix benign race condition in ReplicatedMergeTreeQueue
- Fix Crash in SELECT with Incomplete Aggregate Projection
- Fix coredump issue in distributed table creation with incorrect parameters
- Add Settings Aliases for system.processes Table
- Support S2 Geometry Library: Fix Argument Count for s2RectAdd and s2RectContains Functions
- Fix Invalid Constant Type Conversion for Nullable or LowCardinality Primary Keys
- Fix PREWHERE Handling for Non-Aggregated Columns in GROUP BY
Build/Testing/Packaging Improvements
15- Add FreeBSD Build Support for Aarch64 Machines
- Recursive Submodules Removed for ClickHouse
- ClickHouse Statically Built with Musl: Experimental Support Limited
- Enable Protobuf, Arrow, ORC, and Parquet for AArch64 and Darwin Builds
- Add Cross-Build Support for PowerPC and MySQL Interaction for AArch64 and PowerPC
- Leave Only Required Files in Cross-Compile Toolchains as Submodules
- Implemented Structure-Aware Fuzzing in ClickHouse's Select Statement Parser
- Turning on Experimental Constexpr Expressions Evaluator in Clang for Faster Template Compilation
- Add support for compiling with newer glibc without new symbols
- Reduce Debug Build Binary Size with Clang Optimization
- Images for CI Moved to Separate Dockerhub Repo
- Improve Clang-13 Build Support
- Add Printing of Raw Profile Events to ClickHouse Client
- Add time dependency for clickhouse-server unit in systemd and sysvinit
- Reload Stacktrace Cache on Symbol Reload
Experimental Features
2Improvements
78- Allow User to Change Log Levels Without Restart
- Multiple Enhancements to SQL User Defined Functions in ClickHouse
- Enable Global Per-Query Memory Profiler with 4MiB Step
- Added new columns for data and secondary indices in system tables
- Add table and database aliases to system.tables and system.databases
- Correctly resolve table interdependencies on server startup
- Avoid "Division by zero" Errors in Nullable Denominators for Functions Divide, IntDiv, and Modulo
- Allow Date Parsing in YYYYMMDD Format
- Web UI: Render bars in table cells
- User Can Create Dictionaries and Databases with Comments
- Introduce compiled_expression_cache_elements_size setting
- clickhouse-format now supports --query option
- Support ALTER TABLE for Memory Databases in ClickHouse
- Arrays of Serializable Types Supported by arrayStringConcat
- ClickHouse Accounts for Docker/Cgroups Limitations in Memory Calculation
- Fetched PostgreSQL Table Structure Improved
- Full Support for Positional Arguments in GROUP BY and ORDER BY
- Allow JSONExtractString to Extract Non-String Elements as Strings
- Added FINAL Clause Support in SELECT Queries for GraphiteMergeTree
- Minor Enhancements to Replica Cloning and Fetching Mechanism in Replication Queue
- Allow Symlinks in user_files Directory for File Table Function
- Fixed Date32 Comparison with Other Types
- Allow Removal of SAMPLE BY Expression in MergeTree Tables
- Keeper in ClickHouse Server Now Starts Asynchronously with Node Connection
- ClickHouse Client Introduces Native Multi-Line Editing Support
- polygon dictionaries enhanced for SELECT query method with key column support
- Add ClickHouse Logo to Play UI
- Better Exception Messages for Arrow-Supported Formats
- Fix data race in Buffer tables during flush and startup
- Fix lock-order inversion between DROP TABLE for DatabaseMemory and LiveView
- Fix lock-order inversion in periodic dictionary and config reload
- Update zoneinfo files to version 2021c
- Add configurable retries and delays for clickhouse-copier
- Add shutdown_wait_unfinished_queries Setting for Query Wait Time Management
- Add Memory Peak Usage Tracing to system.trace_log
- PostgreSQL Foreign Tables: Added Partitioned Table Prefix 'p' for Replica Identity Index Query
- Apply memory profiling parameters during mutate/merge operations
- Query Obfuscator Enhancements in ClickHouse
- Fixed clickhouse-format obfuscation for queries with embedded dictionaries
- Fix Nullable Processing in JSON Functions
- Increase listen_backlog by default to align with newer Linux kernel
- Reload Configurations on Server Changes
- Remove Restriction on Projection Name Start with tmp_
- Fixed error in mutations with nested subqueries and restrictions on replicated tables.
- Apply max_concurrent_queries config changes at runtime without restart
- Added use_skip_indexes Setting
- Add Support for FREEZing In-Memory Parts for Backups
- Pass Initial Query ID for ClickHouse Benchmark Queries
- Skip Indexes Enhancements: Support for Array and Map Data Types in tokenbf_v1 and ngrambf_v1
- Function has: Added Map data type support
- Add compress_logs setting for ClickHouse Keeper to enable ZSTD log compression
- Add external_table_strict_query setting to enforce full WHERE expression in foreign database queries
- Disable Projections with ARRAY JOIN to Prevent Alias Issues
- Support Additional Types in MsgPack Input/Output Format
- Allow LowCardinality Columns in ORC Format
- Fix for Incorrect Values in system.distributed_ddl_queue Selection
- Correct Handling of Unknown Methods in HTTP Connection
- Fix Data Loss and Deserialization Bugs in ClickHouse Keeper Converter
- Apply Settings from CREATE ... AS SELECT Queries (Fixes #28810)
- Respect Default Database Setting for ALTER TABLE ON CLUSTER Partition Operations
- gRPC Protocol: Enable Client-Side Control of Server-Side Compression
- Skip "no data" exception for thermal sensors in asynchronous metrics
- Fixed race condition causing Dictionary not found error
- Relax nested function for If-combinator check
- Fix uncaught exception during server termination
- Forbid Cleaning of Active Mutation/Merge tmp Directories
- Allow Optimization of Arithmetic Operations in Aggregate Functions with Alias
- Implement detach_not_byte_identical_parts setting for ReplicatedMergeTree
- Implement max_suspicious_broken_parts_bytes Setting for MergeTree
- Enable Expanding Macros in RabbitMQ Table Settings
- Restore Multi-Threaded Data Reading for Log Engine
- Fix NULL Column Handling in JSON Functions
- Allow Separate Size Settings for Mark/Uncompressed Cache and Columns in Skip Indices
- Allow Mixing JOIN with USING and Other JOIN Types
- Update aws-sdk Submodule for Throttling in Yandex Cloud S3
- Fix releasing query ID and session ID in gRPC query processing
- Fix Shutdown of AccessControlManager to Resolve Flaky Test
- Fix HDFS Assertion Failure and Update libhdfs3 Library
New Features
38- New Asynchronous INSERT Mode for Bulk Data Insertion in ClickHouse
- Add Interactive Mode for clickhouse-local and Merge with clickhouse-client
- Added Support for Executable User Defined Functions (UDFs)
- Predefined Connections to External Data Sources
- Added INFORMATION_SCHEMA Database with SCHEMATA, TABLES, VIEWS, and COLUMNS Views to System Database
- Support EXISTS (subquery) in ClickHouse
- Session Logging Implementation for Audit Trails
- Support for Multidimensional Distance Functions and Tuple Operations in ClickHouse
- Add Compression Support for INTO OUTFILE and FROM INFILE
- Add CORS Support for Serverless Requests in Grafana
- Queries with JOIN ON Now Support Disjunctions (OR)
- Added tokens and ngrams functions for text processing in ClickHouse
- Add Unicode normalization functions: normalizeUTF8NFC, normalizeUTF8NFD, normalizeUTF8NFKC, normalizeUTF8NFKD
- FileLog Table Engine for Streaming Application Log Files in ClickHouse
- Add and Refactor CapnProto Input/Output Format
- Allow Binary Literals in Query Syntax
- Added hashed_array dictionary type for memory efficiency in multi-attribute dictionaries
- Added JSONExtractKeys Function
- Add getOSKernelVersion function to return OS kernel version
- Added MD4 and SHA384 Functions
- Enabling HSTS for ClickHouse HTTP Server
- Huawei OBS Storage Support Implemented
- New functions mapContainsKeyLike and mapExtractKeyLike for regex operations in maps
- Implemented ALTER TABLE x MODIFY COMMENT Feature
- Adds Missing H3 Inspection Functions to ClickHouse
- Allow Non-Replicated ALTER TABLE FETCH and ATTACH in Replicated Databases
- Added CSV Null Representation Setting for Output Format
- Added zookeeperSessionUptime() Function to Retrieve ZooKeeper Session Uptime
- Implements h3ToGeoBoundary function by Ivan Veselov
- Add exponentialMovingAverage aggregate function as a window function
- Allow Subcolumns in DESCRIBE Query Results
- Executable and ExecutablePool now support send_chunk_header option
- Support for String Key Maps in TokenBF with Enhanced Data Skipping in Queries
- Send Profile Events from Server to Client with New Packet Type
- Bit Shift Operations for FixedString and String Data Types
- Support Dynamic Table Management in MaterializedPostgreSQL Replication
- Added accurateCastOrDefault function and resolved issue #21330
- Add Functions for User-Defined Default Values on String Parsing Failures
Performance Improvements
22- Background Merge Scheduling Improvements in ClickHouse
- Allow Asynchronous Reads for Remote Filesystems to Enhance Performance
- Fix Flickering Progress-Bar in ClickHouse for INTO OUTFILE Queries with Multi-Threading
- Reduce Redundant Compressed Data Reads in SELECT Queries for MergeTree Engines
- Remove Redundant Seek Calls in MergeTree Table Engines
- Make url Table Function for Parallel URL Processing
- Improve Aggregation Performance by Optimizing Order of Primary Key
- ClickHouse Implements DNS Caching for External S3 Communication
- Add Pushdown Support for IS NULL/IS NOT NULL in External Databases
- SELECT Queries on Dictionary Tables Now Support Multithreading
- Improve Filtering Performance for Decimal Columns
- Remove Branchy Code in Filter Operation for Improved Performance
- Improve Bytemask Generator for WHERE Operator with SSE/AVX Instructions
- Improve SUM Performance for Nullable Floating Point Numbers
- Speed Up Part Loading Process with Multiple Disks
- Reduce S3 Multipart Upload Part Size for Lower Memory Usage
- Speed up bitmapAnd function #28332
- Removed Inefficient Merge Mutation Notifications in StorageMergeTree
- Improve String Comparison Performance
- Primary Key Index and Partition Filter Functionality in Tuple
- Optimization of Multiple Quantile Aggregate Functions in Queries
- Min-Max Aggregation Optimization for Primary Key Expression