v.21.1
Backward Incompatible Changes
11- Setting input_format_null_as_default Enabled by Default
- Check Profile Settings Constraints in Config to Prevent Server Failures
- Restrict ALTER MODIFY SETTING from Changing Storage Settings Affecting Data Parts
- Set insert_quorum_parallel to 1 by Default for Convenience
- Remove sumburConsistentHash Function, Closes #18120, PR #18656
- Removed Non-Functional Aggregate Functions `timeSeriesGroupSum` and `timeSeriesGroupRateSum`
- Prohibit toUnixTimestamp(Date()) Method Return λ³κ²½
- Allow Extended Integer Types in avg and avgWeighted Functions
- Expression toUUID(N) Replacement with toUUID('00000000-0000-0000-0000-000000000000')
- SSL Certificates with Incorrect Key Usage Rejected
- Removal of implicit substitutions file references from default config in ClickHouse update
Bug Fixes
110- Fix rare connection acceptance issue on server
- Fix Index Analysis for Binary Functions with Constant Arguments
- Fix Wrong Index Analysis for Different Types in Comparison
- Disable AIO Write During Merges to Prevent Rare Data Corruption
- Restrict Merges from Wide to Compact Parts to Prevent Issues
- Fix Incomplete Query Results in MergeTree During Read Backoff
- Fix use after free bug in rocksdb library
- Fix Infinite Reading from ORC Files in ClickHouse
- Fix bug in merge tree data writer affecting mark sizes
- Fix startup bug related to LowCardinality(Nullable(...)) in ClickHouse
- Restrict MODIFY TTL Queries for Legacy MergeTree Tables
- Ensure groupUniqArray Correctly Returns Enum Type
- Fix Error with Ignore Function and LowCardinality Argument
- Fix LowCardinality Column Insertion for TinyLog Engine
- Join Materializes Const Columns Contradicting Code Expectations
- Disable optimize_move_functions_out_of_any Due to Inaccurate Optimization
- Fix QueryPipeline Exception from Merging Expression Steps
- Fixed rare shutdown deadlock issue
- Fix ALTER TABLE DROP PART Behavior to Preserve Deduplication Blocks
- Attach Partition Resets Mutation
- Fix nullptr dereference in bitmapOrCardinality function
- Fix Shutdown Hang in clickhouse-local
- Fix Incorrect Rewriting of Queries for External Databases with IN Expression
- Fix If Combinator with Unary Function and Nullable Types
- Fix Issue with Asynchronous Distributed INSERTs and Network Compression Settings
- Fixed EOF Error in CAST Function for Nullable Decimal Conversion
- Fix minor logging issue
- Fix for Empty Parts Removal in ReplicatedMergeTree Tables
- Fix date overflow bug and set strict limit to 2106-02-07
- Add FixedString Support for MySQL Replication in ClickHouse
- Fix Pipeline Stuck Error with ORDER BY After RIGHT or FULL Join
- Fix ALTER Queries Hanging After Mutation Kill
- Properly Support 12AM in parseDateTimeBestEffort Function
- Fixed Nullable String Parsing Error in toType Functions
- Fix unexpected behavior of SHOW TABLES
- Fix SimpleState Combinator Argument Type Incompatibility
- Fix race condition in concurrent Set/Join table usage and system.tables selects
- Fix issues with system.settings_profile_elements table filling
- Fix Crashes in Aggregate Functions with Distinct in Two-Level Aggregation
- Fixed issues with clickhouse-odbc-bridge regarding dual stack connectivity and malformed ODBC dictionary queries
- Access Control Update: Count Query Execution for Users with Column Access
- Access Control Update: SELECT JOIN Requires SELECT Privilege on Joined Tables
- Fix Enum and Int Key Comparison Issue
- Replication from MySQL: Fixes for Unique Key Conversion Issues
- Fix Inconsistency for Queries with WITH FILL and WITH TIES
- Fix Default Value Insertion on Parsing Error in Last Column
- Fix Unknown Setting Profile Error in Settings Profile Configuration
- Fix TTL Removal Error in MODIFY COLUMN Query
- Fixed std::out_of_range in S3 URL Parsing
- Fix DateTime64 and Date Comparison Issues in ClickHouse
- Replication from MySQL: Fixes for Prefix Index Support in MaterializeMySQL
- Server log rotation issue with large numeric values fixed
- Trivial Query Optimization Error with ARRAY JOIN
- Fix segfault in topK aggregate function
- WAL: Conditional Part Restoration Based on in_memory_parts_enable_wal Setting
- Exception message for max table size drop corrected
- Fixed Segfault in Distributed Table Insertion Due to Insufficient Space
- Fixed ClickHouse MySQL Connection Resumption Issue
- Windows: Fix for RENAME query error in ClickHouse on WSL
- Fix race condition in ON CLUSTER query affecting cluster replication determination
- Fix empty system.stack_trace table in daemon mode
- Exception fmt::v7::format_error Logged in Background for MergeTree Tables
- Fix multiline query comment handling in clickhouse-client interactive mode
- Fix query hang issue when mutation is killed on different replica
- Fix Memory Accounting Issue in ClickHouse with Underestimated Mark Cache Size
- Fix ORDER BY with optimize_redundant_functions_in_order_by setting
- Fix Duplicate Records After DISTINCT Due to Incorrect Optimization
- Fixed High CPU Usage in MergeTree Background Tasks
- Fix Crash When Reading from JOIN Table with LowCardinality Types
- Replication from MySQL: Fix for Header Mismatch in SHOW Statement
- Fix Nondeterministic Functions with Predicate Optimizer
- Fix Unexpected Packet Data Error for Distributed Queries with LIMIT
- Fix Set Index Invalidation for Const Columns in Subquery
- clickhouse-copier fix for non-partitioned tables #15235 #17248 by Qi Chen
- Fixed Mutations for S3-Stored Parts (Experimental Feature)
- Bug Fix for fuzzBits Function Related to Issue #16980
- Fix optimize_distributed_group_by_sharding_key for OFFSET-only queries
- Fix JOIN queries between Merge and Distributed tables
- Fix Order By Optimization with Monotonic Functions
- Fix Type Comparison for DateTime64 with Different Scales
- Fix optimization of group by with optimize_aggregators_of_group_by_keys and joins
- Minor Fix in SHOW ACCESS Query
- Fix optimize_trivial_count_query behavior with partition predicate
- Return Affected Rows for INSERT Queries via MySQL Wire Protocol in ClickHouse
- Fix Inconsistent Behavior of select_sequential_consistency in Count Queries and System Tables
- Throw Error for Non-Existing Column in REPLACE Transformer
- Throw Exception for Non-Equi-Join in RIGHT/FULL JOIN
- BloomFilter Index Crash Fix (#19757, #19884)
- Fix crash in union distinct subquery predicate pushdown
- Fix UInt8 Greater Than 127 Filtering
- Fix crash and infinite loop issues in arrayEnumerateUniq function
- Fixed Stack Overflow in Accurate Comparison of Arithmetic and String Types
- Fix Crash with Nested Column Names in WHERE or PREWHERE Clauses
- Fix segmentation fault in bitmapAndnot function
- Big Integers Experimental Feature May Cause Segfaults
- Fix neighbor function output for LowCardinality argument
- Fix use-after-free in CompressedWriteBuffer after disconnect
- DROP/DETACH TABLE query hang issue fixed
- Query CREATE DICTIONARY ID Expression Fix
- Fix SIGSEGV in merge tree concurrent read settings
- Buffer Overflow Fix in addMonth Function
- Uninitialized Memory Read Fix in Encrypt/Decrypt Functions Due to Empty IV String
- Fix Buffer Overflow in Uber H3 Library
- Fix LOGICAL_ERROR in system.parts _state column due to incorrect order
- Fixed segfault in aggregation with mismatched Materialized View and target table structures
- Fix error converting column now64() due to constant value discrepancies
- Fix Concurrent Query Hang in ReplicatedMergeTree Table
- Fixed "There is no checkpoint" error when inserting data via HTTP interface
- Disable Constant Folding for Uncalculated Subquery Results
- Mutation Hang Issue After Partition Operations Fixed
Build/Testing/Packaging Improvements
30- Add Integrity Check for ClickHouse Binary to Detect Corruption
- Change OpenSSL to BoringSSL to Resolve Multiple Issues
- Simplify Sys/V init script for compatibility with Ubuntu 12.04 and older
- Multiple Improvements in ./clickhouse install Script
- ClickHouse Introduces In-Memory ZooKeeper Emulation
- Fix Dead List Watches Removal in TestKeeperStorage
- Add SYSTEM SUSPEND Command for Fault Injection in Failover Tests
- Generate Build ID for ClickHouse when linked with lld
- Fix shellcheck errors in style check
- Update Timezone Information to 2020e
- Fix codespell warnings and update style checks in Docker image
- Automated Check for Conflict Markers in Documentation
- Enable Thread Fuzzer for Flaky Check in Stateless Tests
- Do not use non thread-safe function strerror
- Update anchore scan-action workflow to main branch
- clickhouse-test implements timed DROP/CREATE databases
- Enable Experimental Pytest Support for Stateless Tests
- Integrate Fresh Docker Daemon Version in Tests
- Send Build and System Info to Sentry if Enabled
- Uninitialized Variable Issue in ClickHouse Copier Code
- Fix MSan Report from Issue #17309
- Fix IPv6 Issue in Arrow Flight Library
- Add Library to Trap and Terminate Process for libc Functions
- Provide Diagnostics for Stack Overflow in Server Logs
- Now Supporting Parallel Execution of Stateless Functional Tests
- Fix Snappy Decompression Corruption in librdkafka for GCC10 Builds
- Server Termination Logging for OOM Killer
- PODArray: Fix UBSan report by avoiding memcpy with nullptr, 0 arguments
- Minor Improvement in Path Concatenation for DDLWorker
- Allow Reloading Symbols from Debug File and Fix Build-ID Issue
Experimental Features
3Improvements
113- Prevent Dependency-Related Dropping of Tables or Dictionaries
- Allow Versioning of Aggregate Function States for Backward Compatibility
- Support PostgreSQL ALTER MODIFY COLUMN Syntax
- Added update_field support for RangeHashedDictionary and ComplexKeyRangeHashedDictionary
- murmurHash3_128 and sipHash128 now accept arbitrary number of arguments
- Support HDFS storage default expression and optimize column-oriented fetching
- Improve Operation Name of OpenTelemetry Span
- Use Content-Type application/x-ndjson for JSONEachRow Output Format
- Improve Skipping Unknown Fields in Template/CustomSeparated Formats
- clickhouse-keeper prevents startups and config changes with duplicate IDs or endpoints
- Set Content-Type in HTTP Packets from URL Engine
- Return JSON Content-Type for JSONEachRow Format When Enabled
- Allow parsing + before Float32/Float64 values
- Allow User Configured HDFS Replication Parameter for DiskHDFS and StorageHDFS
- Added ClickHouse exception fields to OpenTelemetry span log
- Improve OpenTelemetry Span Log Duration on Query Exceptions
- Fix LowCardinality of Int256 Creation Issue
- Recreate system.*_log tables for differing engine/partition_by
- MaterializedMySQL: Fix issue with table named 'table'
- ClickHouse Dictionary Source: Support Predefined Connections
- Allow Predefined Connections for Kafka and RabbitMQ Engines
- Always Re-render Prompt During History Navigation in ClickHouse Client for Improved Usability
- Add Key Bindings for History Navigation
- Improve timeout checks for max_execution_time in ClickHouse
- Better Exception Message for users.xml Loading Error Due to Bad Password Hash
- Use shard and replica names from Replicated database arguments in ReplicatedMergeTree macros
- Better analysis for min/max/count projection with partition key support
- Add --pager support for clickhouse-local
- Fix Editor Waiting Issue During Interactive Query Edition
- Throw Exception for Garbage After Field in JSONCompactStrings Format
- Default timeout settings reduced from 30 minutes to 3 minutes
- MaterializedMySQL Supports CREATE TABLE ... LIKE DDL Queries
- Return Artificial Create Query for Show Create Table on System Tables
- Progress Display Enhanced for numbers_mt Table Function
- Initial User Roles Utilized for Row Policies
- Show warning in system.warnings for outdated setting changes
- Improved Backoff for Background Cleanup Tasks in MergeTree Settings
- Incremental Profile Events Counters for ClickHouse Replicas
- Enable Default Multiline Editing in ClickHouse Client
- Function Name Normalization for ALTER Queries to Prevent Metadata Mismatches
- Support IF EXISTS Modifier for RENAME DATABASE/TABLE/DICTIONARY Query
- Cancel Vertical Merges on Partition Drop
- Local session events in ClickHouse dictionaries no longer send to session log, resolving deadlock issue and fixing flaky test
- Less Locking in ALTER Command
- Fix logging and verbose option in clickhouse-local interactive mode
- New Commands in clickhouse-client: \l, \d, \c Added Like MySQL and PostgreSQL
- Enhancement: Execute Queries Before Entering Interactive Mode in Clickhouse
- Fix potential error with local parts and ZooKeeper during DROP operation
- Avro Format Support Added for Kafka in ClickHouse
- Allow Multiple PostgreSQL Schemas for MaterializedPostgreSQL Database
- Replaced ClickHouse-Keeper Default Ports from 44444 to 9234
- Implement transform function with Decimal arguments
- Fix Debug and Release Server Errors from Bad HDFS URL
- Fix Assert in HDFS Table Function and Add Test
- Avoid Deadlock in INSERT SELECT Operations with TinyLog and Log Engines
- Support SHOW CREATE VIEW syntax in ClickHouse like MySQL
- All Decimal and Float Queries Allowed with Result Types Float32 or Float64
- Improved Web UI: Added History, Sharing Support, and Request Indicators
- clickhouse-server fails to send close request to ZooKeeper
- Avoid Server Termination Due to Low Memory Limits
- Fix non-deterministic results in windowFunnel function for identical event timestamps
- Docker: Set Fixed UID/GID for ClickHouse User and Group in Server Images
- Asynchronous INSERTs to Distributed Tables: New fsync Settings Added
- SYSTEM KILL command implemented in Docker
- Expand Macros in zk Path for FETCH PARTITION Execution
- Apply ALTER TABLE ON CLUSTER to All Replicas
- Allow EXCEPT in Column Transformer to Accept Regex Strings
- Fix SimpleAggregateFunction in SummingMergeTree for Correct Aggregate Behavior
- Fixed NaN Assertion Error in Allocator for Function Bar
- Fix usability issue with missing newline after exception message in tools
- Add Support for Modifying Key Column Types in ClickHouse
- Implement untuple field access in ClickHouse
- Allow CSV Parsing of Serialized Array Fields as Strings
- Make Adaptive Granularity Calculation for Merge Tree Parts More Effective
- ClickHouse Install Now Compatible with Mac
- Better hints for SHOW query syntax
- Array Aggregation Functions Support for Int128, Int256, and UInt256
- Add disk to Set and Join storage settings
- Access Control Update: merge() Function Requires SELECT Privilege on Source Tables
- Temporary Tables Visibility in ClickHouse System Tables Updated
- Fix clickhouse-client rendering issue with terminal window resizing
- Decrease Log Verbosity for Client Connection Drops
- Forcibly Removing Bad Metadata Files for DiskS3 Experimental Feature
- Access Control Changes in ClickHouse: Modifications to Introspection and DDL Granting Permissions
- Usability Improvement: Column Name Hints
- Add Diagnostic Info for Circular Merge Table Reads
- Allow Override of Timeout Value in ClickHouse Docker Image
- Check System Log Tables' Engine Definition Grammar to Prevent Configuration Errors
- Removed Exception Handling for RabbitMQ Table Initialization Without Connection
- Do Not Ignore Server Memory Limits During Buffer Flush
- Switch to Patched RocksDB Version to Fix Use-After-Free Error
- Added Offset to Exception Message for Parallel Parsing
- Don't throw "Too many parts" error during INSERT query
- Allow Query Parameters in UPDATE Statement of ALTER Query
- Query Obfuscator to Avoid SQL Keyword Identifiers
- Export Current Max DDL Entry Executed by DDLWorker via Server Metric
- Export Asynchronous Metrics of Current Server Threads
- Include Dynamic Columns for Wildcard Queries with Configurable Settings
- Allow TTL Specification for System Log Table Entry Removal
- Now Distinct Interface Types for MySQL and PostgreSQL Queries in Query Log
- Fix SETTINGS Clause Parsing in INSERT ... SELECT Query
- Correct memory accounting in RadixSort
- Add EOF Check in receiveHello to Prevent Exception
- Avoid Stack Overflow in Bigint Conversion
- set indices now support GLOBAL IN functionality
- Add Limit for HTTP Redirects in S3 Storage Requests
- When using -OrNull with other combinators, prioritize -OrNull placement
- Support HTTP Proxy and HTTPS S3 Endpoint Configuration
- Added S3 Client Authentication with AssumeRole and Environment Configuration
- Add OpenTelemetry Spans and Zipkin Export Example
- Cache Dictionaries: Eliminate Callbacks and Locks in Key Management
- Fix Experimental Feature: fsync Functions Not Working
- Allow Atomic engine for nested MaterializeMySQL database
New Features
40- Implement gRPC Protocol in ClickHouse
- Allow Multiple Zookeeper Clusters
- Implemented REPLACE TABLE and CREATE OR REPLACE TABLE Queries
- Implement UNION DISTINCT as Default and Add union_default_mode Setting
- Added accurateCastOrNull function and type conversions in x IN (subquery) expressions
- IP Dictionary Supports IPv4 and IPv6 Types
- IP Dictionary Enhances Key Fetching Support
- Add Zstandard Compression Support for Data Import and Export
- Added Statistical Aggregate Functions and Refactored Rank Correlation
- Add countMatches and countMatchesCaseInsensitive functions
- Implement Substring Count Functions in ClickHouse
- Add Database Information and Fields to System Query Log
- Add optimize_on_insert Setting to Transform INSERTed Data
- Kerberos Authentication for HDFS
- Support SHOW SETTINGS Statement and Clause Enhancements in ClickHouse
- Function position now supports POSITION(needle IN haystack) syntax for SQL compatibility
- New Storage Setting for MergeTree Tables: max_partitions_to_read and User Setting force_max_partition_limit
- Add query_id column to system.part_log for inserted parts
- Allow CREATE TABLE AS SELECT with Column Specification
- Added arrayMin, arrayMax, arrayAvg aggregation functions
- Implemented ATTACH TABLE query to create new table and attach data from directory
- Add Mutation Support for StorageMemory
- Support EXISTS DATABASE Syntax in ClickHouse
- Support isIPv4String and isIPv6String Functions Similar to MySQL
- Add Setting for Insertion into Multi-Sharded Distributed Table Without Key
- Add min_compress_block_size and max_compress_block_size to MergeTreeSettings
- Add 64-bit Roaring Bitmap Support
- Extended OPTIMIZE DEDUPLICATE Syntax for Column-Specific Duplicate Checks
- Added functions for converting between Proleptic Gregorian calendar and Modified Julian Day number
- Add Custom TLD List Functionality
- Add PROXYv1 Protocol Support for IP Address Quotas in ClickHouse
- ClickHouse Client Adds EDITOR Support for Command Editing
- Add encodeXMLComponent function for XML character escaping
- Introduce DETACH TABLE/VIEW ... PERMANENTLY Syntax in ClickHouse
- Add Asynchronous Metrics for MergeTree Tables
- Add Limit and Offset Settings for Out-of-SQL Pagination in ClickHouse
- Provide SimpleState Combinator for AggregatingMergeTree in ClickHouse
- Added queries-file parameter for clickhouse-client and clickhouse-local
- Added query parameter for clickhouse-benchmark
- EXPLAIN AST Now Supports Non-SELECT Queries
Performance Improvements
36- Reduce Memory Usage for Remote Data Formats with New Seek Controls
- Add JOIN ON Constant Condition Optimizations
- Support Parallel Formatting for All Text Formats Except Specific Cases
- Speed up count on nullable columns
- Speed Up avg and sumCount Aggregate Functions
- Improve JSON and XML Output Performance
- Improve Data Sync Performance to Block Device
- Fix query performance in LiveView tables
- Speed up query parsing
- Allow Splitting GraphiteMergeTree Rollup Rules for Metrics
- Remove Excessive DESC TABLE Requests for remote() Identifier Usage
- Optimize tupleElement function for subcolumns with enabled optimize_functions_to_subcolumns setting
- Optimize mapContains Function for Subcolumn Key with Settings Enabled
- Add Merge Tree Settings for Concurrent Read on Remote Filesystem
- Skipping Partition Mutations in StorageMergeTree
- New IP Dictionary Implementation Enhances Performance and Reduces Memory Usage
- Parallel Formatting for Data Export
- LDAP Integration: Added verification_cooldown Parameter for Bind Attempt Caching
- Add --no-system-table option for clickhouse-local to improve startup time
- Replace PODArray with PODArrayWithStackMemory in AggregateFunctionWindowFunnelData for Performance Improvement
- Don't Send Empty Blocks to Shards on Synchronous INSERT into Distributed Table
- Optimized StorageMemory Read Implementation
- Using Dragonbox Algorithm for Improved Float to String Conversion Performance
- Speed up IPv6CIDRToRange implementation
- Add remerge_sort_lowered_memory_bytes_ratio setting to manage remerge memory usage
- Improve AggregatingMergeTree Performance with SimpleAggregateFunction in PK
- Now Devirtualized -If Combinator and Properly Vectorized Count
- Fix Performance Issues in Reading from Merge Tables with Many MergeTree Tables
- Improved performance of repeat function
- Slightly Improved Float Parsing Performance
- Add Option to Skip Merged Partitions for OPTIMIZE TABLE FINAL
- Integrate fast_float for parsing floating point numbers in ClickHouse
- Fix max_distributed_connections affecting prefer_localhost_replica and max_threads
- Adaptive Upload Method for S3 with New Setting for Single Part Size
- Support for Async Tasks in PipelineExecutor
- Allow optimize_move_to_prewhere with compact parts for unknown column sizes