v.21.9
Backward Incompatible Changes
7- Do Not Output Trailing Zeros for Decimal Types in ClickHouse
- Do not apply parametric aggregate functions with different parameters for state finalization
- Under clickhouse-local, local addresses with a port are treated as remote
- Fix Bad Cast Issue with Column Aliases in Sophisticated Queries
- Scalar Subquery Now Returns Nullable Result to Prevent Type Compatibility Errors
- Introduce Here Document Syntax in Queries
- Indices Now Support Nullable Types with On-Disk Format Changes
Bug Fix
61- Fix Bugs Causing Replica Divergence
- Fix rare bug in DROP PART causing unexpected error
- Prevent Crashes from NULL Messages in Kafka
- Fix column filtering in subquery with union distinct
- Fix Type Casting Issue with LowCardinality Arrays in ClickHouse
- Fix PostgreSQL table function to resolve non-closing connections
- Fixed Unexpected Merged Part Intersection Error
- Fix Error with Aliased Column in Distributed Table
- Fix issue with resetting max_memory_usage to 0 after setting a non-zero value
- Fixed Time Value Underflow Issue in ClickHouse
- Fix Crash in Projection Materialization Due to Missing Columns
- fix metric BackgroundMessageBrokerSchedulePoolTask typo
- Fix Distributed Queries with Zero Shards and Aggregation
- Compatibility with /proc/meminfo Lacking KB Suffix
- Fix query result issue with row-level security, PREWHERE, and LowCardinality filter
- Fixed Validation of Partition ID for MergeTree Tables
- Fix MySQL Protocol for Parallel Formats (CSV/TSV)
- Fix "Cannot find column" error in queries with sampling
- Fix LowCardinality Query Error Messages and Formatting Issues
- Fix Limitations with Distributed Group By Settings in ClickHouse
- Fix mutation issue in non-replicated MergeTree partitions
- Lambda Functions Prioritize Arguments Over Aliases in Ambiguous Cases
- Fix Column Structure in Merge Join
- Fix incorrect information in system.detached_parts table
- Fix uninitialized memory in multiSearch functions with empty array
- Fix GRPCServer Synchronization Issue #27024
- Fixed Configuration Parsing for Cache Options in ClickHouse
- Fix Mutation Stack Issue Caused by DROP_RANGE Race Condition
- Now validates partition ID in ALTER TABLE queries
- Fix "Unknown column name" error with multiple JOINs
- Fix Handling of Custom TLDs in ClickHouse
- Fix Missing Columns Error When Using DEFAULT Expression
- Fix loading of dictionary keys in library-bridge for library dictionary source
- Aggregate function parameter loss and exception fixes in ClickHouse
- Add event_time_microseconds for REMOVE_PART in system.part_log
- Do Not Remove Data on ReplicatedMergeTree Table Shutdown to Prevent Metadata Inconsistency
- Fix Incorrect Behavior of SET ROLE Functionality
- Some fixes for parallel formatting
- Fix nullptr dereference in window functions, addressing issue #25276
- Fix clickhouse-client history file conversion for empty files during upgrade
- Fix Incorrect Function Names in GroupBitmap Functions
- Update chown cmd check in ClickHouse server Docker entrypoint to fix Kubernetes cluster pod restart issue
- Fix RabbitMQ shutdown crash when setup not started
- Fix CREATE DICTIONARY Query Issues with Quoted Names
- Fix Column Name Resolution After Alias Rewrite
- Fix msan crash and address issue #22517
- Fix infinite non-joined block stream in partial_merge_join
- Fix crash issue when logging in as dropped user
- Fix optimize_distributed_group_by_sharding_key for Multiple Columns
- Fixed rare bug in lost replica recovery to prevent divergence
- Fix zstd Decompression for Internal Buffer Escape Sequences
- Fix logical error on join with totals, closes #26017
- Remove Excessive Newline in thread_name Column of system.stack_trace Table
- Fix potential crash with multiple untuple expressions
- Don't throw exception in toString for Nullable Enum without zero value
- Fixed MySQL Protocol Packet Sequence ID in ClickHouse on Query Execution Exceptions
- Fix Incorrect Type Return in Subdomain Functions Affecting optimize_skip_unused_shards
- Fix Mismatched Header in Normal Projection with Prewhere
- Fix sharding_key issue with remote() function in ClickHouse
- Fixed "Not found column" and "Missing column" errors in MaterializeMySQL
- Fix optimize_skip_unused_shards_rewrite_in for non-UInt64 types
Build/Testing/Packaging Improvement
6- Stateful and Stateless Tests in Random Timezones with Timezone Respect in Protobuf and GraphiteMergeTree
- clickhouse-test now supports SQL tests with Jinja2 templates
- Add support for building with clang-13
- Add CMake Options for CPU Instruction Set Configuration
- Fix linking of auxiliary programs with dynamic libraries
- Update RocksDB to 2021-07-16 master
Experimental Feature
2Improvement
67- Mark Window Functions as Ready for General Use
- Improve Non-Whole-Minute Timezone Offset Compatibility
- Allow Multiple Reads from Regular Files in ClickHouse Local
- Remove Duplicate Index Analysis and Validity Checks during Projection Analysis
- Enable HTTP Request Body to Accept Query Parameters
- Disallow arrayJoin on partition expressions
- Log Client IP on Authentication Failure
- Use Bytes for Binary Data in GRPC Protocol
- Send Error Response for HTTP Requests to TCP Port Without HTTP Port Set
- Add _CAST Function with Modified Nullability Preservation Rules
- Add log_formatted_queries Setting for Enhanced Query Analysis
- Add Settings to Limit Regexp Size in Hyperscan Functions
- Memory Limits Now Account for Bitmap Aggregate Function Memory Consumption
- Add 10 Seconds Cache for S3 Proxy Resolver
- Split Global Mutex for Individual Regexp Construction to Improve Thread Performance
- Support for PostgreSQL Database Engine Schema
- Track Memory Usage in ClickHouse Client
- Try Logging query_kind in system.query_log on Query Failure
- Added replica_is_active column to system.replicas table
- Allow Query Settings via Server URI in Web UI
- Add MaxPushedDDLEntryID Metric for Zookeeper Integration
- Improved znode existence and empty string judgment in clickhouse-keeper
- Merge JOIN Handles Empty Right Set Correctly
- Shard-Level Constants for Functions in Distributed Tables
- Updated extractAllGroupsHorizontal to allow setting upper limit on matches per row
- Expose RocksDB Statistics via system.rocksdb Table in ClickHouse
- Less Verbose RocksDB Logs Integration in ClickHouse
- Changing Default Roles Affects New Sessions Only
- Watchdog Disabled in Docker by Default - Ctrl+C Handling Fix
- SET PROFILE Now Applies Constraints for Passed Profiles
- Improve KILL QUERY Request Handling
- mapPopulatesSeries Function Enhances Map Type Support
- Fix excessive connect attempts with skip_unavailable_shards
- Avoid ClickHouse Benchmark Hang on Connection Failures
- Allow More Threads for Kafka Engine
- Add round-robin support for clickhouse-benchmark
- Executable Dictionaries with DDL Query Support in ClickHouse
- Set Client Query Compatibility for MySQL and PostgreSQL
- Apply LIMIT on Shards with Distributed Push Down Limit for Queries
- Updated protobuf to version 3.17.3
- Enable use_hedged_requests setting to reduce tail latencies in large clusters
- Improve handling of non-existing hosts in user allowed host list
- Add Distributed Directory Monitor Settings via CREATE TABLE
- Save Server Address in Web UI History for Different Origins
- Add Events to Profile Calls for sleep Functions
- Allow Reuse of Shard Connections Across Clusters
- Control Execution Period for Clearing Temporary Directories
- Add `function_range_max_elements_in_block` Setting for Data Volume Safety Threshold
- Check Hash Function at Table Creation for MergeTree Settings
- Added output_format_avro_string_column_pattern setting for Avro string columns
- Add Column Size Information for Log and TinyLog Tables in system.columns
- Don't throw exception when querying system.detached_parts with missing detached directory in custom disk configuration
- Check for Non-Deterministic Functions in Keys
- Convert timestamp and timestamptz to DateTime64 in PostgreSQL table engine
- Apply Aggressive IN Index Analysis for Improved Projections
- Remove GLOBAL Keyword for IN with Scalar Functions
- Add Error IDs to Exception Messages
- Fix Progress Bar Output in clickhouse-local
- Add merge_selecting_sleep_ms setting
- Remove Complicated Linux AIO in Favor of Simple Synchronous IO with O_DIRECT
- Flush Distributed Table on REPLACE TABLE Query and Handle Insertion Failures
- Add Views Column and Enhance Query Logging in system.query_log
- Do Asynchronous Connection Draining for Distributed Queries with New Server Settings
- Support for Multiple Includes in Configuration
- Fix multiple block insertion issue in distributed tables
- Support LowCardinality and FixedString for Map Type
- Enable Local Disk Config Reloading
New Feature
30- Implementation of Short Circuit Function Evaluation with New Configuration Setting
- Add Support for INTERSECT, EXCEPT, ANY, ALL Operators
- Add Virtual File System Encryption Support Using AES-CTR
- Added NLP Functions for Tokenization and Synonyms Search
- Added S2 Geometry Library Integration
- Add SQLite Engine and Table Function to ClickHouse
- Added Support for Custom Query in Multiple Database Sources
- Add Shared Storage for User Data and Settings Profiles via ZooKeeper
- Add Auto-Selecting Compression for INTO OUTFILE
- Add INSERT ... FROM INFILE feature to match SELECT ... INTO OUTFILE
- Added complex_key_range_hashed dictionary, closes #22029
- Support Expressions in JOIN ON Section
- Client Receives Server Warning Information with New system.warnings Table
- Allow Constant Expressions in Aggregate Function Parameters
- Add tupleToNameValuePairs function for named tuples
- Add support for bzip2 compression in import/export
- Added bitmapSubsetOffsetLimit Function for Bitmap Subsets
- Add default_database column to system.users
- Supported cluster macros in table functions 'cluster' and 'clusterAllReplicas'
- Add currentRoles, enabledRoles, and defaultRoles functions
- New functions currentProfiles(), enabledProfiles(), and defaultProfiles() added
- Add functions to return current query's initial query ID, closes #23682
- Add REPLACE GRANT Feature
- EXPLAIN query gains ESTIMATE mode for MergeTree tables
- Added system.zookeeper_log Table to Log ZooKeeper Client Actions
- Zero-copy Replication for ReplicatedMergeTree over HDFS Storage
- Allow Nested Type as Array of Structs in Arrow, ORC, and Parquet Input Formats
- Add Date32 Datatype and toDate32 Function to ClickHouse
- Allow Setting Default Database for Users
- Add SSL Support and Connection String Options to MongoDB Engine
Performance Improvement
12- Improve fast query performance by reducing clock_gettime system calls when max_execution_time is 0
- Specialize date-time comparison for improved performance
- Share File Descriptors for Concurrent Reads to Reduce Opened Files on Linux
- Improve Latency of Short Queries on Wide Tables
- Don't Build Sets for Indices in Query Analysis
- Vectorize SUM of Nullable Integer Types with Native Representation
- Compile Enum Type Expressions in ClickHouse
- Compile aggregate functions groupBitOr, groupBitAnd, groupBitXor
- Improved memory usage with block size prediction for empty DEFAULT columns
- Reduce Memory Usage with ORDER BY Primary Key
- Enable distributed_push_down_limit by default
- Make toTimeZone Monotonic for Constant TimeZones to Support Partition Purging