v.23.5
Bug Fix
83- ActionsDAG: Correcting Optimization Issues
- Correct Handling of Concurrent Snapshots in Keeper
- MergeTreeMarksLoader Stores DataPart Instead of DataPartStorage
- Sequence State Fix #48603 by Ilya Golshtein
- Back/Restore Concurrency Check on Previous Fails
- Fix ReadonlyReplica Metric Not Increasing with Non-Existent ZK Path
- Fix uncaught exception termination issue in ClickHouse
- Fix key not found error for multiple StorageJoin queries
- Fix incorrect query results with nullable primary key
- Fix reinterpretAs*() for Big Endian Support
- Lock zero copy parts more atomically in experimental zero-copy replication
- Fix race condition in Outdated parts loading
- Fix null key values and incorrect group rollup results
- Fix load_factor calculation for hashed dictionaries with shards
- Disallow Configuring Compression CODECs for Alias Columns
- Fix bug in removal of existing part directory
- Properly fix GCS when HMAC is used
- Fix fuzz bug in remote() subquery set handling
- Invert shutdown_wait_unfinished_queries by Konstantin Bogdanov
- Experimental Zero-Copy Replication Bug Fix
- Fix Postgres Database Setting by Mal Curtis
- Correct handling of s3Cluster arguments in ClickHouse
- Fix bug in TraceCollector destructor
- Fix AsynchronousReadIndirectBufferFromRemoteFS Issue with Short Seeks
- Fix Dictionaries Loading Order
- Forbid Data Type Change of Object('json') Column
- Fix logical error in stress test comparison
- Fix bug in DISTINCT #49628
- Fix DISTINCT behavior with zero values in non-sorted columns
- Fix one-off error in big integers identified by UBSan
- Fix Sparse Column Reading After Restart
- Fix assert in SpanHolder::finish() with fibers
- Fix short circuit functions and mutations with sparse arguments
- Fix Appending Files in Incremental Backups
- Fix error during lightweight delete mutation on Object column table
- Fix msan issue in randomStringUTF8 with uneven number
- Fix kolmogorovSmirnovTest Aggregate Function
- Fix settings aliases in native protocol
- Fix arrayMap for Array of Tuples with Single Argument
- Fix per-query IO/BACKUPs Throttling Settings
- Fix NULL Handling in Profile Definition
- Fix Bug in Projections with Aggregate Functions Null for Empty Setting
- Fix Pending Batch Processing for Distributed Async INSERT After Restart
- Fix Assertion in CacheMetadata Cleanup
- fix is_prefix in OptimizeRegularExpression #49919
- Fix WriteBuffer From S3 Metrics
- Fix IPv6 Encoding in Protobuf
- Fix Logical Error in Nullable Parsing for Text Formats
- Add output_format_parquet_compliant_nested_types for enhanced Parquet compatibility
- Fix logical error in stress test "Not enough space to add"
- Avoid Deadlock in Attach Thread of ReplicatedMergeTree
- Fix assert in SpanHolder::finish() with fibers attempt 2
- Add Proper Escaping for DDL OpenTelemetry Context Serialization
- Fix Reporting of Broken Projection Parts #50052
- JIT Compilation NaN Fix by Maksim Kita
- Fix crash in Replicated database without arguments
- Fix crash with multiIf and nullable arguments
- Fix invalid index analysis for date keys
- do not allow modifying order by without order by columns
- Fix Null Constant Argument in Binary Operator Index Analysis
- clickhouse-client: prevent simultaneous use of --query and --queries-file
- Fix UB for INTO OUTFILE Extensions and WATCH EVENTS
- Fix Handling of Trailing Spaces in CustomSeparatedIgnoreSpaces Format
- Fix Iceberg Metadata Parsing
- Fix Nested Distributed SELECT in WITH Clause
- Fix msan issue in keyed siphash
- Fix Bugs in Poco Sockets for True Non-Blocking Mode
- Fix checksum calculation for backup entries
- Comparison Functions NaN Fix by Maksim Kita
- JIT Aggregation Nullable Key Fix by Maksim Kita
- Fix crash in clickhouse-local with empty Arrow or Parquet output
- Fix crash in Pool::Entry::disconnect()
- Improved Fetch with Extended Directory Lock
- Fix bitShift functions with constant arguments
- Fix Keeper Deadlock on Exception During Request Preprocessing
- Fix Hashing of Const Integer Values
- Fix merge tree settings for data skipping indexes
- Limit In-Flight Tasks for Loading Outdated Parts
- Keeper fix: apply uncommitted state after snapshot install
- Fix Incorrect Constant Folding
- Fix logical error in stress test
- Fix Null to LowCardinality(Nullable) Conversion in Values Table Function
- Revert invalid RegExpTreeDictionary optimization #50642
Build/Testing/Packaging Improvement
17- New Features in keeper-bench: Customizable Requests and Enhanced Configuration Options
- Io_uring Not Supported on macOS for Local Tests
- Support Named Fault Injection for Testing
- Allow ClickHouse to Run in Environments Without prctl Syscall
- Fixed Build Conflict Between contrib/isa-l and isa-l in QPL
- Utilities built on explicit request, reducing link times in development builds
- Pull idxd-config build description into separate CMake file
- Add CI Check with Enabled Analyzer in Master
- Switch to LLVM/clang 16
- Allow Building ClickHouse with Clang-17
- ClickHouse Integration Simplified for CMake Projects
- Fix QEMU Logging Issue After #47151
- ClickHouse Support for Linux RISC-V 6.1.22 Implemented
- Bump internal protobuf to v3.18 to fix CVE-2022-1941
- Bump libxml2 to v2.10.4 to fix bogus CVEs
- Bump c-ares to v1.19.1 addressing bogus CVEs
- Fix bogus CVE-2022-2469 in libgsasl
Experimental Feature
5- DEFLATE_QPL Codec Lowers Minimum SIMD Version to SSE 4.2 and Updates ClickHouse CMakefile for QPL Alignment
- Add JOIN support for pure parallel replicas
- More Parallelism in Outdated Parts Removal with Zero-Copy Replication
- Parallel Replicas: Error Fixes and New Configuration Options
- When Parallel Replicas are Enabled, Unavailable Servers are Skipped by Default
New Feature
30- Added AzureBlobStorage Storage Engine and Table Function
- Added ClickHouse Keeper CLI Client
- Add urlCluster Table Function and Refactor Cluster Functions to Reduce Duplication
- Query Cache Now Supports Production Workloads and Totals Modifier
- Geographical Data Types Production-Ready
- Add Schema Inference to PostgreSQL, MySQL, MeiliSearch, and SQLite Table Engines
- Password Type in CREATE USER Queries Automatically Set by default_password_type Configuration
- Add bcrypt password authentication type, closes #34599
- New INTO OUTFILE 'file.txt' APPEND Keyword Introduced
- Added system.zookeeper_connection table for Keeper connection information
- Add generateRandomStructure Function for Random Table Generation
- Allow CASE without ELSE and enhance transform function in ClickHouse
- Added Server-Side Encryption with KMS Keys for S3 Tables and Disks
- Add Memory Tracking for Background Tasks in ClickHouse
- Function dotProduct for array functionality improvement
- Support SHOW INDEX for MySQL Compatibility
- Add Support for Virtual Columns and Improve Error Messages in URL Table Function
- Adding grants field in users.xml for user permissions
- Support Full/Right Join with Grace Hash Join Algorithm
- WITH FILL Modifier for Sorting Prefix in ClickHouse
- Clickhouse-client now supports multiquery mode without a specified query
- Add Handshake Timeout for Hello Packet from Replica
- Added "space" Function for Repeating Spaces
- Added input_format_csv_trim_whitespaces option
- Allow dictGetAll for regexp dictionaries to return multiple matches as arrays
- Added toLastDayOfWeek Function for Rounding Dates to Nearest Weekend
- Ability to Ignore Skip Index with ignore_data_skipping_indices
- Add system.user_processes table and SHOW USER PROCESSES query for user-level memory info and ProfileEvents
- Add Server Settings for Displaying Secrets and User Privilege Control
- Allow ROW POLICY Setup for All Database Tables
Improvement
68- BACKUP Command Does Not Decrypt Data from Encrypted Disks
- Added Support for Temporary Tables in ATTACH and REPLACE PARTITION Queries
- Added async_insert Setting for MergeTree Tables
- Add Size Suffix Support in Quota Creation Parameters
- Extend first_value and last_value to Accept NULL
- Add aliases str_to_map and mapFromString to extractKeyValuePairs
- Add CGroup v2 Support for Asynchronous Memory Metrics
- Cluster Table Functions to Skip Unavailable Shards
- Allow Empty Columns in CSV Header
- Add Google Cloud Storage S3 Compatible Table Function gcs
- Add Strict Parts Size Support for S3 Compatibility with CloudFlare R2
- Added new columns for Replicated database replicas in system.clusters and enhanced SYSTEM DROP DATABASE REPLICA query
- Add zookeeper_name Column to system.replicas for Metadata Storage Indication
- IN Operator Supports Comparison of Date and Date32
- Support for Erasure Codes in HDFS
- Implement SYSTEM DROP REPLICA from Auxiliary ZooKeeper Clusters
- Add Array Data Type to MongoDB
- Support for Storing Interval Data Types in Tables
- Allow ntile window function without explicit frame definition
- Added Settings for Delaying or Throwing ALTER Queries in ClickHouse
- Catch Exception in create_directories for Filesystem Cache
- Copies embedded examples to the example field in system.functions
- Enable MongoDB Connection Options for Dictionary Source
- Added alias for asymp method in kolmogorovSmirnovTest and improved documentation
- Aggregation Functions Updated for Signed Integers in ClickHouse
- Split Function Documentation into Fine-Granular Fields
- Use Shared Threads for Loading Outdated Data in ClickHouse
- Don't Overestimate Processed Data Size for LowCardinality Columns with Shared Dictionaries
- Parquet Writer Optimizes Row Group Size with OUTFILE Usage
- Allow Quoted Restricted Keywords as Aliases in ClickHouse
- Data Parts Loading and Deletion Jobs Moved to Shared Server-Wide Pools
- Allow password in Play UI URL with history replacement
- Allow Reading Zero-Size Objects from Remote Filesystems
- Attach MemoryTracker to total_memory_tracker After ThreadGroup Detachment
- Fix Parameterized Views for Multiple Query Parameter Usage
- Release Memory for Last ProfileEvents Snapshot in Query Context
- Function "makeDate" Now Supports MySQL-Compatible Overload
- Support dictionary table function for RegExpTreeDictionary
- Added Weighted Fair IO Scheduling and Dynamic Resource Management in ClickHouse
- Add Compose Request After Multipart Upload to GCS
- Enhance extractKeyValuePairs Function to Accept Key-Value Delimiter in Values
- Add initial_query_id field to processors_profile_log
- System log tables support custom sorting keys
- New 'partitions' Field Added to system.query_log for Calculation Participation Tracking
- Added enable_the_endpoint_id_with_zookeeper_name_prefix setting to ReplicatedMergeTree to prevent Duplicate interserver IO endpoint errors
- Add Query Parameters to clickhouse-local
- Allow Default Loading of Dictionaries and Functions from YAML
- Kafka Table Engine Supports Alias Columns
- Add Memory Limit Setting for Key-Value Pair Extraction
- Add support for single-element tuples in IN operator
- bitHammingDistance function now supports String and FixedString data types
- Fix OS X Client Timeout Resetting Errors
- Add Support for Big Integers in bitCount Function for Hamming Distance in AI Applications
- Fingerprints Replace Key IDs for Simplified Encrypted Disk Configuration
- Add UUID Data Type to PostgreSQL
- Function toUnixTimestamp Updated to Accept Date and Date32 Arguments
- Charge Server Memory for Dictionaries
- Server to Support SQL_* Settings as No-Ops for MySQL Compatibility
- Preserve initial_query_id for ON CLUSTER queries for introspection
- Preserve Backward Incompatibility with Aliases for Renamed Settings
- Support FQDN for Cluster Node Registration and Multi-Compute Group Invisible Setting
- Fix PostgreSQL Data Retrieval Ignoring LIMIT Clause
- Add Profile Events for Queries with Subqueries
- Adding Roles Field to users.xml for Configurable Permissions
- Report CGroupCpuCfsPeriod and CGroupCpuCfsQuota in AsynchronousMetrics with cgroup v2 memory limits compliance during startup
- Add SIGQUIT signal handler to match SIGINT behavior
- JSON Parsing Failure Handling for Large Object Sizes
- Support Variable-Size Decimals, Closes #49130
Performance Improvement
24- Compress Marks and Primary Key Enabled by Default in Version 22.9
- New Setting for Concurrently Loaded Parts in Multipart Uploads
- When reading from multiple files, reduce parallel parsing threads per file. Resolves issue #42192.
- Use Aggregate Projection for Fewer Granules in ClickHouse Queries
- Do not store blocks in ANY hash join if nothing is inserted
- Fix JIT Compilation for Aggregate Combinator -If and Functions
- Optimizing Remote Table Reads with Smaller Tasks and 1MB Buffers
- Introduced New Settings for Memory Management and Vertical Merges in ClickHouse
- Default Read Buffer Size Improved with New Settings for Local and Remote Filesystems
- Improve memory efficiency and performance of SPARSE_HASHED and HASHED dictionaries
- Optimize query_log tables with LowCardinality for improved performance
- Improved Local Parquet File Performance with Parallel Reading
- Improve RIGHT/FULL JOIN performance by up to 2x in specific scenarios
- Improve BLAKE3 Performance by 11% with Rust LTO
- Optimize structure of system.opentelemetry_span_log with LowCardinality usage
- Reserve hash table size in grace_hash join
- Parallel Merge of uniqExactIf States, Closes #49885
- Keeper Enhancement: Add CheckNotExists Request for Improved Replicated Table Performance
- Keeper Performance Enhancements: Improved Request Serialization and Caching
- Reduced ZooKeeper Requests for Merging Parts in ClickHouse
- Rework Locking in FS Cache
- Disable Pure Parallel Replicas When Trivial Count Optimization is Possible
- Optimize Head Requests in Iceberg Schema Inference
- Setting enable_memory_bound_merging_of_aggregation_results Enabled by Default
upgrade-notes
8- Compress Marks and Primary Key by Default: Upgrade Notes for Version 22.9 and Newer
- Make local object storage compatible with S3 and introduce configuration for independent storage, resolving appending issues.
- In-memory Data Parts Experimental Feature Removed
- Changed Default Settings to Improve ClickHouse Row Reading Performance
- Make Projections Production-Ready with New Settings
- Mark joinGet and dictGet as Non-Deterministic for Mutations
- Revert change preventing nullable returns in groupArray functions due to binary compatibility issues
- Setting enable_memory_bound_merging_of_aggregation_results to false during update from versions prior to 22.12