v.23.11
Backward Incompatible Changes
5- Default ClickHouse Configuration Enables Access Management and Named Collection Control for Default User
- Multiple Improvements for RESPECT NULLS/IGNORE NULLS in Window Functions
- Remove optimization optimize_move_functions_out_of_any
- Formatters in parseDateTime and formatDateTime Now Support Hours/Months Without Leading Zeros
- Changes to Aggregate Functions and Bug Fixes in ClickHouse
Bug Fix
70- Fix Experimental Analyzer for Subquery Insertion Processing
- Fix Bug in str_to_map Function
- Keeper reconfig: add leadership timeout
- Fix header in grace hash join and filter pushdown
- Select from System Tables Using Table Function
- RFC: Fix "Cannot find column X" for Distributed Queries with LIMIT BY
- Fix 'Cannot read from file:' Error in Background Client Execution
- Fix clickhouse-local exit on invalid send_logs_level μ€μ
- Bug fix: Explain AST with parameterized view
- Fix crash during table loading on startup
- Fix ClickHouse-sourced Dictionaries with Explicit Query
- Fix segfault in signal handler for Keeper
- Fix UNION query result in view() function
- Fix DateTime64 Casting Inconsistency
- Fix race condition in memory allocation failure
- Fix Backup Restoration with flatten_nested and data_type_default_nullable
- Fix crash when adding Object(JSON) column
- Fix Crash in filterPushDown Function
- Fix Backup Restore with Materialized View and Dropped Source Table
- Fix Segfault During Kerberos Initialization
- Fix buffer overflow in T64 #56434
- Fix nullable primary key in final 2
- Fix ON CLUSTER Queries without Database on Initial Node
- Fix startup failure caused by TTL dependency
- Fix ALTER COMMENT Queries ON CLUSTER#56491
- Fix ALTER COLUMN with ALIAS in ClickHouse
- Fix Empty Named Collections in ClickHouse
- Fix projection analysis issues in ClickHouse
- Fix Aliases Handling in Query Cache
- Fix Nullable Enum to Nullable String Conversion
- More Reliable Log Handling in Keeper
- Fix Configuration Merge for Nodes with Substitution Attributes
- Fix duplicate usage of table function input()
- Fix RabbitMQ OpenSSL Dynamic Loading Issue
- Fix GCD codec crash with zero data presence
- Fix mutex lock failure in clickhouse-local during insert into function
- Fix Date Text Parsing in Optimistic Path
- Fix Crash in FPC Codec
- DatabaseReplicated: Fix DDL Query Timeout After Replica Recovery
- Fix Nullable Columns Reporting in MySQL Binary Protocol
- Support Iceberg Metadata Files for Metastore Tables
- Fix TSAN report under transform #56817
- Fix formatting in SET query and SETTINGS
- Fix startup failure due to table dependency in joinGet
- Fix Nested Column Flattening in ADD COLUMN
- Fix CSV End of Line Handling for CR Characters
- Fix tryBase64Decode for Invalid Input
- Fix Deep Nested Column Generation in CapnProto/Protobuf Schemas
- Prevent incompatible ALTER of projection columns
- Fix SQLite File Path Validation #56984
- S3Queue: Fix Metadata Reference Increment
- S3Queue Minor Fix by Kseniia Sumarokova
- Fix DatabaseFileSystem File Path Validation
- Fix fuzzBits with ARRAY JOIN
- Fix Nullptr Dereference in Partial Merge Join with Joined Subquery
- Fix race condition in RemoteSource
- Implement bitHammingDistance for Big Integers
- S3-style links bug fix by Yarik Briukhovetskyi
- Fix JSON_QUERY Function for Multiple Numeric Paths
- Fix buffer overflow in Gorilla codec
- Close Interserver Connection on Exception Before Authentication
- Fix segfault after ALTER UPDATE with Nullable MATERIALIZED column
- Fix JOIN Plan Optimization for Partially Materialized Normal Projection
- Ignore Comments in Column Description Comparison
- Fix ReadonlyReplica Metric for All Cases
- Background Merges Use Temporary Data Storage in Cache
- Keeper fix for changelog and snapshots
- Ignore Finished ON CLUSTER Tasks if Hostname Changed
- MergeTree Mutations Reuse Source Part Index Granularity
- FS cache: Limit for background download
Build/Testing/Packaging Improvement
15- Add SQLLogic Test by Han Fei
- Make clickhouse-local and clickhouse-client Accessible via Short Names for Usability
- Optimized build size by removing unused code from external libraries
- Add automatic check for large translation units
- Lower Single-Binary Distribution Size
- Information on Translation Unit and Binary File Sizes Sent to CI Database
- Fix continuous rebuild of non-essential Apache Arrow files in ClickHouse
- Avoid Recompiling Translation Units Based on Autogenerated Version Source
- Linker Invocation Data Tracing Added to ClickHouse Cloud CI Database
- Use DWARF 5 Debug Symbols for ClickHouse Binary
- Add SANITIZE_COVERAGE Build Option for Code Coverage Tracking in ClickHouse
- Some stack frames may be unresolved; raw address can be useful
- Add option to disable libssh
- Enable temporary data caching in S3 CI tests
- Set max memory usage for clickhouse-client to 1G in CI
Improvement
58- Add Implicit Sign Column Constraint for CollapsingMergeTree Tables
- Enable Adding New Disk to Storage Configuration Without Restart
- Support Index Creation and TTL Modification in a Single ALTER Query
- Add fuzzJSON Function for Perturbed JSON Variations
- Engine Merge Filters Records by Row Policies
- Add Settings for Execution Time and Timeout Behavior in Distributed Queries
- Add Setting to Disable Tunneling for HTTPS Requests over HTTP Proxy in ClickHouse
- Set background fetch and schedule pool sizes for optimal production performance
- CSV Parsing Error in ClickHouse Due to Improper Line Endings
- Update Arrow Library to Release 13.0.0 for New Encodings
- Improve ON CLUSTER Query Performance by Eliminating Heavy System Calls
- Fixed Memory Accounting for Threads in Queries
- Add LARGE_LIST support in Apache Arrow formats
- Allow Manual Compaction of EmbeddedRocksDB via OPTIMIZE Query
- Add BlockBasedTableOptions Support for EmbeddedRocksDB Tables
- SHOW COLUMNS Displays MySQL Data Type Names by Default in MySQL Protocol Connections
- Fixed error related to parts mismatch after server restart following TRUNCATE or DROP PARTITION
- Fixed Query String Handling and Added OrNull Variants in formatQuery Functions
- Allow backup of materialized view with dropped inner table
- Deduplicate Requests to ZooKeeper from Concurrent Queries in system.replicas
- Fix MySQL Compatible Query for External Database Translation
- Add Backup and Restore Support for Tables with KeeperMap Engine
- 404 Response Issue for CompleteMultipartUpload Requires Reevaluation
- Enable HTTP OPTIONS Method by Default for Simplified ClickHouse Requests from Browsers
- Fix DNS Configuration and Enhance HTTP Keep-Alive Timeout
- Load Base Backups Lazily with Logging and Profiling Updates
- Setting query_cache_store_results_of_queries_with_nondeterministic_functions Marked Obsolete and Replaced with query_cache_nondeterministic_function_handling
- Rewrite JOIN ON equality with is null check in Analyzer only
- Function concat now supports arbitrary argument types, aligning with MySQL behavior
- Allow Cache Configuration from Named Collections in Config or SQL
- PostgreSQL: Moderate Outdated Table Removal with Failed Connections
- Issue with Slow Connection to PG Causing Query Cancellation Due to Incorrect URL
- Keeper: Disable Compressed Logs by Default
- Add config setting wait_dictionaries_load_at_startup
- Potential Vulnerability in ClickHouse: Unauthenticated Exploitation via Ignored Packets
- Fetching Full Parts from Remote Replicas in ClickHouse
- Fix PostgreSQL replication error in MaterializedPostgreSQL
- Implement Asynchronous Execution for ALTER TABLE Partition Moves in ClickHouse
- Filter by Engine When Scanning System Tables to Improve Efficiency
- Show total_bytes and total_rows in system tables for RocksDB storage
- Allow Basic Commands in ALTER for TEMPORARY Tables
- LZ4 Compression Enhancements for Buffer Management
- Add metrics for queued jobs in IO thread pool
- Add PostgreSQL Table Engine Setting to Config File
- Function concat now accepts a single argument for consistency with MySQL
- Signs all x-amz-* headers as required by AWS S3 docs
- Function fromDaysSinceYearZero Now Supports Unsigned and Signed Integer Types
- Add system.s3queue_log to Default Config
- Change Default Setting for wait_dictionaries_load_at_startup to True
- Check Dictionary Source Type During Creation with Lazy Load Enabled
- Plan-level optimizations can now be individually enabled/disabled in ClickHouse
- Server Exit Codes Updated to Reflect Specific Exceptions
- Do not demangle and symbolize stack frames from functional C++ header
- HTTP Server Page /dashboard Supports Multi-Line Charts
- max_memory_usage_in_client command line option supports string suffix values
- Bump Intel QPL to v1.3.1 and Fix BOF Handling Bug
- Increase Default replicated_deduplication_window in MergeTree Settings to 1k
- Stop Excessive Use of INCONSISTENT_METADATA_FOR_BACKUP
New Feature
24- Added async_load_databases Setting for Faster Database Loading
- Add blob_storage_log table for auditing data in S3 and object storages
- Use Statistics to Improve Prewhere Condition Ordering
- Added Compression Support in Keeper's Protocol for ClickHouse
- Introduce storage_metadata_write_full_object_key Feature for Enhanced Metadata Flexibility
- Add Protection for Named Collections' Fields to Prevent Unauthorized Access
- Add hostname column to system log tables for replication and distribution support
- Add CHECK ALL TABLES query
- Added fromDaysSinceYearZero Function Similar to MySQL's FROM_DAYS
- Add External Python Tool for Viewing Backups
- Implement preferred_optimize_projection_name setting in ClickHouse
- Add 4-Letter Command for Leadership Resignation
- Added arrayRandomSample SQL Function for Simplified Element Sampling
- Added Float16 Support for .npy Files
- Added information_schema.statistics for Tableau Online Compatibility
- Add system.symbols table for binary introspection
- Configurable Dashboards with New Query Table System
- Introduce fileCluster Table Function for Shared Filesystems
- Add _size virtual column for file size in bytes to storage engines
- Expose Error Counts by Code Since Last Server Restart
- ClickHouse Keeper Reports Availability Zone Configuration
- Make ALTER materialized_view MODIFY QUERY Non-Experimental and Deprecate allow_experimental_alter_materialized_view_structure Setting
- Setting join_algorithm Respects Specified Order
- Add Support for Well-Known Protobuf Types in Protobuf Format
Performance Improvement
23- Adaptive Timeouts for S3 Interaction
- Increase max_concurrent_queries from 100 to 1000 and enable concurrency control by default
- Support Parallel Evaluation of Window Functions
- Numbers table engine enhances condition analysis for data subset generation
- Improved IN filtering performance for Merge table engine
- Improvement in Filesystem Cache Handling for Large Reads
- Add option to disable S3 checksums with s3_disable_checksum setting
- Improved Synchronous Reads from Remote Tables in Page Cache
- Optimization for map and arrayElement performance with 30% speedup
- Optimization of Multi-Stage Filtering with AVX-512 on Intel Xeon Platinum 8380
- Limit Threading in Query Profiler for ClickHouse
- Decrease Virtual Function Calls in Window Functions
- Allow Recursive Tuple Field Pruning in ORC for Faster Scanning
- Trivial Count Optimization for Npy Data Format with Result Caching
- Memory Optimization for Aggregation Queries with Multiple Streams
- Improve Query Performance for Highly Concurrent Users in ClickHouse
- Trivial Improvement on Array Join Reusing Intermediate Results
- Stack Unwinding Performance Improvement
- Default Read Pool Utilized for External Storage with max_streams = 1
- Keeper: Optimize Memory Usage During Startup by Delaying Log Preprocessing
- Improved Glob Matching Performance for File and HDFS Storages
- Posting Lists in Experimental Full Text Indexes Now Compressed by 10-30%
- Parallelise BackupEntriesCollector in Backups