v.22.4Experimental Feature
Enhancements for Remote Filesystem Cache Management in ClickHouse
Allow to write remote FS cache on all write operations. Addsystem.remote_filesystem_cachetable. Adddrop remote filesystem cachequery. Add introspection for s3 metadata withsystem.remote_data_pathstable. Closes #34021. Add cache option for merges by adding moderead_from_filesystem_cache_if_exists_otherwise_bypass_cache(turned on by default for merges and can also be turned on by query setting with the same name). Rename cache related settings (remote_fs_enable_cache -> enable_filesystem_cache, etc). #35475 (Kseniia Sumarokova).
Why it matters
The feature addresses the need for improved caching of remote filesystem data by enabling cache writes on all write operations, which helps optimize read performance and resource usage. It simplifies cache management by providing introspection tables and commands to drop remote FS cache, enhancing monitoring and control over remote data caching.How to use it
Users can leverage the new remote filesystem cache functionality by using thesystem.remote_filesystem_cache table for cache status and management, and the system.remote_data_paths table for introspection of S3 metadata. The cache write on all write operations is enabled by default for merges via the mode read_from_filesystem_cache_if_exists_otherwise_bypass_cache, which can also be controlled per query using a query setting of the same name. To clear the cache, users can execute the DROP REMOTE FILESYSTEM CACHE query. Additionally, cache-related settings have been renamed for clarity, e.g., remote_fs_enable_cache is now enable_filesystem_cache.