v.23.7New Feature
Added New Database Engines: Overlay, Filesystem, S3, and HDFS
AddedOverlaydatabase engine to combine multiple databases into one. AddedFilesystemdatabase engine to represent a directory in the filesystem as a set of implicitly available tables with auto-detected formats and structures. A newS3database engine allows to read-only interact with s3 storage by representing a prefix as a set of tables. A newHDFSdatabase engine allows to interact with HDFS storage in the same way. #48821 (alekseygolub).
Why it matters
These new engines solve the problem of accessing and combining data across diverse storage backends and formats seamlessly.Overlay allows combining multiple databases into a single unified view. Filesystem enables automatic table creation by interpreting directory contents as tables with auto-detected formats. S3 and HDFS engines provide read-only access to external S3 and HDFS storage respectively, simplifying data analysis without requiring data migration.How to use it
Users can create databases using the new engines with theCREATE DATABASE statement specifying the desired engine and configuration parameters. For example, to create an Overlay database combining multiple databases or a Filesystem database pointing to a local directory. Similarly, S3 and HDFS databases are created by setting endpoint details, prefixes, and access credentials to expose storage content as tables.