FileLog Table Engine for Streaming Application Log Files in ClickHouse

Why it matters

This feature addresses the need to efficiently process and analyze application log files in real-time within ClickHouse, enabling seamless ingestion of append-only and rotated logs without relying on external message brokers. It simplifies the workflow for users who want to directly consume local log files as streaming data sources.

How to use it

Create a table with the FileLog engine specifying the log file path and format. ClickHouse will then stream new entries from these append-only or rotated log files for continuous querying. Usage is similar to engines like Kafka, but operates directly on local file-based logs.