v.25.6New Feature

Function tokens now supports a new tokenizer

Official CH changelog text
Function tokens now supports a new tokenizer, named split, which is good for logs. #80195 (Robert Schulze).
The tokens function in ClickHouse now supports a new tokenizer called split, optimized for parsing log data.

Why it matters

This feature addresses the need for a simple and effective way to tokenize log strings by splitting them on whitespace or specified separators. It improves log data processing by enabling more efficient extraction of tokens from log entries.

How to use it

Use the tokens function with the split tokenizer by specifying it as an argument. For example:

sql<br>SELECT tokens('your log string', 'split')<br>FROM your_table<br>

Related resources

PreviousNext