v.21.11New Features

Support for String Key Maps in TokenBF with Enhanced Data Skipping in Queries

Official CH changelog text
tokenbf_v1 and ngram support Map with key of String of FixedSring type. It enhance data skipping in query with map key filter. sql CREATE TABLE map_tokenbf ( row_id UInt32, map Map(String, String), INDEX map_tokenbf map TYPE ngrambf_v1(4,256,2,0) GRANULARITY 1 ) Engine=MergeTree() Order by id With table above, the query select * from map_tokebf where map['K']='V' will skip the granule that doesn't contain key A . Of course, how many rows will skipped is depended on the granularity and index_granularity you set. #28511 (凌涛).
Added support for Map data type with keys of String or FixedString type in tokenbf_v1 and ngram indexes, enhancing data skipping in queries filtering by map keys.

Why it matters

This feature improves query performance by enabling more efficient granule skipping when filtering on Map keys of String or FixedString types. It allows indexes on map keys to reduce the amount of data scanned during query execution, thereby speeding up lookups and saving resources.

How to use it

Define a Map column with String or FixedString keys and create an index on it using ngrambf_v1 or tokenbf_v1 index types with appropriate granularity. For example:

CREATE TABLE map_tokenbf (
  row_id UInt32,
  map Map(String, String),
  INDEX map_tokenbf_map TYPE ngrambf_v1(4,256,2,0) GRANULARITY 1
) ENGINE=MergeTree()
ORDER BY row_id


Then queries filtering by map keys like map['K'] = 'V' will benefit from skipping granules without the specified key/value.

Related resources

PreviousNext