v.25.9Improvement

Allow it to survive zookeeper connection loss

Official CH changelog text
Improved S3(Azure)Queue table engine to allow it to survive zookeeper connection loss without potential duplicates. Requires enabling S3Queue setting use_persistent_processing_nodes (changeable by ALTER TABLE MODIFY SETTING). #85995 (Kseniia Sumarokova).
Improved the S3 (Azure) Queue table engine to handle Zookeeper connection losses without producing potential duplicate entries.

Why it matters

This feature addresses the issue of potential duplicates occurring when the connection to Zookeeper is lost during queue processing. By making the queue engine more resilient to such disruptions, it ensures more reliable and consistent data processing in distributed environments.

How to use it

Enable the feature by setting the use_persistent_processing_nodes setting for the S3Queue table engine. This can be done or modified dynamically using the following SQL command:

ALTER TABLE <table_name> MODIFY SETTING use_persistent_processing_nodes = 1

Related resources

PreviousNext