Check the Time Limit Every (flush_interval / Poll_timeout) Number of Rows from Kafka

Why it matters

This feature addresses the issue of delayed time limit checks during Kafka consumption by enabling more frequent interruptions in reading. It ensures that top-level stream time limits are enforced more promptly, improving control over Kafka data ingestion and resource management.

How to use it

The system automatically checks the time limit every (flush_interval / poll_timeout) number of rows read from Kafka. Users can configure flush_interval and poll_timeout settings to adjust the frequency of these checks according to their workload requirements.