Release Memory for Early Aggregation to Optimize Performance

Release memory for aggregation earlier. This may avoid unnecessary external aggregation. #57691 (Nikolai Kochetov).