Fix Join Performance Regression from #35616

Fix significant join performance regression which was introduced in #35616. It's interesting that common join queries such as ssb queries have been 10 times slower for almost 3 months while no one complains. #38052 (Amos Bird).