Fix rare connection acceptance issue on server

Fix the issue when server can stop accepting connections in very rare cases. #17542 (Amos Bird, alexey-milovidov).