Fixed rare server connection acceptance issue

Fixed the issue when server can stop accepting connections in very rare cases. #17542 (alexey-milovidov).