Fixed rare server connection issue

Fixed the issue when server can stop accepting connections in very rare cases. #17542 (alexey-milovidov).