Fix rare server connection issue

Fix the issue when server can stop accepting connections in very rare cases. #17542 (Amos Bird).