Fatal error on SSL transport about monocle

[Noctem]

I'm not quite sure what the problem is here. Does it happen in the middle of sessions or while exiting? On every request, many, or just a few? Is your internet connection stable?

[Memurame]

Yes my internet connection is stable.

Here a littlebit more.

[2017-02-06 00:25:43,582][ WARNING][worker-89] Login attempt timed out.
[2017-02-06 00:25:43,585][ WARNING][worker-431] Login attempt timed out.
[2017-02-06 00:25:43,587][ WARNING][worker-106] Login attempt timed out.
[2017-02-06 00:25:43,590][ WARNING][worker-409] Login attempt timed out.
[2017-02-06 00:25:50,880][ WARNING][worker-89] ranxavelxy249 is not authenticated.
[2017-02-06 00:25:50,882][ WARNING][worker-431] ranxavelxy352 is not authenticated.
[2017-02-06 00:25:50,882][ WARNING][worker-106] sanfunx87yy68 is not authenticated.
[2017-02-06 00:25:50,883][ WARNING][worker-409] ranxavelxy238 is not authenticated.
[2017-02-06 00:25:51,394][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:51,394][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4babea8198>, 'transport': <_SelectorSocketTransport closing fd=67 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:51,395][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:51,395][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4babe6d9e8>, 'transport': <_SelectorSocketTransport closing fd=68 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:51,395][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:51,395][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4babf3fdd8>, 'transport': <_SelectorSocketTransport closing fd=71 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:51,395][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:51,395][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4bc4042f28>, 'transport': <_SelectorSocketTransport closing fd=74 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:54,407][ WARNING][worker-89] Swapping out ranxavelxy249 because login failed.
[2017-02-06 00:25:54,411][ WARNING][worker-431] Swapping out ranxavelxy352 because login failed.
[2017-02-06 00:25:54,415][ WARNING][worker-106] Swapping out sanfunx87yy68 because login failed.
[2017-02-06 00:25:54,419][ WARNING][worker-409] Swapping out ranxavelxy238 because login failed.
[2017-02-06 00:25:57,931][ WARNING][worker-235] Login attempt timed out.
[2017-02-06 00:25:57,937][ WARNING][worker-253] Login attempt timed out.
[2017-02-06 00:25:57,941][ WARNING][worker-193] Login attempt timed out.
[2017-02-06 00:25:57,945][ WARNING][worker-362] Login attempt timed out.
[2017-02-06 00:25:57,985][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:57,986][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4bc44cec88>, 'transport': <_SelectorSocketTransport closing fd=71 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:57,986][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:57,986][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4bc4038cf8>, 'transport': <_SelectorSocketTransport closing fd=68 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:57,986][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:57,986][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4babe73b00>, 'transport': <_SelectorSocketTransport closing fd=86 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:25:57,986][   ERROR][eventloop] A wild exception appeared!
[2017-02-06 00:25:57,986][   ERROR][eventloop] {'protocol': <asyncio.sslproto.SSLProtocol object at 0x7f4babec9c50>, 'transport': <_SelectorSocketTransport closing fd=74 read=idle write=<idle, bufsize=0>>, 'message': 'Fatal error on SSL transport', 'exception': RuntimeError('shutdown in progress',)}
[2017-02-06 00:26:03,745][ WARNING][worker-235] Login attempt timed out.
[2017-02-06 00:26:03,748][ WARNING][worker-253] Login attempt timed out.
[2017-02-06 00:26:03,750][ WARNING][worker-193] Login attempt timed out.
[2017-02-06 00:26:03,753][ WARNING][worker-362] Login attempt timed out.
[2017-02-06 00:26:09,173][ WARNING][worker-235] Login attempt timed out.
[2017-02-06 00:26:09,176][ WARNING][worker-253] Login attempt timed out.
[2017-02-06 00:26:09,179][ WARNING][worker-193] Login attempt timed out.
[2017-02-06 00:26:09,183][ WARNING][worker-362] Login attempt timed out.
[2017-02-06 00:26:14,499][ WARNING][worker-253] pokegobern192 is not authenticated.
[2017-02-06 00:26:14,500][ WARNING][worker-235] sanfunx87yy156 is not authenticated.
[2017-02-06 00:26:14,501][ WARNING][worker-362] ranxavelxy328 is not authenticated.
[2017-02-06 00:26:14,501][ WARNING][worker-193] pokegobern281 is not authenticated.
[2017-02-06 00:26:16,441][ WARNING][worker-253] Swapping out pokegobern192 because login failed.
[2017-02-06 00:26:16,452][ WARNING][worker-235] Swapping out sanfunx87yy156 because login failed.
[2017-02-06 00:26:16,455][ WARNING][worker-362] Swapping out ranxavelxy328 because login failed.
[2017-02-06 00:26:16,459][ WARNING][worker-193] Swapping out pokegobern281 because login failed.

[MonkeyTedBaker]

I figured out what this is.
asyncio.sslproto.SSLProtocol and other TCP related errors tend to happen when the server can't process TCP in a timely manner. While there could be various reasons for this (mostly infra related), one common reason is the maximum CPU usage of server.

Usually, Monocle is very conservative in terms of CPU usage but when doing certain operations like bootstrap, MORE_POINTS, etc, it maxes out CPU and when that happens, server fails to properly process TCP connections and you get those errors.

[jeffpeiyt]

@MonkeyTedBaker great findings. We find similar issuing on SSLProtocol with the Fatal error on SSL transport. Do you have suggestions to mitigate the issue other than to lower cpu usage? Seems when that happens, many IO/TCP requests are failing