ai.py sometimes crashes when communicating with lab server about aliro

[hjwilli]

A different error message. Ai autostarted from common.env, error occurred while navigating the experiments page:

 | 2018 01:37:28 AM UTC : checking results...
lab_1      | requesting from : %s http://lab:5080/api/experiments
lab_1      | foo
lab_1      | Traceback (most recent call last):
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 1318, in do_open
lab_1      |     encode_chunked=req.has_header('Transfer-encoding'))
lab_1      |   File "/opt/conda/lib/python3.6/http/client.py", line 1239, in request
lab_1      |     self._send_request(method, url, body, headers, encode_chunked)
lab_1      |   File "/opt/conda/lib/python3.6/http/client.py", line 1285, in _send_request
lab_1      |     self.endheaders(body, encode_chunked=encode_chunked)
lab_1      |   File "/opt/conda/lib/python3.6/http/client.py", line 1234, in endheaders
lab_1      |     self._send_output(message_body, encode_chunked=encode_chunked)
lab_1      |   File "/opt/conda/lib/python3.6/http/client.py", line 1026, in _send_output
lab_1      |     self.send(msg)
lab_1      |   File "/opt/conda/lib/python3.6/http/client.py", line 964, in send
lab_1      |     self.connect()
lab_1      |   File "/opt/conda/lib/python3.6/http/client.py", line 936, in connect
lab_1      |     (self.host,self.port), self.timeout, self.source_address)
lab_1      |   File "/opt/conda/lib/python3.6/socket.py", line 724, in create_connection
lab_1      |     raise err
lab_1      |   File "/opt/conda/lib/python3.6/socket.py", line 713, in create_connection
lab_1      |     sock.connect(sa)
lab_1      | ConnectionRefusedError: [Errno 111] Connection refused
lab_1      |
lab_1      | During handling of the above exception, another exception occurred:
lab_1      |
lab_1      | Traceback (most recent call last):
lab_1      |   File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
lab_1      |     "__main__", mod_spec)
lab_1      |   File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
lab_1      |     exec(code, run_globals)
lab_1      |   File "/appsrc/ai/ai.py", line 460, in <module>
lab_1      |     main()
lab_1      |   File "/appsrc/ai/ai.py", line 443, in main
lab_1      |     if pennai.check_results():
lab_1      |   File "/appsrc/ai/ai.py", line 212, in check_results
lab_1      |     r = urllib.request.urlopen(req)
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 223, in urlopen
lab_1      |     return opener.open(url, data, timeout)
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 526, in open
lab_1      |     response = self._open(req, data)
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 544, in _open
lab_1      |     '_open', req)
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 504, in _call_chain
lab_1      |     result = func(*args)
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 1346, in http_open
lab_1      |     return self.do_open(http.client.HTTPConnection, req)
lab_1      |   File "/opt/conda/lib/python3.6/urllib/request.py", line 1320, in do_open
lab_1      |     raise URLError(err)
lab_1      | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
lab_1      | [TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
lab_1      | /root/.pm2/pm2.log last 15 lines:
lab_1      | PM2        | [2018-08-02T01:35:42.329Z] PM2 log: PM2 home             : /root/.pm2
lab_1      | PM2        | [2018-08-02T01:35:42.329Z] PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
lab_1      | PM2        | [2018-08-02T01:35:42.335Z] PM2 log: RPC socket file      : /root/.pm2/rpc.sock
lab_1      | PM2        | [2018-08-02T01:35:42.335Z] PM2 log: BUS socket file      : /root/.pm2/pub.sock
lab_1      | PM2        | [2018-08-02T01:35:42.335Z] PM2 log: Application log path : /root/.pm2/logs
lab_1      | PM2        | [2018-08-02T01:35:42.347Z] PM2 log: Process dump file    : /root/.pm2/dump.pm2
lab_1      | PM2        | [2018-08-02T01:35:42.348Z] PM2 log: Concurrent actions   : 2
lab_1      | PM2        | [2018-08-02T01:35:42.348Z] PM2 log: SIGTERM timeout      : 1600
lab_1      | PM2        | [2018-08-02T01:35:42.349Z] PM2 log: ===============================================================================
lab_1      | PM2        | [2018-08-02T01:35:42.701Z] PM2 log: [Watch] Start watching lab
lab_1      | PM2        | [2018-08-02T01:35:42.703Z] PM2 log: Starting execution sequence in -fork mode- for app name:lab id:0
lab_1      | PM2        | [2018-08-02T01:35:42.740Z] PM2 log: App name:lab id:0 online
lab_1      | PM2        | [2018-08-02T01:37:28.051Z] PM2 log: App [lab] with id [0] and pid [50], exited with code [1] via signal [SIGINT]
lab_1      | PM2        | [2018-08-02T01:37:28.068Z] PM2 log: Starting execution sequence in -fork mode- for app name:lab id:0
lab_1      | PM2        | [2018-08-02T01:37:28.085Z] PM2 log: App name:lab id:0 online
lab_1      |
lab_1      | /root/.pm2/logs/lab-out.log last 15 lines:
lab_1      | 0|lab      |   user:
lab_1      | 0|lab      |    { _id: 5b625fbf4f79ab8cec2a1c46,
lab_1      | 0|lab      |      username: 'testuser',
lab_1      | 0|lab      |      firstname: 'Test',
lab_1      | 0|lab      |      lastname: 'User',
lab_1      | 0|lab      |      roles: [ 'admin', 'beginner' ],
lab_1      | 0|lab      |      algorithms:
lab_1      | 0|lab      |       [ 'DecisionTreeClassifier',
lab_1      | 0|lab      |         'GradientBoostingClassifier',
lab_1      | 0|lab      |         'KNeighborsClassifier',
lab_1      | 0|lab      |         'SVC',
lab_1      | 0|lab      |         'LogisticRegression',
lab_1      | 0|lab      |         'RandomForestClassifier' ] } }
lab_1      | 0|lab      | GET /api/userdatasets 304 - - 108.344 ms
lab_1      | 0|lab      | GET /api/preferences 200 - - 70.848 ms
lab_1      |
lab_1      | /root/.pm2/logs/lab-error.log last 15 lines:
lab_1      | 0|lab      | Error: invalid compressed data
lab_1      | 0|lab      |     at Receiver.error (/appsrc/lab/node_modules/ws/lib/Receiver.js:321:18)
lab_1      | 0|lab      |     at /appsrc/lab/node_modules/ws/lib/Receiver.js:513:25
lab_1      | 0|lab      |     at /appsrc/lab/node_modules/ws/lib/Receiver.js:365:9
lab_1      | 0|lab      |     at InflateRaw.onError (/appsrc/lab/node_modules/ws/lib/PerMessageDeflate.js:254:5)
lab_1      | 0|lab      |     at emitOne (events.js:96:13)
lab_1      | 0|lab      |     at InflateRaw.emit (events.js:188:7)
lab_1      | 0|lab      |     at Zlib._handle.onerror (zlib.js:374:10)
lab_1      |

[hjwilli]

The core issue is probably on the lab server side; it looks like some requests that the ai is making of lab are not returning or are returning unexpected results. AI should be updated to handle these error a little more gracefully (log what call failed; if an unexpected result was received log the result, log if the call never returned, etc.). This will help to debug this and future issues.

On the lab server side, the issue seems to be related to socket server.

[hjwilli]

Seems like this have been resolved by disabling webSockets in the lab server (both webSocket and socket.io were running before)

[hjwilli]

No longer seeing this, marking as resolved.