Flush gets stuck on Linux about applicationinsights-python

[turowicz]

Still an issue. Does anyone knows why the endpoint is not accepting/responding?

[smereczynski]

It works for me:

(venv)michal@michal:~/flush$ python test.py 
None
Done

What You have in telemetry variable? It should be smth like that:

tc = TelemetryClient('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')

[enpo]

I have the same issue. It happens when the quota for application insights is exceeded on running python manage.py check in a django application.

[nicodeslandes]

I've had this issue occasionally on Linux; haven't been able to track it down yet. Here's the call stack I get when the process is frozen:

  File "/var/lib/app-monitor/connectionMonitor.py", line 52, in <module>
        tc.flush()
  File "/usr/local/lib/python2.7/dist-packages/applicationinsights/TelemetryClient.py", line 55, in flush
        self._channel.flush()
  File "/usr/local/lib/python2.7/dist-packages/applicationinsights/channel/TelemetryChannel.py", line 71, in flush
        self._queue.flush()
  File "/usr/local/lib/python2.7/dist-packages/applicationinsights/channel/SynchronousQueue.py", line 39, in flush
        local_sender.send(data)
  File "/usr/local/lib/python2.7/dist-packages/applicationinsights/channel/SenderBase.py", line 118, in send
        response = HTTPClient.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
        return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
        response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
        '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
        result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1222, in https_open
        return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
        r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1089, in getresponse
        response.begin()
  File "/usr/lib/python2.7/httplib.py", line 444, in begin
        version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 400, in _read_status
        line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
        data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 341, in recv
        return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 260, in read
        return self._sslobj.read(len)

The process is basically stuck waiting for a reply from the Azure service. I'm thinking or getting a PR together to allow for a timeout value; at least that will allow for some retry logic if such a situation occurs.

[nicodeslandes]

I have a temporary workaround for the problem.
Setting up a global timeout for socket calls allows the self.read call to fail, and the API can then make another try immediately.
Adding this at the beginning of the script will do that:

import socket
socket.setdefaulttimeout(30)

This is of course not ideal as this will apply to any HTTP calls made in the script, but that should be a useful workaround in a lot of cases.

[lostromb]

I would personally pin the root cause of this issue as a lack of fallback logic on SynchronousQueue. If you look in a code it's a very rudimentary "while True:" loop that will go forever until the sender has successfully sent all its data. If the web call blocks or if the internet connection drops, this loop will hang your program. I realize that it is intended to be a synchronous queue so blocking is a natural consequence, but this code should be more resilient to cases where the remote service never responds and fallback or try again later.

For others who are affected by this issue I would recommend using an AsynchronousQueue in your telemetry client constructor:
instrumentation = TelemetryClient(instrumentation_key, TelemetryChannel(queue = AsynchronousQueue(AsynchronousSender())))

[nicodeslandes]

No, in practice this infinite loop in the Synchronous queue is not the problem. I've added a lot of logs on my fork of the lib (https://github.com/nicodeslandes/ApplicationInsights-Python), and the issue is that for some reason sometimes the App Insights service doesn't answer anything, even after 10 minutes. That cause the web call to block forever, so the while True: is not even evaluated.
(By the way the fact that the socket timeout fixes the problem does indeed indicate that the infinite loop is not at fault here; I haven't seen cases where Azure repeatedly return an error, so we always get out of the loop after 2 or 3 attempts in case of a failure).

I haven't tried it, but looking at the code, my guess is that the AsynchronousSender will also see the same problem with the web call, as no timeout will be set either in that case. That will result in all the requests to be enqueued in the AsynchronousQueue, but never actually sent to the server because the sending thread will be blocked. That might actually be worst, as there won't be any way for the caller to tell that nothing is being sent to the server when that happens.

It seems to me that the proper fix to this issue is to set a timeout (ideally configurable) in the call to HTTPClient.urlopen() in the SenderBase call. That should fix the problem in both synchronous and asynchronous cases.