TypeError exceptions from wsgi app are silently absorbed about wsgi-intercept

[hmoffatt]

Result is as follows; as you can see the simple_app is called twice;

connect: foo, 80
INTERCEPTING call to foo:80
>>> GET / HTTP/1.1
Host: foo
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.2.1 CPython/2.7.3 Linux/3.13-0.bpo.1-amd64

 >>>
HEADER: Host foo
HEADER: Accept-Encoding gzip, deflate, compress
HEADER: Accept */*
HEADER: User-Agent python-requests/2.2.1 CPython/2.7.3 Linux/3.13-0.bpo.1-amd64
METHOD LINE: GET / HTTP/1.1

method: GET; script_name: ; path_info: /; query_string: 
WSGI environ dictionary: {'wsgi.multithread': 0, 'SCRIPT_NAME': '', 'wsgi.input': <_io.BytesIO object at 0x226e950>, 'REQUEST_METHOD': 'GET', u'HTTP_HOST': u'foo', 'PATH_INFO': '/', 'SERVER_PROTOCOL': 'HTTP/1.1\r\n', u'HTTP_ACCEPT': u'*/*', u'HTTP_USER_AGENT': u'python-requests/2.2.1 CPython/2.7.3 Linux/3.13-0.bpo.1-amd64', 'wsgi.version': (1, 0), 'SERVER_NAME': 'foo', 'REMOTE_ADDR': '127.0.0.1', 'wsgi.run_once': 0, 'wsgi.errors': <_io.BytesIO object at 0x226e9b0>, 'wsgi.multiprocess': 0, 'wsgi.url_scheme': 'http', 'SERVER_PORT': 80, u'HTTP_ACCEPT_ENCODING': u'gzip, deflate, compress'}
HEADER: Host foo
HEADER: Accept-Encoding gzip, deflate, compress
HEADER: Accept */*
HEADER: User-Agent python-requests/2.2.1 CPython/2.7.3 Linux/3.13-0.bpo.1-amd64
METHOD LINE: GET / HTTP/1.1

method: GET; script_name: ; path_info: /; query_string: 
WSGI environ dictionary: {'wsgi.multithread': 0, 'SCRIPT_NAME': '', 'wsgi.input': <_io.BytesIO object at 0x226ead0>, 'REQUEST_METHOD': 'GET', u'HTTP_HOST': u'foo', 'PATH_INFO': '/', 'SERVER_PROTOCOL': 'HTTP/1.1\r\n', u'HTTP_ACCEPT': u'*/*', u'HTTP_USER_AGENT': u'python-requests/2.2.1 CPython/2.7.3 Linux/3.13-0.bpo.1-amd64', 'wsgi.version': (1, 0), 'SERVER_NAME': 'foo', 'REMOTE_ADDR': '127.0.0.1', 'wsgi.run_once': 0, 'wsgi.errors': <_io.BytesIO object at 0x226eb30>, 'wsgi.multiprocess': 0, 'wsgi.url_scheme': 'http', 'SERVER_PORT': 80, u'HTTP_ACCEPT_ENCODING': u'gzip, deflate, compress'}
Traceback (most recent call last):
  File "except_test.py", line 18, in <module>
    print requests.get('http://foo/')
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/adapters.py", line 330, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.2.1-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 315, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1032, in getresponse
    response = self.response_class(*args, **kwds)
  File "/usr/lib/python2.7/httplib.py", line 346, in __init__
    self.fp = sock.makefile('rb', 0)
  File "/usr/local/lib/python2.7/dist-packages/wsgi_intercept-0.6.2-py2.7.egg/wsgi_intercept/__init__.py", line 356, in makefile
    app_result = self.app(environ, start_response)
  File "except_test.py", line 12, in simple_app
    raise TypeError, "bah"
TypeError: bah

[sashahart]

Trying your code, I locally saw two calls as well (although I don't certify it's the same problem)...

In the tracebacks, I see the calls emitting from requests' vendored urllib3 (for me this is requests/packages/urllib3/connectionpool.py, lines 313 and 315).

        try: # Python 2.7+, use buffering of HTTP responses
            httplib_response = conn.getresponse(buffering=True)
        except TypeError: # Python 2.6 and older
            httplib_response = conn.getresponse()

I am getting the intentionally-raised TypeError from conn.getresponse(buffering=True), so this try/except catches it and calls conn.getresponse() a second time, in the except clause.

urllib3 could be patched to detect the version difference in a less corny way (or 2.6 deprecated so it isn't even necessary to use this try/except... ahem) but that will be more trouble to get fixed - and anyway there are going to be other instances of the more abstract problem here. The app's TypeError probably shouldn't be caught by requests' TypeError handler, which can't reasonably suppose that exceptions raised by code in "the server" will reach it, that expectation is being violated by wsgi-intercept of course.

Does solving this require intentionally catching non-profound errors at some barrier before they bubble up to requests?

[cdent]

Thanks for the detailed report.

Does solving this require intentionally catching non-profound errors at some barrier before they bubble up to requests?

That makes some sense. That barrier would act in the same capacity as an actual network. Of course the barrier couldn't simply re-raise, it would need to encapsulate the exception as something special to avoid the existing problem.

Thoughts?

[hmoffatt]

I agree. I think you would need to re-raise the exception from the WSGI app as a new exception of a different type. This could either just be for TypeError, or perhaps better just wrap all errors from the WSGI app and pass their details along as parameters of a new wsgi_intercept exception.

[sashahart]

So you are collecting the app's error somewhere, and then at some time after requests.get is not on the stack, you want to raise (so that the exception reaches e.g. the unit test calling requests.get). Some other exception which holds stale traceback info from the app's error. My question is what triggers that. It is overly clever and intrusive to replace requests.get with a wrapper that does the raise right afterward. So now you are somehow relying on the user to somehow trigger a poll for whether an app error just happened, which they are likely to forget.

Here's one brainstorm that doesn't seem optimal to me, but could be a fallback position: use a context manager as recommended API for installing/uninstalling the intercept. User's calls like request.get happen inside the with-block. On exit from the with-block, the context manager can poll for the app exception. This means the user doesn't have to do anything except use recommended API for install/uninstall. If you could guarantee user would run the uninstall thingy, you could poll there.

What bothers me about this is that a lot can happen between the end of requests.get and the uninstall, so then you are in the position of collecting and reporting any number of past app errors... also this is a pretty big change

[hmoffatt]

I was thinking that any exceptions from the app would be re-raised immediately as a different type that isn't caught by requests' exception handler. That might be a bit fragile though.

[cdent]

A context manager is a good idea (for more than just the reasons stated), but yes would be a quite large change. The original design of wsgi-intercept predates context managers by quite a bit. Probably worth doing in the long run.

In the short term I think wrapping any exception from the app and raising it as a special WSGIInterceptException (or similar) is the way to go. This would be the equivalent of getting a 500 from the server.

I might be able to do this on Sunday, but if anybody wants to get to it before then feel free.

[sashahart]

Ah, so the idea is to catch TypeError("bah") and then reraise OurOwnError(), knowing that requests.get may catch any normal error but not OurOwnError? That seems simple enough.

[sashahart]

I believe this is addressed by the merged PR #6.