urllib3 intercept cause TypeError with https URL about wsgi-intercept

[cdent]

I guess there must be a missing test. I'll look into this and make sure there's a test that covers this, fix it, and make a new release.

Thanks for reporting the problem.

[cdent]

I'm unable to replicate this problem. Can you provide a pointer to a failing test or at least more information on your environment where you are seeing the problem (version of requests, version of Python, operating system, etc)?

[boralyl]

This is within a unix docker container. We are using python2.7. The requests lib is version 2.10.0. I also tried with the latest (2.13.0). Here is a simple script to reproduce it.

import requests
from wsgi_intercept import (http_client_intercept, requests_intercept,
                            urllib3_intercept, add_wsgi_intercept)


def app(environ, start_response):
    url = 'https://play.google.com/store/apps/details'
    params = {
        'hl': 'en',
        'id': 'com.disney.WMW',
    }
    r = requests.get(url, params=params, timeout=3.0)
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [r.content]


def make_app():
    return app


if __name__ == '__main__':
    try:
        from wsgiref.simple_server import make_server
        httpd = make_server('', 8080, app)
        print('Serving on port 8080...')
        requests_intercept.install()
        urllib3_intercept.install()
        http_client_intercept.install()
        add_wsgi_intercept('play.google.com', 443, make_app)
        httpd.serve_forever()
    except KeyboardInterrupt:
        requests_intercept.uninstall()
        urllib3_intercept.uninstall()
        http_client_intercept.uninstall()
        print('Goodbye.')

$ python2.7 server.py &
$ curl localhost:8080
Traceback (most recent call last):
  File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "server.py", line 12, in app
    r = requests.get(url, params=params, timeout=3.0)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 250, in _get_conn
    return conn or self._new_conn()
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 832, in _new_conn
    strict=self.strict, **self.conn_kw)
  File "/usr/lib/python2.7/site-packages/wsgi_intercept/_urllib3.py", line 28, in __init__
    WSGI_HTTPSConnection.__init__(self, *args, **kwargs)
  File "/usr/lib64/python2.7/httplib.py", line 1254, in __init__
    source_address)
TypeError: unbound method __init__() must be called with HTTP_WSGIInterceptor instance as first argument (got HTTPS_WSGIInterceptor instance instead)
127.0.0.1 - - [14/Mar/2017 16:16:30] "GET / HTTP/1.1" 500 59

[cdent]

Hmmm. Couple of things here. One is that you only need to install one intercept. In this case requests_intercept.install() ought to do the trick. In your own environment if you fix that, you might be able to get things working.

In the example you've provided, however, once you are only using one interceptor you'll see that you're still getting an error, but a different one that your proposed fix does not fix. Instead we get an error in the application which eventually reveals itself to be a maximum recursion problem.

That's because what you're doing here is not something that wsgi-intercept is really expecting: You are intercepting a hostname used by the application that you are intercepting, so the app calls itself, which calls itself, which calls itself...

Does that make any sense?

So my advice at this stage would be to try using just one interceptor and see where that gets you. Also consider using the context manager style of interceptors if that can work for you.

[boralyl]

Regarding the max recursion issue, yes that makes sense. Using a single interceptor for requests gives me the results I would expect, the problem is when you use a requests interceptor along with an http client interceptor. That's what causes the error. You can use any combination of the other interceptors, you just can't use both HttpClientInterceptor and RequestsInterceptor at the same time.

For our use case the app under test uses 2 or more underlying libraries (e.g. requests, urllib3, httplib). So we need to intercept all of them for mocking http calls.

Here's an example using the context, which gives the same result as using the old style manual install/add_wsgi_itercept.

import requests
from wsgi_intercept.interceptor import RequestsInterceptor, HttpClientInterceptor


def foo1(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [b'foo']


def foo():
    return foo1


if __name__ == '__main__':
    with HttpClientInterceptor(foo, 'somedomain.com'):
        with RequestsInterceptor(foo, 'play.google.com', 443):
            url = 'https://play.google.com/store/apps/details'
            params = {
                'hl': 'en',
                'id': 'com.disney.WMW',
            }
            r = requests.get(url, params=params)
            print r.content

$ python server.py
Traceback (most recent call last):
  File "server.py", line 35, in <module>
    r = requests.get(url, params=params)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 250, in _get_conn
    return conn or self._new_conn()
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 832, in _new_conn
    strict=self.strict, **self.conn_kw)
  File "/usr/lib/python2.7/site-packages/wsgi_intercept/_urllib3.py", line 28, in __init__
    WSGI_HTTPSConnection.__init__(self, *args, **kwargs)
  File "/usr/lib64/python2.7/httplib.py", line 1254, in __init__
    source_address)
TypeError: unbound method __init__() must be called with HTTP_WSGIInterceptor instance as first argument (got HTTPS_WSGIInterceptor instance instead)

If you replace HttpClientInteceptor with any of the others like Urllib3Interceptor it works as expected.

[cdent]

Interesting. Your use of wsgi-intercept (using different interceptors at the same time) is a bit different from the common use case (where you use the interceptor simply to allow mounting a wsgi app on a fake socket and test it with one client type at any given time). But since your suggested change fixes it and doesn't seem to break anything else I'll go ahead and make it and do a new release. Thanks for persisting in answering my questions.

[cdent]

doesn't seem to break anything else

This isn't the case. When I make your suggested change it causes several failures in the test suite under py27 and in py35 the tests give up immediately because of

E   TypeError: Cannot create a consistent method resolution
E   order (MRO) for bases HTTPSConnection, HTTP_WSGIInterceptor, WSGI_HTTPConnection

The underlying issue here is that all the usual client libraries end up eventually using the http.client or httplib code so if you have already monkeypatched in requests and then want to monkey patch in http.client too, the thing being monkey patched is the wrong type: it has already been patched by the other interceptor.

So the end result of this is what I said earlier: use one interceptor at a time. If you need to test different clients, only intercept the one you are testing during any given test.

If I'm still missing something please let me know, but please keep in mind that the existing tests and use cases need to continue working, so try your ideas out on the code and the tests that already exist. Thanks.

[boralyl]

Thanks for the feedback and quick responses!