cdent / wsgi-intercept Goto Github PK

This fails:

def test_intercept_by_url():
    hostname = str(uuid4())
    url = 'http://%s/foobar' % hostname
    interceptor = Httplib2Interceptor(app=app, url=url)
    assert isinstance(interceptor, Interceptor)
    assert interceptor.app == app
    assert interceptor.host == hostname
    assert interceptor.port == 80
    assert interceptor.script_name == '/foobar'
    assert interceptor.url == 'http://%s:%s/foobar' % (hostname, port)
    assert interceptor.url == url

with

wsgi_intercept/interceptor.py:35: in __init__
    self._init_from_url(url)
wsgi_intercept/interceptor.py:70: in _init_from_url
    host, port = parsed_url.netloc.split(':')
E   ValueError: need more than 1 value to unpack

Basically that split needs to be safer.

I am packaging wsgi-intercept python module for fedora.
As per setup.py file, License is MIT. But License file is missing from the code repository.
Please include it.

I understand that http.client doesn't offer such an obvious way to monkeypatch it (like setting .ConnectionCls in requests, etc.) because it's just not made to be pluggable this way. So of course the current implementation does the obvious thing and replaces HTTPConnection/HTTPSConnection on the imported module object. But this also makes it sensitive to how you import and access the classes, at minimum you shouldn't access attributes of the http.client module object until after our install() ran. Just by accident I've found several ways to make the intercept installation fail.
It's all understandable but maybe not so nice for users and I'm not yet sure what to do with it. More cosmetically, it would be nice to be able to toss that snippet in the doc about how it's only somewhat implemented, and the warning snippet I put in the doc example for http.client.

I don't know if it is worth doing a deeper dive into http.client to find a different extension point (messing with its imported socket module or something like that) so that we can change it in-place to avoid some of the import order issue.

Alternatively, wsgi_intercept could provide a fake 'patched' module to use instead of the real one, and we just recommend that usage. This doesn't address uses where you want to run unmodified third-party code with the intercept (in which case, you can probably afford to manage the fiddly import order?), but it does provide a natural usage which is less fragile for things like test code under direct control of the user.

But directly providing fake modules (imported from wsgi_intercept) is something I consider providing for all the intercepts, so if that is the way to resolve the http.client issue then maybe we should do nothing on http.client and let that bigger change mostly-resolve this case too.

For example, when mod_wsgi gets an outgoing header value that is unicode, it explodes. wsgi-intercept does not, which means that tests can pass in wsgi-intercept that would fail in integration or production. That's a bummer.

Could you move the buildsystem to PEP517?

I recently found out about wsgi-intercept and it looks extremely useful. I would like to convert one of my own projects, Flask-Loopback (https://github.com/vmalloc/flask-loopback) to use it, but am missing a small feature - the ability to register callbacks to take place before and after intercepted requests. It is useful in some testing scenarios, where you want to simulate time passage or other aspects of mocked environments. It would be also useful to provide those callbacks with as much data as is available about the request that was intercepted.

Thanks in advance!

No idea what else I should write, but I had to replace gaierror with error to get the test to succeed.

Hi!
It would be great to also have README.rst in the pypi sdist tarball, as for consistency I'm using that upstream, when packaging for Arch Linux.
Thanks!

Relevant traceback:

File "/usr/lib/python2.7/site-packages/store_scraper/_fetch.py", line 21, in _fetch_raw
    r = requests.get(url, params=params, timeout=3.0, **request_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 566, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 256, in _get_conn
    return conn or self._new_conn()
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 802, in _new_conn
    strict=self.strict, **self.conn_kw)
  File "/usr/lib/python2.7/site-packages/wsgi_intercept/_urllib3.py", line 28, in __init__
    WSGI_HTTPSConnection.__init__(self, *args, **kwargs)
  File "/usr/lib64/python2.7/httplib.py", line 1254, in __init__
    source_address)
TypeError: unbound method __init__() must be called with HTTP_WSGIInterceptor instance as first argument (got HTTPS_WSGIInterceptor instance instead)

It appears the order of classes is wrong for the WSGI_HTTPSConnection class. If you swap them around the error goes away and https URLs are mocked as expected.

class WSGI_HTTPSConnection(WSGI_HTTPConnection, HTTPSConnection):

It's about time.

I've done an initial pass at this (as part of working on #74 ) and it seems to be relatively clean.

Hey there and thanks for a fancy tool!

I've noticed that interceptor puts a io.BytestIO to wsgi.errors.
That causes some trouble:

• it is never read -- so logs from my flask app crashes are lost
• flask attempts to write a unicode there and fails with

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/logging/__init__.py", line 882, in emit
    stream.write(fs % msg)
TypeError: 'unicode' does not have the buffer interface
Logged from file app.py, line 1560

IMO putting a sys.stderr there would actually improve the situation.

If it is a viable option I can go ahead and produce a pull request.

A while ago we pinned to urlib3<2 because the interface changed a bit.

It turns out that updating to >2 is harder than expected, seeing errors in tests like

_____________________________________________________________________________ test_https _____________________________________________________________________________
wsgi_intercept/tests/test_urllib3.py:59: in test_https
    resp = http.request(
.tox/py311/lib/python3.11/site-packages/urllib3/_request_methods.py:136: in request
    return self.request_encode_url(
.tox/py311/lib/python3.11/site-packages/urllib3/_request_methods.py:183: in request_encode_url
    return self.urlopen(method, url, **extra_kw)
.tox/py311/lib/python3.11/site-packages/urllib3/poolmanager.py:444: in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
.tox/py311/lib/python3.11/site-packages/urllib3/connectionpool.py:772: in urlopen
    conn = self._get_conn(timeout=pool_timeout)
.tox/py311/lib/python3.11/site-packages/urllib3/connectionpool.py:295: in _get_conn
    return conn or self._new_conn()
.tox/py311/lib/python3.11/site-packages/urllib3/connectionpool.py:1073: in _new_conn
    return self.ConnectionCls(
wsgi_intercept/_urllib3.py:53: in __init__
    WSGI_HTTPSConnection.__init__(self, *args, **kwargs)
wsgi_intercept/__init__.py:584: in __init__
    super().__init__(*args, **kwargs)
/opt/homebrew/Cellar/[email protected]/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py:1425: in __init__
    super(HTTPSConnection, self).__init__(host, port, timeout,
E   TypeError: HTTPSConnection.__init__() takes from 2 to 3 positional arguments but 5 positional arguments (and 1 keyword-only argument) were given

which I've yet to untangle. Because wsgi-intercept does heinous monkey-patching, it's hard to trace all the pieces.

[wsgi-intercept==1.2.2]
I'm using requests.Session() -> wsgi_intecept -> falcon
I'm trying to use cookies, but I can not. wsgi_intercept.make_environ method turns headers into unicode, and this is not working with SimpleCookie.load:

def load(self, rawdata):
    if type(rawdata) == type(""):
        self.__ParseString(rawdata)
    else:
        # self.update() wouldn't call our custom __setitem__
        for k, v in rawdata.items():
            self[k] = v
    return

As you can see, the SimpleCookie check if rawdata is type of "", which is not when the rawdata is an unicode, and this result in error: AttributeError: 'unicode' object has no attribute 'items'

The simplest way to "fix" this is to remove line https://github.com/cdent/wsgi-intercept/blob/master/wsgi_intercept/__init__.py#L223 but this line is there for a reason and I do not know why.

Hi,

I experienced an issue while using requests and wsgi-intercept to test a WSGI app, with Python 2.
I could simplify the code down to this:

#! /usr/bin/env python

from __future__ import print_function

import wsgiref.simple_server

import requests

from wsgi_intercept.interceptor import RequestsInterceptor

def load_app():
    return wsgiref.simple_server.demo_app

with RequestsInterceptor(load_app, host='www.example.net', port=80):
    r = requests.get('http://www.example.net')
    print(r.text)

That code works fine with Python 3 and fails with Python 2.
As I understand it, the problem is that wsgi-intercept builds the environ dict, with some keys as unicode strings. It seems to me that all keys for the environ dict have to be native strings.
The following patch fixes the issue for my trivial example:

--- wsgi_intercept/__init__.py	2022-05-13 23:30:34.051461656 +0200
+++ wsgi_intercept/__init__.py	2022-05-13 23:31:03.175107230 +0200
@@ -260,7 +260,7 @@
         else:
             h = k.upper()
             h = h.replace(b'-', b'_')
-            environ['HTTP_' + h.decode('ISO-8859-1')] = v
+            environ['HTTP_' + str(h.decode('ISO-8859-1'))] = v
 
         if debuglevel >= 2:
             print('HEADER:', k, v)

However, I’m not 100%-confident that patch is correct.
Any thoughts on this?

This line has a check for whether the first return value of the application function's returned iterable is falsey: https://github.com/cdent/wsgi-intercept/blob/ac5f41a/wsgi_intercept/__init__.py#L490

try:
    generator_data = None
    try:
        generator_data = next(self.result)

    finally:
        for data in self.write_results:
            self.output.write(data)

    if generator_data:
        try:
            self.output.write(generator_data)
        except TypeError as exc:
            raise TypeError('bytes required in response: %s' % exc)

        while 1:
            data = next(self.result)
            self.output.write(data)

except StopIteration:
    pass

I ran into this bug because the first item in my application's returned iterable was an empty bytestring, and this caused the body to be skipped entirely. This seems incorrect, as PEP-3333 includes specific references to empty byte strings being present in the returned iterable.

Was working on some local tests that use verify=False in the code to do calls using the Requests library, and any calls to external resources with this flag were getting SSL verification issues regardless.

In WSGI_HTTPSConnection, it doesn't handle the change in the SSL context when calling it for external resources. I've got a patch locally that works for this + with tests. Example test:

def test_https_no_ssl_verification_not_intercepted():
    with InstalledApp(wsgi_app.simple_app, host=HOST, port=443) as app:
        resp = requests.get('https://self-signed.badssl.com/', verify=False)
        assert resp.status_code >= 200 and resp.status_code < 300
        assert not app.success()

Hello. pytest 7.2.0 no longer depends on py so lines like this one

no longer works with pytest 7.2.0.

http://wsgi-intercept.readthedocs.org/en/latest/

$ echo $http_proxy
some_proxy.com:1234

and run the requests example, I got this error:

Traceback (most recent call last):
  File "test.py", line 20, in <module>
    resp = requests.get(url)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 511, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 231, in _get_conn
    return conn or self._new_conn()
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 192, in _new_conn
    strict=self.strict, **self.conn_kw)
  File "/usr/local/lib/python2.7/site-packages/wsgi_intercept/requests_intercept.py", line 22, in __init__
    WSGI_HTTPConnection.__init__(self, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'socket_options'

and the kwargs of wsgi_intercept/requests_intercept.py:HTTP_WSGIInterceptor.init is

{'strict': False, 'host': 'some_proxy.com', 'socket_options': [], 'timeout': <object object at 0x1028490e0>, 'port': 1234}

I'm testing my WSGI app using Requests and wsgi_intercept. My app was throwing a TypeError, which was being silently caught once and the WSGI request repeated.

It appears to be happening in wsgi_intercept. I wonder it's because the wsgi_fake_socket.sendall method catches a TypeError and then tries the request again with a different encoding... but I can't confirm this from the stack trace.

import requests
import wsgi_intercept
from wsgi_intercept import requests_intercept

wsgi_intercept.debuglevel = 2

def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type', 'text/plain')]
    start_response(status, response_headers)
    raise TypeError, "bah"
    #raise Exception, "bah"

if __name__ == '__main__':
    requests_intercept.install()
    wsgi_intercept.add_wsgi_intercept('foo', 80, lambda s = simple_app: s)
    print requests.get('http://foo/')

When building the package in Debian, I had to patch out all tests doing external network connections (to google.com in this case), because buildd do not have network access (on purpose). The issue is that on each new upstream release, the patch to remove these tests must be rebased, which is annoying.

It'd be nice to have an option set through environment variable to disable these tests when running pytest. Something like this:
WSGI_INTERCEPT_DO_NOT_TEST_WITH_INTERNET=yes

Basically self-explanatory. If I access my app via wsgi_intercept using HTTPS and receive a redirect, the URLs returned begin with "http://" instead of "https://".

I traced this down to the wsgi.url_scheme environment parameter. Based on the current trunk, if you look at init.py line 245 it appears the value is hard-coded into make_environ().

I haven't spent time to get a good feel for the coding style of this library; here is a quick and dirty patch for the issue, please rework as necessary. (I don't actually modify make_environ() here, but rather the code around it.)

--- a/wsgi_intercept/__init__.py    2014-10-31 13:13:15.000000000 -0700
+++ b/wsgi_intercept/__init__.py    2014-10-31 13:10:25.399835500 -0700
@@ -326,7 +326,7 @@
         data has been sent to the socket by the request class;
      2. non-persistent (i.e. non-HTTP/1.1) connections.
     """
-    def __init__(self, app, host, port, script_name):
+    def __init__(self, app, host, port, script_name, https=False):
         self.app = app                  # WSGI app object
         self.host = host
         self.port = port
@@ -336,6 +336,7 @@
         self.write_results = []          # results from the 'write_fn'
         self.results = None             # results from running the app
         self.output = BytesIO()        # all output from the app, incl headers
+        self.https = https

     def makefile(self, *args, **kwargs):
         """
@@ -382,6 +383,8 @@

         # build the environ dictionary.
         environ = make_environ(inp, self.host, self.port, self.script_name)
+        if self.https:
+            environ["wsgi.url_scheme"] = "https"

         # run the application.
         try:
@@ -539,7 +542,7 @@
                     sys.stderr.write('INTERCEPTING call to %s:%s\n' %
                                      (self.host, self.port,))
                 self.sock = wsgi_fake_socket(app, self.host, self.port,
-                                             script_name)
+                                             script_name, https=True)
             else:
                 HTTPSConnection.connect(self)

Hi!
I'm trying to run my gabbits tests using wsgi-interceptor against a django 2.2 application
Thats my load_tests function, where I import my django application in order to pass it to the interceptor

# For pathname munging
import os

# The module that build_tests comes from.
from gabbi import driver

from django_api.wsgi import application
# We need access to the WSGI application that hosts our service

# We're using fixtures in the YAML files, we need to know where to
# load them from.

# By convention the YAML files are put in a directory named
# "gabbits" that is in the same directory as the Python test file.
TESTS_DIR = 'gabbits'

def load_tests(loader, tests, pattern):
    """Provide a TestSuite to the discovery process."""
    test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
    # Pass "require_ssl=True" as an argument to force all tests
    # to use SSL in requests.
    return driver.build_tests(test_dir, loader,
                              intercept=application)

This is the contain of django_api/wsgi.py (which comes by default with django)

"""
WSGI config for django_api project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/1.9/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_api.settings")
application = get_wsgi_application()

When I run my gabbits tests using python -m testtools.run -v test/test_racotest.py I'm getting the following traceback for each test:

Traceback (most recent call last):
  File "/dades/python3/lib/python3.6/site-packages/gabbi/suitemaker.py", line 95, in do_test
    return test_method(*args, **kwargs)
  File "/dades/python3/lib/python3.6/site-packages/gabbi/case.py", line 94, in wrapper
    func(self)
  File "/dades/python3/lib/python3.6/site-packages/gabbi/case.py", line 148, in test_request
    self._run_test()
  File "/dades/python3/lib/python3.6/site-packages/gabbi/case.py", line 554, in _run_test
    redirect=test['redirects'])
  File "/dades/python3/lib/python3.6/site-packages/gabbi/case.py", line 463, in _run_request
    redirect=redirect
  File "/dades/python3/lib/python3.6/site-packages/gabbi/httpclient.py", line 44, in request
    method, absolute_uri, body=body, headers=headers, retries=retry)
  File "/dades/python3/lib/python3.6/site-packages/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/dades/python3/lib/python3.6/site-packages/urllib3/request.py", line 89, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/dades/python3/lib/python3.6/site-packages/urllib3/poolmanager.py", line 326, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/dades/python3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/dades/python3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/soft/python-3.6.4/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/soft/python-3.6.4/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/soft/python-3.6.4/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/soft/python-3.6.4/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/home/soft/python-3.6.4/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/dades/python3/lib/python3.6/site-packages/wsgi_intercept/__init__.py", line 564, in connect
    (app, script_name) = self.get_app(self.host, self.port)
  File "/dades/python3/lib/python3.6/site-packages/wsgi_intercept/__init__.py", line 548, in get_app
    app = app_fn()
TypeError: __call__() missing 2 required positional arguments: 'environ' and 'start_response'

Does anyone have any idea of where is the error? Could it be a compatibility error between django wsgi default application and wsgi-intercept?
Thank you in advance! :)

Hi,

Trying to build wsgi-intercept in Debian Sid, in order to upgrade to version 0.8.0 and fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=755315, I get the below unit test errors. It'd be really cool if you could have a look so that I can upload and fix this package.

Cheers,

Thomas Goirand (zigo)

debian/rules override_dh_auto_test
make[1]: Entering directory '/home/zigo/sources/openstack/juno/python-wsgi-intercept/build-area/python-wsgi-intercept-0.8.0'
http_proxy= https_proxy= dh_auto_test
I: pybuild base:170: cd /home/zigo/sources/openstack/juno/python-wsgi-intercept/build-area/python-wsgi-intercept-0.8.0/.pybuild/pythonX.Y_2.7/build; python2.7 -m pytest
============================= test session starts ==============================
platform linux2 -- Python 2.7.8 -- py-1.4.22 -- pytest-2.6.0
collected 33 items

test/test_http_client.py .....x
test/test_httplib2.py .......
test/test_requests.py .........
test/test_urllib.py ......
test/test_wsgi_compliance.py .....

===================== 32 passed, 1 xfailed in 1.38 seconds =====================
I: pybuild base:170: cd /home/zigo/sources/openstack/juno/python-wsgi-intercept/build-area/python-wsgi-intercept-0.8.0/.pybuild/pythonX.Y_3.4/build; python3.4 -m pytest
============================= test session starts ==============================
platform linux -- Python 3.4.1 -- py-1.4.22 -- pytest-2.6.0
collected 33 items

test/test_http_client.py .....x
test/test_httplib2.py .......
test/test_requests.py ....FF..F
test/test_urllib.py ......
test/test_wsgi_compliance.py .....

=================================== FAILURES ===================================
__________________________________ test_https __________________________________

def test_https():
    with InstalledApp(wsgi_app.simple_app, host=HOST, port=443) as app:

      resp = requests.get('https://some_hopefully_nonexistant_domain:443/')

test/test_requests.py:52:

/usr/lib/python3/dist-packages/requests/api.py:55: in get
return request('get', url, *_kwargs)
/usr/lib/python3/dist-packages/requests/api.py:44: in request
return session.request(method=method, url=url, *_kwargs)
/usr/lib/python3/dist-packages/requests/sessions.py:456: in request
resp = self.send(prep, *_send_kwargs)
/usr/lib/python3/dist-packages/requests/sessions.py:559: in send
r = adapter.send(request, *_kwargs)
/usr/lib/python3/dist-packages/requests/adapters.py:327: in send
timeout=timeout
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:491: in urlopen
conn = self._get_conn(timeout=pool_timeout)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:230: in _get_conn
return conn or self._new_conn()

self = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f9263cfb5f8>

def _new_conn(self):
    """
        Return a fresh :class:`httplib.HTTPSConnection`.
        """
    self.num_connections += 1
    log.info("Starting new HTTPS connection (%d): %s"
             % (self.num_connections, self.host))

    if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
        # Platform-specific: Python without ssl
        raise SSLError("Can't connect to HTTPS URL because the SSL "
                       "module is not available.")

    actual_host = self.host
    actual_port = self.port
    if self.proxy is not None:
        actual_host = self.proxy.host
        actual_port = self.proxy.port

    conn = self.ConnectionCls(host=actual_host, port=actual_port,
                              timeout=self.timeout.connect_timeout,

                            strict=self.strict, **self.conn_kw)

E TypeError: init() got an unexpected keyword argument 'strict'

/usr/lib/python3/dist-packages/urllib3/connectionpool.py:678: TypeError
___________________________ test_https_default_port ____________________________

def test_https_default_port():
    with InstalledApp(wsgi_app.simple_app, host=HOST, port=443) as app:

      resp = requests.get('https://some_hopefully_nonexistant_domain/')

test/test_requests.py:59:

/usr/lib/python3/dist-packages/requests/api.py:55: in get
return request('get', url, *_kwargs)
/usr/lib/python3/dist-packages/requests/api.py:44: in request
return session.request(method=method, url=url, *_kwargs)
/usr/lib/python3/dist-packages/requests/sessions.py:456: in request
resp = self.send(prep, *_send_kwargs)
/usr/lib/python3/dist-packages/requests/sessions.py:559: in send
r = adapter.send(request, *_kwargs)
/usr/lib/python3/dist-packages/requests/adapters.py:327: in send
timeout=timeout
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:491: in urlopen
conn = self._get_conn(timeout=pool_timeout)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:230: in _get_conn
return conn or self._new_conn()

self = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f926354e710>

def _new_conn(self):
    """
        Return a fresh :class:`httplib.HTTPSConnection`.
        """
    self.num_connections += 1
    log.info("Starting new HTTPS connection (%d): %s"
             % (self.num_connections, self.host))

    if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
        # Platform-specific: Python without ssl
        raise SSLError("Can't connect to HTTPS URL because the SSL "
                       "module is not available.")

    actual_host = self.host
    actual_port = self.port
    if self.proxy is not None:
        actual_host = self.proxy.host
        actual_port = self.proxy.port

    conn = self.ConnectionCls(host=actual_host, port=actual_port,
                              timeout=self.timeout.connect_timeout,

                            strict=self.strict, **self.conn_kw)

E TypeError: init() got an unexpected keyword argument 'strict'

/usr/lib/python3/dist-packages/urllib3/connectionpool.py:678: TypeError
__________________________ test_https_not_intercepted __________________________

def test_https_not_intercepted():
    with InstalledApp(wsgi_app.raises_app, host=HOST, port=80):
        try:

          resp = requests.get("https://google.com")

test/test_requests.py:84:

/usr/lib/python3/dist-packages/requests/api.py:55: in get
return request('get', url, *_kwargs)
/usr/lib/python3/dist-packages/requests/api.py:44: in request
return session.request(method=method, url=url, *_kwargs)
/usr/lib/python3/dist-packages/requests/sessions.py:456: in request
resp = self.send(prep, *_send_kwargs)
/usr/lib/python3/dist-packages/requests/sessions.py:559: in send
r = adapter.send(request, *_kwargs)
/usr/lib/python3/dist-packages/requests/adapters.py:327: in send
timeout=timeout
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:491: in urlopen
conn = self._get_conn(timeout=pool_timeout)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:230: in _get_conn
return conn or self._new_conn()

self = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f9262c91b70>

def _new_conn(self):
    """
        Return a fresh :class:`httplib.HTTPSConnection`.
        """
    self.num_connections += 1
    log.info("Starting new HTTPS connection (%d): %s"
             % (self.num_connections, self.host))

    if not self.ConnectionCls or self.ConnectionCls is DummyConnection:
        # Platform-specific: Python without ssl
        raise SSLError("Can't connect to HTTPS URL because the SSL "
                       "module is not available.")

    actual_host = self.host
    actual_port = self.port
    if self.proxy is not None:
        actual_host = self.proxy.host
        actual_port = self.proxy.port

    conn = self.ConnectionCls(host=actual_host, port=actual_port,
                              timeout=self.timeout.connect_timeout,

                            strict=self.strict, **self.conn_kw)

E TypeError: init() got an unexpected keyword argument 'strict'

/usr/lib/python3/dist-packages/urllib3/connectionpool.py:678: TypeError
================ 3 failed, 29 passed, 1 xfailed in 1.45 seconds ================
E: pybuild pybuild:256: test: plugin distutils failed with: exit code=1: cd /home/zigo/sources/openstack/juno/python-wsgi-intercept/build-area/python-wsgi-intercept-0.8.0/.pybuild/pythonX.Y_3.4/build; python3.4 -m pytest
dh_auto_test: pybuild --test -i python{version} -p 3.4 --dir . returned exit code 13
debian/rules:11: recipe for target 'override_dh_auto_test' failed
make[1]: *** [override_dh_auto_test] Error 13
make[1]: Leaving directory '/home/zigo/sources/openstack/juno/python-wsgi-intercept/build-area/python-wsgi-intercept-0.8.0'
debian/rules:8: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2

http_client_intercept does not work if non-intercepted HTTPS connections are going to be used by client.

  File "/.../lib/python2.7/site-packages/oauth2/__init__.py", line 682, in request
    connection_type=connection_type)
  File "/.../lib/python2.7/site-packages/httplib2/__init__.py", line 1450, in request
    self.disable_ssl_certificate_validation)
  File "/.../lib/python2.7/site-packages/httplib2/__init__.py", line 929, in __init__
    cert_file=cert_file, strict=strict)
  File "/.../lib/python2.7/site-packages/wsgi_intercept/http_client_intercept.py", line 39, in __init__
    HTTP_WSGIInterceptor.__init__(self, host, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'cert_file'

• Python 2.7.6 (default, Nov 21 2013, 15:55:38) [GCC] on linux2

If intercepts are created for more than one wsgi application in the same process, using the same interceptor module, uninstalling one of them will turn off intercept for the others because the intercept module's socket class overrides will be removed. This is not desirable.

Since version 1.0.0 wsgi-intercept fails with httplib2_intercept request.
With version 0.10.3 work properly. Here are my test:
wsgi-intercept-issue.txt

Congratulations for your project. Thanks!

It seems URLs with character escapes (e.g. %3F for an "escaped" ? in a URL) are not being unescaped, as is expected by some libraries (bottle in my case).

This was traced to the WSGI environment variable PATH_INFO.

PEP 333 does not seem to explicitly mandate that PATH_INFO be unescaped when extracted from a URL. However, it is implied via its docs on how to get a URL from the WSGI environment variables.

Here is my patch (based upon the 0.8.1 release):

--- a/__init__.py   2014-11-03 10:56:56.570455800 -0800
+++ b/__init__.py   2014-11-03 10:59:48.904455800 -0800
@@ -106,6 +106,11 @@
 except ImportError:
     from StringIO import StringIO as BytesIO

+try:
+    from urllib.parse import unquote as url_unquote
+except ImportError:
+    from urllib import unquote as url_unquote
+
 import traceback

 debuglevel = 0
@@ -224,7 +229,7 @@
         url = url[len(script_name):]

     url = url.split('?', 1)
-    path_info = url[0]
+    path_info = url_unquote(url[0])
     query_string = ""
     if len(url) == 2:
         query_string = url[1]

I found this while trying out some restructuring of the tests to get rid of the globally shared state between different tests, and found that doing that caused this one test to start failing on python2. Other tests seemed to be working.

To reproduce, suppose I disable every other test in test_httplib2: for example, if I go into test_httplib2.py and change 'def test_success' to 'def _test_success' and 'def test_bogus_domain' to 'def _test_bogus_domain'. I see that test_https_success fails on python2 (python2.6, python2.7, pypy's 2.7) with ServerNotFoundError: Unable to find the server at some_hopefully_nonexistant_domain. Still seems OK on python3.

Re-enable either of the other tests, and it succeeds for all the versions.
As expected, same thing if you use py.test -k https_success to only run that one test without running the others in test_httplib2.

It stands to reason that this is because of the calls to install() affecting global state to install the intercept, and that state not being reset between tests, creating the implicit dependency of test_https_success on the others' installation of the intercept, and the failure mode of actually trying to do something with the hostname.

It depends on install() running at all - with all tests running, test_https_success fails in python2 depending on whether or not another test has called install() - tested by commenting out those lines.

It also depends on what argument is passed to install(). Again with all tests but test_https_success disabled, I modify test_https_success so that install(443) says instead install(). And I see the same test fails on python3 (python3.3, python3.4) with httplib2.ServerNotFoundError: Unable to find the server at some_hopefully_nonexistant_domain. Still seems OK on python2.

To summarize: in isolation from the other tests, calling install() lets test_https_success pass on python2 but fail on python3, while calling install(443) lets it pass on python3 but fail on python2. The question is why the intercept installation is failing to take in either case, almost certainly involving some python2 vs. python3 branch.

To be continued with more info