Unit test errors with Python 3.4 about wsgi-intercept

[cdent]

I'm unable to replicate this locally or on travis (with python 3.4.1 for both).

It appears you have an unclean environment, probably with an old version of urllib3. Can you check that and get back?

[thomasgoirand]

Well, I do use urllib3 from Sid, which currently is version 1.8.3 (not: as a package maintainer, I only install Python modules using apt-get...). Why is this a problem? Anyway, I tried packaging and installing version 1.9 of urllib3, and there's the same errors. So that's not the issue. If you want to try, maybe you should just install Debian Sid, and install the dependencies with apt:

apt-get install python-httplib2 python-pytest python-requests python-setuptools python-urllib3 python3-all python3-httplib2 python3-pytest python3-requests python3-setuptools python3-urllib3

Then just run: python3.4 -m pytest

[cdent]

It appears the issue you are seeing is with urllib3: urllib3/urllib3#238 and the way in which urllib3 is used by requests (which packages it own version of urllib3). Either the python3-urllib3 or python3-requests packages are not up to date with the fix linked in that bug report.

So I'd suggest pip installing requests for the python3 env and seeing if that gets it. If it does then perhaps, there's yet another update Sid needs.

[thomasgoirand]

Well, the issue in urllib3 that you are referencing is said to be fixed in version 1.7.1. Though I'm running with version 1.8.3. As for requests, Sid has version 2.3.0, which is the latest available. As I told you, I also updated urllib3 to version 1.9 (latest on PyPi), and I still have the error. So, finally, no, the issue is not that I'm using an old version of urllib3 or requests.

[cdent]

I'm not sure what you want me to do.

I have virtualenvs, running Python 3.4, where I am unable to replicate the problem. My local virtualenv where I just ran successful tests has the following modules:

$ pip list
httplib2 (0.9)
pip (1.5.6)
py (1.4.23)
pytest (2.6.0)
requests (2.3.0)
setuptools (2.1)

This tells me the problem is somewhere other than wsgi-intercept and also somewhere other than the most modern versions of the required modules. In other words it is either something in Sid or something in your environment, neither of which I have any control over, so I don't know what I can do.

If you have some suggestions, please let me know, but as I've said before I think you need to look to your environment.

Can you confirm that things are working when you do not use debs for the Python module requirements? Something like the following ought to get you a clean test run:

cd <the repo>
pyvenv-3.4 venv4
source venv34/bin/activate
pip install `python -c 'from setup import META; print(" ".join(META["extras_require"]["testing"]))'`
make test

[thomasgoirand]

Hi. I just tried with a cowbuilder (eg: a clean chroot) and the issue is exactly the same. I tried what you wrote above, but it didn't work. First, there's no Makefile, so "make test" doesn't work. Then running py.test-3.4 didn't work either (I get even more errors).

If you want, I can provide you with a build environment using Sid, so that you can try by yourself. If so, please get in touch with me on IRC (nickname zigo on both OFTC and Freenode).

[sashahart]

I made a VM of debian sid (starting from a clean jessie, then editing /etc/apt/sources.list to point to sid repos, then apt-get dist-upgrade), logged in as a non-root user, installed pip and virtualenv with the --user scheme, git cloned the repo, made a virtualenv with python3.4 and pip installed pytest httplib2 requests, as well as wsgi-intercept from the clone (not -e). 'make test' ran clean.
pip freeze output:
httplib2==0.9
py==1.4.23
pytest==2.6.0
requests==2.3.0
wsgi-intercept==0.8.0

another try in a new virtualenv, this time using minimum versions where specified, also ran clean
httplib2==0.9
py==1.4.23
pytest==2.4.0
requests==2.0.1

another try not in a virtualenv, with pip install --user for everything, had the same result

in another try, I made a new virtualenv, and ran pip install with the clone (again not -e)
then I went into the virtualenv, in lib/python3.4/site-packages/wsgi_intercept and there were no tests there to run
in that same virtualenv, using python3.4 -m pytest from the clone root, tests ran clean

So I think it's safe to guess that the problem is with sid versions or the specific environment, not the package as it exists on github and PyPI.

I then as root used apt-get install python3-{httplib2,pytest,requests} and verified that python3.4 was importing these from the installed path of /usr/lib/python3/dist-packages
Then I did a pip install --user of wsgi_intercept from the clone
this resulted in the following versions
httplib2==0.9
py==1.4.22
pytest==2.6.0
requests==2.3.0
urllib3==1.8.3
virtualenv==0.8.0

here on running python3.4 -m pytest, I got an error collecting test/test_requests.py because it imported wsgi_intercept at line 2, which in line 5 referenced requests.packages -
ImportError: No module named 'requests.packages'
but this doesn't look like the error reported at the top of this issue

new virtualenv, install exact same versions from PyPI, again tests run clean.

Despite some fiddling I still haven't been able to replicate the reported errors so clearly there are some variables we haven't dragged out yet.

[cdent]

@thomasgoirand has identified a bug in the packaging of wsgi-intercept, but I don't see that it has anything to do with this problem: There's no MANIFEST and thus the Makefile and some associated files are not in the tarball that ends up on PyPI. I'll fix that.

@sashahart: Thanks again for going the extra mile to try and work things out. I had hoped to try some of the same things but Monday was my wedding anniversary and Tuesday I had a followup appointment for the rock climbing accident I had a while ago, so I was away from any network availability.

I'd very much like to see us get this resolved. My own lack of expertise with Debian and Debian packaging is a big of handicap. @thomasgoirand, perhaps you can provide a recipe for the steps you've been following (just the steps, not the output) so we can see about replicating exactly those steps and narrow down the problem. I'm home for the rest of the week which gives me some leeway in terms of setting up or using a Debian VM. I'll look for zigo on IRC when I've got a moment and we can go from there. Onwards.

[cdent]

I've released a 0.8.1 with somewhat better packaging.

[thomasgoirand]

Hi. I have dropped the source package here:
http://xen01.node0184.gplhost.com/python-wsgi-intercept/

To build it, you must do:
apt-get install devscripts build-essential fakeroot
apt-get install build-dep python-wsgi-intercept
dget http://xen01.node0184.gplhost.com/python-wsgi-intercept/python-wsgi-intercept_0.8.1-2.dsc
dpkg-source -x python-wsgi-intercept_0.8.1-2.dsc
cd python-wsgi-intercept-0.8.1
dpkg-buildpackage

Anyway, one of the possible reasons for the issue could be that your package is using the embedded version of urllib3 from requests, which is removed from Debian. Therefore, you don't get the error that you should be having with the latest urllib3. You can have a look at:
debian/patches/01-system-urllib3.patch
from the source package above. As you can see, the package patches wsgi_intercept to use the real urllib3. Note that there's no other ways to do things in Debian, because we removed embedded copies. We do that all the time, because otherwise, it's a nightmare to maintain security with hundreds of copies everywhere (for which you don't even know what version is where). So in Debian, requests doesn't have a copy of urllib3. I am not certain that this is the problem, but that's what my nose is telling me. I would by the way warmly recommend that you do apply this patch upstream.

I am currently unsure about other patches (eg: old-urllib3.patch and run-test-offline.patch), but it doesn't look like they are influencing anything. I would also recommend that you have a look at them. The run-test-offline.patch should IMO be applied upstream too, as when packages are building, it's forbidden by the Debian policy to do any network access.

I hope this help, and that you'll be able to reproduce the issue. In any ways, I'd like to thank you a lot for your support, I can see that you seem to care about the Debian package, and it feels really good. Not everyone is trying to help as much as you did. Let's kill this @)(#$&*@ bug! :)

[sashahart]

a minor correction, apt-get install build-dep python-wsgi-intercept will not work, that probably should be:

apt-get build-dep python-wsgi-intercept

If we are going to spend time debugging the process used for building a Debian package and its accompanying metadata, that process needs to be described in sufficient detail that it can be independently reproduced rather than just copying flawed products. (example: what process did the .dsc file come from?)

I'm only speaking for myself here, but I don't think we should make blind changes to wsgi_intercept code (reproducibly passing its tests for some time) in order to address tracebacks seen in the Debian package build environment, without understanding what those tracebacks mean. For example, reflexively removing the mechanize intercept because the word 'mechanize' appeared in a traceback during a Debian package build with unknown parameters, and that was misunderstood to mean that mechanize was a production dependency for using wsgi_intercept. I have no attachment to mechanize, but making that kind of decision just to shut up a traceback without understanding its causes is a mistake.

There is not one bug discussed here, but two different tracebacks, both generated by issues in the downstream environment rather than wsgi_intercept code itself.

The issue with requests.packages that I saw isn't the originally reported issue, and it isn't upstream of the originally reported issue. wsgi_intercept imports requests.packages in order to do a monkeypatch that gets underneath requests itself; requests keeps its dependencies vendored under requests.packages because Kenneth Reitz's packages make a habit of vendoring dependencies internally rather than pinning versioned external dependencies, which we can't do anything about; Debian has a policy against vendoring we can't do anything about, so its version of requests is mangled, and the best we can do about that is introduce a dirty workaround, fine. But it does not clarify or resolve the cause of the other traceback, it's just a different subject.

The errors at the top of this issue have a different profile, in which urllib3 was throwing TypeError: init() got an unexpected keyword argument 'strict' - when one of its functions was being invoked by requests, inside the tests of the requests intercept. In other words, the urllib3 interface assumed by requests differed from the interface urllib3 was exposing. This is exactly the kind of thing you'd see if the version of urllib3 were way behind or way ahead of the version of requests. Which is an issue in the Debian package build environment that wsgi_intercept has no possible way of addressing!

Please, let's not bury the build-environment issue with urllib3 vs. requests versions (which wsgi_intercept didn't cause, but cannot do anything about) by conflating it with the requests.packages issue (which wsgi_intercept didn't cause, but could accommodate). If the build-environment issue has spontaneously disappeared and can't be reproduced due to some local ad hoc changes that's okay, though it would be best to know what happened, at least we need to know that it disappeared! But on the other hand, without further information then we can assume the reported build bug persists, and we definitely can't assume it will go away when we do a workaround for requests.packages. It is no good to arbitrarily mutate the working code for wsgi_intercept in response to bugs in a build process.

[cdent]

It is no good to arbitrarily mutate the working code for wsgi_intercept in response to bugs in a build process.

I agree, and unless I'm misreading the above, I hope noone is suggesting anything to contrary. Instead people just want it to work. My feeling is that wsgi-intercept works as it stands, in alignment with established Python modules. If it doesn't work in Debian then that is Debian's job to fix, in Debian.

It's not in keeping with Python standards to patch dependencies, you just use them. requests is weird in what it does and it seems to be causing some difficulties, but I'm not going to "patch the upstream" to deal with that in wsgi-intercept itself. This code follows the norms and will continue to do so.

In spite of all, I'd like to provide what aid I can to bring the problem to some useful resolution, thus the continued discussion. If there is a bug in the packaging of wsgi-intercept or its code or its tests I want us to fix that. If there's a problem elsewhere then I am happy to facilitate finding that, but I'm not going to change wsgi-intercept itself to accommodate bugs elsewhere.

[sashahart]

So cdent was exactly right in:
"it appears the issue you are seeing is with urllib3: urllib3/urllib3#238"
since strict= does not exist in python3.4 http.client.HTTPConnection but does exist in python2.7 httplib.HTTPConnection, and that difference is being propagated down to urllib3's interface

[sashahart]

... but then that was fixed in 1.7.1 and Sid is using 1.8.3, so supposedly the exact cited issue against urllib3 was fixed, it's just the same general area.

Our intercept is replacing ConnectionCls used by urllib3, and our replacement inherits from e.g. http.client.HTTPConnection/httplib.HTTPConnection. urllib3's "fix" in 1.8.3 is in its urllib3.connection.HTTPConnection.init which strips out the strict= kwarg. We have not duplicated that workaround in our replacement which inherits directly from http.client.HTTPConnection. So urllib3.connectionpool.HTTPConnectionPool._new_conn is invoking self.ConnectionCls with the expectation that it is urllib3.connection.HTTPConnection and will strip out the strict=, but instead we are giving it our own replacement which does not do that.
What's less clear right now is why this isn't happening in other environments where urllib3 is vendored under requests.

[sashahart]

urllib3.connectionpool.HTTPSConnectionPool from urllib3 vendored in recent requests uses this sequence

extra_params = {}
if not six.PY3:   # Python 2
    extra_params['strict'] = self.strict
extra_params.update(self.conn_kw)

conn = self.ConnectionCls(host=actual_host, port=actual_port,
 [... snip ...]
**extra_params)

the one in Debian's urllib3 uses this in the same position

conn = self.ConnectionCls(host=actual_host, port=actual_port,
 [... snip ...]
strict=self.strict, **self.conn_kw)

so in other words, the newer version of urllib3 fixes THIS problem and not just the problem in urllib3/urllib3#238

So perhaps Sid needs a newer urllib3?
Please let me know if I made a mistake here.

[cdent]

Okay, it seems we are getting somewhere, thanks @sashahart. I agree with you that there appears to be a problem with urllib3 in Sid. The latest pypi version is 1.9. If that version is not available, this patch to wsgi-intercept may fix it:

diff --git a/wsgi_intercept/__init__.py b/wsgi_intercept/__init__.py
index 9c4b94f..1e03474 100644
--- a/wsgi_intercept/__init__.py
+++ b/wsgi_intercept/__init__.py
@@ -507,6 +507,12 @@ class WSGI_HTTPSConnection(HTTPSConnection, WSGI_HTTPConnec
     Intercept all traffic to certain hosts & redirect into a WSGI
     application object.
     """
+    def __init__(self, *args, **kwargs):
+        if 'strict' in kwargs:
+            del kwargs['strict']
+
+        HTTPSConnection.__init__(self, *args, **kwargs)
+
     def get_app(self, host, port):
         """
         Return the app object for the given (host, port).

(can't use super on old versions, apparently there are some old-style classes in there).

I'd rather not make this workaround in the wsgi-intercept code itself, but if the patch works in the debian packaging, go for it?

@thomasgoirand Does any of this help?

[sashahart]

I did a PR for this part of it (which I don't really expect to merge) but you may be right that super won't work out, I did not take the time yet to inspect the whole hierarchy :/

I don't know what we want to do if anything about the requests.packages bit, we could do a try/except ImportError workaround but I think it's probably best to leave this to Debian to patch rather than doing this upstream of them, because it is Debian's specific policy which unvendored urllib3 from requests.

[cdent]

It blew up for me on testing with 2.7, but that may have been because of something else I had going on, I was moving quickly.

Let's wait to hear from @thomasgoirand and see where to go from here.

[thomasgoirand]

@sashahart
first sorry for the typo, indeed, it's "apt-get build-dep" and not "apt-get install". Then please let's not try to point fingers at who's wrong, and let's try to find a solution here. You're basically writing: "we shouldn't care, it's a Debian issue". Well, if you don't care, fine, but please let other help if they want. I'm just trying to do my best to keep wsg-intercept in good shape in Debian, nothing more, and there's nothing you or I can do about the bad vendoring practice of requests: we can just deal with it.

As for the .dsc, it's simply generated when you build with dpkg-buildpackage. Giving you the link to the .dsc file is just giving you a link to a source package, with dget as a wrapper around curl or wget to be able to get the 3 files which form a Debian source package (eg: the .orig.tar.gz, which is the archive from upstream, and the .dsc and the .debian.tar.gz, which form the Debian modifications). By the way, I could have explain the workflow using git and git-buildpackage (which is what I use every day), but I just thought it was easier for you to use a source package directly.

Also, no, installing the latest version of urllib3 (eg: verison 1.9) did not fix the issue, so it looks like you are wrong sashahart, that Sid would need the latest version of urllib3 (as I mentioned in earlier posts).

@cdent
Thanks a lot for your patch, it does fix the problem indeed! :)
Note that I have no problem keeping this patch as Debian specific, if it's too much a problem for you to have it upstream. The real issue we're having here is in fact requests vendoring python modules, and nothing else, and I think this is the perfect example of why this is a very bad thing to do, with sometimes unexpected consequences. Maybe I should try to convince upstream for requests that he should stop doing so.

This issue can be closed now. I've uploaded wsg-intercept 0.8.1 to Debian Sid. This upload prevents wsgi-intercept from being removed from Debian testing, which is a very good thing, especially close to the freeze of Jessie.

I'd like to thank you a lot for insisting, finding and fixing the issue. Really, you've been awesome!

[sashahart]

Mr. Goirand, you have been pointing fingers since you were flaming about this project on multiple public mailing lists and issuing ultimatums that you would blacklist wsgi-intercept and declare it as unmaintainable unless it made random changes that you demanded based on your personal misunderstandings, instead of asking questions first to understand things that may not be in your area of expertise.

I never said and would never say "we shouldn't care." I am actually happy to help anyone who wants to produce packages or whatever it is they want to do. But it is impossible to adequately address problems without understanding them first and it is certainly wrong to make arbitrary changes based on misunderstandings. I don't believe that the people working on this project, should be cowed into issuing an endless stream of otherwise unnecessary kludges and mea culpas to cover for weird issues specific to the Debian environment. It actually matters when these are bugs in the Debian environment or your build process because it affects the search for solutions. And when you want help with something, you could nicely ask for help with whatever you are doing rather than asserting that upstream itself has a fatal bug and throwing your weight around.

As it is, I spent hours setting up a Debian Sid VM and did most of the work of producing a Debian package myself just to reproduce your issue because you did not really document what you did, then I spent a few more hours tracking down the cause to make an informed solution and a little time after that on feature branches aimed at your issues. Insofar as I am doing a major part of your job for you and happily letting you take the credit for it, I would appreciate at least a basic level of courtesy.

I actually found the source of the strict= issue, and it is indeed fixed in newest urllib3, so it is really incredible that you would bother to tell me that I am wrong about that and you know more based on the latest underdescribed "failure" emerging from your underdescribed build environment, why am I not surprised?

This is cdent's repo, he deserves tons of credit for resurrecting wsgi-intercept and doing a lot of work on it on a continuing basis, and he's a very nice guy, so all around he certainly deserves thanks. But just the same it's very rude to insult me and then assign him all the credit for the footwork I have done.

[cdent]

I think before this gets too out of hand I'd like to say a couple of things that may help avoid this tension in the future:

• @sashahart has done a ton of awesome work on wsgi-intercept in the past few months. In fact pretty much all the substantial work. If there was an effective way to recognize that I would. There isn't, though, so really all I can do is make sure the contributors list reflects the work and say thanks: So, Sasha, sincerely: thank you very much.
• Generally speaking if someone reports an issue here my feeling is that I (or whoever) else should try to replicate it and if they can, fine, it will likely be easy to fix. If, however, it is hard or even impossible to replicate then the onus is on the reporter of the issue to narrow things down, effectively creating a minimal test case that demonstrates the problem. In this case I don't think that process happened as well as it could have: Sasha was too willing to do too much work too soon instead of waiting for Thomas to do the legwork.
• When the reporting process gets stuck we should help them, but that doesn't mean we should try to recreate their environment for them. This is especially the case when in the standard environment (straight up Python modules loaded from PyPI) the issue does not exist. This means we should answer questions and explore possibilities.
• I want wsgi-intercept to be packaged in distributions. I find it to be an extremely useful tool, that's why I resurrected it. My hope is that other people will find it useful. I don't, however, wants its essentially simplicity to be convoluted by the needs of the various distributions. In this case the "problem" is outside of wsgi-intercept, either in Debian or in Requests depending on your point of view. Those are the places where it should be resolved. If Thomas is happy to package with a patch to hack things to working order, that's fine with me, as along as it doesn't break the functionality of the tool.
• This project is pretty small-time. There's not going to be a lot of glory and credit that is gained by participating. We're here because we're either scratching an itch or have some other personal goal (learning, staving off boredom, fleshing out a github profile, being generous, who knows).

Thanks to you both for moving things along and making wsgi-intercept better.

[thomasgoirand]

sashahart, probably "ultimatum" is a bit of a strong word. I raised concerns that mechanize was hardly maintained, and impossible to deal with at the distribution level. It is IMO very important to keep python module maintainable. Having embedded copies (or as some say "vendorized") of python modules is a very real concern, especially when they are updated. This very issue shows how much trouble this can bring.

Though I'm sorry if you didn't like the way I expressed myself. I have to admit that I'm often the source of this kind of tensions, but please trust me, this isn't at all something I intend to do. I probably don't think enough about the consequences of my words, and how they may be perceived by the involved parties. I didn't intend to piss you off, and I'd like to apologies for it. I don't believe I have been insulting, but if you feel that way, then it's my fault. Thanks to you as well for your work, Sasha.