ipython fails to install with pip-accel: pkg_resources.DistributionNotFound: gnureadline about pip-accel

[jquast]

also, this is on OSX

(xxx)$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10
BuildVersion:   14A389

[xolox]

Thanks for the bug report and sorry for not following up on this earlier. I did investigate a bit but haven't found anything conclusive. I suspect that this issue is the first time that I see that a user of pip-accel runs into "cache invalidation" issues. If that doesn't make sense to you, consider the following:

• pip-accel is basically pip + caching
• caching happens on several levels and one of those levels is the "binary cache" containing Python packages that use *.so files
• to use those *.so files your system needs to be in a state that is ABI compatible with your system at the time that the *.so files were compiled (this is an oversimplification but it explains the basic idea)

To actually verify that this is what happened in your case I need to be able to reproduce it. My wife is running Mac OS X as well so with some luck I'll be able to reproduce this issue on her system. To be continued :-)

[jquast]

Also reproduces on "Mavericks",

ProductName:    Mac OS X
ProductVersion: 10.9.5
BuildVersion:   13F34

e-mailed you an offer to provide ssh access to such machine.

[xolox]

Problem analysis

The SSH access helped to quickly narrow down the cause of this problem, so thanks for that :-). It's not at all related to cached binary distributions becoming invalid, instead it has to do with the (IMHO) strange behavior of the setup.py script of IPython:

1. When pip downloads and unpacks the IPython source distribution, the setup.py script of IPython reports a set of requirements in the install_requires keyword to the setup() function. Based on this list of requirements pip extends the requirement set and downloads missing source distributions. This process continues until all dependencies are satisfied.
2. When step 1 finishes pip-accel assumes that the requirement set is complete, i.e. all requirements have been satisfied. To me this seems like a reasonable assumption to make, because pip is also under the assumption that the dependency set is complete (this is the point where pip starts installing packages if it weren't for pip-accel).
3. What pip-accel adds in behavior compared to plain pip is that after step 1 pip uses python setup.py install to install packages while pip-accel instead runs python setup.py bdist_dumb (or python setup.py bdist) and caches the resulting binary distribution. This is what makes pip-accel faster. When the setup.py script of IPython sees bdist* in sys.argv (the command line arguments) it unconditionally adds gnureadline to install_requires. Because pip never runs any bdist* commands it never sees this dependency, so the dependency is not part of the dependency set. But it is written to the metadata included with the binary distribution, so the first time IPython is used after pip-accel installs it, setuptools notices the missing requirement and complains with:

Traceback (most recent call last):
  File "/tmp/issue-34-env/bin/ipython", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 2829, in <module>
    working_set = WorkingSet._build_master()
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 449, in _build_master
    ws.require(__requires__)
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 742, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gnureadline

Coming up with a solution

It might be possible for pip-accel to work around this issue but it would be a nasty hack, not an elegant and general solution. I think our best option is to convince the IPython developers to change their setup.py script. I just created a pull request to accomplish this, see ipython/ipython#7047.

[xolox]

@jquast: As a short term workaround you can add gnureadline to the pip command line or requirements file. It's a bit pragmatic but it will solve the immediate problem until ipython/ipython#7047 is merged (assuming the IPython developers are willing to merge it :-).

[xolox]

@jquast: My pull request ipython/ipython#7047 has been merged so the changes will eventually find their way to PyPI and until that time there is the (short term) workaround of adding gnureadline to the pip command line or requirements file. I believe this resolves the issue you reported.

[jquast]

Great job, I'm satisfied. Thank you @xolox