Do not run `egg_info` to each package in requirements list before installing the previous packages about pip

[hltbra]

There was a new report of this issue. The issue is #272.

[carljm]

The suggestion made here (installing each package fully before moving on to the next package) is not an option. One of the core advantages of pip over easy_install is "look before you leap" - pip ensures that all dependencies can be found, unpacked, and have working setup.py before it installs any of them. This avoids leaving the system in a half-broken state when an error is hit after three-quarters of the requirements have already been installed.

This does cause an issue when one package depends on another package in order to even run its setup.py. Really, this is a broken setup.py, and should be fixed in scipy.

For a workaround in pip, the idea presented in #272 is the only one I've seen so far that I think might be an ok option -- allowing some kind of per-line marker in a requirements file that means "install this in a separate first round of installs". I'm not sure I like this, because it feels like going down the road of turning requirements files into a mini-scripting-language for pip, and I'm not sure we want to go that way. But if someone actually turned this proposal into a patch, I'd be interested in seeing how it looks.

[hltbra]

Hi Carl,

I agree with not adding this kind of behavior to pip. Now I see if we do it or stuff like this, we are not fixing the right place, but only hiding the real problem.

Maybe SciPy folks don't know about this problem... It would be good to send them an email talking about this issue.

I would not like to do it these days. Can someone do it?

[edwardabraham]

Thanks for the response ... this makes sense to me. I have raised a ticket on scipy
http://projects.scipy.org/scipy/ticket/1429

In the meantime I simply split my requirements file in two and ran pip twice. Not too arduous!

[cournape]

@carljm: this is not a bug in scipy: scipy setup.py requires numpy, that is numpy is a build dependency of scipy. This problem is inherent to any package which requires another package to build - without pip supporting this, there is nothing we can do in scipy to solve this.

[carljm]

Hi David,

scipy setup.py requires numpy, that is numpy is a build dependency of scipy.

The two clauses there are not equivalent. A build dependency is not the same thing as a running-setup.py dependency.
Even if scipy requires numpy to build, it still should not require it unconditionally in setup.py -- it should be possible to get package metadata out of setup.py (which is all pip is doing here) without having numpy available.

That said, I see your point that even if that were fixed, the scipy install would still fail, just later on at the build phase.

The only currently-available means for explicitly expressing a build dependency in Python packaging is setuptools' "setup_requires" keyword, which pip already supports. I understand that using setuptools and setup_requires may not be feasible for scipy, but I'm not sure what else pip can or should do to support this case.

Closing this bug -- it's clear that serial installation is the only answer here, and I'm not convinced that pip's requirements file format should be extended to support multiple serial installation within a single requirements file. What we really need is a standard way to express build dependencies in Python packaging - I'll inquire on the distutils2 mailing list about that.

[carljm]

Actually, I think if scipy really only had a build dependency on numpy, rather than an importing-setup.py dependency, it would work fine in pip, as long as numpy was listed above scipy in requirements.txt. We don't separate build and install steps, and IIRC current pip versions now process installations in order, so if the egg_info step for scipy could succeed, numpy would be installed before pip tries to build/install scipy.

[cournape]

I don't understand the distinction you make between setup.py dependency vs build dependency. setup.py is used to build scipy, and the setup.py certainly needs to import numpy.distutils for monkey patching to work. This is not particular to scipy, but will be the same for any package requiring distutils extensions provided by 3rd party tools (starting from setuptools, BTW).

[carljm]

@cournape - setup.py is used to build scipy, but it is not only used to build scipy. It is also used for much simpler things, such as e.g. discovering the version of scipy via "python setup.py --version". What pip is doing ("python setup.py egg_info") is simply an extended version of that: metadata discovery.

Scipy should not require an external package in order to make its core metadata discoverable via setup.py. The import of numpy.distutils should be conditional on the use of a setup.py command (e.g. build or install) that actually requires it.

[merwok]

Scipy should not require an external package in order to make its core metadata discoverable via setup.py. The import of
numpy.distutils should be conditional on the use of a setup.py command (e.g. build or install) that actually requires it.

Then you would have the setup script look at sys.argv? It would be messy.

[carljm]

Then you would have the setup script look at sys.argv? It would be messy.

Yes, it sure would be messy. Packaging is hard - this is why we are
moving to static metadata in d2 :-) But it's better than having a
package which can't even construct its own metadata without external
dependencies. There's no reasonable workaround pip can be expected to
make in this case.

[cournape]

Someone can try to add more hacks to our distutils extensions if they want to support this.

This (and many other things) will go away once we use our own packaging infrastructure in scipy, so I am not really interested in adding more workaround for distutils limitations.

[astrofrog]

Would it not be possible to have an option to make pip install packages sequentially for problematic cases? Note that I'm not suggesting that it be the default behavior, but more the equivalent of --force or something like that that only people who know what they are doing will use?

[carljm]

@astrofrog Not inherently opposed to that, feel free to work on an implementation and see what it looks like.

[ilblackdragon]

FYI: Just proposed a change in setup.py in scipy: scipy/scipy#453
Hopefully they will accept it. For now using my branch.
Without this fix, it's really annoying to work with tox, when scipy is in requirements.

[wrought]

What about a more helpful error response that recommends to manually install dependencies and then run through requirements.txt a la http://stackoverflow.com/a/11015904/1445241

[ilblackdragon]

mattsenate: To show this message pip will need to parse ImportError exception output from python setup.py egg_info command it executes - this may lead to suppressing issue of misspelled module name and some other caveats.

If the world was perfect - everybody would not import external modules if egg_info command is executed.

[bjodah]

I want to second what @wrought said. I got bitten here - maybe you could just print a hint that this might be the issue and re-raise the exception?

EDIT: is it enough to check for 'build', 'build_ext', 'install' in sys.argv and only for these cases import from dependencies?

[ilblackdragon]

@bjodah It's better to handle egg_info, clean and help commands as special case (at least this is what has been done in scipy (https://github.com/scipy/scipy/blob/master/setup.py) and some other libraries that depend on numpy):

if '--help' in sys.argv[1:] or \
    sys.argv[1] in ('--help-commands', 'egg_info', 'clean', '--version'):
    ...
else:
    import numpy
    ...

[griesmey]

Was this ever fixed? I'm having the same issue when trying to install scipy from the requirements file using wheel...

[shindere]

Robert Griesmeyer (2016/03/03 15:56 -0800):

Was this ever fixed? I'm having the same issue when trying to install
scipy from the requirements file using wheel...

No idea, sorry.

[xavfernandez]

I don't think this needs fixing on pip's end.
Package should declare their setup_requires and/or users should either provide them or expect setuptools to do so.