buildout / buildout.wheel Goto Github PK
View Code? Open in Web Editor NEWBuildout extension to provide wheel support
License: MIT License
Buildout extension to provide wheel support
License: MIT License
Some malformed wheels can be installed with pip
directly but not with wheel.
Also some people might prefer to only trust wheels from their own wheelhouses instead of directly from pip (say, they use a different compiler, or don't trust binaries coming from PyPI, only sources).
It would be nice if we could blacklist certain wheels or even whole hosts.
Considering we're monkey-patching distros_for_location
, this should be easy to implement.
I have enabled this extension on a project with a lot of dependencies, since a new dependency only provides wheels on pypi.
However, this seems to result in various odd bugs and glitches in some of the other dependencies, which are now being sourced from wheels, rather than the (s)dist that was being used previously).
Would it be possible to make this extension only try to install from a wheel if there is no other source available, or if the 'traditional' methods of install failed, rather than doing it for all deps? #14 could be an option - though this would require explicit white or blacklisting in advance.
I'm using Python 3.6.1, zc.buildout 2.9.3 and buildout.wheel 0.2.0.
I activated the buildout.wheel extension and tried to build an application that needs binary distributions; for each downloaded wheel, buildout errors out the following traceback:
Getting distribution for 'reportlab==3.4.0'.
While:
Installing sol.
Getting distribution for 'reportlab==3.4.0'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\buildout.py", line 2123, in main
getattr(buildout, command)(args)
File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\buildout.py", line 796, in install
installed_files = self[part]._call(recipe.install)
File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\buildout.py", line 1553, in _call
return f()
File "f:\solista\eggs\zc.recipe.egg-2.0.3-py3.6.egg\zc\recipe\egg\egg.py", line 126, in install
reqs, ws = self.working_set()
File "f:\solista\eggs\zc.recipe.egg-2.0.3-py3.6.egg\zc\recipe\egg\egg.py", line 84, in working_set
allow_hosts=self.allow_hosts)
File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\easy_install.py", line 913, in install
return installer.install(specs, working_set)
File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\easy_install.py", line 714, in install
for dist in self._get_dist(req, ws):
File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\easy_install.py", line 567, in _get_dist
shutil.rmtree(tmp)
File "c:\Python36\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\Python36\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "c:\Python36\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprdfug7p3get_dist\\reportlab-3.4.0-cp36-cp36m-win32.whl'
Apparently the wheel got installed under eggs/reportlab-3.4.0-cp36-cp36m-win32.ovo
, and indeed re-executing the buildout it goes on fetching the remaining packages, failing again on the next wheel.
Using buildout.wheel 0.2.0 with zc.buildout 2.9.3, github bash on Windows 10.
Steps to reproduce follow.
First Prepare a buildout.cfg in empty directory, bootstrap buildout into a new virtualenv, download the Django wheel to local directory.
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ ls -l
total 1
-rw-r--r-- 1 day.barr 1049089 139 Apr 23 21:16 buildout.cfg
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ cat buildout.cfg
[buildout]
extensions = buildout.wheel
parts = django
[django]
recipe = zc.recipe.egg
eggs = Django==1.11
index =
find-links = .
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ mkvirtualenv buildout.wheel.test
New python executable in C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\python.exe
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\predeactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\postdeactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\preactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\postactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\get_env_details
virtualenvwrapper.user_scripts could not run "C:\Users\day.barr\.virtualenvs\premkvirtualenv": [Error 2] The system cannot find the file specified
virtualenvwrapper.user_scripts could not run "C:\Users\day.barr\.virtualenvs\preactivate": [Error 2] The system cannot find the file specified
virtualenvwrapper.user_scripts could not run "C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\preactivate": [Error 2] The system cannot find the file specified
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ curl -O 'https://raw.githubusercontent.com/buildout/buildout/2.9.3/bootstrap/bootstrap.py'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7467 100 7467 0 0 17047 0 --:--:-- --:--:-- --:--:-- 17694
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ python bootstrap.py
ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check https://github.com/pypa/setuptools/issues/581 for more info; use pip to install setuptools
Downloading https://pypi.io/packages/source/s/setuptools/setuptools-33.1.1.zip
Extracting in c:\users\day~1.bar\appdata\local\temp\tmpuagfwz
Now working in c:\users\day~1.bar\appdata\local\temp\tmpuagfwz\setuptools-33.1.1
Building a Setuptools egg in c:\users\day~1.bar\appdata\local\temp\bootstrap-d93yfz
warning: no files found matching '*' under directory 'setuptools\_vendor'
c:\users\day~1.bar\appdata\local\temp\bootstrap-d93yfz\setuptools-33.1.1-py2.7.egg
no previously-included directories found matching 'doc'
no previously-included directories found matching 'old-tutorial'
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\bin'.
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\parts'.
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\develop-eggs'.
Generated script 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\bin\\buildout'.
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ pip download --only-binary :all: Django==1.11
Collecting Django==1.11
Downloading Django-1.11-py2.py3-none-any.whl (6.9MB)
100% |################################| 6.9MB 2.3MB/s
Saved c:\users\day.barr\dev\buildout.wheel.test\django-1.11-py2.py3-none-any.whl
Collecting pytz (from Django==1.11)
Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
100% |################################| 491kB 1.6MB/s
Saved c:\users\day.barr\dev\buildout.wheel.test\pytz-2017.2-py2.py3-none-any.whl
Successfully downloaded Django pytz
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ ls -la
total 7293
drwxr-xr-x 1 day.barr 1049089 0 Apr 24 09:38 ./
drwxr-xr-x 1 day.barr 1049089 0 Apr 24 09:30 ../
-rw-r--r-- 1 day.barr 1049089 6942445 Apr 24 09:38 Django-1.11-py2.py3-none-any.whl
drwxr-xr-x 1 day.barr 1049089 0 Apr 24 09:33 bin/
-rw-r--r-- 1 day.barr 1049089 7467 Apr 24 09:33 bootstrap.py
-rw-r--r-- 1 day.barr 1049089 139 Apr 23 21:16 buildout.cfg
drwxr-xr-x 1 day.barr 1049089 0 Apr 24 09:33 develop-eggs/
drwxr-xr-x 1 day.barr 1049089 0 Apr 24 09:33 parts/
-rw-r--r-- 1 day.barr 1049089 484917 Apr 24 09:38 pytz-2017.2-py2.py3-none-any.whl
Now try to buildout install
(-U
to be sure I'm not using my user prefs (cache etc)):
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ bin/buildout -U
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\eggs'.
Getting distribution for 'buildout.wheel'.
warning: no files found matching 'shootout.cfg'
Got buildout.wheel 0.2.0.
Getting distribution for 'six'.
no previously-included directories found matching 'documentation\_build'
zip_safe flag not set; analyzing archive contents...
six: module references __path__
Got six 1.10.0.
Getting distribution for 'zc.recipe.egg>=2.0.0a3'.
Got zc.recipe.egg 2.0.3.
Installing django.
Not found: /Django/
Not found: /Django/
Getting distribution for 'Django==1.11'.
While:
Installing django.
Getting distribution for 'Django==1.11'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\buildout.py", line 2123, in main
getattr(buildout, command)(args)
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\buildout.py", line 796, in install
installed_files = self[part]._call(recipe.install)
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\buildout.py", line 1553, in _call
return f()
File "c:\users\day.barr\dev\buildout.wheel.test\eggs\zc.recipe.egg-2.0.3-py2.7.egg\zc\recipe\egg\egg.py", line 126, in install
reqs, ws = self.working_set()
File "c:\users\day.barr\dev\buildout.wheel.test\eggs\zc.recipe.egg-2.0.3-py2.7.egg\zc\recipe\egg\egg.py", line 84, in working_set
allow_hosts=self.allow_hosts)
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 913, in install
return installer.install(specs, working_set)
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 665, in install
for dist in self._get_dist(requirement, ws):
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 563, in _get_dist
dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 1698, in _move_to_eggs_dir_and_compile
unpacker(dist.location, tmp_dest)
File "c:\users\day.barr\dev\buildout.wheel.test\eggs\buildout.wheel-0.2.0-py2.7.egg\buildout\wheel\__init__.py", line 25, in unpack_wheel
WheelInstaller(spec).install_into(dest)
File "c:\users\day.barr\dev\buildout.wheel.test\eggs\buildout.wheel-0.2.0-py2.7.egg\buildout\wheel\__init__.py", line 67, in install_into
self.wheel.install(overrides=overrides)
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 268, in install
if self.parsed_wheel_info['Root-Is-Purelib'] == 'true':
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\decorator.py", line 17, in __get__
val = self.wrapped(inst)
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 238, in parsed_wheel_info
return read_pkg_info_bytes(self.zipfile.read(self.wheelinfo_name))
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\decorator.py", line 17, in __get__
val = self.wrapped(inst)
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 232, in zipfile
self.verify(vzf)
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 385, in verify
record = zipfile.read(record_name)
File "c:\python27\Lib\zipfile.py", line 938, in read
return self.open(name, "r", pwd).read()
File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 437, in open
ef = zipfile.ZipFile.open(self, name_or_info, mode, pwd)
File "c:\python27\Lib\zipfile.py", line 964, in open
zinfo = self.getinfo(name)
File "c:\python27\Lib\zipfile.py", line 912, in getinfo
'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'django-1.11.dist-info/RECORD' in the archive"
Note that the path in the .whl
archive that we should be looking for is 'Django-1.11.dist-info/RECORD' (note uppercase D
).
I am getting the exception below when loading extensions in buildout using zc.buildout-2.11.3 and pip 10.0.0
Thanks
Carlos
Got buildout.wheel 0.2.0.
While:
Installing.
Loading extensions.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/zc.buildout-2.11.3-py2.7.egg/zc/buildout/buildout.py", line 2127, in main
getattr(buildout, command)(args)
File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/zc.buildout-2.11.3-py2.7.egg/zc/buildout/buildout.py", line 638, in install
self._load_extensions()
File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/zc.buildout-2.11.3-py2.7.egg/zc/buildout/buildout.py", line 1174, in _load_extensions
ep.load()(self)
File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 2324, in load
return self.resolve()
File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 2330, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/buildout.wheel-0.2.0-py2.7.egg/buildout/wheel/__init__.py", line 9, in <module>
import pip.pep425tags
ImportError: No module named pep425tags
Running ./bin/buildout
gave this error:-
Using unpack_wheel() shim over the deprecated wheel_to_egg() hook.
Please update your wheel extension implementation for one that installs a .whl
handler in zc.buildout.easy_install.UNPACKERS
While:
Installing.
Loading extensions.
Getting distribution for 'mr.developer'.
Error: Wheels are not supported
Using setuptools v38.1.0 still work. Probably caused by pypa/setuptools#1200.
Create a manylinux distro of the extension sample and check it in.
Make a new test (probably that extends/calls the existing one) that only runs on linux and that uses the anylinux wheel rather than building one.
Otherwise I can't have a fork of this and of the PYPA wheel at the same time in github.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.