buildout / buildout.wheel

Buildout extension to provide wheel support

Buildout

Buildout is a project designed to solve 2 problems:

1. Application-centric assembly and deployment

   Assembly runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.).

   Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done.

   Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done.

   Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby.

2. Repeatable assembly of programs from Python software distributions

   Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout.

   Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed.

To learn more about buildout, including how to use it, see http://docs.buildout.org/.

buildout.wheel's Issues

Only fall back to wheel if no other dists

I have enabled this extension on a project with a lot of dependencies, since a new dependency only provides wheels on pypi.

However, this seems to result in various odd bugs and glitches in some of the other dependencies, which are now being sourced from wheels, rather than the (s)dist that was being used previously).

Would it be possible to make this extension only try to install from a wheel if there is no other source available, or if the 'traditional' methods of install failed, rather than doing it for all deps? #14 could be an option - though this would require explicit white or blacklisting in advance.

Cannot install wheel from local file system if name not all lowercase on Windows

Using buildout.wheel 0.2.0 with zc.buildout 2.9.3, github bash on Windows 10.

Steps to reproduce follow.

First Prepare a buildout.cfg in empty directory, bootstrap buildout into a new virtualenv, download the Django wheel to local directory.

day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ ls -l
total 1
-rw-r--r-- 1 day.barr 1049089 139 Apr 23 21:16 buildout.cfg

day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ cat buildout.cfg
[buildout]
extensions = buildout.wheel
parts = django

[django]
recipe = zc.recipe.egg
eggs = Django==1.11
index =
find-links = .
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test

$ mkvirtualenv buildout.wheel.test
New python executable in C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\python.exe
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\predeactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\postdeactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\preactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\postactivate
virtualenvwrapper.user_scripts creating C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\get_env_details
virtualenvwrapper.user_scripts could not run "C:\Users\day.barr\.virtualenvs\premkvirtualenv": [Error 2] The system cannot find the file specified
virtualenvwrapper.user_scripts could not run "C:\Users\day.barr\.virtualenvs\preactivate": [Error 2] The system cannot find the file specified
virtualenvwrapper.user_scripts could not run "C:\Users\day.barr\.virtualenvs\buildout.wheel.test\Scripts\preactivate": [Error 2] The system cannot find the file specified
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ curl -O 'https://raw.githubusercontent.com/buildout/buildout/2.9.3/bootstrap/bootstrap.py'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7467  100  7467    0     0  17047      0 --:--:-- --:--:-- --:--:-- 17694
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ python bootstrap.py
ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check https://github.com/pypa/setuptools/issues/581 for more info; use pip to install setuptools
Downloading https://pypi.io/packages/source/s/setuptools/setuptools-33.1.1.zip
Extracting in c:\users\day~1.bar\appdata\local\temp\tmpuagfwz
Now working in c:\users\day~1.bar\appdata\local\temp\tmpuagfwz\setuptools-33.1.1
Building a Setuptools egg in c:\users\day~1.bar\appdata\local\temp\bootstrap-d93yfz
warning: no files found matching '*' under directory 'setuptools\_vendor'
c:\users\day~1.bar\appdata\local\temp\bootstrap-d93yfz\setuptools-33.1.1-py2.7.egg
no previously-included directories found matching 'doc'
no previously-included directories found matching 'old-tutorial'
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\bin'.
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\parts'.
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\develop-eggs'.
Generated script 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\bin\\buildout'.
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ pip download --only-binary :all: Django==1.11
Collecting Django==1.11
  Downloading Django-1.11-py2.py3-none-any.whl (6.9MB)
    100% |################################| 6.9MB 2.3MB/s
  Saved c:\users\day.barr\dev\buildout.wheel.test\django-1.11-py2.py3-none-any.whl
Collecting pytz (from Django==1.11)
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |################################| 491kB 1.6MB/s
  Saved c:\users\day.barr\dev\buildout.wheel.test\pytz-2017.2-py2.py3-none-any.whl
Successfully downloaded Django pytz
(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ ls -la
total 7293
drwxr-xr-x 1 day.barr 1049089       0 Apr 24 09:38 ./
drwxr-xr-x 1 day.barr 1049089       0 Apr 24 09:30 ../
-rw-r--r-- 1 day.barr 1049089 6942445 Apr 24 09:38 Django-1.11-py2.py3-none-any.whl
drwxr-xr-x 1 day.barr 1049089       0 Apr 24 09:33 bin/
-rw-r--r-- 1 day.barr 1049089    7467 Apr 24 09:33 bootstrap.py
-rw-r--r-- 1 day.barr 1049089     139 Apr 23 21:16 buildout.cfg
drwxr-xr-x 1 day.barr 1049089       0 Apr 24 09:33 develop-eggs/
drwxr-xr-x 1 day.barr 1049089       0 Apr 24 09:33 parts/
-rw-r--r-- 1 day.barr 1049089  484917 Apr 24 09:38 pytz-2017.2-py2.py3-none-any.whl

Now try to buildout install (-U to be sure I'm not using my user prefs (cache etc)):

(buildout.wheel.test)
day.barr@win10-laptop MINGW64 ~/dev/buildout.wheel.test
$ bin/buildout -U
Creating directory 'C:\\Users\\day.barr\\dev\\buildout.wheel.test\\eggs'.
Getting distribution for 'buildout.wheel'.
warning: no files found matching 'shootout.cfg'
Got buildout.wheel 0.2.0.
Getting distribution for 'six'.
no previously-included directories found matching 'documentation\_build'
zip_safe flag not set; analyzing archive contents...
six: module references __path__
Got six 1.10.0.
Getting distribution for 'zc.recipe.egg>=2.0.0a3'.
Got zc.recipe.egg 2.0.3.
Installing django.
Not found: /Django/
Not found: /Django/
Getting distribution for 'Django==1.11'.
While:
  Installing django.
  Getting distribution for 'Django==1.11'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\buildout.py", line 2123, in main
    getattr(buildout, command)(args)
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\buildout.py", line 796, in install
    installed_files = self[part]._call(recipe.install)
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\buildout.py", line 1553, in _call
    return f()
  File "c:\users\day.barr\dev\buildout.wheel.test\eggs\zc.recipe.egg-2.0.3-py2.7.egg\zc\recipe\egg\egg.py", line 126, in install
    reqs, ws = self.working_set()
  File "c:\users\day.barr\dev\buildout.wheel.test\eggs\zc.recipe.egg-2.0.3-py2.7.egg\zc\recipe\egg\egg.py", line 84, in working_set
    allow_hosts=self.allow_hosts)
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 913, in install
    return installer.install(specs, working_set)
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 665, in install
    for dist in self._get_dist(requirement, ws):
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 563, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "c:\users\day.barr\.buildout\eggs\zc.buildout-2.9.3-py2.7.egg\zc\buildout\easy_install.py", line 1698, in _move_to_eggs_dir_and_compile
    unpacker(dist.location, tmp_dest)
  File "c:\users\day.barr\dev\buildout.wheel.test\eggs\buildout.wheel-0.2.0-py2.7.egg\buildout\wheel\__init__.py", line 25, in unpack_wheel
    WheelInstaller(spec).install_into(dest)
  File "c:\users\day.barr\dev\buildout.wheel.test\eggs\buildout.wheel-0.2.0-py2.7.egg\buildout\wheel\__init__.py", line 67, in install_into
    self.wheel.install(overrides=overrides)
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 268, in install
    if self.parsed_wheel_info['Root-Is-Purelib'] == 'true':
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\decorator.py", line 17, in __get__
    val = self.wrapped(inst)
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 238, in parsed_wheel_info
    return read_pkg_info_bytes(self.zipfile.read(self.wheelinfo_name))
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\decorator.py", line 17, in __get__
    val = self.wrapped(inst)
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 232, in zipfile
    self.verify(vzf)
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 385, in verify
    record = zipfile.read(record_name)
  File "c:\python27\Lib\zipfile.py", line 938, in read
    return self.open(name, "r", pwd).read()
  File "c:\users\day.barr\.virtualenvs\buildout.wheel.test\lib\site-packages\wheel\install.py", line 437, in open
    ef = zipfile.ZipFile.open(self, name_or_info, mode, pwd)
  File "c:\python27\Lib\zipfile.py", line 964, in open
    zinfo = self.getinfo(name)
  File "c:\python27\Lib\zipfile.py", line 912, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'django-1.11.dist-info/RECORD' in the archive"

Note that the path in the .whl archive that we should be looking for is 'Django-1.11.dist-info/RECORD' (note uppercase D).

ImportError: No module named pip.pep425tags

I am getting the exception below when loading extensions in buildout using zc.buildout-2.11.3 and pip 10.0.0

Thanks

Carlos

Got buildout.wheel 0.2.0.
While:
  Installing.
  Loading extensions.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/zc.buildout-2.11.3-py2.7.egg/zc/buildout/buildout.py", line 2127, in main
    getattr(buildout, command)(args)
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/zc.buildout-2.11.3-py2.7.egg/zc/buildout/buildout.py", line 638, in install
    self._load_extensions()
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/zc.buildout-2.11.3-py2.7.egg/zc/buildout/buildout.py", line 1174, in _load_extensions
    ep.load()(self)
  File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "build/bdist.macosx-10.13-x86_64/egg/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/buildout.wheel-0.2.0-py2.7.egg/buildout/wheel/__init__.py", line 9, in <module>
    import pip.pep425tags
ImportError: No module named pep425tags

Please rename repo to buildout.wheel

Otherwise I can't have a fork of this and of the PYPA wheel at the same time in github.

Add manylinux test

• Create a manylinux distro of the extension sample and check it in.

• Make a new test (probably that extends/calls the existing one) that only runs on linux and that uses the anylinux wheel rather than building one.

Wheel blacklisting

Some malformed wheels can be installed with pip directly but not with wheel.

Also some people might prefer to only trust wheels from their own wheelhouses instead of directly from pip (say, they use a different compiler, or don't trust binaries coming from PyPI, only sources).

It would be nice if we could blacklist certain wheels or even whole hosts.

Considering we're monkey-patching distros_for_location, this should be easy to implement.

Installing from wheel failed after upgrading to setuptools v38.2.0

Running ./bin/buildout gave this error:-

Using unpack_wheel() shim over the deprecated wheel_to_egg() hook.
Please update your wheel extension implementation for one that installs a .whl
handler in zc.buildout.easy_install.UNPACKERS
While:
  Installing.
  Loading extensions.
  Getting distribution for 'mr.developer'.
Error: Wheels are not supported

Using setuptools v38.1.0 still work. Probably caused by pypa/setuptools#1200.

Error when installing a wheel on Windows

I'm using Python 3.6.1, zc.buildout 2.9.3 and buildout.wheel 0.2.0.

I activated the buildout.wheel extension and tried to build an application that needs binary distributions; for each downloaded wheel, buildout errors out the following traceback:

Getting distribution for 'reportlab==3.4.0'.
While:
  Installing sol.
  Getting distribution for 'reportlab==3.4.0'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\buildout.py", line 2123, in main
    getattr(buildout, command)(args)
  File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\buildout.py", line 796, in install
    installed_files = self[part]._call(recipe.install)
  File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\buildout.py", line 1553, in _call
    return f()
  File "f:\solista\eggs\zc.recipe.egg-2.0.3-py3.6.egg\zc\recipe\egg\egg.py", line 126, in install
    reqs, ws = self.working_set()
  File "f:\solista\eggs\zc.recipe.egg-2.0.3-py3.6.egg\zc\recipe\egg\egg.py", line 84, in working_set
    allow_hosts=self.allow_hosts)
  File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\easy_install.py", line 913, in install
    return installer.install(specs, working_set)
  File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\easy_install.py", line 714, in install
    for dist in self._get_dist(req, ws):
  File "f:\solista\eggs\zc.buildout-2.9.3-py3.6.egg\zc\buildout\easy_install.py", line 567, in _get_dist
    shutil.rmtree(tmp)
  File "c:\Python36\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\Python36\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\Python36\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\IEUser\\AppData\\Local\\Temp\\tmprdfug7p3get_dist\\reportlab-3.4.0-cp36-cp36m-win32.whl'

Apparently the wheel got installed under eggs/reportlab-3.4.0-cp36-cp36m-win32.ovo, and indeed re-executing the buildout it goes on fetching the remaining packages, failing again on the next wheel.