pypa / cibuildwheel Goto Github PK

We've just finished transitioning our repository (afdko) to using cibuildwheel. The process of building and testing the wheels is now much simple — thank you! — but for some reason the file size of the Windows wheel is now noticeably larger (2.65 MB → 3.76 MB). Our wheels contain both Python-based tools and C-based tools (not Python extensions). The binaries of the latter is what grew. Generating the wheel on a local machine also yields a size comparable to the cibuildwheel output.

In our quest to understand the cause of the size difference, we compared build logs, environment variables, and even the hex data of the binaries themselves, to no avail.

We're out of ideas, so if anyone has a clue of why the Windows wheel file is larger, please share.
Thank you.

--

Before cibuildwheel config:

• wheels (v2.7.3a5)
• appveyor.yml

After cibuildwheel config:

• wheels (v2.7.3b0)
• appveyor.yml

Currently, the integration tests are checking for success by seeing if a hardcoded number of wheels is built: https://github.com/joerick/cibuildwheel/blob/master/bin/run_test.py#L25.

This should be changed into a condition that each test can impose independently.

See also #101 (comment) for more details.

I'm setting up Twisted with cibuildwheel over in twisted/twisted#1051. I haven't really tested anything but I got Windows wheels building on AppVeyor and Linux wheels on Travis so far. Next step? CircleCI for OSX wheels... Before I dive in, any comments on any other efforts to do this? Interest in trying to do some of the work? Suggestions how I get started? I honestly haven't done anything quite yet but I like to raise the topic publicly first in case there's advice or opinions worth sharing.

Cheers,
-kyle

P.S. Thanks so much for cibuildwheel, it certainly made this all go a lot quicker so far.

I'm trying to run tests from the tests directory like so: CIBW_TEST_COMMAND="cd {project}/tests && pytest --cov". This works on Travis' Linux CI but not on the macOS CI.

See e.g. https://travis-ci.org/adobe-type-tools/psautohint/jobs/400028035#L401

The upshot is that the command silently does nothing on the default macOS sh (bash would complain about too many args to cd).

https://github.com/joerick/cibuildwheel/blob/9372e37b1886be448ca6b67a14dc6708657ac4a8/README.md#L35

It will try to install packages globally; this is no good.

I haven't looked and this isn't any priority for me but... couldn't this installation be optional? I might tend towards having it disabled by default but I get there will be arguments both ways.

Questions from @zfrenchee in #4 - I'm posting these to a new thread for better visibility

1. You say in the "Manual Deploy" instructions:

# Go and download your wheel files from wherever you put them.

but I'm not sure in my current config where I'm even putting my wheels. My mental model for travis is that it's spinning up a VM, doing all these tasks, then throwing away all the data and reporting back whether it succeeded. It's putting all the wheels in wheelhouse/ but I'm not sure how to access that directory, and I'm assuming it's removed once the travis job completes. Is that mental model incorrect? How would I go about downloading my wheels from wheelhouse/?

2. Wheelhouse uploader, which you recommend as the semi-automatic method, seems to mainly be built to upload wheels to clouds like s3, and then perhaps download them later to be uploaded to pypi. But it seems like s3 buckets are required to make it work. Am I reading that wrong -- Is there a way to use wheelhouse uploader to upload wheels from the travis VM?

Hello,

although my package builds normally on Linux under Travis, when running it on OSX I get the strange error:

(most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/delocate-listdeps", line 12, in
from delocate import tree_libs, wheel_libs, version
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delocate/init.py", line 3, in
from .delocating import delocate_path, delocate_wheel, patch_wheel
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delocate/delocating.py", line 18, in
from .wheeltools import rewrite_record, InWheel
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/delocate/wheeltools.py", line 14, in
from wheel.util import urlsafe_b64encode, open_for_csv, native
ImportError: cannot import name open_for_csv

for which I couldn't find a reference online.
Is it because a library is downgraded on OSX (as another one get's upgraded)?
Maybe delocate should be upgraded or wheel, although I don't have a clue :)
I am using the default osx image of travis.

Thanks for the support and this amazingly helpful package!

I have the following:

  - CIBW_TEST_COMMAND='cd {project}; python -m pytest -v test'

And on Linux, it works, but on macOS, it's broken, because:

>>> shlex.split('cd foo; python -m pytest -v test')
['cd', 'foo;', 'python', '-m', 'pytest', '-v', 'test']

And

$ cd '/Users/travis/build/admesh/python-admesh;' python -m pytest -v test
/usr/bin/cd: line 4: cd: /Users/travis/build/admesh/python-admesh;: No such file or directory

I will now fix my tests not to assume running from the projects directory and will workaround this, but there might be more use cases, where multiple CIBW_TEST_COMMAND commands are desired.

Hi,

I'm using this CIBW with travis, and it's working great! (for this repo) I tried to set up appveyor today, but ran into an issue:

'cstdint': No such file or directory

This might be unique to 32-bit windows? If so, I'll just need to skip that build. Otherwise, if you have another suggestion, let me know!

This would allow things like CIBW_BEFORE_BUILD=command && command2

It would be nice to have a way to build wheels based on an existing source release on PyPI without having to manually download the tar file, expand it, etc. This could look like:

cibuildwheel --from-pypi=mypackage

It would be nice to be able to optionally specify a version (when building wheels for old versions of packages).

Another related feature that would be nice is the ability to also skip the building of wheels when a wheel with the same name exists on PyPI. This would then allow a workflow of having a dedicated repository with a bunch of cibuildwheel commands for different packages, and have it run e.g. as a daily cron job, building new wheels whenever necessary.

Thanks a lot for putting together this package, I think it's great idea to take the hassle out of building binary wheeels for multiple platforms.

I ran into the issue that the tests have an import problem, links to the detailed build log and the source code

I want to know what exactly is done. How is the binary wheel installed? Perhaps this could be clarified in the docs. I tried to recreate the problem on my development machine and did

pip wheel . -w /tmp/built_wheel --no-deps # taken from the build log
pip install --user dist/pypmc-1.1.1-cp27-cp27mu-linux_x86_64.whl

I then ran the tests both in the source directory and the installed directory

nosetests 
nosetests /home/beaujean/.local/lib/python2.7/site-packages/pypmc

and both worked fine. So I don't understand why the tests don't work. Any idea?

...
+ sh -c 'nosetests /project'
E
======================================================================
ERROR: Failure: ImportError (cannot import name gauss)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/_internal/cpython-2.7.14-ucs2/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/opt/_internal/cpython-2.7.14-ucs2/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/opt/_internal/cpython-2.7.14-ucs2/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/project/pypmc/__init__.py", line 9, in <module>
    from . import mix_adapt, density, sampler, tools
  File "/project/pypmc/mix_adapt/__init__.py", line 7, in <module>
    from . import hierarchical, variational, pmc, r_value
  File "/project/pypmc/mix_adapt/r_value.py", line 7, in <module>
    from ..tools._doc import _add_to_docstring
  File "/project/pypmc/tools/__init__.py", line 6, in <module>
    from ._partition import partition, patch_data
  File "/project/pypmc/tools/_partition.py", line 6, in <module>
    from ..density.gauss import Gauss
  File "/project/pypmc/density/__init__.py", line 6, in <module>
    from . import gauss, student_t, mixture, base
ImportError: cannot import name gauss

On Linux wheels are correctly built for all supported python versions. Windows however stops at python 3.6.

Looking at windows.py there seem to be no references to python 3.7

For my workflow I need to install a C++14 capable compiler onto the manylinux1 image, then build a C++ library with that compiler, then link my pybind11 project against the library. (I may not be sane.)

Parselmouth does something similar. It imports a version of gcc-7.2 toolchain built for centos5, compiles ccache, aims ccache at the /host directory and then calls cibuildwheel. This allows both the newer compiler and then by statically linking libstdc++ the ABI should be compatible.

I am thinking it might be better to fork the official manylinux1 image and install what I need. The dependencies don't change often. If so, I'd need an option to use different Docker images, which doesn't look hard to add.

Do you think swapping out the quay.io images actually makes sense for my use case or are there reasons I shouldn't do that?

Trying to get binary wheels built, tested and deployed from ci.appveyor.com, most of the things are going smoothly, except:

1. In the appveyor log there are several fatal: BUG: setup_git_env called without repository messages in row, each taking as long as a whole minute to appear, just after cibuildwheel starts: is this something related to the tool, some bad setting on my side, or simply something I should not care about?
2. Even if the CIBW_TEST_COMMAND appears in log (see here for the configuration), the command does not seem to be executed (on GNU/Linux the test suite takes several tens of seconds, I cannot believe that Windows can be so much faster 😈), nor it does produce some output
3. The final twine upload exits with an error, even though it actually uploaded the win32 and win_amd64 wheels

Do you have any hint on how I could solve any of above?

Thank you in advance!

Hi,

I tried following the instructions here, which work fine for Linux, but seem to fail on OSX. Example: https://travis-ci.org/sintefmath/Splipy/jobs/370590848

This seems related to #57 which claims to be fixed, so I am sure I'm just doing something wrong here.

.travis.yml file is

language: python

matrix:
  include:
    - sudo: required
      services:
        - docker
      env:
        - PIP=pip
        - CIBW_SKIP="cp27-* cp33-*"
    - os: osx
      language: generic
      env:
        - PIP=pip2
        - CIBW_SKIP="cp27-* cp33-*"

script:
  - $PIP install cibuildwheel==0.7.1
  - cibuildwheel --output-dir wheelhouse

Is there a reason why the shell() method in windows.py uses check_output() instead of check_call() to execute subcommands? It was changed in commit f26da27 (actually PR #18), but it's not clear if it was fixing a problem or what...

I found this while investigating on one of the problems I annotated in issue #22: changing it back to check_call() magically solved the fact that the pytest call wasn't apparently executed, see https://ci.appveyor.com/project/lelit/python-rapidjson/build/1.0.29.

If @tgarc could check and confirm it wasn't intentional, I will open a PR.

While trying to use a vanilla cibuildwheel configuration with AppVeyor, I kept running into this error,

cibuildwheel: Failed to get name of the package. Command was ['c:\\python27\\python.exe', '.\\setup.py', '--name']

It was only after running that command (python .\\setup.py --name) ahead of the cibuildwheel --output-dir wheelhouse that I finally got a clue of what was going on,

setup requires that the Python package 'wheel' be installed. Try the command 'pip install wheel'.

To resolve this problem I've added 'wheel' to setup.py install_requires list in our fork.

I think that two changes are necessary:

1. On line 111 of __main__.py, print the output of CalledProcessError, and exit with the value of returncode.
2. Add 'wheel' to setup.py install_requires list

Do you want a PR?

I have been trying to troubleshoot a unicodedecode error that I get: https://ci.appveyor.com/project/kavvkon/gams-api/build/1.0.8

For this reason I created a new branch and simplified the setup.py. However it seems that a previous version of setup.py is used.

Is it using some kind of cache? Any ideas on the above unicodedecode error? I wasn't able to reproduce locally. Thanks for the nice work

I noticed that Python 2.6 is still listed under the linux.py. I have added CIBW_SKIP=cp26-* but it doesn't seem to skip it. (Edit: that was my mistake, it skips now)

I get this build error which I assume is a problem with py26 being left in the linux.py file, as it doesn't exist in the windows or macos files.

DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Processing /project
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-koHRvT-build/setup.py", line 91, in <module>
        package_info = get_package_info()
      File "/tmp/pip-koHRvT-build/astropy_helpers/astropy_helpers/setup_helpers.py", line 401, in get_package_info
        ext_modules.extend(setuppkg.get_extensions())
      File "sunpy/io/setup_package.py", line 15, in get_extensions
        if platform.system() == 'Windows' or sys.version_info.major == 3:
    AttributeError: 'tuple' object has no attribute 'major'

That attribute exists for py27 and py3, so I assume that it doesn't exist for py26.

What is the right way to build (make && make install) dependencies only once at the beginning? If I run a script in CIBW_BEFORE_BUILD, everything gets built for every py version etc.

The API for this feature is designed (as in README), but it needs implementation.

If this feature would be useful to you, comment below and I might prioritise it over other work.

Check out these travis docs. I'm probably wrong, but would it be possible to upload these wheels directly, in the same step?

I'm working on leveraging cibuildwheel to build the wheels for afdko — (WIP branch). The tool kit currently only supports Python 2.7, but we've been working on modernizing it.

I'm really enjoying cibuildwheel as it greatly simplifies the process of building and testing our wheels, but I think I've hit a snag: I can't find a way to tell it to NOT build the -cp27m- Linux wheel. For Linux we're only interested in the -cp27mu- wheel.

The complete afdko's test suite takes several minutes to run, so we'd really like to reduce the total build & test time to the minimum.

Is it possible to skip Linux builds depending on their abi tag? Thank you.

Is it possible to access the folders of the cloned repository from the CIBW_TEST_COMMAND ?
I run the cibuildwheels in a subfolder, so some scripts in the root are not accessible in the container.

Could {host} work similar to {project} ?

With enforced TLS 1.2 connection, pip fails to collect dependencies on macOS.

c.f. #57

Hi @joerick

I could use some help.
The wheels I'm trying to build are a binding for a C library. The setup.py file is located in a sub folder (python) where the python build takes place (as part of the overall build or seperatly).
However, during build the compiler needs access to additional header and source file located in other sub folders (src include) located on the same level as the python folder.

cibuildwheel seems to only import the folder holding the setup.py file. And the build sadly fails. I tried everything. Setting global environment variables, adding the whole src for as project and adding cibuildwheel pre-build commands.

How can I use cibuildwheel in such a setup?

Thanks a lot

Project: https://github.com/sgeto/libdnet
Travis: https://travis-ci.org/sgeto/libdnet/

What's the reason letting it fail?

Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.9/bin/cibuildwheel", line 9, in <module>
    load_entry_point('cibuildwheel==0.2.0', 'console_scripts', 'cibuildwheel')()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/cibuildwheel/__main__.py", line 110, in main
    cibuildwheel.linux.build(**build_options)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/cibuildwheel/linux.py", line 104, in build
    stdin=subprocess.PIPE)
  File "/opt/python/2.7.9/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/opt/python/2.7.9/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

.travis.yml is

language: python
python:
  - "2.7"
  - "3.4"
  - "3.5"
  - "3.6"
install: pip install -r requirements-appveyor.txt tox-travis
script:
  - tox
  - pip install urllib cibuildwheel==0.2.0
  - cibuildwheel --platform linux --output-dir wheelhouse

I noticed while looking at the code today that the package_name property isn't actually used anywhere. We should probably remove it, unless anyone has particular designs for it?

Also, a few of the test cases have stuff like

if sys.argv[-1] != '--name':

which could be removed at the same time

This is a more a "should we do this" question rather than an actual issue.

Background : Python 3.2 and up offer a subset of the C API which is guaranteed to be stable at the ABI level. For packages that only make use of this limited API, it is possible to build wheels which can be used across multiple Python versions.

https://docs.python.org/3/c-api/stable.html

For instance cryptography switched to Py_LIMITED_API a few months ago:

pyca/cryptography#3307

Adding support for this would for instance have avoid the need to rebuild wheels for Python 3.7.

I'm not sure if it helps but the homebrew formula for python switched to Python 3.7 today.

I'm trying to setup automatic wheel deploy from Travis to PyPI for a couple of projects of myne.

Everything goes well, the packages get built within the manylinux1 docker image, and tests pass, but when twine tries to upload to PyPI, it fails with

Uploading pg_query-0.2-cp36-cp36m-manylinux1_i686.whl
HTTPError: 403 Client Error: Invalid or non-existent authentication information. for url: https://upload.pypi.org/legacy/

I followed the examples and read the available documentation, but as this is my first time at that, it may be that I'm missing the obvious.

My .travis.yml sets TWINE_USERNAME, and I set the TWINE_PASSWORD in the Travis project settings to the value obtained by executing travis encrypt 'my-pypi-password' as explained here. I tried also with travis encrypt 'TWINE_PASSWORD=my-pypi-password' with the same result.

What am I doing wrong (or not doing at all)?

Can you either shed some light on me, or point me to whatever place where I can learn more on the process?

Thanks in advance!

Hey,

First of all great work, this is a really abstraction and takes care care of a lot of stuff.

I'm compiling DLLs in Go and I want to import them with Python and it work, so far so good. The thing is that to build wheels I also need to install Go. I saw there is a CIBW_BEFORE_BUILD parameter so I tried to curl -O https://storage.googleapis.com/golang/go1.9.1.linux-amd64.tar.gz but it didn't work because curl isn't installed. I tried to install it with yum isn't available either.

As far as I understand the Docker image uses Centos 5; can you enlighten me as to what is available and how I could potentially install Go?

EDIT: here is my .travis.yml.

When using setup_requires, the 3.6 version of Python on macOS will blow up.

See actual Travis CI log from joerick/cibuildwheel-autopypi-example#4

This can be workarounded by manually pip installing the requirement in CIBW_BEFORE_BUILD. But given it only blows up with 3.6, I think it's a problem with Python installation.

When using cibuildwheel in OSX and linux, inside travis, the OSX deployment seems to behave differently to the before build variables.

Cython is required for running setup.py and although CIBW_BEFORE_BUILD variable
has the same content both in linux and osx environments (installing in one of its lines Cython), in the second case the setup.py crashes while not having Cython and while in the linux environment cython
is been normally installed and all the procedure follows accordingly.

To see the error see the linux execution behavior and the osx.

Thanks a lot for taking this issue into consideration, while I hope it is my fault (although I have been fighting with it for days..).

PS: I used 0.7.1 version, but even upgrading to 0.7.2 the problem is still there.

After printing that Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture, there are two red errors in my AppVeyor build:

The system cannot find the path specified.
The system cannot find the path specified.

(cfr. https://ci.appveyor.com/project/YannickJadoul/parselmouth/build/1.0.13/job/4cofnsd0y05jgtb9)

Having had a quick look, it seems as if that's due to the lines

"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release

during the execution of https://raw.githubusercontent.com/ogrisel/python-appveyor-demo/09a1c8672e5015a74d8f69d07add6ee803c176ec/appveyor/run_with_env.cmd

This is not urgent, since my build still continues and finishes building everything, I think. But it might be nice to get rid of it (if possible) since the red color dóes make it seems as if there's a problem.

Besides, the python --version call is also highlighted in red, but I'm guessing that's because Python is printing to the stderr. I'll have a look if this is easily resolvable and submit a PR, if I manage?

Hello!

I've been looking around for something just like this. I'm a beginner with travis so I hope you'll bear with me.

My code is here and the relevant log file can be found here. I can build wheels locally, upload to pypi and download and install, but my friend can't, so I'm hoping to build many-platform wheels. I think the issue has to do with travis' isolated environments

The command "pip install cibuildwheel==0.1.3 pybind11==2.1.0" exited with 0.
80.32s$ cibuildwheel --output-dir wheelhouse
+ curl -L -o /tmp/Python.pkg https://www.python.org/ftp/python/2.7.13/python-2.7.13-macosx10.6.pkg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.4M  100 21.4M    0     0  2950k      0  0:00:07  0:00:07 --:--:-- 3594k
+ sudo installer -pkg /tmp/Python.pkg -target /
installer: Package name is Python
installer: Installing at base path /
installer: The install was successful.
+ which python2
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2
+ python2 --version
Python 2.7.13
+ python2 -m ensurepip --upgrade
Requirement already up-to-date: setuptools in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
+ pip2 --version
pip 9.0.1 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)
+ pip2 install wheel
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 1.0MB/s 
Installing collected packages: wheel
Successfully installed wheel-0.29.0
+ pip2 install delocate
Collecting delocate
  Downloading delocate-0.6.4-py2-none-any.whl (85kB)
    100% |████████████████████████████████| 92kB 2.5MB/s 
Requirement already satisfied: wheel in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (from delocate)
Installing collected packages: delocate
Successfully installed delocate-0.6.4
+ pip2 wheel . -w /tmp/tmpwheel2.7
Processing /Users/travis/build/fraenkel-lab/pcst_fast
Collecting pybind11==2.1.0 (from pcst-fast==1.0.2)
  Using cached pybind11-2.1.0-py2.py3-none-any.whl
  Saved /private/tmp/tmpwheel2.7/pybind11-2.1.0-py2.py3-none-any.whl
Skipping pybind11, due to already being wheel.
Building wheels for collected packages: pcst-fast
  Running setup.py bdist_wheel for pcst-fast ... error
  Complete output from command /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -u -c "import setuptools, tokenize;__file__='/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmppPIxUUpip-wheel-:
  running bdist_wheel
  running build
  running build_py
  package init file 'src/__init__.py' not found (or not a regular file)
  creating build
  creating build/lib.macosx-10.6-intel-2.7
  creating build/lib.macosx-10.6-intel-2.7/pcst_fast
  copying src/test_pcst_fast.py -> build/lib.macosx-10.6-intel-2.7/pcst_fast
  running build_ext
  creating var
  creating var/folders
  creating var/folders/my
  creating var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn
  creating var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T
  /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpgO7CCO.cpp -o var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpgO7CCO.o -std=c++14
  /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpOnS03A.cpp -o var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/tmpOnS03A.o -fvisibility=hidden
  building 'pcst_fast' extension
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 103, in <module>
      zip_safe=False,
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
      dist.run_commands()
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
      self.run_command(cmd)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 179, in run
      self.run_command('build')
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py", line 127, in run
      self.run_command(cmd_name)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 75, in run
      _build_ext.run(self)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 88, in build_extensions
      build_ext.build_extensions(self)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
      self.build_extension(ext)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
      _build_ext.build_extension(self, ext)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
      depends=ext.depends)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 566, in compile
      depends, extra_postargs)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 355, in _setup_compile
      pp_opts = gen_preprocess_options(macros, incdirs)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 1052, in gen_preprocess_options
      pp_opts.append ("-I%s" % dir)
    File "/private/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/pip-vBbxQd-build/setup.py", line 18, in __str__
      import pybind11
  ImportError: No module named pybind11
  
  ----------------------------------------
  Failed building wheel for pcst-fast

It looks like it installs pybind11, then installs python, then installs pybind11 again? But then can't find pybind11 when it tries to make the wheel. What do you make of this?

Thanks!

I keep finding myself checking the lists of build configurations and their identifiers when adapting CIBW_SKIP (and now introducing CIBW_BUILD! ;-) )

Could/should we either add an explicit list (or table) to the README, or add a subcommand that has cibuildwheel list all known configurations? Or is this already documented and am I just overlooking it?

I got a very well known error this days, concerning OSX and openssl which is:

URLError: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)>

see from urllib2 in python@2 and urllib in python3.5.
After searching a lot I found out that this is because of deprecated version of open-ssl used by python: "OpenSSL 0.9.8zh 14 Jan 2016"
[You can see that by running {python} -c "from __future__ import print_function; import ssl; print(ssl.OPENSSL_VERSION)" - see from urllib2 in python@2 and urllib in see]

Instead, a version >= 1.0.2 is needed to avoid the error and although the system has it (as you can see when brew installing openssl), it is not the one executed from path. Even if you change the path and append the location of the desired binary in front, this is not a workaround for python as the correct libraries are not linked. An optional workaround is to brew install {python} --with-brewed-openssl and afterwards replace the one in path marked by {python} tagger. For this workaround you can see here.

I am stuck and I don't know how to proceed because of aliases of python version and mysticism of where is the python executable is and if and how it can be replaced by a new one (if there are predependencies installed that are not the ones installed by user). That's why I am opening an issue and suggesting adding an installation of python in osx with the current version of openssl in your package, driven by the fact that this is not a problem of the project that I am testing and a normal user would be able to solve any dependency issue on his machine.

Thanks in advance for your awesome package and I hope the problem was made clear :)

Add support for manylinux2010

c.f. pypa/manylinux#179

Dear,

Tanks for your awesome tools.
I would like to know if it is possible to add teamcity support on your tools ?

Anyway thanks again for your great tools.

Best regards

Presently the build list is effectively everything supported. It'd be nice to be able to easily request a single wheel build to fit into CI schemes with a build per Python version. This is coming up in my Twisted PR's as I try to set it up to run all tests against the built wheels. Travis doesn't save artifacts or even files between jobs (without using external storage accounts) so I'm shooting for building in the individual job.

I would expect the built items to be effectively the set(CIBW_BUILD) - set(CIBW_SKIP).

Of course there we'll be some overhead to spinning up docker more times. We'll see what that is after I get my PR working. twisted/twisted#1067

I might work on this soon, I might not. We'll see... :]

Hi,

I just pushed the first commit in a long time to my project, and cibw ran into a couple issues. The first was, on osx:

$ pip install cibuildwheel==0.6.0
/Users/travis/.travis/job_stages: line 57: pip: command not found

This might be related to this issue: travis-ci/travis-ci#8829 and perhaps fixed with this comment: travis-ci/travis-ci#8829 (comment)

thanks for working on this project! The windows build works for me right out of the box.

However, the manylinux builds don't work for my project as it requires the libffi-dev package. (simply: yum install libffi-dev). It's nice that this 'just works' but is there any room for customization in cases like this? BTW, I believe this is an issue with the osx build as well.

For the command:

- CIBW_TEST_COMMAND="{python} -m unittest {project}/tests/test_simple"

I'm getting KeyError: 'python'

How do I access current python version, like in CIBW_BEFORE_BUILD?
Thank you.

Hi,

trying to build PyHyphen which has a C extension, the first build passes, but the second one fails due to shutil error. See the log below.

If PyHyphen's setup.cfg should be changed to explicitly guide the wheel build process, pls let me know. This should then be documented.

Leo

1
Build started
2
hg clone -r 1312cc107dbc6c378bac95936c934a60f4299c40 https://bitbucket.org/fhaxbox66/pyhyphen C:\projects\pyhyphen
3
adding changesets
4
adding manifests
5
adding file changes
6
added 76 changesets with 225 changes to 55 files
7
updating to branch default
8
42 files updated, 0 files merged, 0 files removed, 0 files unresolved
9
pip install cibuildwheel==0.4.1
10
Collecting cibuildwheel==0.4.1
11
Downloading cibuildwheel-0.4.1.tar.gz
12
Installing collected packages: cibuildwheel
13
Running setup.py install for cibuildwheel: started
14
Running setup.py install for cibuildwheel: finished with status 'done'
15
Successfully installed cibuildwheel-0.4.1
16
cibuildwheel --output-dir wheelhouse
17
18
_ _ _ _ _ _ _
19
|| | _ || || |_ _ | | ___ | |
20|
| | . | | | | | . | | | | | -| -| |
21
||||||||||||_||
22
23
cibuildwheel version 0.4.1
24
25
Build options:
26
platform: 'windows'
27
output_dir: 'wheelhouse'
28
package_name: 'PyHyphen'
29
before_build: None
30
skip: BuildSkipper('')
31
project_dir: '.'
32
test_requires: []
33
test_command: None
34
35
Here we go!
36
37

• python --version
  38
  Using default MSVC build environment for 32 bit architecture
  39
  Executing: python --version
  40
  Python 2.7.13
  41
• python -c "import struct; print(struct.calcsize('P') * 8)"
  42
  Using default MSVC build environment for 32 bit architecture
  43
  Executing: python -c "import struct; print(struct.calcsize('P') * 8)"
  44
  32
  45
• pip install --disable-pip-version-check --user --upgrade pip
  46
  Using default MSVC build environment for 32 bit architecture
  47
  Executing: pip install --disable-pip-version-check --user --upgrade pip
  48
  Requirement already up-to-date: pip in c:\python27\lib\site-packages
  49
• pip install wheel
  50
  Using default MSVC build environment for 32 bit architecture
  51
  Executing: pip install wheel
  52
  Collecting wheel
  53
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
  54
  Installing collected packages: wheel
  55
  Successfully installed wheel-0.29.0
  56
• pip wheel . -w c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel --no-deps
  57
  Using default MSVC build environment for 32 bit architecture
  58
  Executing: pip wheel . -w c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel --no-deps
  59
  Processing c:\projects\pyhyphen
  60
  Building wheels for collected packages: PyHyphen
  61
  Running setup.py bdist_wheel for PyHyphen: started
  62
  Running setup.py bdist_wheel for PyHyphen: finished with status 'done'
  63
  Stored in directory: c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel
  64
  Successfully built PyHyphen
  65
• pip install c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel\PyHyphen-3.0.0-py2-none-any.whl
  66
  Using default MSVC build environment for 32 bit architecture
  67
  Executing: pip install c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel\PyHyphen-3.0.0-py2-none-any.whl
  68
  Processing c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel\pyhyphen-3.0.0-py2-none-any.whl
  69
  Collecting appdirs (from PyHyphen==3.0.0)
  70
  Downloading appdirs-1.4.3-py2.py3-none-any.whl
  71
  Collecting six (from PyHyphen==3.0.0)
  72
  Downloading six-1.10.0-py2.py3-none-any.whl
  73
  Installing collected packages: appdirs, six, PyHyphen
  74
  Successfully installed PyHyphen-3.0.0 appdirs-1.4.3 six-1.10.0
  75
• python --version
  76
  Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture
  77
  Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.0.
  78
  Targeting Windows Server 2008 x64 RELEASE
  79
  80
  Executing: python --version
  81
  Python 2.7.13
  82
• python -c "import struct; print(struct.calcsize('P') * 8)"
  83
  Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture
  84
  Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.0.
  85
  Targeting Windows Server 2008 x64 RELEASE
  86
  87
  Executing: python -c "import struct; print(struct.calcsize('P') * 8)"
  88
  64
  89
• pip install --disable-pip-version-check --user --upgrade pip
  90
  Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture
  91
  Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.0.
  92
  Targeting Windows Server 2008 x64 RELEASE
  93
  94
  Executing: pip install --disable-pip-version-check --user --upgrade pip
  95
  Requirement already up-to-date: pip in c:\python27-x64\lib\site-packages
  96
• pip install wheel
  97
  Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture
  98
  Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.0.
  99
  Targeting Windows Server 2008 x64 RELEASE
  100
  101
  Executing: pip install wheel
  102
  Collecting wheel
  103
  Using cached wheel-0.29.0-py2.py3-none-any.whl
  104
  Installing collected packages: wheel
  105
  Successfully installed wheel-0.29.0
  106
• pip wheel . -w c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel --no-deps
  107
  Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture
  108
  Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.0.
  109
  Targeting Windows Server 2008 x64 RELEASE
  110
  111
  Executing: pip wheel . -w c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel --no-deps
  112
  Processing c:\projects\pyhyphen
  113
  Building wheels for collected packages: PyHyphen
  114
  Running setup.py bdist_wheel for PyHyphen: started
  115
  Running setup.py bdist_wheel for PyHyphen: finished with status 'done'
  116
  Stored in directory: c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel
  117
  Successfully built PyHyphen
  118
• pip install c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel\PyHyphen-3.0.0-py2-none-any.whl
  119
  Configuring Windows SDK "v7.0" for Python 2 on a 64 bit architecture
  120
  Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.0.
  121
  Targeting Windows Server 2008 x64 RELEASE
  122
  123
  Executing: pip install c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel\PyHyphen-3.0.0-py2-none-any.whl
  124
  Processing c:\users\appveyor\appdata\local\temp\1\cibuildwheel4izxch\built_wheel\pyhyphen-3.0.0-py2-none-any.whl
  125
  Collecting six (from PyHyphen==3.0.0)
  126
  Using cached six-1.10.0-py2.py3-none-any.whl
  127
  Collecting appdirs (from PyHyphen==3.0.0)
  128
  Using cached appdirs-1.4.3-py2.py3-none-any.whl
  129
  Installing collected packages: six, appdirs, PyHyphen
  130
  Successfully installed PyHyphen-3.0.0 appdirs-1.4.3 six-1.10.0
  131
  Traceback (most recent call last):
  132
  File "C:\Python27\Scripts\cibuildwheel-script.py", line 11, in
  133
  load_entry_point('cibuildwheel==0.4.1', 'console_scripts', 'cibuildwheel')()
  134
  File "c:\python27\lib\site-packages\cibuildwheel_main_.py", line 116, in main
  135
  cibuildwheel.windows.build(**build_options)
  136
  File "c:\python27\lib\site-packages\cibuildwheel\windows.py", line 101, in build
  137
  shutil.move(built_wheel, output_dir)
  138
  File "c:\python27\lib\shutil.py", line 292, in move
  139
  raise Error, "Destination path '%s' already exists" % real_dst
  140
  shutil.Error: Destination path 'wheelhouse\PyHyphen-3.0.0-py2-none-any.whl' already exists
  141
  Command exited with code 1

Hi,

thanks for this very useful project.

I am trying to use CIBuildWheel to build wheels for PyHyphen, a hyphenation lib for Python. Se https://bitbucket.org/fhaxbox66/pyhyphen.

It has a C extension. I want to use cibuildwheel to generate wheels for all common Windows and Python versions. I no nothing about CI configuration and find the appveyor docs too concise. So I naively copied the demo appveyor.yml file from the cibuild docs. Here is the build error:

Error parsing appveyor.yml: (Line: 4, Col: 5, Idx: 106) - (Line: 4, Col: 5, Idx: 106): While parsing a block mapping, did not find expected key

Any hint would be much appreciated.

Leo

Currently the linux build works by piping a bash script into /bin/bash inside the docker container.

A couple good ideas have come out of the discussion in #97, so I wanted to keep them in an open issue!

1. @altendky: maybe a good way would be a Python script with a CLI (probably small enough to comfortably do with stdlib and 2/3 compatibility) and we pass in the info via the command. No config file to write and copy, no template to fill out. Just copy the exact Python script over to the container and then docker run it with proper arguments.

2. @altendky: Or, maybe we just replace the string/file with python code calling docker run for each command?

Add some repos at ./test that would build every push using travis/appveyor. Would be very helpful for testing contributions.

Sorry to open yet another issue, but in my efforts to try getting my wheels built on the manylinux docker image, I'm running into the problem that there seems no way in cibuildwheel to prepare the container (i.e. install stuff and set environment variables).

In my particular case, for example, I want to install a newer GCC toolchain (https://github.com/squeaky-pl/centos-devtools/releases), set the PATH to use it, and set the CXXFLAGS to be picked up by my build.

Am I overlooking a possibilty, or is this currently not possible?

(Probably also related to #13)