macpython / numpy-wheels Goto Github PK

Cython 0,27 has some numpy issues, see numpy/numpy#9754. We should probably stick at 0.26 until they are fixed.

fixes numpy/numpy#13764

In numpy/numpy#13109 we preload any dlls as a way around other methods of solving dll hell. That code should live here, as part of a _distributor_init.py file. In addition, the win32 CI run should run a sanity check on the wheel it builds in a clean environment.

The transferred code should also add a try except for the case where __file__ does not exist.

It seems we should be able to use multibuild to build osx wheels for python3.8 with the following diff. Multibuild supports 3.8 on OSX since matthew-brett/multibuild#262 (which is badly named)

diff --git a/.travis.yml b/.travis.yml
index 6b19a35..7eaae8d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -63,6 +63,11 @@ matrix:
       env:
         - MB_PYTHON_VERSION=3.7
         - MB_PYTHON_OSX_VER=10.9
+    - os: osx
+      language: generic
+      env:
+        - MB_PYTHON_VERSION=3.8
+        - MB_PYTHON_OSX_VER=10.9
 
 before_install:
     - if [ "$TRAVIS_BRANCH" == "master" ]; then

For 3.8rc1, downloads were attempted from

https://www.python.org/ftp/python/3.8.0rc1/python-3.8.0rc1.exe

instead of

https://www.python.org/ftp/python/3.8.0/python-3.8.0rc1.exe

This should no longer be needed now that we're shipping wheels with OpenBLAS. And it's an actual problem, here's a report of a user that ran into a gfortran runtime incompatibility: https://mail.python.org/pipermail/numpy-discussion/2019-January/079151.html

Let's try to fix this before the next release.

They have not been showing up in the nightly build repos for several weeks.

I am aware that Python 3.11 is still a few months away, but can I expect any NumPy wheels to become available for alphas or betas of the aforementioned version?

See numpy/numpy#12939

This came up in pandas-dev/pandas#26546

Both NumPy and SciPy only do scheduled builds on Travis (Linux / Mac) but not on Appveyor (Windows), so regular master branch wheels / binaries are not available consistently on Windows for i.e., debugging downstream, etc.

It looks like appveyor team wants to be contacted directly by OSS projects that want scheduled builds to avoid wasted cycles on abandoned projects with scheduled builds.

See also: https://www.appveyor.com/docs/build-configuration/#scheduled-builds

Not sure if it is much easier on Azure or not.

The C compiler detection of f2py is failing, hence all tests in test_array_from_pyobj.py are being skipped. This is not new with pytest.

It would be nice to be able to use the newest python versions to build wheels. For linux, these are available via the manylinux wheels, which are updated pretty quickly after a python release. But for windows and macOS, we need python executables via the CI. Azure tends to lag behind github actions, so perhaps we should move the CI builds to github actions before the release of Python 3.10

I think we should archive this repo once we are done with 1.23 releases. 1.24 uses cibuildwheel.

For the last several weeks the aarch nightly wheels have been uploading to the staging repo.

Implement the SciPy changes at MacPython/scipy-wheels#19 .

I was looking at wheel sizes and was surprised that the 1.20 wheels were barely larger than the 1.19 ones. I can't remember we have taken a decision on what SIMD architectures to include in wheels, however I'd expect it to include at least AVX2. This is what I see on macOS right now:

>>> np.__version__
'1.20.1'
>>> np.core._multiarray_umath.__cpu_baseline__
['SSE', 'SSE2', 'SSE3']

I think it's similar on other platforms, we didn't update this repo with a new baseline. Thoughts @seiko2plus, @mattip?

I assume that this is the result of using osx 10.14 for the target version. Both the 10_14 x86_64 and 11_0 arm64 wheels are produced, but no universal wheel combining the two. When 10.9 was the target version we also got both to those, along with 10_9 x86_64 wheels and universal2 wheels. It looks like the universal builds used to produce both the 10_14 and 11.0 wheels. The relevant PR that led to this problem is probably #149. @isuruf Any thoughts about what might be going on here? There are no error messages.

@matthew-brett

Failing builds: https://ci.appveyor.com/project/matthew-brett/numpy-wheels/history

Successful builds: https://ci.appveyor.com/project/charris/numpy-wheels/history

The daily builds have gone missing.

On the next release, please update the multibuild submodule to matthew-brett/multibuild@b2748e5 or newer. This should reduce the size of the numpy wheels from 16,1M to 12M by excluding unnecessary debug symbols.

I may be forgetting some bit of history here, but right now http://wheels.scipy.org/ contains only outdated wheels (till Dec 2018 for numpy-1.17.0.dev) while https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com/ is up to date.

Do we need to point wheels.scipy.org to the new place? Can anyone summarize what happened here?

@matthew-brett @mattip Were any changes made to the travis trigger that you know of? Thoughts?

 curl https://pypi.org/pypi/numpy/json | jq | grep filename | grep whl | grep aarch64

There are no wheels for pypy on aarch64, any change to add the support for it?

BTW, the numpy can installed by source successfully, but it's very slow.

The numpy submodule is part of this repo. Do we need it?

Since auto-generated file _numpyconfig.h has NPY_SIZEOF_LONGDOUBLE and NPY_SIZEOF_COMPLEX_LONGDOUBLE generated, it's wrong for x86_64 as the universal2 wheel distributes the arm64 version of it.

Easiest option would be to stop distributing universal2 wheels and only distribute thin wheels.

@mattip The secrets have disappeared. Do you know if there was a time limit on them? I'm also wondering if trying to build 1.18.5 broke things? Anyway, the first attempt failed on travis, so I am trying again with azure, which I suspect is going to fail for Python 3.5 on Mac, but one doesn't know until they try.

I have deleted the TravisCI cron job, so that should no longer run on Saturday nights. Note that those jobs are set up on the TravisCI website, not in the yml file.

Probably just needs a UserAgent for wheeluploader. I think we may want to build our own downloader and put it in the numpy tools directory.

Summary

The 1.22 series are using the latest VS2019 toolchain (version 142). This makes the built objects and static libraries incompatible with software built with older toolchains. Would you accept a PR to use the older VS2017 version 141 toolchain instead?

Detail

The Numpy 1.22 series wheels are breaking the Scipy MinGW-w64 builds.

I believe this is because y'all have updated from the previous VS2017 toolchain (version 141) to the latest VS2019 toolchain (version 142).

As MS point out, using version 142 to build the static libraries means that anyone linking against those libraries must also be using (at least) this toolchain.

This means that projects like Scipy can't link to Numpy without also updating to version 142.

In fact, for Scipy, it also, and presumably for similar reasons, breaks our ability to link using MinGW-w64:

matthew-brett/dll_investigation#1

I believe, for these reasons, conda-forge has standardized to version 141 for their builds.

Would y'all consider following conda-forge, and using the 141 toolchain for builds, so that more of us can safely
link to Numpy - particularly - npymath?

@isuruf tells me that this just requires the following lines for a Visual Studio 2019 image:

call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.16
set DISTUTILS_USE_SDK=1

@isuruf points out that

Last part is to make sure that distutils doesn't try to override that if you are using distutils.

@rgommers @carlkl

Looks like the NUMPY_STAGING_UPLOAD_TOKEN is missing. The builds are triggered, they just don't upload. See https://travis-ci.org/github/MacPython/numpy-wheels.

@mattip @tylerjereddy

I need to start preparing numpy-wheels master for the 1.19 release. There are a couple of things that need fixing or decisions.

• Should wheel uploads go to the new anaconda repo?
• Do we need https://anaconda.org/multibuild-wheels-staging/numpy?
• What needs to be changed to use the libraries on anaconda?
• Should 1.19 stick with manylinux2010 or fall back to manylinux1 for working 32 bit NumPy?

Pandas looks to have uploaded the recent pandas to anaconda, maybe we can steal some code from them.