Comments (17)
For me, the observation shared above raises those questions:
- Why does the procedure try to invoke the compiler at all and not pull binary wheels from PyPI instead? They should be available. Here:
greenlet
, which is a dependency of SQLAlchemy. - Why does the procedure not see the
gcc
compiler when it is falling back to it? - Why does it only happen with Python 3.10?
from buildout.
Thanks for the kind words.
Are you having those problems on a M1 ? Or classic Intel CPU ?
from buildout.
Dear @gotcha,
thanks for your quick reply. CI/GHA on behalf of test-buildout-python310 is running on classic Intel. @proddata also might want to tell us about what his workstation is running on.
On my workstation, also Intel, it is not a problem because I am still on macOS 10.15.7 (Catalina), which also works on GHA. The flaw is only present on newer versions of macOS >= 11.
With kind regards,
Andreas.
from buildout.
Hi, since a week I also see failing GHA runs for Python 3.10 when C code is involved. It is only 3.10, older and newer versions are okay and only MacOS. There is not even zc.buildout
involved. (Locally on Intel this cannot be reproduced.)
See zopefoundation/zope.security#87 for details even if they might have nothing to do with this possible issue in buildout and show another traceback.
from buildout.
Dear @gotcha,
thanks for your quick reply. CI/GHA on behalf of [test-buildout-python310] is running on classic Intel. @proddata also might want to tell us about what his workstation is running on.
Only tested it with an M1 on macos12
from buildout.
Hi again,
thank you for sharing your observations, @icemac. I was a bit hesitant first, but now I've also shared them with the people at GHA, see actions/runner-images#5791.
With kind regards,
Andreas.
from buildout.
Did you try to use the patch as mentioned in pypa/setuptools#2514 (comment) only to see if it works and maybe we can narrow the problem? Or running a minimal buildout as specified in pypa/setuptools#2514 (comment)? This may be a setuptools issue.
from buildout.
Dear @idgserpro,
running a minimal buildout
A reproduction repository containing a minimal buildout has been created at test-buildout-python310. It can by used by anyone to investigate this issue.
Did you try to use the patch as mentioned in pypa/setuptools#2514 (comment)?
I haven't had the chance to do so. I will try and report back, thank you again!
With kind regards,
Andreas.
from buildout.
@idgserpro Thank you for the link to pypa/setuptools#2514. The comment from @goschtl there helped me in another project where installing of packages containing C code suddenly stopped in my local MacOS (Python 2.7 on zc.buildout 2.13).
from buildout.
Hey, setting _PYTHON_HOST_PLATFORM="macosx-11-x86_64"
also helped to fix the GHA build in zopefoundation/zope.security#87. (At least partially.) – That's interesting.
from buildout.
Hi again,
regarding my question at #609 (comment):
For me, the observation shared above raises those questions:
- Why does the procedure try to invoke the compiler at all and not pull binary wheels from PyPI instead? They should be available. Here:
greenlet
, which is a dependency of SQLAlchemy.
I just discovered #544 which I believe shows the same symptom that is happening here, only that now, Buildout fails to use wheels only when running on Python 3.10 and macOS=>11.
With kind regards,
Andreas.
from buildout.
Hey, setting
_PYTHON_HOST_PLATFORM="macosx-11-x86_64"
also helped to fix the GHA build in zopefoundation/zope.security#87. (At least partially.) – That's interesting.
On the Plone core development buildout since recently we also have a problem with gh-actions on Mac with Python 3.10: an error installing MarkupSafe
. Setting _PYTHON_HOST_PLATFORM
helps here as well.
from buildout.
Dear Maurits,
thank you for also sharing your observations on this matter. For the lifetime of the GHA run logs, and for the sake of completeness, I would like to reference the one where Python 3.10 croaked on macOS on buildout.coredev
[1].
On our end, it was enough to invoke sudo xcode-select -switch /Library/Developer/CommandLineTools
in the workflow definition file, as suggested by @mikhailkoliada at actions/runner-images#5791 (comment). See crate-workbench/test-buildout-python310#2. Maybe both things are fixing different things? In any case, it is good to have them referenced here as a reference for others.
On the other aspect from my previous post: May I humbly ask you if you also would have expected that a corresponding MarkupSafe
binary wheel package would have been selected from the package files published on PyPI at [2]? In this case, a compiler would not be needed and invoked at all, right? As mentioned above, I am looking at #544 here and wonder if Buildout's wheels-use machinery is broken in this specific condition as well.
With kind regards,
Andreas.
[1] https://github.com/plone/buildout.coredev/runs/7099812545
[2] https://pypi.org/project/MarkupSafe/#files
from buildout.
I am seeing issues that may at least be related.
There's a very insidious bug in how buildout selects a suitable package from a list of packages provided from the package index in the method zc.buildout.easy_install._obtain
that specifically affects macOS. It took me a long time to find it.
buildout/src/zc/buildout/easy_install.py
Lines 497 to 523 in a8d182c
The code takes a list of distributions returned from a query to the package index for a package and version number. Example: I am running Python 3.10 on macOS/arm64 and want to install BTrees==4.11.2
. The index query will return the following possible matches:
[BTrees 4.11.2 (https://files.pythonhosted.org/packages/90/4f/29127b14926568848416ba3c51a58d1310161749c9ed883bc38dd63e5a45/BTrees-4.11.2.tar.gz#sha256=98be22e387b4aeebb6381bd1ceec0311c250762dd8ba936216ae55c9419024ad),
BTrees 4.11.2 (https://files.pythonhosted.org/packages/29/6c/cfc13b4374717c13f1b8c1ba07d01a8720b4a4c2f051a176aa133a300250/BTrees-4.11.2-cp310-cp310-macosx_10_9_universal2.whl#sha256=71333d8d2cc2bbd44427097c547e73e76b7ef462273f50344bb6f1ce1434f675),
BTrees 4.11.2 (https://files.pythonhosted.org/packages/95/79/f9ede135b35763af1dcbd603fbe36f8e8ac0991332aacc51b9a9fb5557d2/BTrees-4.11.2-cp310-cp310-macosx_11_0_arm64.whl#sha256=d4f9f441ac760900dbce8d283032eb3100306dd7c6054787148dbc629cf98c58)]
One is the source distribution, and there are two matching binary distributions, one specifically for arm64
and the other a "fat" universal2
wheel that contains binary code with both arm64
and x86_64
compatibility. I am guessing the next steps (-> sort the resulting list and then simply return the last element) is based on the assumptions that sorting will always put the source distribution first and there is only one binary distribution. So in this particular case the arm64
wheel is returned.
This simplistic algorithm fails when Python is compiled with universal2
architecture, as the later Python versions on GitHub Actions runners are. It expects to get a universal2
wheel, but gets an arm64
wheel, leading to issues correctly converting the wheel to an egg that is usable:
While:
Installing alltests.
Getting distribution for 'zope.container==4.8'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 2252, in main
getattr(buildout, command)(args)
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 856, in install
installed_files = self[part]._call(recipe.install)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 1652, in _call
return f()
^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/eggs/zc.recipe.testrunner-2.2-py3.11.egg/zc/recipe/testrunner/__init__.py", line 44, in install
eggs, ws = self.egg.working_set(('zope.testrunner', ))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/eggs/zc.recipe.egg-2.0.7-py3.11-macosx-12.6-universal2.egg/zc/recipe/egg/egg.py", line 78, in working_set
ws = self._working_set(
^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/eggs/zc.recipe.egg-2.0.7-py3.11-macosx-12.6-universal2.egg/zc/recipe/egg/egg.py", line 161, in _working_set
ws = zc.buildout.easy_install.install(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 971, in install
return installer.install(specs, working_set)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 744, in install
for dist in self._get_dist(req, ws):
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 581, in _get_dist
dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 1948, in _move_to_eggs_dir_and_compile
assert newdist is not None # newloc above is missing our dist?!
^^^^^^^^^^^^^^^^^^^^^^^^^^
from buildout.
@amotl I tried your repo https://github.com/crate-workbench/test-buildout-python310
One small thing: in the setup.py
I need to add packages=[]
otherwise setuptools complains, but this may depend on where you add a virtualenv.
Last night I updated my Mac from 12.6 to 13.5. When I try your buildout with Python 3.10.12 now I get the same error. I installed this with pyenv a month or so ago. When I reinstall this python (pyenv install -f 3.10.12
) and try again, the buildout works.
@davisagli reported this behaviour as well a few months ago.
Reinstalling Python 3.11 fixed the same problem for me in a different buildout. I will do that with all my pyenv-installed Pythons now, and remove all *macosx-12*
eggs from my eggs cache.
from buildout.
from buildout.
I think we can consider this issue as resolved then?
At least for me your example no longer gives an error.
There may still be subtle bugs lingering, for example I now have a problem installing zope.container 4.10
on Python 3.8, but we can discuss more where needed in issue #606.
from buildout.
Related Issues (20)
- setuptools vendoring of distutils breaks `zc.buildout` (and `dev.py`) HOT 8
- buildout 3: python3.8 install compiles scipy 1.7.3 but whl HOT 12
- buildout 2/3: Wheel file mode installed via buildout != pip install HOT 10
- buildout3 rc3 problem HOT 16
- ERROR: Package 'collective-jsonmigrator' requires a different Python: 3.8.10 not in '==2.7,>=3.7' HOT 1
- Private pypi repository with basic auth used to work in previous versions HOT 5
- Where does `buildout.exe` come from? HOT 4
- distutils-sig has closed HOT 1
- Error message with pip 22.2+ HOT 4
- gh-actions are run twice for PRs HOT 3
- Recipe feature error, depending on part order HOT 1
- Support PEP 508 markers in section expressions/conditions
- zc.buildout 3.0.0 broken HOT 3
- Buildout may be slow with pip 23 and setuptools 67 HOT 4
- vscode python extension passes `-I` option to the designated python interpreter which is unknown to the python interpreter wrapper installed by zc.recipe.egg
- Parts += bug with multiple extends HOT 4
- be explicit when egg cannot be installed due to python-requires conflict
- Could be nice if a conditional section expression could evaluate a buildout variable
- Bootstrap.py unable to install zc.buildout HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buildout.