setuptools vendoring of distutils breaks `zc.buildout` (and `dev.py`) about buildout

[gotcha]

Fix released in https://pypi.org/project/zc.buildout/3.0.0rc3/

[mauritsvanrees]

Jenkins has succeeded as well. I have merged my PR: the Plone 6.0 coredev buildout is now using release candidate 3.

[gotcha]

My current understanding is the following:

setuptools is vendoring distutils. It installs a hack to ensure that its copy of distutils is used instead of the stdlib version. Unfortunately, there is still an issue with pip (pypa/pip#8761) where the mechanism fails and where setuptools allow pip to use the stdlib version.

We are in the dark spot of the problem with zc.buildout>3 since it combines setuptoolsand pip.

[gotcha]

  ...
  File "/foo/env/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 57, in __init__
    raise TypeError("dist must be a Distribution instance")
TypeError: dist must be a Distribution instance

This is another symptom of the same issue with setuptools and pip.

[gotcha]

@mauritsvanrees @icemac Would you try that latest release ?

I hope it is the last release candidate.

[mauritsvanrees]

I am trying it out in plone/buildout.coredev#786 with latest setuptools 62.0.0.

So far so good:

• All GitHub Actions have succeeded, on all combinations of Python 3.7/3.8/3.9 and Ubuntu/Mac/Windows. It can be that all needed packages were already cached though, so nothing needed to be downloaded or installed. But sounds good already.
• I started Jenkins tests, but those take longer.
• Locally a buildout run using my eggs cache worked fine.
• A local buildout run without downloads or eggs cache is still running. Only problem so far is that I had to restart a few times, but this is due to timeouts when downloading from files.pythonhosted.org, so it has nothing to do with the release candidate.

Thanks a lot!

[mauritsvanrees]

A local buildout run without downloads or eggs cache is still running.

Update: it has finished successfully, after installing 309 packages:

$ ls -1 downloads/dist/ | wc -l
     309
$ ls -1 eggs/cp39/ | wc -l
     309
$ du -sh downloads/dist/ eggs/cp39/
 52M	downloads/dist/
205M	eggs/cp39/

[gotcha]

@mauritsvanrees

Thanks for testing it ! And for Plone 6 coredev.

Before cutting a 3.0 release, I still would like to test a buildout that pins a lower version of setuptools than the one installed in the virtualenv.

My intuition tells me that it is one of the cases where the race condition mentioned in pypa/pip#8761 could happen. But I could be wrong.