[feature request] Parallelise building parts about buildout

[gotcha]

PR welcome ;-)

[arterrey]

I may be able to do something next year, but it is worth discussing first.

A solution I was thinking about. Would it work if there was a way to accept a buildout config as stdin and return .installed.cfg as stdout?

That way it could be possible to chop up a larger buildout configuration into a subset configurations, subprocess them out, then merge the installed.cfg results as they come back.

Another issue that i was thinking about is - I'm not sure how to co-ordinate multi process python package installations. I suspect there are in-memory records about what is installed and not installed. Secondly, I think there may need to be some kind of locking to prevent two processes installing the same python package at the same time. 🤔 let me know if you have insight on that.

[arterrey]

In terms of config and an master process/executer - that may actually be handleable inside a recipe, like such...

[buildout]
parts =
   my-app
   frontend
   my-script

[frontend]
recipe = zc.buildout.parallel
workers = 2
parts =
  nginx
  haproxy
...

The other benefit of doing that way would be that it preserves dependencies that are expressed as the parts order.

[gotcha]

And that you could experiment without touching buildout at first