buildout / buildout

Buildout is a deployment automation tool written in and extended with Python

Buildout

Buildout is a project designed to solve 2 problems:

1. Application-centric assembly and deployment

   Assembly runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.).

   Buildout might be confused with build tools like make or ant, but it is a little higher level and might invoke systems like make or ant to get its work done.

   Buildout might be confused with systems like puppet or chef, but it is more application focused. Systems like puppet or chef might use buildout to get their work done.

   Buildout is also somewhat Python-centric, even though it can be used to assemble and deploy non-python applications. It has some special features for assembling Python programs. It's scripted with Python, unlike, say puppet or chef, which are scripted with Ruby.

2. Repeatable assembly of programs from Python software distributions

   Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren't locked down, or in a deployment environment where dependency versions are fully specified. You should be able to check buildout into a VCS and later check it out. Two checkouts built at the same time in the same environment should always give the same result, regardless of their history. Among other things, after a buildout, all dependencies should be at the most recent version consistent with any version specifications expressed in the buildout.

   Buildout supports applications consisting of multiple programs, with different programs in an application free to use different versions of Python distributions. This is in contrast with a Python installation (real or virtual), where, for any given distribution, there can only be one installed.

To learn more about buildout, including how to use it, see http://docs.buildout.org/.

buildout's Issues

script initialization broken for broken -S

Not sure if this is what was intended, but at:
https://github.com/buildout/buildout/blob/1.6.x/src/zc/buildout/easy_install.py#L1249

Then if has_broken_dash_S is true, the script_initialization parameter is completely discarded. This breaks zc.recipe.testrunnner in this environment.

buildout uses sys.path, not PYTHONHOME to insert libraries. means subprocesses fail, or use wrong libs.

We've been experiencing https://bugs.launchpad.net/zc.buildout/+bug/492321 (copied below for easier reference) since buildout 1.5 on FreeBSD. I'm willing to work on a patch for this, since I would really appreciate the speed improvements made in 1.6, but I need a little guidance on where I should be looking for the code to make this change.

From @claytron in the launchpad ticket:

The test and pythons created by buildout do not use PYTHONHOME to insert the egg paths, but instead hack sys.path.

This is ok if the pythons do not call other processes... but if they do, the libs used will be wrong (probably will use system libs).

A work around is to create a PYTHONHOME from the sys.path to pass to sub processes. However, maybe just setting the PYTHONHOME in the bin/python and bin/test etc would be better and more reliable.

I started running into this issue with the 1.5.x bootstrapping process on FreeBSD (doesn't seem to affect OS X). When I bootstrap a project using 1.4.3, the files get written out using the new 1.5.x process. I ran into the above mentioned issue when a recipe does a subprocess.call that in turns calls python code expecting to have the proper PYTHONPATH setup.

I had to temporarily fix a recipe that I wrote to insert the proper path as follows:

http://dev.plone.org/collective/changeset/145988/buildout/collective.recipe.plonesite/trunk

I did a write up about this on my company blog:

http://www.sixfeetup.com/blog/bootstrapping-buildout-killing-pythonpath

zc.buildout fails behind a cntlm proxy

When behind a cntlm proxy [1] used to authenticate to forefront [2], zc.buildout fails downloading files as reported in https://bugs.launchpad.net/zc.buildout/+bug/484735.
The patch in the bug report fixes the problem [3].

[1] http://cntlm.sourceforge.net
[2] http://technet.microsoft.com/en-us/forefront/ee807302
[3] the patch.

diff --git a/src/zc/buildout/download.py b/src/zc/buildout/download.py
index ee6865a..c0d5c3a 100644
--- a/src/zc/buildout/download.py
+++ b/src/zc/buildout/download.py
@@ -25,6 +25,7 @@ import re
 import shutil
 import tempfile
 import urllib
+import urllib2
 import urlparse
 import zc.buildout

@@ -177,7 +178,10 @@ class Download(object):

         try:
             try:
-                tmp_path, headers = urllib.urlretrieve(url, tmp_path)
+                tmp_sock = urllib2.urlopen(url)
+                tmp_file = open(tmp_path, 'w')
+                tmp_file.write(tmp_sock.read())
+                tmp_file.close()
                 if not check_md5sum(tmp_path, md5sum):
                     raise ChecksumError(
                         'MD5 checksum mismatch downloading %r' % url)

Unused options for buildout: 'unzip'.

Since buildout 2.0, I keep getting this message printed in the screen.
Then I looked at the documentation and I did not notice that the option is removed.
https://pypi.python.org/pypi/zc.buildout

Could you clarify whether this option is no longer needed?

Thanks.

`allow_hosts` != * prevents requirement satisfaction via `develop`

or at least when using mr.developer.

No idea why. I suspect it may even be a bug in distribute, but can't be sure.

This is using 2.0.1 on Python 3.2 with distribute 0.6.34.

error while removing tmp file on windows when downloading extends fails

when running on windows, and the buildout fails to download the extends, the windows error is raised because the tmp file is locked:

here is the code that failes, download.py, line 179 - 187:

        try:
            tmp_path, headers = urllib.urlretrieve(url, tmp_path)
            if not check_md5sum(tmp_path, md5sum):
                raise ChecksumError(
                    'MD5 checksum mismatch downloading %r' % url)
        except IOError, e:
            os.remove(tmp_path)
            raise zc.buildout.UserError("Error downloading extends for URL"
                                        " %s: %r" % (url, e[1:3]))

the os.remove(tmp_path) raises following error on windows:

Traceback (most recent call last):
File "C:\Buildouts\plone43wsgi\lib\site-packages\zc\buildout\buildout.py", line 1851, in main
command)
File "C:\Buildouts\plone43wsgi\lib\site-packages\zc\buildout\buildout.py", line 203, in init
data['buildout'].copy(), override, set()))
File "C:\Buildouts\plone43wsgi\lib\site-packages\zc\buildout\buildout.py", line 1489, in _open
downloaded)
File "C:\Buildouts\plone43wsgi\lib\site-packages\zc\buildout\buildout.py", line 1434, in _open
path, is_temp = download(filename)
File "C:\Buildouts\plone43wsgi\lib\site-packages\zc\buildout\download.py", line 99, in call
local_path, is_temp = self.download(url, md5sum, path)
File "C:\Buildouts\plone43wsgi\lib\site-packages\zc\buildout\download.py", line 185, in download
os.remove(tmp_path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\users\rlacko~1.tem\appdata\local\temp\buildout-1qrvkz'

Due this error the real error message [1] is not shown to user

[1] raise zc.buildout.UserError("Error downloading extends for URL"...

update-versions-file without `show-picked-version`

buildout.dumppickedversions doesn't print the list of picked versions if an output file is given. I like that behavior but it's not possible with buildout 2.0.

Right now you have to set show-picked-versions = true for update-versions-file to work. (https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py#L999). (b.t.w. This is not documented clearly).

I suggest that if show-picked-versions = false but update-versions-file is set to a filename the file will get updated.

version 2.x generates wrong formatted .installed.cfg

The following setup (which works fine with v <= 1.7.0) brakes .installed.cfg:

[debug-instance]
<= instance-1
[...]
eggs +=  
    Products.PrintingMailHost
    Products.PDBDebugMode
environment-vars +=
    ENABLE_PRINTING_MAILHOST True
[...]

Running buildout with such setup causes:

ParsingError: File contains parsing errors: /path/to/my/buildout/.installed.cfg
    [line 375]: 'eggs + = Products.PrintingMailHost\n'
    [line 378]: 'environment-vars + = ENABLE_PRINTING_MAILHOST True\n'

If you replace '+ =' with '+=' (with no white space) directly into .installed.cfg the buildout works again but next time you'll get the same error cause .installed.cfg is broken again.

bootstrap.py is no longer Python 2.4 compatible

Is this intended? Did I miss mention of dropping support for 2.4 somewhere in the docs?

find_links = os.environ.get(
    'bootstrap-testing-find-links',
    options.find_links or
    ('http://downloads.buildout.org/'
     if options.accept_buildout_test_releases else None)
    )

Lost script_initialization when _has_broken_dash_S is true in easy_install.sitepackage_safe_scripts function

Hi,

in buildout 1.7.0, when interpreter is here :

https://github.com/buildout/buildout/blob/1.7.0/src/zc/buildout/easy_install.py#L1259

def sitepackage_safe_scripts(
    dest, working_set, executable, site_py_dest,
    reqs=(),
    scripts=None,
    interpreter=None,
    extra_paths=(),
    initialization='',
    include_site_packages=False,
    exec_sitecustomize=False,
    relative_paths=False,
    script_arguments='',
    script_initialization='',
    ):
    """Generate scripts and/or an interpreter from a system Python.

This accomplishes the same job as the ``scripts`` function, above,
but it does so in an alternative way that allows safely including
Python site packages, if desired, and choosing to execute the Python's
sitecustomize.
"""
    if _has_broken_dash_S(executable):
        if not include_site_packages:
            warnings.warn(BROKEN_DASH_S_WARNING)
        return _original_scripts_function(
            reqs, working_set, executable, dest, scripts, extra_paths,
            script_arguments, interpreter, initialization, relative_paths)

the _original_scripts_function is executed without "script_initialization" parameter.
This parameter is lost :(

WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\temp\\buildout-saqtfs'

On winbot.zope.org we had a hard to diagnose problem during bootstrap:

Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg
While:
  Initializing.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "c:\temp\tmpzovmuj\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 1865, in main
    command, args)
  File "c:\temp\tmpzovmuj\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 203, in __init__
    data['buildout'].copy(), override, set()))
  File "c:\temp\tmpzovmuj\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 1510, in _open
    downloaded))
  File "c:\temp\tmpzovmuj\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 1452, in _open
    path, is_temp = download(filename)
  File "c:\temp\tmpzovmuj\zc.buildout-1.7.1-py2.6.egg\zc\buildout\download.py", line 99, in __call__
    local_path, is_temp = self.download(url, md5sum, path)
  File "c:\temp\tmpzovmuj\zc.buildout-1.7.1-py2.6.egg\zc\buildout\download.py", line 185, in download
    os.remove(tmp_path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\temp\\buildout-saqtfs'

Turns out this was caused by a buildout.cfg that had

[buildout]
extends = 
    http://svn.zope.org/repos/main/zopetoolkit/trunk/ztk.cfg?p=109981 
    http://svn.zope.org/repos/main/zopetoolkit/trunk/zopeapp.cfg?p=109918 

These URLs went away and started causing 404 errors. On Linux you could see the download error correctly:

Error: Error downloading extends for URL http://svn.zope.org/repos/main/zopetoolkit/trunk/zopeapp.cfg?p=109918: (404, 'Not Found')

On Windows it never got that far thanks to that WindowsError.

Buildout should take care to close temporary files before attempting to remove them.

Add python version checking (buildout-versions functionality)

Originally part of #46, python version checking was removed because it wasn't perfect yet.

The removed code: https://github.com/reinout/buildout/commit/fc1bc5d0ff034ff11df87a0748721fcfca384f1b

So... this needs to get in again. Changes needed compared to that original code:

• New buildout option: python-version that restricts the Python version, with the same semantics as buildout-version provides now.
• Properly parse the version instead of comparing a version against the whole of sys.version as happened in the original code. That string includes OS versions and so on, so too many chances of false positives.

bootstrap from http://downloads.buildout.org/2/bootstrap.py still picks 1.7.0

well, or something like that:

buildout is here: https://github.com/Simplistix/testfixtures/tree/py3k

Having worked around #59, I now get:

buzzkill:testfixtures chris$ /src/Python-3.3.0/python.exe bootstrap.py
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.34.tar.gz
Extracting in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmphwk3q_
Now working in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmphwk3q_/distribute-0.6.34
Building a Distribute egg in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpzcrsq0
creating build
...
Skipping implicit fixer: ws_comma
/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpzcrsq0/distribute-0.6.34-py3.3.egg
File "build/bdist.macosx-10.7-x86_64/egg/zc/buildout/buildout.py", line 82
print "Annotated sections"
^
...lots more like the above...
File "/private/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpzcrsq0/zc.buildout-1.7.0-py3.3.egg/zc/buildout/buildout.py", line 82
print "Annotated sections"
^
SyntaxError: invalid syntax
...lots more like the above..
...and then get dumped back a the shell with no other message.

PermissionError attempting to install coverage on Windows

An internal error occured due to a bug in either zc.buildout or in a
recipe being used:

Traceback (most recent call last):
  File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", line 1808, in main
    getattr(buildout, command)(args)
  File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", line 606, in install
    installed_files = self[part]._call(recipe.install)
  File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\buildout.py", line 1333, in _call
    return f()
  File "c:\users\jenkins\.buildout\eggs\zc.recipe.egg-2.0.0a3-py3.3.egg\zc\recipe\egg\egg.py", line 162, in install
    relative_paths=self._relative_paths,
  File "c:\users\jenkins\.buildout\eggs\zc.buildout-2.0.0-py3.3.egg\zc\buildout\easy_install.py", line 916, in scripts
    contents = dist.get_metadata('scripts/' + name)
  File "c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py", line 1217, in get_metadata
    return self._get(self._fn(self.egg_info,name)).decode("utf-8")
  File "c:\users\jenkins\.buildout\eggs\distribute-0.6.34-py3.3.egg\pkg_resources.py", line 1327, in _get
    stream = open(path, 'rb')
PermissionError: [Errno 13] Permission denied: 'c:\\users\\jenkins\\.buildout\\eggs\\coverage-3.6-py3.3-win32.egg\\EGG-INFO\\scripts\\__pycache__' 

Buildout does not install setup.scripts, even with zc.recipe.egg:scripts

http://pypi.python.org/pypi/zc.recipe.egg/1.3.2#script-generation does not work.

You can verify this with the following test package: http://pypi.python.org/pypi/tompoes/0.0.2.

It seems zc.buildout.easy_install isn't doing it's work properly.

There is one more issue, which might or might not be related. In any case, the example package has only 1 module; tompoes.py. This module is NOT installed. That means the endpoint tom isn't working!

Since I've got no idea if this bug is caused by the failure to install scripts, I added it to this ticket instead of creating a separate issue.

buildout with python 3.3 attempts to write where it shouldn't

I'm not sure if this is actually buildout's fault, let me know if you can spot the blame for any other package.

So, the buildout is here:
https://github.com/Simplistix/testfixtures/tree/py3k

Attempting to run bootstrap:

buzzkill:testfixtures chris$ /src/Python-3.3.0/python.exe bootstrap.py
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.34.tar.gz
Extracting in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmptsekm9
Now working in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmptsekm9/distribute-0.6.34
Building a Distribute egg in /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpkgrkdb
creating build
...
root: Generating grammar tables from /src/Python-3.3.0/Lib/lib2to3/PatternGrammar.txt
root: Writing grammar tables to /src/Python-3.3.0/Lib/lib2to3/PatternGrammar3.3.0.final.0.pickle
root: Writing failed:[Errno 13] Permission denied: '/src/Python-3.3.0/Lib/lib2to3/PatternGrammar3.3.0.final.0.pickle'

/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpkgrkdb/distribute-0.6.34-py3.3.egg
error: Setup script exited with error: SandboxViolation: open('/src/Python-3.3.0/Lib/lib2to3/Grammar3.3.0.final.0.pickle', 'wb') {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand. Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.
Traceback (most recent call last):
File "bootstrap.py", line 148, in
repr(cmd)[1:-1])
Exception: ('Failed to execute command:\n%s', "'/src/Python-3.3.0/python.exe', '-c', 'from setuptools.command.easy_install import main; main()', '-mZqNxd', '/var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpkgrkdb', 'zc.buildout==1.7.0'")

For now, I've worked around this with:

buzzkill:testfixtures chris$ sudo chown chris /src/Python-3.3.0/Lib/lib2to3/

zc.buildout-1.7.0 incorrect exception when trying to download and cache a non-existent url (404)

updates:

• the exception below is only raised when trying to download and cache a non-existent (404) url using the zc.buildout.download.Download.download_cached() method

exception:

• WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\users\dev\appdata\local\temp\buildout-z8yjgw'
• at https://github.com/buildout/buildout/blob/1/src/zc/buildout/download.py#L185

using:

• bootstrap v1.7 (http://downloads.buildout.org/1/bootstrap.py)
• win7 x64, python 2.7.3 x32

buildout.cfg with non-existent (404) url to be downloaded:

[buildout]
parts =
    gae_sdk
    gae_tools

[gae_sdk]
recipe = appfy.recipe.gae:sdk
url = http://googleappengine.googlecode.com/files/google_appengine_1.7.5_preview.zip

[gae_tools]
recipe = appfy.recipe.gae:tools

works ok with the correct url in bootstrap.cfg:

url = http://googleappengine.googlecode.com/files/google_appengine_1.7.5_prerelease.zip

log:

[st 13.02.2013 18:18:59,15] # Setting Workdir = C:\Users\dev\test 
[st 13.02.2013 18:18:59,21] # Setting Options = 
[st 13.02.2013 18:18:59,29] # Setting PythonVersion = 2.7 
[st 13.02.2013 18:18:59,36] # Setting PythonPath = 
[st 13.02.2013 18:18:59,44] # Setting Builder = C:\Users\dev\test\bin\buildout.exe 
[st 13.02.2013 18:18:59,52] # Setting BuildOptions = -v 
[st 13.02.2013 18:18:59,60] # Setting AutoBuild = true 
[st 13.02.2013 18:18:59,69] # Setting SkipBootstrap = true 
[st 13.02.2013 18:18:59,75] # Loading configuration from environment 'Bootstrap' 
[st 13.02.2013 18:19:00,20] # Getting "C:\Windows\System32\reg.exe" query "HKLM\SOFTWARE\Python\PythonCore\2.7\InstallPath" /ve 
[st 13.02.2013 18:19:00,98] # Setting CachedAppLocations = "C:\Program Files" "C:\Program Files (x86)" "C:" 
[st 13.02.2013 18:19:01,07] # Setting PythonPath = C:\Python27 
[st 13.02.2013 18:19:01,25] === [C:\Users\dev\test] === :logged C:\Python27\python.exe C:\Users\dev\test\bin\bootstrap.py 
Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Creating directory 'C:\\Users\\dev\\test\\parts'.
Creating directory 'C:\\Users\\dev\\test\\develop-eggs'.
Generated script 'C:\\Users\\dev\\test\\bin\\buildout'.
[st 13.02.2013 18:19:20,52] === [C:\Users\dev\test] === :logged C:\Users\dev\test\bin\buildout.exe -v 
Installing 'zc.buildout >=1.7.0, <2dev', 'distribute'.
We have the best distribution that satisfies 'zc.buildout>=1.7.0,<2dev'.
Picked: zc.buildout = 1.7.0
We have a develop egg: distribute 0.6.34
Installing 'appfy.recipe.gae'.
We have no distributions for appfy.recipe.gae that satisfies 'appfy.recipe.gae'.
Getting distribution for 'appfy.recipe.gae'.
Running easy_install:
C:\Python27\python.exe "-S" "-c" ""import sys,os;p = sys.path[:];import site;sys.path[:] = p;[sys.modules.pop(k) for k, v in sys.modules.items() if hasattr(v, '__path__') and len(v.__path__)==1 and not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];from setuptools.command.easy_install import main;main()"" "-mUNxd" ""C:\Users\dev\test\eggs\tmpvnizk5"" "-q" ""c:\users\dev\appdata\local\temp\tmp_w06f7get_dist\appfy.recipe.gae-0.9.3.zip""
path=c:\python27\lib\site-packages

Got appfy.recipe.gae 0.9.3.
Picked: appfy.recipe.gae = 0.9.3
Getting required 'z3c.recipe.scripts>=1.0.1'
  required by appfy.recipe.gae 0.9.3.
We have no distributions for z3c.recipe.scripts that satisfies 'z3c.recipe.scripts>=1.0.1'.
Getting distribution for 'z3c.recipe.scripts>=1.0.1'.
Running easy_install:
C:\Python27\python.exe "-S" "-c" ""import sys,os;p = sys.path[:];import site;sys.path[:] = p;[sys.modules.pop(k) for k, v in sys.modules.items() if hasattr(v, '__path__') and len(v.__path__)==1 and not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];from setuptools.command.easy_install import main;main()"" "-mUNxd" ""C:\Users\dev\test\eggs\tmpi_fjud"" "-q" ""c:\users\dev\appdata\local\temp\tmpcmc6jxget_dist\z3c.recipe.scripts-1.0.1.tar.gz""
path=c:\python27\lib\site-packages

Got z3c.recipe.scripts 1.0.1.
Picked: z3c.recipe.scripts = 1.0.1
Getting required 'zc.recipe.egg>1.3.0,==1.3.0,<2'
We have no distributions for zc.recipe.egg that satisfies 'zc.recipe.egg>1.3.0,==1.3.0,<2'.
Getting distribution for 'zc.recipe.egg>1.3.0,==1.3.0,<2'.
Running easy_install:
C:\Python27\python.exe "-S" "-c" ""import sys,os;p = sys.path[:];import site;sys.path[:] = p;[sys.modules.pop(k) for k, v in sys.modules.items() if hasattr(v, '__path__') and len(v.__path__)==1 and not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];from setuptools.command.easy_install import main;main()"" "-mUNxd" ""C:\Users\dev\test\eggs\tmpucfhu7"" "-q" ""c:\users\dev\appdata\local\temp\tmpjuhmxaget_dist\zc.recipe.egg-1.3.2.tar.gz""
path=c:\python27\lib\site-packages

Got zc.recipe.egg 1.3.2.
Picked: zc.recipe.egg = 1.3.2
Unused options for buildout: 'show-picked-versions'.
Installing gae_sdk.
Searching cache at c:\users\dev\test\downloads\
Cache miss; will cache http://googleappengine.googlecode.com/files/google_appengine_1.7.5_preview.zip as c:\users\dev\test\downloads\google_appengine_1.7.5_preview.zip
Downloading http://googleappengine.googlecode.com/files/google_appengine_1.7.5_preview.zip
While:
  Installing gae_sdk.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "c:\users\dev\test\eggs\zc.buildout-1.7.0-py2.7.egg\zc\buildout\buildout.py", line 1866, in main
    getattr(buildout, command)(args)
  File "c:\users\dev\test\eggs\zc.buildout-1.7.0-py2.7.egg\zc\buildout\buildout.py", line 625, in install
    installed_files = self[part]._call(recipe.install)
  File "c:\users\dev\test\eggs\zc.buildout-1.7.0-py2.7.egg\zc\buildout\buildout.py", line 1345, in _call
    return f()
  File "c:\users\dev\test\eggs\appfy.recipe.gae-0.9.3-py2.7.egg\appfy\recipe\download.py", line 51, in install
    cached_path, is_temp = self.download()
  File "c:\users\dev\test\eggs\appfy.recipe.gae-0.9.3-py2.7.egg\appfy\recipe\download.py", line 154, in download
    cached_path, is_temp = d(self.option_url, md5sum=self.option_md5sum)
  File "c:\users\dev\test\eggs\zc.buildout-1.7.0-py2.7.egg\zc\buildout\download.py", line 97, in __call__
    local_path, is_temp = self.download_cached(url, md5sum)
  File "c:\users\dev\test\eggs\zc.buildout-1.7.0-py2.7.egg\zc\buildout\download.py", line 142, in download_cached
    _, is_temp = self.download(url, md5sum, cached_path)
  File "c:\users\dev\test\eggs\zc.buildout-1.7.0-py2.7.egg\zc\buildout\download.py", line 185, in download
    os.remove(tmp_path)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\dev\\appdata\\local\\temp\\buildout-z8yjgw'

add some versioning info to bootstrap.py

It's a pain that you have no idea whether you have an old/new version of it.

allow-picked-versions=false throws wierd message

• when installing a package that doesn't have pinned version, error is confusing and doesn't include information why it happened
• installing pyramid_marrowmailer throws error pyramid-marrowmailer couldn't be picked, which is really confusing

MD5 Sum check

Is it possible to implement a md5sum check from the files downloated during buildout?
Many times I had some problems because the file downloaded was corrupt..
I think an extra check would be a nice soluction.

dev.py doesn't work with Python 2.6 or 2.7 on windows

This makes development a bit hard. :(

A work around is to run with Python 2.5 first, then with 2.6. Weird.
This isn't a long term solution, obviously, but is OK for now, as
there are bigger fish to fry.

bootstrap.py: add option to use local distribute_setup.py

bootstrap.py currently urlopens the latest distribute_setup.py and therefore ends up building the latest distribute. My package ships with a distribute_setup.py where I've held off upgrading to 0.6.35 due to python 2.4 compatibility issues that were introduced:

Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.gz
Extracting in /tmp/end/tmpqXij9Q
Now working in /tmp/end/tmpqXij9Q/distribute-0.6.35
Building a Distribute egg in /tmp/end/tmpHfd8L2
  File "build/bdist.linux-x86_64/egg/setuptools/tests/test_easy_install.py", line 315
    finally:
          ^
SyntaxError: invalid syntax

I have changed my local copy of bootstrap.py to instead read('distribute_setup.py').

Maintaining support for 2.4 is a pain, I know. Unfortunately the default version is still 2.4 where I deploy packages (CERN).

Update boostrap.py on: http://downloads.buildout.org/2/bootstrap.py

Bootstrap.py on github is newer then the one on: http://downloads.buildout.org/2/bootstrap.py

Please upload a new one to buildout.org.

buildout 2 uses wrong filename for scripts on windows

With the following buildout.cfg on windows

[buildout]
parts = tools

[tools]
recipe = zc.recipe.egg
eggs =
     pypi-tools

[versions]
pypi-tools = 0.0.2

buildout2 generates a bin directory containing

pypi.exe
pypi

Calling bin/pypi.exe results in an error "Cannot open ....\bin\pypi-script.py".

It looks like scripts should have a "-script.py" suffix on windows.

pypi-tools setup contains

setup(...
    scripts = ['pypi'],
    ...)

cache workingset evaluation

@rpatterson had as part of his branch an optimisation that meant if you had two zc.recipe.egg parts with the same set of requirements the 2nd one took much less time. This seemed to work really well. Our buildouts have 16 zope instance sections and each take equally long. Can we bring back this optimsation?

The document for zc.buildout-1.6.x was deprecated on pypi

1. Maybe newest bootstrap.py only exist at github, but the document point to svn.zope.org.
2. $ python bootstrap.py --distribute was removed at newest bootstrap.py on github.

can't get bootstrap.py directly from https://github.com/buildout/buildout/raw/1.6.x/bootstrap/bootstrap.py

PyPI say::

Use the new bootstrap.py (available from https://github.com/buildout/buildout/raw/1.6.x/bootstrap/bootstrap.py).

But "curl -O [url]" get HTML file content that say::

<html><body>You are being <a href="https://raw.github.com/buildout/buildout/1.6.x/bootstrap/bootstrap.py">redirected</a>.</body></html>

Link to "https://raw.github.com/buildout/buildout/1.6.x/bootstrap/bootstrap.py" is better on PyPI?

bootstrap.py refuse some options suddenly (on zc.buildout-1.6.x)

http://svn.zope.org/repos/main/zc.buildout/tags/1.5.2/bootstrap/bootstrap.py accepts options:

• -h or --help
• -v or --version
• -d or --distribute
• --setup-source
• --download-base
• --eggs
• -t or --accept-buildout-test-releases
• -c

but new buildout-1.6.x's bootstrap.py at https://github.com/buildout/buildout/blob/master/bootstrap/bootstrap.py accepts only:

• -h or --help
• -v or --version
• -t or --accept-buildout-test-releases
• -c

then, --distribute, --setup-source, --download-base, --eggs options cause error.

Please accept these options for a while and emission deprecation warnings.

Buildout does not exit with non-zero exit code on timeout

When buildout hits a timeout it aborts with a traceback, but still uses a 0 exit code. This breaks the use of buildout in makefiles.

Support for setup_requires

There seems to exist no support on zc.buildout or zc.recipe.egg for inspecting and providing egg paths for eggs required for setting up a package. Namely, a setup.py file like the one described in this stackoverflow thread can not be built by zc.buildout.

Discuss: provide bootstrap.py that uses 2.0.0 branch for py3k

https://github.com/Pylons/webtest/blob/master/bootstrap-py3k.py uses zc.buildout 2.0.0a2 by default when running py3k. Maybe we should do the same as soon as we have a new alpha release out?

Removed IncompatibleVersionError leads to ImportError in buildout-versions

I use buildout-versions to report versions that buildout pins. With buildout 2.0a5, this extension no longer works because of an import error:

from zc.buildout.easy_install import IncompatibleVersionError

The line in buildout-versions where it fails: https://github.com/Simplistix/buildout-versions/blob/master/src/buildout_versions/__init__.py#L12

In zc.buildout.easy_install, there is a IncompatibleConstraintError.

• Should the old name be re-added in buildout? Perhaps as a BBB alias?
• Or should it be fixed (try/except import, probably) in buildout-versions?

(I think the first option is the best on first sight).

behaviour leaves buildout unusable and needing a re-bootstrap

Here's the shell session:

buzzkill:checker chris$ bin/buildout -o
Develop: '/Users/chris/LocalGIT/checker/.'
Updating py.
While:
Updating py.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/Users/chris/buildout-eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 1866, in main
getattr(buildout, command)(args)
File "/Users/chris/buildout-eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 595, in install
installed_files = self[part]._call(update)
File "/Users/chris/buildout-eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 1345, in _call
return f()
File "/Users/chris/buildout-eggs/zc.recipe.egg-2.0.0a3-py2.7.egg/zc/recipe/egg/egg.py", line 126, in install
reqs, ws = self.working_set()
File "/Users/chris/buildout-eggs/zc.recipe.egg-2.0.0a3-py2.7.egg/zc/recipe/egg/egg.py", line 75, in working_set
[options['develop-eggs-directory'], options['eggs-directory']]
TypeError: working_set() takes at least 3 arguments (2 given)

This leaves you with no buildout:

buzzkill:checker chris$ bin/buildout
-bash: bin/buildout: No such file or directory

make simpler simple use-cases such as easy_install or virtualenv

Aclark suggested I put up my ideas now, since decisions are being made for buildout 2.0.
When I learnt buildout it took me ages to workout that

[buildout]
parts = myegg

[myegg]
recipe = zc.recipe.egg
eggs = myegg
interpreter = python

is roughly the same as

easy_install myegg

I had thought that there should be simpler syntax that bridged the gap between those using easy_install or virtualenv to using buildout. I'm not 100% on what that syntax would be but I put some ideas below. First was to just rename zc.recipe.egg to something more like easy_install since thats what it does. But then I thought how we can it simpler still.

[buildout]
parts = *

[recipe:recipe.easyinstall]
requires = myegg

or with some new defaults this could become

[recipe:easyinstall]
requires = myegg

or maybe

[recipe.easyinstall:myegg]

ie.

• the default of parts = becomes "*"
• there is no need to have a [buildout] section
• there is no need to "name" a part if you don't want to.
• or you can use a shorthand of [recipe:name] to so you don't have to do recipe = all the time.
• the default to always create a bin/python unless there is more than one easyinstall part?
• or maybe a default to create bin/python-myegg?

Maybe there are some more backwards compatible ways to achieve these same goals?

SyntaxError in zc.recipe.egg-generated bin/py when eggs is empty

$ cat buildout.cfg

[buildout]
parts = py

[py]
recipe = zc.recipe.egg
eggs =
interpreter = py

$ wget http://downloads.buildout.org/2/bootstrap.py
$ python3.2 bootstrap.py

Creating directory '/tmp/bout/bin'.
Creating directory '/tmp/bout/parts'.
Creating directory '/tmp/bout/develop-eggs'.
Generated script '/tmp/bout/bin/buildout'.

$ bin/buildout

Installing py.
Generated interpreter '/tmp/bout/bin/py'.

$ bin/py

  File "bin/py", line 6
    ,
    ^
SyntaxError: invalid syntax

$ head bin/py

#!/usr/bin/python3.2

import sys

sys.path[0:0] = [
  ,
  ]


_interactive = True

Console script with # -*- coding: utf-8 -*- on top fails after installing it with zc.recipe.egg

When trying to install the transifex-client console script with buildout 2.0, I noticed that the # -- coding: utf-8 -- is placed under the sys.path declaration and therefore throws an encoding exception because of an utf-8 single quote in the comment just below the if __name__ == '__main__': line, i.e. the single quote in # sys.argv[0] is the name of the script that we’re running.

Add support to overwrite instead of update the versions-file

buildout.dumppickedversions has the option overwrite-picked-versions-file to overwrite instead of appending to the versions file. (https://github.com/collective/buildout.dumppickedversions/blob/master/src/buildout/dumppickedversions/__init__.py#L63)

Buildout 2.0 only provides update-versions-file. When using this option the picked versions file just grows each buildout run.

@reinout Do you think it's possible to add this behavior to buildout 2.0?

Cannot install eggs and extensions in offline mode

The code at https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py#L1047 set the dest to None when doing an offline buildout.

As a result, an offline build will fail, trying to join path with an empty dest with AttributeError: 'NoneType' object has no attribute 'endswith' here:
https://github.com/buildout/buildout/blob/master/src/zc/buildout/easy_install.py#L328
newloc = os.path.join(dest, os.path.basename(d.location))

if you set:
dest = buildout_options['eggs-directory']
at https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py#L1047
this solves the problem

The net effect is that its is not possible to run an offline buildout that uses a download-cache where you keep cached downloads. The buildout will attempt to process eggs from the cache alright, but the install step of that eggs will fail, even if all dependencies are satisfied and available in a cache

bin/buildout (2.0.1) fails with Unicode named directories in dist-packages

I use zc.buildout for a Django project on Debian (Wheezy). I just ran into trouble on a machine that has the system package python-pyramid installed. The package contains a file named /usr/lib/python2.7/dist-packages/pyramid/tests/fixtures/static/héhé (the accented és are stored UTF-8 encoded on the file system). This leads to the following problem with running bin/buildout:

While:
  Installing.

An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/jan/.buildout/eggs/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 1808, in main
    getattr(buildout, command)(args)
  File "/home/jan/.buildout/eggs/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 505, in install
    self._compute_part_signatures(install_parts)
  File "/home/jan/.buildout/eggs/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 749, in _compute_part_signatures
    sig = _dists_sig(pkg_resources.working_set.resolve([req]))
  File "/home/jan/.buildout/eggs/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 1515, in _dists_sig
    result.append(dist.project_name + '-' + _dir_hash(location))
  File "/home/jan/.buildout/eggs/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 1487, in _dir_hash
    hash.update(' '.join(dirnames).encode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Uninstalling python-pyramid helps (running python -S bootstrap.py / python -S bin/buildout does not). Uninstalling python-pyramid is not really an option because there are packaged system wide available applications that require it.

Variable expression evaluation fails when "-c" is used

The following 2 command outputs show that {buildout:directory} is not evaluated when "-c" is used (the first command) but is evaluated when no "-c" is used (the 2nd command)

(1)
aclark@Alexs-MacBook-Pro:/var/folders/ks/sh5b4vj13q7_clr_h8tt0vj00000gn/T/tmp1XFa1v/ > bin/buildout -c https://gist.github.com/raw/4668162/fd26feeb24148999ef08cab0ffda85f07f2078bf/deployment-minimal.cfg
While:
  Initializing.
Error: The directory:
'/private/var/folders/ks/sh5b4vj13q7_clr_h8tt0vj00000gn/T/tmp1XFa1v/${buildout:directory}/var/cache/extends'
to be used as a download cache doesn't exist.

(2)
aclark@Alexs-MacBook-Pro:/var/folders/ks/sh5b4vj13q7_clr_h8tt0vj00000gn/T/tmp1XFa1v/ > bin/buildout 
While:
  Initializing.
Error: The specified download cache:
'/private/var/folders/ks/sh5b4vj13q7_clr_h8tt0vj00000gn/T/tmp1XFa1v/var/cache/downloads'
Doesn't exist.

Verbosity level bug with buildout v2 (coming back from the dead?)

Somehow, this bug has made its way to v2. Is it a fact?

No variable substitutions in extends

A conf example:

[release]
nova = dev
vendor = dev
branch = master

[buildout]
develop = .
index = http://pypi.rd.securactive.lan/
extends = http://release.rd.securactive.lan/vendor/{release:vendor}/versions.cfg

$ ./bin/buildout
While:
Initializing.
Error: Error downloading extends for URL http://release.rd.securactive.lan/vendor/{release:vendor}/versions.cfg: (404, 'Not Found')

It works with find-links.

put back is_distribute for backwards-compatibility?

Currently, at least the buildout-versions extension fails with zc.buildout master because of removed zc.buildout.easy_install.is_distribute that was still there in at least zc.buildout 2.0.0a1, defined thus in easy_install.py:

is_distribute = (
    pkg_resources.Requirement.parse('setuptools').key=='distribute')

Just reporting this - I don't have sufficient knowledge of packaging and zc.buildout internals to understand what should be considered part of the external API etc. Should is_distribute be put back to zc.buildout or should I file an issue on buildout-versions instead?

Weird Error: Referenced option does not exist

zc.buildout 1.7.0...using the following configuration for creating configuration files from templates. Running buldout fails for me with

While:
Installing.
Getting section ini-file2.
Initializing part ini-file2.
Error: Referenced option does not exist: ini-file2 ...
macyet@lgcyapp01:~/sandboxes/mib.pyramid$

The error message is nonsense since the part ini-file2 exists.

Debugging somewhat deeper reveals that the template file abc.in
contains a line

by placing them in ${...} notation. For example:

causing the trouble. Removing the line from the abc.in file resolves
the problem. So there is possibly an error inside the recipe
causing the confusion of zc.buildout.

---

[buildout]
extends = buildout.cfg

parts +=

es-yml

ini-file2
ini-file

[settings]
pyramid_port = 6000
postgres_dsn = postgresql://user:password@localhost/dbname
elasticsearch_port = 6001
elasticsearch_url = http://localhost:6001

[ini-file]
recipe = collective.recipe.template
input = templates/development.ini.in
output = devel.ini

[ini-file2]
recipe = collective.recipe.template
input = templates/abc.in
output = config/elasticsearch.yml

buildout breaks on Ubuntu 12.04

Trying to bootstrap a buildout using a current bootstrap gives me this error:

/usr/bin/python2.7 bootstrap.py
Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Traceback (most recent call last):
  File "bootstrap.py", line 257, in <module>
    import zc.buildout.buildout
  File "/tmp/tmp_keZR4/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 18, in <module>
    import zc.buildout.easy_install
  File "/tmp/tmp_keZR4/zc.buildout-2.0.1-py2.7.egg/zc/buildout/easy_install.py", line 61, in <module>
    pkg_resources.Requirement.parse('distribute')
AttributeError: 'NoneType' object has no attribute 'location'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
ImportError: No module named apport.fileutils

Original exception was:
Traceback (most recent call last):
  File "bootstrap.py", line 257, in <module>
    import zc.buildout.buildout
  File "/tmp/tmp_keZR4/zc.buildout-2.0.1-py2.7.egg/zc/buildout/buildout.py", line 18, in <module>
    import zc.buildout.easy_install
  File "/tmp/tmp_keZR4/zc.buildout-2.0.1-py2.7.egg/zc/buildout/easy_install.py", line 61, in <module>
    pkg_resources.Requirement.parse('distribute')
AttributeError: 'NoneType' object has no attribute 'location'
make: *** [bin/buildout] Error 1

Support referencing environment variables in buildout

See https://bugs.launchpad.net/zc.buildout/+bug/557769

Deprecate Searching External Links

As a counterpart to my post on catalog-sig (http://mail.python.org/pipermail/catalog-sig/2013-February/005330.html) I would like to propose that buildout follows a similar set of steps as described in my catalog-sig post.

1. Ensure that whenever a particular version is available on both the index and externally it selects the version available on the index.
2. Issue a warning whenever a package is downloaded from something other than the index.
3. Allow a command line option to disable external url fetching completely.
4. In some amount of time (1 release?) switch to external url fetching being disabled by default, and allow a flag to re-enable it.
5. In some more time (1 more release?) remove the option to fetch external urls completely.

(The pip counterpart to this issue is located at: pypa/pip#818)

bootstrap.py does not work if buildout.org is down

At this moment (Mi 26. Sep 10:23:28 CEST 2012) buildout.org is down.
Wanted to setup a buildout, "python bootstrap.py" refuses to work with:

Download error on http://buildout.org: [Errno -2] Name or service not known -- Some packages may not be found!

tried different -v options, -t etc. nothing works.
Why does it want to resolve buildout.org when all zc.buildout-packages reside on pypi.python.org ?

Seems that on http://pypi.python.org/simple/zc.buildout/ some "home_page" links point to buildout.org. dunno what exactly happens. Any idea?

AttributeError: 'NoneType' object has no attribute 'group'

When you try to use fancy version constraints like foo = < 4.0dev in your buildout.cfg, and you accidentally run the 1.7 bootstrap, instead of a clear error message saying this feauture is unsupported you get this internal error:

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 1866, in main
    getattr(buildout, command)(args)
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 625, in install
    installed_files = self[part]._call(recipe.install)
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\buildout.py", line 1345, in _call
    return f()
  File "build\bdist.win32\egg\zc\recipe\testrunner\__init__.py", line 46, in install
    eggs, ws = self._delegated.working_set(('zope.testrunner', ))
  File "c:\eggs\zc.recipe.egg-1.3.2-py2.6.egg\zc\recipe\egg\egg.py", line 101, in working_set
    **kw)
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\easy_install.py", line 1079, in install
    return installer.install(specs, working_set)
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\easy_install.py", line 905, in install
    req = self._constrain(requirements.pop(0))
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\easy_install.py", line 861, in _constrain
    requirement = _constrained_requirement(constraint, requirement)
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\easy_install.py", line 1910, in _constrained_requirement
    _constrained_requirement_constraint(constraint, requirement)
  File "c:\eggs\zc.buildout-1.7.1-py2.6.egg\zc\buildout\easy_install.py", line 1959, in _constrained_requirement_constraint
    cop, cv = _parse_constraint(constraint).group(1, 2)
AttributeError: 'NoneType' object has no attribute 'group'

(See this failed build for a live example)

Buildout 2 should have a clearer warning when run with setuptools

Buildout 2 requires distribute instead of setuptools. The bootstrap guarantees distribute, but we cannot guarantee that the correct bootstrap is being used.

See #81 for how easy it is to miss it and how unclear the error message is: AttributeError: 'NoneType' object has no attribute 'location'.

Would a special try/except like the following be a good idea?

something = pkg_resources.Requirement.parse('distribute')
if something is None:
    if pkg_resources.Requirement.parse('setuptoos') is not None:
        raise WhatAreYouDoingException