Straight Ornamental

edit

Adventures with buildout

Posted by Graham Stratton Wed, 27 Jun 2007 22:22:00 GMT

Recently I wanted to try some modifications to a Pylons project against the latest Pylons trunk without upgrading the system-installed Pylons, so I decided it was about time to try Buildout. Jim Fulton’s buildout (or zc.buildout, since Jim likes to put things in namespaces), is a package to create development and deployment versions of software. It’s not in any way Zope-specific, and whilst it is written in Python primarily for installing Python programs, it’s not really Python-specific either.

It is quite complex. Maybe very complex. But I think that this is only because deployment is a complex business. I can imagine using nearly all the features in the buildout docs, and a few more besides. But I did find a lack of really simple examples for us mortals.

The example I’m going to give here is this: suppose you have a project in a repository. You can check it out, but when you’ve done so you want a way to install all the dependencies, without affecting what’s already installed on your system.

Buildout makes this easy. You need to add two files to your repository:

• bootstrap.py which installs the latest buildout and setuptools

• buildout.cfg which contains the configuration for your install

You can get bootstrap.py from the Zope Corp subversion server at http://svn.zope.org/zc.buildout/trunk/bootstrap/

Here’s my buildout.cfg:

[buildout]
parts = foxtrot

[foxtrot]
recipe = zc.recipe.egg
interpreter = python2.5
eggs = python-cjson
       nose
       twill
       pastescript
       pylons == 0.9.5
       sqlalchemy
#       pymssql
find-links = http://python.cx.hu/python-cjson/python-cjson-1.0.3x5.tar.gz
             http://www.mirrorservice.org/sites/download.sourceforge.net/pub/sourceforge/p/py/pymssql/

So my buildout is simple. It only has one part, and the recipe for that is the simple egg recipe zc.recipe.egg. By specifying the interpreter option, I will get a python executable in my bin directory. When I run it, the environment will contain all the specified packages. I have commented out pymssql because it has too many system dependencies.

Packages which are explicitly specified (not installed as dependencies of other packages) and which define scripts to be installed will have those scripts written to ./bin/ instead of the system directory, and will work in the new environment. So I’ve explicitly specified nose and pastescript, so that I end up with ./bin/nosetests and ./bin/paster working with this new environment. Cool, eh?

The ‘find-links’ option is just a list of locations other than the cheeseshop in which to search for packages to install.

Once you have your configuration, setting up the environment is simple, if a little slow (mainly due to pypi being rather lethargic at the moment). First, run

python bootstrap.py

, to install buildout itself. Then run ./bin/buildout, and everything should magically install. If you want to run it again, it will be much faster with the -N option, which tells buildout not to look for new versions of packages if it already has a version matching the requirements. You can specify version for packages as you would with easy_install.

Well, I thought it all worked. But actually buildout doesn’t seem to use the setuptools that it installs, despite it modifying the path:

$ ./bin/buildout -N
Uninstalling foxtrot.
Installing foxtrot.
Getting distribution for 'Routes>=1.6.3'.
The required version of setuptools (>=0.6c6) is not available, and
can't be installed while this script is running. Please install
 a more recent version first.

(Currently using setuptools 0.6c5 (/usr/local/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg))
error: Setup script exited with 2

Which is really odd, given that bin/buildout begins with:

#!/usr/local/bin/python2.5

import sys
sys.path[0:0] = [
  '/Users/graham/development/pylons/foxtrot/eggs/zc.buildout-1.0.0b27-py2.5.egg',
  '/Users/graham/development/pylons/foxtrot/eggs/setuptools-0.6c6-py2.5.egg',
  ]

And I had another problem. I wanted to install the development version of pylons into my buildout. So I specified pylons==dev. But buildout gave an error since the version which got installed (0.9.6dev-r2373) wasn’t the version requested. But then specifying pylons >= 0.9.6dev-r2373 worked.

Once you have everything running nicely in your buildout environment, you probably want to then deploy it somewhere. Creating consistent environments between systems in buildout’s main purpose, so how do we do it? It’s described in detail in the buildout docs in the section controlling eggs used.

The configuration file buildout.cfg can contain sections specifying the versions of packages required. To find out what versions are being used, buildout can be run in verbose mode, using the -v option. Strangely, it seems that you then need to manually write these versions into the config file.

Hopefully this simple example will be enough to show the potential of buildout. The complete docs are very thorough and describe all the possible options.