Adventures with buildout

Posted by Graham Stratton Wed, 27 Jun 2007 22:22:00 GMT

Recently I wanted to try some modifications to a Pylons project against the latest Pylons trunk without upgrading the system-installed Pylons, so I decided it was about time to try Buildout. Jim Fulton’s buildout (or zc.buildout, since Jim likes to put things in namespaces), is a package to create development and deployment versions of software. It’s not in any way Zope-specific, and whilst it is written in Python primarily for installing Python programs, it’s not really Python-specific either.

It is quite complex. Maybe very complex. But I think that this is only because deployment is a complex business. I can imagine using nearly all the features in the buildout docs, and a few more besides. But I did find a lack of really simple examples for us mortals.

The example I’m going to give here is this: suppose you have a project in a repository. You can check it out, but when you’ve done so you want a way to install all the dependencies, without affecting what’s already installed on your system.

Buildout makes this easy. You need to add two files to your repository:

  • bootstrap.py which installs the latest buildout and setuptools
  • buildout.cfg which contains the configuration for your install

You can get bootstrap.py from the Zope Corp subversion server at http://svn.zope.org/zc.buildout/trunk/bootstrap/

Here’s my buildout.cfg:

[buildout]
parts = foxtrot

[foxtrot]
recipe = zc.recipe.egg
interpreter = python2.5
eggs = python-cjson
       nose
       twill
       pastescript
       pylons == 0.9.5
       sqlalchemy
#       pymssql
find-links = http://python.cx.hu/python-cjson/python-cjson-1.0.3x5.tar.gz
             http://www.mirrorservice.org/sites/download.sourceforge.net/pub/sourceforge/p/py/pymssql/

So my buildout is simple. It only has one part, and the recipe for that is the simple egg recipe zc.recipe.egg. By specifying the interpreter option, I will get a python executable in my bin directory. When I run it, the environment will contain all the specified packages. I have commented out pymssql because it has too many system dependencies.

Packages which are explicitly specified (not installed as dependencies of other packages) and which define scripts to be installed will have those scripts written to ./bin/ instead of the system directory, and will work in the new environment. So I’ve explicitly specified nose and pastescript, so that I end up with ./bin/nosetests and ./bin/paster working with this new environment. Cool, eh?

The ‘find-links’ option is just a list of locations other than the cheeseshop in which to search for packages to install.

Once you have your configuration, setting up the environment is simple, if a little slow (mainly due to pypi being rather lethargic at the moment). First, run 

python bootstrap.py
, to install buildout itself. Then run ./bin/buildout, and everything should magically install. If you want to run it again, it will be much faster with the -N option, which tells buildout not to look for new versions of packages if it already has a version matching the requirements. You can specify version for packages as you would with easy_install.

Well, I thought it all worked. But actually buildout doesn’t seem to use the setuptools that it installs, despite it modifying the path:

$ ./bin/buildout -N
Uninstalling foxtrot.
Installing foxtrot.
Getting distribution for 'Routes>=1.6.3'.
The required version of setuptools (>=0.6c6) is not available, and
can't be installed while this script is running. Please install
 a more recent version first.

(Currently using setuptools 0.6c5 (/usr/local/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg))
error: Setup script exited with 2

Which is really odd, given that bin/buildout begins with:

#!/usr/local/bin/python2.5

import sys
sys.path[0:0] = [
  '/Users/graham/development/pylons/foxtrot/eggs/zc.buildout-1.0.0b27-py2.5.egg',
  '/Users/graham/development/pylons/foxtrot/eggs/setuptools-0.6c6-py2.5.egg',
  ]

And I had another problem. I wanted to install the development version of pylons into my buildout. So I specified pylons==dev. But buildout gave an error since the version which got installed (0.9.6dev-r2373) wasn’t the version requested. But then specifying pylons >= 0.9.6dev-r2373 worked.

Once you have everything running nicely in your buildout environment, you probably want to then deploy it somewhere. Creating consistent environments between systems in buildout’s main purpose, so how do we do it? It’s described in detail in the buildout docs in the section controlling eggs used.

The configuration file buildout.cfg can contain sections specifying the versions of packages required. To find out what versions are being used, buildout can be run in verbose mode, using the -v option. Strangely, it seems that you then need to manually write these versions into the config file.

Hopefully this simple example will be enough to show the potential of buildout. The complete docs are very thorough and describe all the possible options.

Posted in , ,  | 1 comment

Functional testing in Pylons, Twill and wsgi_intercept

Posted by Graham Stratton Sun, 13 May 2007 20:08:00 GMT

When trying to do some functional testing in Pylons, I initially tried Paste’s solution, which worked a bit, but I got stuck on trying to select multiple options in a select box. So I decided to move over to Twill, which I am already familiar with anyway.

In order to use Twill for in-process testing, Titus Brown, the developer of Twill, has provided a cunning module called wsgi_intercept. Yes, there is a clue in the name. wsgi_intercept replaces httplib.HTTPConnection, redirecting specified requests to the wsgi application. Titus has an article on testing with Twill and wsgi_intercept.

Using the module with Pylons is quite easy. I created a new project called squirrel, with a controller called nuts. In squirrel/tests/init.py, I added:

import twill

class TwillTestController(TestCase):

    wsgi_app = loadapp('config:test.ini', relative_to=conf_dir)

    def setUp(self):
        def build_app():
            return self.wsgi_app

        twill.add_wsgi_intercept('localhost', 8080, build_app)

    def tearDown(self):
        twill.remove_wsgi_intercept('localhost', 8080)

Then I changed squirrel/tests/functional/test_nuts.py to

from squirrel.tests import TwillTestController
from twill.commands import *

class TestNutsController(TwillTestController):
    def test_index(self):
        go('http://localhost:8080/')
        find('World')
        notfind('Universe')

And running nosetests works quite nicely for me. Unless, that is, http_proxy is set, which it is on my work machine. I haven’t yet traced this problem to see whether it can be fixed or not, though I suspect there won’t be a simple solution.

Posted in  | no comments

SAP and Python

Posted by Graham Stratton Thu, 31 Aug 2006 21:33:47 GMT

There is growing interest in scripting languages amongst SAP developers. As a python enthusiast, I naturally wanted to use python to interface to SAP.

Scripting languages have to connect to SAP via RFC calls. saprfc is a library to do this in python created by Piers Harding. It’s available on the cheese shop.

For web interfaces there are a number of python web frameworks available. Django and Turbogears are possibly the best known of these. Like Rails, Django is very much centred on database-driven applications. Since interfacing to SAP is not what your average web framework is designed for, I wanted a framework where all the components could be easily replaced. Pylons was suggested, and I have to say I’m really enjoying using it.

Pylons is well documented, flexible (for example, you can easily change templating engines or even mix templates in a controller), and generally feels very clean. For database apps you can use SQLAlchemy or SQLObject for ORM; SQLAlchemy is highly recommended for its flexibility. Form validation is easy using FormEncode, without requiring automated form generation, leaving you in full control of page design.

Posted in ,  | no comments

mod_python grief

Posted by Graham Stratton Thu, 10 Aug 2006 00:27:11 GMT

I decided that I wanted to put a Pylons application behind Apache. After the grief I had with Rails, I wasn’t keen to use fcgi. So I decided to try the mod_python approach, which is generally significantly faster anyway.

Since a Pylons app is a WSGI app, this should be trivial. The first thing to do was to install mod_python, a standard download, configure, compile and install. Then the WSGI app must be made suitable for deployment via mod_python. There is a project to create a mod_python gateway, but for some reason this wasn’t included in the latest version of mod_python. So I downloaded it, and saved it as mod_python/wsgi.py.

Then I created a two-liner python program startup.py:

from paste.deploy import loadapp
app = loadapp("config:/home/graham/flightsearch/deployment.ini")

Then I had to configure apache.

<Location /flightsearch>
  SetHandler mod_python
  PythonHandler mod_python.wsgi
  PythonPath "['/home/graham/flightsearch'] + sys.path" 
  PythonOption wsgi.application startup::app
</Location>

I did all this and, erm, it didn’t work. In fact, I had segmentation faults in my Apache error log. A lot of Googling suggested a lot of possible reasons. It turned out to be that Apache uses the version of libexpat installed on the system, whereas python 2.4 ships with its own (later) version. This could just be an issue because the system is old (Fedora Core 1). Anyway, for some reason replacing the libraries on the system with the same version as python was using, and sym-linking them appropriately was enough to get things working.

Finally, Apache needs to be given access to some data and lock files associated with the application.

Posted in , ,  | no comments