Straight Ornamental

New blog has RSS

Graham Stratton — Sun, 20 Jan 2008 03:03:20 +0000

Any subscribers to this blog might like to subscribe to my new blog instead. The feed is at http://grahamstratton.org/straightornamental/feed.rss

New Blog

Graham Stratton — Tue, 04 Dec 2007 17:18:00 +0000

I haven’t posted much recently, mainly because I’ve found using Textile markup a pain. So I’m moving to a new Python-based blog.

It’s currently living at http://grahamstratton.org/straightornamental/ until I get the old posts moved over

Software for Python presentations

Graham Stratton — Sat, 30 Jun 2007 12:16:04 +0000

Whilst trying to prepare my EuroPython presentation, I encountered the severe lack of a presentation program for the task. I certainly don’t have time to create such a program at the moment, but I thought I’d write down the requirements anyway. I suspect most of the necessary components are available, so it would just be a matter of integrating them.

What I’d like is a program that understands that certain sections are code snippets, allowing doctest to be run on them and providing syntax highlighting. That’s probably about it, really. I guess the obvious solution would be an extension to ReST doctests, in order to indicate presentation-specific information such as slide boundaries (is that it?). That would probably be enough to be useful, and would make it very easy to make a document out of a slideshow and vice versa.

Any advice on components to create such a system would be gratefully received!

Adventures with buildout

Graham Stratton — Wed, 27 Jun 2007 22:22:00 +0000

Recently I wanted to try some modifications to a Pylons project against the latest Pylons trunk without upgrading the system-installed Pylons, so I decided it was about time to try Buildout. Jim Fulton’s buildout (or zc.buildout, since Jim likes to put things in namespaces), is a package to create development and deployment versions of software. It’s not in any way Zope-specific, and whilst it is written in Python primarily for installing Python programs, it’s not really Python-specific either.

It is quite complex. Maybe very complex. But I think that this is only because deployment is a complex business. I can imagine using nearly all the features in the buildout docs, and a few more besides. But I did find a lack of really simple examples for us mortals.

The example I’m going to give here is this: suppose you have a project in a repository. You can check it out, but when you’ve done so you want a way to install all the dependencies, without affecting what’s already installed on your system.

Buildout makes this easy. You need to add two files to your repository:

bootstrap.py which installs the latest buildout and setuptools

buildout.cfg which contains the configuration for your install

You can get bootstrap.py from the Zope Corp subversion server at http://svn.zope.org/zc.buildout/trunk/bootstrap/

Here’s my buildout.cfg:

[buildout]
parts = foxtrot

[foxtrot]
recipe = zc.recipe.egg
interpreter = python2.5
eggs = python-cjson
       nose
       twill
       pastescript
       pylons == 0.9.5
       sqlalchemy
#       pymssql
find-links = http://python.cx.hu/python-cjson/python-cjson-1.0.3x5.tar.gz
             http://www.mirrorservice.org/sites/download.sourceforge.net/pub/sourceforge/p/py/pymssql/

So my buildout is simple. It only has one part, and the recipe for that is the simple egg recipe zc.recipe.egg. By specifying the interpreter option, I will get a python executable in my bin directory. When I run it, the environment will contain all the specified packages. I have commented out pymssql because it has too many system dependencies.

Packages which are explicitly specified (not installed as dependencies of other packages) and which define scripts to be installed will have those scripts written to ./bin/ instead of the system directory, and will work in the new environment. So I’ve explicitly specified nose and pastescript, so that I end up with ./bin/nosetests and ./bin/paster working with this new environment. Cool, eh?

The ‘find-links’ option is just a list of locations other than the cheeseshop in which to search for packages to install.

Once you have your configuration, setting up the environment is simple, if a little slow (mainly due to pypi being rather lethargic at the moment). First, run

python bootstrap.py

, to install buildout itself. Then run ./bin/buildout, and everything should magically install. If you want to run it again, it will be much faster with the -N option, which tells buildout not to look for new versions of packages if it already has a version matching the requirements. You can specify version for packages as you would with easy_install.

Well, I thought it all worked. But actually buildout doesn’t seem to use the setuptools that it installs, despite it modifying the path:

$ ./bin/buildout -N
Uninstalling foxtrot.
Installing foxtrot.
Getting distribution for 'Routes>=1.6.3'.
The required version of setuptools (>=0.6c6) is not available, and
can't be installed while this script is running. Please install
 a more recent version first.

(Currently using setuptools 0.6c5 (/usr/local/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg))
error: Setup script exited with 2

Which is really odd, given that bin/buildout begins with:

#!/usr/local/bin/python2.5

import sys
sys.path[0:0] = [
  '/Users/graham/development/pylons/foxtrot/eggs/zc.buildout-1.0.0b27-py2.5.egg',
  '/Users/graham/development/pylons/foxtrot/eggs/setuptools-0.6c6-py2.5.egg',
  ]

And I had another problem. I wanted to install the development version of pylons into my buildout. So I specified pylons==dev. But buildout gave an error since the version which got installed (0.9.6dev-r2373) wasn’t the version requested. But then specifying pylons >= 0.9.6dev-r2373 worked.

Once you have everything running nicely in your buildout environment, you probably want to then deploy it somewhere. Creating consistent environments between systems in buildout’s main purpose, so how do we do it? It’s described in detail in the buildout docs in the section controlling eggs used.

The configuration file buildout.cfg can contain sections specifying the versions of packages required. To find out what versions are being used, buildout can be run in verbose mode, using the -v option. Strangely, it seems that you then need to manually write these versions into the config file.

Hopefully this simple example will be enough to show the potential of buildout. The complete docs are very thorough and describe all the possible options.

Shared sign-on across web applications

Graham Stratton — Fri, 25 May 2007 16:35:00 +0000

I’ve spent the day on what I can only assume is a really common problem with a distinct lack of a solution.

We have a number of web applications, some Zope-based and also a Pylons app served using the Paste server. We are using Apache as a proxy in front of all of them.

We’d like users to be able to sign in to any of those applications and then access any of the others. To implement their security the applications will need to know what groups the user is a member of.

To complicate matters slightly, users need to be authenticated by NLTM if possible, failing that looked up in an LDAP directory, and if that fails verified against a relational database. For NTML users we’ll need to get the groups out at some point, presumably from LDAP.

Each of the applications will have its own way of keeping track of whether the user is logged in, probably by means of a cookie.

I think there are a few potential ways of sharing the sign-on information, given the restrictions of HTTP:

1) Something similar to OpenID, maybe actually using the OpenID protocol. Write a server which the web applications redirect to if authorization fails for some request. There are OpenID interfaces for plone and AuthKit. But there is a question as to whether the OpenID spec would allow returning of the list of groups a user belongs to.

2) Have the proxying Apache authenticate users and put this information in the environment. This would work if we only wanted to protect whole directories. An Apache module to help with this is mod_auth_tkt, which does rather more. Unfortunately it still doesn’t allow you to redirect users when authorization is required. What is needed is some way of catching a 401, as is done by AuthKit, and providing a sign-on there.

What I think mod_auth_tkt does is to hash the username with a secret code and store it in a cookie. The secret code is share between all the applications. If the browser sends a request claiming to be fred and hashing ‘fred’ with the secret code produces the code passed by the browser, the app1 can conclude that app2 successfully authenticated fred and gave him the code. This is really quite neat and I wouldn’t have thought of it.

3) Use paste.proxy as an extra proxy layer. This makes it easy to add middleware which will intercept status 401 messages and replace them with a login form. Since the proxy would not have a session, it would have to use set cookies in the same way as mod_auth_tkt to communicate user verification.

The problem is further complicated by external and internal Apache instances. The internal one proxies services mainly used internally, and only accepts requests which come either from machines internally of from the external Apache instance. The external one proxies the organisational website, and also allows external access to the internal services.

Functional testing in Pylons, Twill and wsgi_intercept

Graham Stratton — Sun, 13 May 2007 20:08:00 +0000

When trying to do some functional testing in Pylons, I initially tried Paste’s solution, which worked a bit, but I got stuck on trying to select multiple options in a select box. So I decided to move over to Twill, which I am already familiar with anyway.

In order to use Twill for in-process testing, Titus Brown, the developer of Twill, has provided a cunning module called wsgi_intercept. Yes, there is a clue in the name. wsgi_intercept replaces httplib.HTTPConnection, redirecting specified requests to the wsgi application. Titus has an article on testing with Twill and wsgi_intercept.

Using the module with Pylons is quite easy. I created a new project called squirrel, with a controller called nuts. In squirrel/tests/init.py, I added:

import twill

class TwillTestController(TestCase):

    wsgi_app = loadapp('config:test.ini', relative_to=conf_dir)

    def setUp(self):
        def build_app():
            return self.wsgi_app

        twill.add_wsgi_intercept('localhost', 8080, build_app)

    def tearDown(self):
        twill.remove_wsgi_intercept('localhost', 8080)

Then I changed squirrel/tests/functional/test_nuts.py to

from squirrel.tests import TwillTestController
from twill.commands import *

class TestNutsController(TwillTestController):
    def test_index(self):
        go('http://localhost:8080/')
        find('World')
        notfind('Universe')

And running nosetests works quite nicely for me. Unless, that is, http_proxy is set, which it is on my work machine. I haven’t yet traced this problem to see whether it can be fixed or not, though I suspect there won’t be a simple solution.

Setting up Python 2.5 on OS X

Graham Stratton — Sun, 13 May 2007 13:47:00 +0000

Sadly, this isn’t quite as simple as it might be. First, set up /usr/local/src/ to contain the source of the packages to be installed:

mkdir /usr/local/src
sudo mkdir /usr/local/src
sudo chgrp admin  /usr/local/src
sudo chmod -R 775 /usr/local/src
cd /usr/local/src

Custom packages are generally installed to /usr/local/, hence /usr/local/src is a sensible place to put the source.

Now install readline:

url -O ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
tar -xzvf readline-5.2.tar.gz
cd readline-5.2
./configure --prefix=/usr/local
make
sudo make install
cd ..

One of the exciting new batteries included with Python 2.5 is bindings for Sqlite. So, we’d better install that, too:

curl -O http://www.sqlite.org/sqlite-3.3.17.tar.gz
./configure --prefix=/usr/local --with-readline-dir=/usr/local
tar xzvf sqlite-3.3.17.tar.gz 
cd sqlite-3.3.17
./configure --prefix=/usr/local --with-readline-dir=/usr/local
make
sudo make install
cd ..

And finally we can actually build Python:

wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tar.bz2
tar xjvf Python-2.5.1.tar.bz2
cd Python-2.5.1/
./configure --prefix=/usr/local --with-readline-dir=/usr/local --with-sqlite3=/usr/local
make
sudo make install
cd ..

Finally, enable tab completion in the interactive interpreter. Set ~/.pythonstartup to contain

import readline, rlcompleter
readline.parse_and_bind("tab: complete")

and set the environment variable PYTHONSTARTUP to point to this file,for example by adding

export PYTHONSTARTUP=~/.pythonstartup

to ~/.bashrc

Phew. Now I can actually do something useful! Maybe my next post will be able to contain some more significant substance!

A quick trick I came across today. I had code working in the interactive interpreter, but I wanted to copy into an editor. So I copied the block into TextMate, and needed to strip out the >>>s, the …s and the output. Text -> Filter Through Command, setting the command to be

egrep ">>>|..." | cut -c5-

and all sorted. Another instance of a nice blend of GUI and command line in OS X.

Virtualisation

Graham Stratton — Mon, 09 Apr 2007 15:49:28 +0000

Virtualisation is the current big thing, and it’s easy to see why. But in many ways it opens up such a flood of possibilities that it’s hard to understand where to start. Virtualisation also has interesting implications for hardware and software vendors.

Products

By far the largest player in the virtualisation market is VMware. Microsoft are clearly not comfortable with this situation, and since their Virtual Server product cannot compete and they will not have anything new until at least 2008, they have announced support for VMWare’s main rival, XenSource, a company built around the Open Source Xen virtualisation platform.

The ways that Xen and VMWare work are quite fundamentally different. VMWare’s solution requires slightly more overhead, but means that pretty much any x86 OS will run in a VMWare virtual machine, and these virtual machines can be hosted on a variety of other OSes. Having got a bit of experience with VMWare’s VMWare Server, (which has become free for download since the increase in competition from the likes of Xen), I have to say that it is very professional and reliable. Install was straightforward and painless, and downloaded virtual machines just worked as you would hope.

Wikipedia has a useful overview of virtualisation. As a rough summary, the difficulty in implementing virtualisation is with a small set of instructions which need to be handled differently on a virtual operating system to how they would be on an OS running natively. VMWare works by replacing these calls with system calls before they are run. This is responsible for the overhead when using VMWare.

Xen requires guest OSes to be modified in advance in order to be able to run, taking out offending instructions replacing them with calls to Xen itself. This means that to run under Xen, operating systems have to be available for modification (for example open source ones), and have to actually have been modified (often only the most recent version). This clearly limits Xen’s potential.

However, Intel and AMD have recently added dedicated virtualisation extensions to their chips, in the form of VT-x and AMD-V extensions. These are similar extensions, and Xen manages to add a single abstration layer above both of them called HVM. These extensions enable an unmodified OS to be started as a virtual machine, after which the OS will call the hypervisor (the code which runs the virtual machine, eg Xen) whenever a ‘problem instruction’ is encountered. This enables many more operating systems, notably including Windows, to be run, and already compatible OSes to be run without modification.

VT-x technology is available in most Intel chips from the Core Duo onwards. The exception is a few budget and low-power chips. However, some hardware vendors disable the functionality, particularly in laptops.

Apple and Vistualisation

Apple are doing very well at the moment, and part of that is because since they moved to Intel processors it has been possible to run Windows on Macs. The means that the ‘I like Macs but won’t buy one in case I need to run some PC software’ crowd were all suddenly potential customers. But some people need to use Windows applications pretty much continuously, and for them the pain of having to reboot to a different OS for that is still too much. Along comes virtualisation, and suddenly they can run Windows and OSX simultaneously. Great. So now they’re potential customers.

But OS X on Intel is (somehow) restricted to only be able to run on Apple hardware. And currently this means that you can’t have an OS X virtual machine, even running on a Mac. So whilst I can have a few Linux virtual machines lying around for testing different things, I currently can’t have a second OS X install. Maybe this is a critical thing, maybe not. It does seem to limit Apple’s potential in the server market, though. Parallels does support Intel’s VT-x virtualisation extenstion, which enables guest OSes to run with no modification directly on the hardware, but it seems that this still isn’t enough.

I was just browsing the Parallels website, and encountered their new Coherence technology, allowing you to run Windows ‘in the background’ with Windows application windows floating around your desktop alongside you Mac apps. Very cool, if an obvious step on a route to confusion!

Applications of Virtualisation

So, what uses does virtualisation have? There are a few obvious savings where a number of different OSes need to be run simultaneously, for example for testing. And it can be useful in order to try out a different OS. But more interesting is to ask what problems currently exist, and consider whether virtualisation is a good solution.

As a web application developer, I spend a surprising amount of my time doing sys admin stuff, and I’m still far from having a good setup. I’d like a more solid backup strategy, and some sort of continual testing.

I offer my clients a hosting service, but at the moment if my server fails it will take a few hours of work on my part to get the sites hosted there back up and running (relying on some notes which I will have to pull out of Google’s cache!). If I had all of the sites backed up to a virtual machine, I could just copy the virtual machine onto another server and be ready to go.

Of course, the obvious next step here is to replace my server with a virtual server which is continually backed up. This doesn’t seem to be as popular an option as I was expected. None of the virtualisation vendors seem to be pushing the ability to do this. Possibly part of the reason is that even if you have a continual backup of the disk image, you are far from having a continual backup, as writes may be buffered. If you snapshot a disk image of a running system and run it, the OS will assume that it crashed. To take a snapshot of a running virtual machine, you have to snapshot the memory as well, which is entirely possible, but means you might have to back up more data (and certainly have more changes in incremental backups) than you might expect.

One feature which the Xen guys like to promote is the ability to transfer a virtual machine seamlessly from one host to another. This is a very nice and useful feature. The downtime is estimated at about 0.1 seconds, so provided that your network gear switches over quickly enough everything should go unnoticed. However, this requires a large amount of bandwidth, and therefore having a server continuously in a state where it could instantaneously fail-over is not practical, though maybe mirroring every ten seconds would not be unreasonable within a private network.

Overdoing it?

After first encountering virtualisation, I soon wanted to run everything in a separate machine. Then I began to ask why. Surely most operating systems are perfectly capable of running many separate services, and having a separate install of an operating system and lots of standard packages for one small task seems rather wasteful, particularly of memory (which isn’t quite free yet, unlike disk space).

So, is there a limit? It does make sense to run each service on a separate machine (say mailserver, webserver, firewall (virtual firewall??)), but what about individual web applications? Certainly some of my deployed apps require different libraries to others, and it might be handy to be able to upgrade them individually. But there’s workingenv to create a completely separate python environment, so only system libraries are an issue.

A few random thoughts

I haven’t yet seen anyone saying that they can dynamically move memory between systems. I wonder whether that’s possible?

End.

This post is getting a bit long, so I think I’ll return to ‘what virtualisation can do for me’ another time.

Backup tools

Graham Stratton — Thu, 15 Feb 2007 18:37:28 +0000

I was investigating backing up track when I came across some useful tools. First is backupninja, which allows you to take a backup of lots of different applications such as databases whilst they’re running.

More interesting to me was rdiff-backup, at http://www.nongnu.org/rdiff-backup/ rdiff-backup basically performs an rsync mirroring (it uses librsync), but actually performs an incremental backup allowing you to restore older versions of files, and has some other nice features as well.

Trac setup

Graham Stratton — Thu, 15 Feb 2007 17:39:06 +0000

Just in case I haven’t already expressed my opinion on this: trac is wonderful. It’s a simple integration of subversion with an issue tracker and a wiki. It’s probably not that difficult to set up by default, but for some reason I chose to use Postgres instead of SQLite, so there’s a little bit to it. I covered installing everything elsewhere, so here I’m just going to describe setting up the database and configuring it all.

First we need to create a subversion repository. This needs to be writable by the user which Apache is running as, probably www-data or similar.

sudo svnadmin create /var/svn/hippocms
sudo chown -R www-data /var/svn/hippocms/

Now we need to configure Apache:

  <Location /svn>
    DAV svn
    SVNParentPath /var/svn
    AuthType Basic
    AuthName "Subversion" 
    AuthUserFile /etc/svn-auth
    Require valid-user
  </Location>

The SVNParentPath variable makes all the subdirectories of this directory which should be repositories available, but denies access to the directory itself.

We’re only protecting the repository with BasicAuth here; you’ll need to use https to make this secure. We need to create the htpasswd file:

sudo htpasswd -c /etc/svn-auth graham

So now, you should be able to access the repository through the web at http://your.server/svn/hippocms/

And you should be able to check out its emptyness with

svn co http://cambridgewebdevelopment.co.uk/svn/hippocms/

Next we set up the Apache to host trac:

  Alias /trac "/usr/share/trac/htdocs" 

  ScriptAlias /hippocms /usr/share/trac/cgi-bin/trac.cgi
  <Location "/hippocms">
    SetEnv TRAC_ENV "/var/trac/hippocms" 
    AuthType Basic
    AuthName "hippocms" 
    AuthUserFile /var/trac/hippocms/.htpasswd
    Require valid-user
  </Location>

And we need to add a user for Apache to authenticate:

sudo htpasswd -c /var/trac/hippocms/.htpasswd graham

Finally we need to create a database and create a trac user:

sudo su - postgres -c createdb hippocms
sudo su - postgres -c psql hippocms
# create user trac with password 'secret';

Now we’re ready to initialise our trac environment:

sudo trac-admin /var/trac/hippocms initenv

And everything should Just Work.