Boxing

One of the expected ones was the boxing of low-level types. If you want to play with Java libraries, you might need to pass unboxed values around. But as soon as you touch anything in Clojure, it's automatically boxed. Calling libraries with values is fine: Clojure does some magic, but if you need to pass, say, an array of floats, you need to convert your Clojure vector explicitly. I made the mistake of expecting this to work:

(into-array [(double 1) 2.0 3.0])

Since into-array is a function which creates an array in which everything is the same type as the first item. However, what actually happens is that the double has to be boxed to be put into the Clojure vector, so you get an array of boxed doubles. The correct solution is:

(double-array [1 2 3])

Java libraries

In order to use ChartDirector with Clojure I had to download a couple of web libraries (servlet-api and jsp-api) and add them to my classpath. This wasn't too painful when I worked out what to get, but there are multiple jars out there containing the classes I needed, and I don't know what might be an official source of them.

Functional programming

I'm still to be convinced that the purists from either the functional or the OO world have a point. Generally I've liked the functional side of Clojure, but today I wanted to create an exponentially weighted moving average function. It took me a long while! Here's what I came up with:

(defn ema
([f values]
  (ema f (rest values) (first values)))
([f values running]
  (if (empty? values)
    (list running)
    (conj (ema f (rest values) (+ (* f (first values))
                                  (* (- 1 f) running))) running))))

Although this seems to work, it won't work with very large lists: I need to use recur since the JVM doesn't support tail-call optimisation. I also haven't thought very carefully about whether I should be using lists or vectors. Probably the performance is good enough with both, but it's another thing to think about.

After a while writing this, I wonder how much effort it would take in Python. Answer, about two minutes:

def ema(f, values):
   running = values[0]
   for v in values:
       running = f*v + (1-f)*running
       yield running

A lot simpler. (I'm not arguing that Python is always this much clearer; I'm just making the point that some things don't seem to be suited to the functional approach).

Maybe the Clojure code could be more like the Python by using the lazy-seq macro. I couldn't see how immediately, though.

Brain size

I'm not sure whether Clojure could ever fit in my brain. There are sooo many functions and macros. And often they are synonyms which do different things. For example vector returns a vector containing its arguments, whilst vec takes a collection as an argument and returns a vector containing its contents. I haven't found the equivalent of vec for lists yet!

Inlining

I spent ages early on trying to figure out why some code (which wasn't correct) behaved in inconsistent ways. It turned out to be related to function inlining. I haven't looked into this in depth, but I'm not the only person who's been confused by it. Looking at the source for + you can see the inline option:

(defn +
 "Returns the sum of nums. (+) returns 0."
 {:inline (fn [x y] `(. clojure.lang.Numbers (add ~x ~y)))
  :inline-arities #{2}}
 ([] 0)
 ([x] (cast Number x))
 ([x y] (. clojure.lang.Numbers (add x y)))
 ([x y & more]
  (reduce + (+ x y) more)))

Installing the software

Here, I'm going to assume that we're going to use zc.buildout. This is mainly because I have familiarity with it, it's extensible, it does what I want roughly how I think it should be done, and if there was anything better out there I suspect Jim Fulton would have found it.

Running the services

There are a few tools out there to run a program and monitor it to ensure that it keeps running. The two main ones I considered were zdaemon and supervisor. However, to confuse matters there is also D J Bernstein's daemontools, which covers pretty similar ground, but also provides a start-up system which works across most unix-based systems. Once you get there you also come across runit, which is meant to be an enhanced daemontools. Runit would really like to be run as process 1 and replace init, but it's not necessary in order to use it. There's a good article explaining the whats, whys and wherefores of runit.

Configuring boot-up scripts

This is possibly the trickiest bit. In our existing set-up, all custom things to be started on boot-up are in /etc/rc.local. In order to make it easy to install and upgrade and install different applications separately, we would ideally like to just place a start-up script in a directory and know that it will be run.

When daemontools installs itself it makes sure that it gets started by appending a line to /etc/inittab if it exists. From Fedora Core 9 this file still exists, but is not automatically executed on boot-up; Fedora has moved to the new upstart system. OS X has its own startup system called launchd.

Puppet

On my way to getting all this working, I took a look at Puppet. Puppet is a tool for managing systems. It will create files, install packages, configure services and all those other things that one usually writes flakey scripts to do. A Puppet configuration, called a manifest, can be run repeatedly and will update the necessary components. Components can depend on one another, and it's all cool. Puppet is written in Ruby. The test coverage is high, and they use Trac and Buildbot. I can't help having a very positive feel about the project.

Puppet will also do some level of supervision. All that is required is a process which daemonises itself, and a set of commands to start/stop/status it. There are buildout recipes to install both zdaemon and supervisor with an application. However, following slightly the philosophy of Daemontools, I decided it made more sense to install the application and install something to daemonise it entirely separately.

I used zdaemon to do daemonise my process, as it produces an executable with start, stop and status commands. Supervisor doesn't install an init-style script. There's one for Debian in the respository here, but that's all, and that doesn't include a status command. There was a slight snag; the zdaemon status command returns exit status 0 even if it's not running. Since this is how Puppet tells whether the process is running, I hacked zdaemon to return 1 if the supervisor process is not running.

I should add that Puppet can either check the status of a process every time it updates itself (by default on boot-up and then every half hour), and start the process if it's not running, or it can configure the process to be started by init. It doesn't really make that much sense to do both, and I opted for the former since it makes it cleaner if I decide that I don't want the process running.

Puppet Introduction

Puppet manifests are written in a custom language which borrows heavily from Ruby syntax. The code below demonstrates a lot of the features of Puppet, and I concluded that there are too many to attempt to explain them all!

But as a quick overview, I have defined a, erm, definition which takes a buildout config file and installs the application. I also created a definition to take an executable and make in into a service. (Unfortunately this only works under linux as it puts a script in /etc/init.d/. It would also be possible to write the control script to the application directory and then specify the start stop and status command explicitly in the service resource.)

Below is the Puppet configuration file which controls all the above. It requires Puppet to be configured to serve some files, including the application buildout file and a separate buildout which just installs zdaemon. (zdaemon could have just been installed globally, but this felt a little cleaner).

Exec { path => "/usr/bin:/usr/sbin" }

define setuptools($executable) {
    file { ezsetup:
        name => '/tmp/ez_setup.py',
        source => 'puppet://server.example.net/files/ez_setup.py',
    }

    exec { install:
        require => File[ezsetup],
        command => "$executable /tmp/ez_setup.py",
    }
}

class setuptools25 {
    setuptools { setuptools25:
        executable => 'python2.5',
    }
}

class zdaemon25 {
    buildout { zdaemon:
        require => File['/cwd'],
        path => '/cwd/zdaemon/',
        config_file => "puppet://server.example.net/files/zdaemon.cfg",
        python => 'python2.5',
    }
}


define service_daemon($exe, $dir, $service_name) {
    include zdaemon25

    $python = 'python2.5'

    $init_script = "#!/bin/sh
/cwd/zdaemon/bin/zdaemon -C ${dir}zdaemon.conf \$1
"

    $zdaemon_conf = "<runner>
 directory $dir
 program $exe
 transcript ${dir}log/transcript.log
</runner>

<eventlog>
 <logfile>
  path ${dir}log/zdaemon.log
 </logfile>
</eventlog>
"

    file { "${dir}log":
        ensure => directory,
    }

    file { "/etc/init.d/$service_name":
        content => $init_script,
        mode => 755,
    }

    file { "${dir}zdaemon.conf":
        content => $zdaemon_conf,
    }
}

define buildout($path, $config_file, $python) {
    include setuptools25

    file { $path:
        ensure => directory,
    }
    file { "$path/bootstrap.py":
        source => "puppet://server.example.net/files/bootstrap.py"
    }
    file { "$path/buildout.cfg":
        source => $config_file
    }
    exec { "bootstrap_$path":
        require => [File["$path/bootstrap.py"], File["$path/buildout.cfg"]],
        command => "cd $path; python2.5 bootstrap.py",
        unless => "test -d $path/bin",
    }
    exec { "build_$path":
        require => Exec["bootstrap_$path"],
        command => "cd $path; ./bin/buildout",
        subscribe => File["$path/buildout.cfg"],
        refreshonly => true,
    }
}

file { "/cwd":
    ensure => directory,
}

$python25headers = $operatingsystem ? {
    fedora => "python-devel",
    default => "python2.5-dev"
}

package { $python25headers:
    alias => pydev,
}

#Buildout, wrap with daemoniser, start myapp
buildout { myapp:
    require => [File['/cwd'], Package[pydev]],
    path => '/cwd/myapp/',
    config_file => "puppet://server.example.net/files/myapp.cfg",
    python => 'python2.5',
}

service_daemon { myapp:
    service_name => myapp,
    dir => '/cwd/myapp/',
    exe => '/cwd/myapp/bin/myapp',
}

service { myapp:
    require => [Buildout[myapp], Service_daemon[myapp]],
    ensure => running,
}

The unsolved problems

In some ways you might argue that the above fails to get anywhere near solving the problem that I set out to solve. In particular, there is still no simple way to decide which version of myapp is deployed. You would need to find a buildout configuration file for the correct version and copy it to the Puppet file serving directory. That's not too bad for performing an upgrade, but it's pretty messy for performing a downgrade or, more commonly, seeing what version is currently deployed.

It would be much better if I could just specify the version of myapp in the Puppet configuration. Hopefully that will happen soon!

Python

Python rocks. I'm not sure whether I could stand programming in any other language. Everything else is so redundant. Python is simple: I don't have to think about how to code something. I don't need a massive reference manual next to me just for the language itself. It makes me happy. Well, less annoyed, anyway.

So, recursing slightly, what's so good about Python?

Component Architecture

Zope 3's component architecture has lead to a level of cleanliness, reusability and testability of code that I would not have thought possible. The component architecture forces you to think about how your code works, without restricting you in any way. I increasingly notice that I write better code through trying to take advantage of the component architecture. What does that mean? Well, usually it means adding new functionality without modifying existing code at all. How? Use an adapter.

Object Publishing

I have to admit, I really don't get this MVC business for web applications, and I don't see how the standard /model/controller/id url scheme is going to make sense very often. Choosing a URL structure for a site forces it to be made into a hierarchy of some sort, and this fits with having some tree of objects (not all of which are necessarily persistent). I often don't have many views per object, often just a view and an edit view, making my applications really quite ReST-like (which isn't necessarily good, but is fashionable at the moment).

I also like having a separate object created for every view and viewlet; it feels right to me. I guess it is more like multi-model MVC.

zc.buildout

zc.buildout is unfortunately quite tricky to get started with, but is well worth a lot of investment. It enables the construction of repeatable environments for development, testing or deployment, without affecting the existing Python installation. How we got by without it I'll never know.

The ZODB

In my opinion, the ZODB is Zope's biggest asset. Change your classes to inherit from persistent.Persistent, use persistent lists and mappings, and that's it. You can completely ignore the problem of data storage. People continue to claim great things of object-relational mappers, and they continue to fail to live up to the promise, mainly due to the fundamental impedance mismatch.

I'm encouraged to see that new tools are under development for making the ZODB less scary, for example a new object browser.

And the bad bits

Okay, so what's not so good? Well, all the experimentation which goes on to develop all these great technologies also leads to a lot of less good ideas, and it can be very difficult to keep ahead and to know which ones to go for. It makes life interesting, but can be pretty frustrating when you just want to get a job done.

Python 3 is probably not that far away now, and is rather scary. There is no significant common subset between Python 2 and Python 3 (mainly due to the change in string type, which will default to unicode instead of raw bytes). What the transition will do to the Zope world remains to be seen, but there is certainly a question as to whether the cost of the change is really worth it.

Finally, Zope uses some C-level code to implement some of its security, which I believe is the sticking point getting it to run on PyPy/Jython/Google App Engine. Hopefully these problems will be navigated somehow. Zope on App Engine sounds like a very attractive proposition!

Getting it

Currently, z3c.tutorialtest is only available from Assembla:

svn co http://svn.assembla.com/svn/z3c_tutorialtest/trunk z3c.tutorialtest

If other people use it I will move it to svn.zope.org and make a release.

Using it

z3c.tutorialtest defines the http://namespaces.zope.org/tutorialtest zcml namespace with a single directive, test. The test directive registers an existing doctest as a TutorialTest, for example:

<tutorialtest:test
    id="businessdevelopment"
    title="Business development introduction"
    test_path="web/ftests/businessdev.txt"
    />

The TutorialTest test runner will set a browser object in the test namespace very similar to an instance of zc.testbrowser.real.Browser. To run the same test as part of your test suite, you'll therefore likely to want to create an instance a different browser class in your setUp code:

from zope.testbrowser.testing import Browser
from dsrscheduler.testing import FunctionalDocFileSuite

def setUp(test):
    test.globs = {'browser':Browser()}

def test_suite():
    return FunctionalDocFileSuite('web/ftests/businessdev.txt', setUp=setUp)

This package creates a new traversal namespace for tests. To view an index of all the available tests, visit /++tutorialtest++index Individual tests can be accessed at /++tutorialtest++<testid>.

To run a test, you will first need to start MozRepl (MozLab 0.1.9 on Firefox 3 works) on port 4242, and open a browser window. If the web application you are testing is running on a different machine, you will need to select "Allow outside connections" before starting MozRepl. Be aware of the security implications of this: if you're not behind a firewall, don't do it!

Then, navigate to ++tutorialtest++index and choose the test you want. The test should now play in your browser. Comments from the doctest are turned into Javascript alert boxes. (Obviously there's room for improvement there!)

Because z3c.tutorialtest runs tests against a live application, it uses the live ZODB and modifies real data. So far I've got round this by creating a new instance of the application, but that's only easy since I'm using Grok. I'm not sure what might be considered good practice here; it certainly requires some thought.

Showing the current branch in the prompt

To show the current git branch in your prompt, edit it to look something like this:

export PS1='\[\033[1;36m\]\u@\h:\w\[\033[0m\]$(__git_ps1 "(%s)")$ '

Autocompletion

git has proper command completion which you'll need to enable, like this:

ln -s /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_git-core/work/git-1.5.3.7/contrib/completion/git-completion.bash ~/
. ~/.git-completion.bash

And whilst we're on the completion theme, here's how to get sudo to do something sensible:

complete -c -f command sudo

Desktop development

Google are providing an SDK which enables you to run the same platform on your desktop in order to develop your app. I haven't looked into this is detail yet. Can you download you DB to test new changes? How do you set up a staging environment? What about running tests?

The datastore

I'm glad to see Google using something which is basically an Object database with simple queries on object types and back-references. Hopefully this will start to spread the concept of object databases into the mainstream, though I suspect that a lot of people are going to complain about the lack of an RDB. In fact, quite a few already are doing.

Google's stated reason for not using an RDB (my feeling is that 'this is better' would be good enough) is that relational databases can't be made to scale in the same way as the datastore. I can't find any reference to how the datastore deals with conflicts, so this doesn't really make any sense. If a user in Australia and a user in the UK make changes to the data, something still needs to check that those changes are not in conflict before both parties can see the results of the changes they made. Maybe instead of a having a single master server App Engine allows for multiple master servers each responsible for part of the database. For a discussion of the issues of distributed RDBs, I highly recommend Scalable Internet Architectures (and the chapter on databases is available for preview), which does what it says on the tin, though it also covers reliability and redundancy, whilst wisely giving optimisation a wide berth.

Effect on the industry

There's an interesting question as to how big an effect App Engine will have on the industry. I can imagine that this changes the landscape significantly for Paul Graham-style web start-ups (without even considering AppJet, who I think it leaves in a significantly worse situation than they were already in). App Engine significantly reduces the complexity of launching a web start-up, and possibly removes the need for investment from the business model completely.

A few commentators are suggesting that this is going to be the start of the PaaS (Platform as a Service) wars. Having control of someone's database is a really good way to tie people to service.

I see that within a few hours of the launch, someone had already produced an OpenID provider application, using the authentication against Google accounts to produce an OpenID provider based on Google logins. Very neat.

Interestingly (to me), Google have included an yaml parser and use yml as the configuration language for applications. When I encountered yaml in my brief visit to the world of Ruby I quite liked it and thought it was quite Pythonic, and I continue to be surprised that it isn't supported by the standard library. Okay, I just looked the the yaml specs. Now I understand.

In summary, I think Google did everything right; including PIL in App Engine would make it really attractive, and overall I'm quite happy. As ever, though, I can't immediately think of any uses for it!