Application deployment

3 January 2009 02:56

We have a number of Linux machines onto which we want to deploy a number of applications which will run as services. There will likely need to be some amount of data transfer between the applications at some point, too, and possibly some shared configuration.

The applications which we are deploying need to be started at boot-up, restarted if they crash (and preferably also if they misbehave, such as by using too many system resources), and need to be manually controllable. We want it to be possible to easily install new versions of any of the services we have running on any of the systems.

Of course, it's also important that it's easy to install versions of the software for development and testing. In particular, it would be nice if installing on my Mac worked too.

Our standard existing deployment platform is Fedora Core 8. It would be good to have a solution that works on later versions of Fedora, but also Debian, Ubuntu and other distributions. Working on the custom Linux distributions found on some SCCs which we would like to use as embedded devices would also be an advantage.

Installing the software

Here, I'm going to assume that we're going to use zc.buildout. This is mainly because I have familiarity with it, it's extensible, it does what I want roughly how I think it should be done, and if there was anything better out there I suspect Jim Fulton would have found it.

Running the services

There are a few tools out there to run a program and monitor it to ensure that it keeps running. The two main ones I considered were zdaemon and supervisor. However, to confuse matters there is also D J Bernstein's daemontools, which covers pretty similar ground, but also provides a start-up system which works across most unix-based systems. Once you get there you also come across runit, which is meant to be an enhanced daemontools. Runit would really like to be run as process 1 and replace init, but it's not necessary in order to use it. There's a good article explaining the whats, whys and wherefores of runit.

Configuring boot-up scripts

This is possibly the trickiest bit. In our existing set-up, all custom things to be started on boot-up are in /etc/rc.local. In order to make it easy to install and upgrade and install different applications separately, we would ideally like to just place a start-up script in a directory and know that it will be run.

When daemontools installs itself it makes sure that it gets started by appending a line to /etc/inittab if it exists. From Fedora Core 9 this file still exists, but is not automatically executed on boot-up; Fedora has moved to the new upstart system. OS X has its own startup system called launchd.

zdaemon compared to supervisor

zdaemon and supervisor fill almost the same rĂ´le, so it makes sense to compare them. This thread is a good comparison.

Puppet

On my way to getting all this working, I took a look at Puppet. Puppet is a tool for managing systems. It will create files, install packages, configure services and all those other things that one usually writes flakey scripts to do. A Puppet configuration, called a manifest, can be run repeatedly and will update the necessary components. Components can depend on one another, and it's all cool. Puppet is written in Ruby. The test coverage is high, and they use Trac and Buildbot. I can't help having a very positive feel about the project.

Puppet will also do some level of supervision. All that is required is a process which daemonises itself, and a set of commands to start/stop/status it. There are buildout recipes to install both zdaemon and supervisor with an application. However, following slightly the philosophy of Daemontools, I decided it made more sense to install the application and install something to daemonise it entirely separately.

I used zdaemon to do daemonise my process, as it produces an executable with start, stop and status commands. Supervisor doesn't install an init-style script. There's one for Debian in the respository here, but that's all, and that doesn't include a status command. There was a slight snag; the zdaemon status command returns exit status 0 even if it's not running. Since this is how Puppet tells whether the process is running, I hacked zdaemon to return 1 if the supervisor process is not running.

I should add that Puppet can either check the status of a process every time it updates itself (by default on boot-up and then every half hour), and start the process if it's not running, or it can configure the process to be started by init. It doesn't really make that much sense to do both, and I opted for the former since it makes it cleaner if I decide that I don't want the process running.

Not doing any of that

You might think that it would make sense for Puppet to do roughly what zdaemon does, so that you can just install an application and tell Puppet to make in into a service. But I have something which works, so I'm happy.

Another improvement at some point may be to move to using Supervisor instead of zdaemon. Supervisor is much more full-featured, including being able to configure notifications on failure, clever restarting logic and memory monitoring. But until I need one of these features, I'm quite happy.

Puppet Introduction

Puppet manifests are written in a custom language which borrows heavily from Ruby syntax. The code below demonstrates a lot of the features of Puppet, and I concluded that there are too many to attempt to explain them all!

But as a quick overview, I have defined a, erm, definition which takes a buildout config file and installs the application. I also created a definition to take an executable and make in into a service. (Unfortunately this only works under linux as it puts a script in /etc/init.d/. It would also be possible to write the control script to the application directory and then specify the start stop and status command explicitly in the service resource.)

Below is the Puppet configuration file which controls all the above. It requires Puppet to be configured to serve some files, including the application buildout file and a separate buildout which just installs zdaemon. (zdaemon could have just been installed globally, but this felt a little cleaner).

Exec { path => "/usr/bin:/usr/sbin" }

define setuptools($executable) {
    file { ezsetup:
        name => '/tmp/ez_setup.py',
        source => 'puppet://server.example.net/files/ez_setup.py',
    }

    exec { install:
        require => File[ezsetup],
        command => "$executable /tmp/ez_setup.py",
    }
}

class setuptools25 {
    setuptools { setuptools25:
        executable => 'python2.5',
    }
}

class zdaemon25 {
    buildout { zdaemon:
        require => File['/cwd'],
        path => '/cwd/zdaemon/',
        config_file => "puppet://server.example.net/files/zdaemon.cfg",
        python => 'python2.5',
    }
}


define service_daemon($exe, $dir, $service_name) {
    include zdaemon25

    $python = 'python2.5'

    $init_script = "#!/bin/sh
/cwd/zdaemon/bin/zdaemon -C ${dir}zdaemon.conf \$1
"

    $zdaemon_conf = "<runner>
 directory $dir
 program $exe
 transcript ${dir}log/transcript.log
</runner>

<eventlog>
 <logfile>
  path ${dir}log/zdaemon.log
 </logfile>
</eventlog>
"

    file { "${dir}log":
        ensure => directory,
    }

    file { "/etc/init.d/$service_name":
        content => $init_script,
        mode => 755,
    }

    file { "${dir}zdaemon.conf":
        content => $zdaemon_conf,
    }
}

define buildout($path, $config_file, $python) {
    include setuptools25

    file { $path:
        ensure => directory,
    }
    file { "$path/bootstrap.py":
        source => "puppet://server.example.net/files/bootstrap.py"
    }
    file { "$path/buildout.cfg":
        source => $config_file
    }
    exec { "bootstrap_$path":
        require => [File["$path/bootstrap.py"], File["$path/buildout.cfg"]],
        command => "cd $path; python2.5 bootstrap.py",
        unless => "test -d $path/bin",
    }
    exec { "build_$path":
        require => Exec["bootstrap_$path"],
        command => "cd $path; ./bin/buildout",
        subscribe => File["$path/buildout.cfg"],
        refreshonly => true,
    }
}

file { "/cwd":
    ensure => directory,
}

$python25headers = $operatingsystem ? {
    fedora => "python-devel",
    default => "python2.5-dev"
}

package { $python25headers:
    alias => pydev,
}

#Buildout, wrap with daemoniser, start myapp
buildout { myapp:
    require => [File['/cwd'], Package[pydev]],
    path => '/cwd/myapp/',
    config_file => "puppet://server.example.net/files/myapp.cfg",
    python => 'python2.5',
}

service_daemon { myapp:
    service_name => myapp,
    dir => '/cwd/myapp/',
    exe => '/cwd/myapp/bin/myapp',
}

service { myapp:
    require => [Buildout[myapp], Service_daemon[myapp]],
    ensure => running,
}

The unsolved problems

In some ways you might argue that the above fails to get anywhere near solving the problem that I set out to solve. In particular, there is still no simple way to decide which version of myapp is deployed. You would need to find a buildout configuration file for the correct version and copy it to the Puppet file serving directory. That's not too bad for performing an upgrade, but it's pretty messy for performing a downgrade or, more commonly, seeing what version is currently deployed.

It would be much better if I could just specify the version of myapp in the Puppet configuration. Hopefully that will happen soon!

Comments

Trevor wrote on 14 December 2009:

First comment ever on the Internet by a sheep

Leave a comment