Posted by Graham Stratton
Wed, 27 Jun 2007 22:22:00 GMT
Recently I wanted to try some modifications to a Pylons project against the latest Pylons trunk without upgrading the system-installed Pylons, so I decided it was about time to try Buildout. Jim Fulton’s buildout (or zc.buildout, since Jim likes to put things in namespaces), is a package to create development and deployment versions of software. It’s not in any way Zope-specific, and whilst it is written in Python primarily for installing Python programs, it’s not really Python-specific either.
It is quite complex. Maybe very complex. But I think that this is only because deployment is a complex business. I can imagine using nearly all the features in the buildout docs, and a few more besides. But I did find a lack of really simple examples for us mortals.
The example I’m going to give here is this: suppose you have a project in a repository. You can check it out, but when you’ve done so you want a way to install all the dependencies, without affecting what’s already installed on your system.
Buildout makes this easy. You need to add two files to your repository:
- bootstrap.py which installs the latest buildout and setuptools
- buildout.cfg which contains the configuration for your install
You can get bootstrap.py from the Zope Corp subversion server at http://svn.zope.org/zc.buildout/trunk/bootstrap/
Here’s my buildout.cfg:
[buildout]
parts = foxtrot
[foxtrot]
recipe = zc.recipe.egg
interpreter = python2.5
eggs = python-cjson
nose
twill
pastescript
pylons == 0.9.5
sqlalchemy
# pymssql
find-links = http://python.cx.hu/python-cjson/python-cjson-1.0.3x5.tar.gz
http://www.mirrorservice.org/sites/download.sourceforge.net/pub/sourceforge/p/py/pymssql/
So my buildout is simple. It only has one part, and the recipe for that is the simple egg recipe zc.recipe.egg. By specifying the interpreter option, I will get a python executable in my bin directory. When I run it, the environment will contain all the specified packages. I have commented out pymssql because it has too many system dependencies.
Packages which are explicitly specified (not installed as dependencies of other packages) and which define scripts to be installed will have those scripts written to ./bin/ instead of the system directory, and will work in the new environment. So I’ve explicitly specified nose and pastescript, so that I end up with ./bin/nosetests and ./bin/paster working with this new environment. Cool, eh?
The ‘find-links’ option is just a list of locations other than the cheeseshop in which to search for packages to install.
Once you have your configuration, setting up the environment is simple, if a little slow (mainly due to pypi being rather lethargic at the moment). First, run
python bootstrap.py
, to install buildout itself. Then run ./bin/buildout, and everything should magically install. If you want to run it again, it will be
much faster with the -N option, which tells buildout not to look for new versions of packages if it already has a version matching the requirements. You can specify version for packages as you would with easy_install.
Well, I thought it all worked. But actually buildout doesn’t seem to use the setuptools that it installs, despite it modifying the path:
$ ./bin/buildout -N
Uninstalling foxtrot.
Installing foxtrot.
Getting distribution for 'Routes>=1.6.3'.
The required version of setuptools (>=0.6c6) is not available, and
can't be installed while this script is running. Please install
a more recent version first.
(Currently using setuptools 0.6c5 (/usr/local/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg))
error: Setup script exited with 2
Which is really odd, given that bin/buildout begins with:
#!/usr/local/bin/python2.5
import sys
sys.path[0:0] = [
'/Users/graham/development/pylons/foxtrot/eggs/zc.buildout-1.0.0b27-py2.5.egg',
'/Users/graham/development/pylons/foxtrot/eggs/setuptools-0.6c6-py2.5.egg',
]
And I had another problem. I wanted to install the development version of pylons into my buildout. So I specified pylons==dev. But buildout gave an error since the version which got installed (0.9.6dev-r2373) wasn’t the version requested. But then specifying pylons >= 0.9.6dev-r2373 worked.
Once you have everything running nicely in your buildout environment, you probably want to then deploy it somewhere. Creating consistent environments between systems in buildout’s main purpose, so how do we do it?
It’s described in detail in the buildout docs in the section controlling eggs used.
The configuration file buildout.cfg can contain sections specifying the versions of packages required. To find out what versions are being used, buildout can be run in verbose mode, using the -v option. Strangely, it seems that you then need to manually write these versions into the config file.
Hopefully this simple example will be enough to show the potential of buildout. The complete docs are very thorough and describe all the possible options.
Posted in Python, Software development, Pylons | 2 comments
Posted by Graham Stratton
Thu, 15 Feb 2007 17:39:06 GMT
Just in case I haven’t already expressed my opinion on this: trac is wonderful. It’s a simple integration of subversion with an issue tracker and a wiki. It’s probably not that difficult to set up by default, but for some reason I chose to use Postgres instead of SQLite, so there’s a little bit to it. I covered installing everything elsewhere, so here I’m just going to describe setting up the database and configuring it all.
First we need to create a subversion repository. This needs to be writable by the user which Apache is running as, probably www-data or similar.
sudo svnadmin create /var/svn/hippocms
sudo chown -R www-data /var/svn/hippocms/
Now we need to configure Apache:
<Location /svn>
DAV svn
SVNParentPath /var/svn
AuthType Basic
AuthName "Subversion"
AuthUserFile /etc/svn-auth
Require valid-user
</Location>
The SVNParentPath variable makes all the subdirectories of this directory which should be repositories available, but denies access to the directory itself.
We’re only protecting the repository with BasicAuth here; you’ll need to use https to make this secure. We need to create the htpasswd file:
sudo htpasswd -c /etc/svn-auth graham
So now, you should be able to access the repository through the web at http://your.server/svn/hippocms/
And you should be able to check out its emptyness with
svn co http://cambridgewebdevelopment.co.uk/svn/hippocms/
Next we set up the Apache to host trac:
Alias /trac "/usr/share/trac/htdocs"
ScriptAlias /hippocms /usr/share/trac/cgi-bin/trac.cgi
<Location "/hippocms">
SetEnv TRAC_ENV "/var/trac/hippocms"
AuthType Basic
AuthName "hippocms"
AuthUserFile /var/trac/hippocms/.htpasswd
Require valid-user
</Location>
And we need to add a user for Apache to authenticate:
sudo htpasswd -c /var/trac/hippocms/.htpasswd graham
Finally we need to create a database and create a trac user:
sudo su - postgres -c createdb hippocms
sudo su - postgres -c psql hippocms
# create user trac with password 'secret';
Now we’re ready to initialise our trac environment:
sudo trac-admin /var/trac/hippocms initenv
And everything should Just Work.
Posted in Software development | 3 comments
Posted by Graham Stratton
Thu, 13 Jul 2006 21:56:00 GMT
I wrote most of this pretty much the day after I returned from Europython, but I haven't gotten round to posting it until now.
<sidenote>
I don't like markdown. Apparently it's meant to be as much like plain text as possible, but which do you do:
Heading
=======
or
###Heading###
?
Hashes being so much effort to type on Mac doesn't make them any more endearing, either. We need a Zope 3 blog application!
</sidenote>
Introduction
Notes from Europython 2006, at CERN in Switzerland. As last year, the conference was great fun, really interesting and left my brain crammed with ideas. I hope I got most of them written down!
I think that there were slightly fewer people there this year, but the enthusiam and number of interesting projects seemed to still be very high. I'm happy python's still en route to taking over the world. Not quickly, but it's getting there. Ironically the IT industry seems to be one of the slowest moving out there.
Web Frameworks
In a stark contrast to last year, when there were oodles of Zope talks and no other Python web frameworks, this year most of the talk was about the three new Python web frameworks: Django, TurboGears and Pylons.
Zope
Europython definitely left me with the impression that Zope is losing momentum. This doesn't necessarily the end for Zope; firstly, it could just be an illusion, as people were too busy being enthusiastic about other frameworks. But more importantly, there never have been that many sites built with raw Zope; Zope's success has been with the use of Plone and other CMSs built on it.
However, I do feel that the Zope 3 project is losing momentum. Most of the ideas have been experimented with, and what remains is the hard slog to make existing applications reasonably compatible with it. There's a limit to how many more major architectural changes people will put up with once major projects start to be built with it.
Zope 3 in general feels too complex to me at the moment. There were few major changes in the last release; most of them were removals of 'convenience ZCML' directives, which, whilst I agree with Philipp that the changes make Zope 3 a bit easier to understand, don't really make life any easier. The component architecture is wonderful, but there's a big step between that and a working web application.
I think that either Zope 3 is too complex, and most people will never understand it well enough to use it, or there's just far too big a gap from the books to what you need to know and understand to write an application. Maybe not on the implementation side, but in terms of data structuring and things like that. There have been a lot of discussions on the mailing lists about how to do many to many relations, whether one should do a direct object reference, and things like that. Most users could probably manage to do one way or another, but get stuck on trying to work out which one to do.
Tarek's talk on zope-cookbook.org was quite interesting for me. I had assumed that it was just a place people could upload random docs, but it's actually very carefully planned and structured. If it gets a lot more recipes in it, it should become a very valuable resource. I think it probably needs a lot more design guides. Working out how to do things is hard, but nowhere near as hard as working out what should be done in the first place.
The new boys
I attended talks on TurboGears, Django and Pylons, but didn't really take in too much, and I can't entirely remember which is which. Maybe someone will draw up a comparison table showing which ORM, templating engine, etc, each of them use. I was impressed by the number of things Pylons does using WSGI; adding functionality using WSGI demonstrates that components are genuinely easily pluggable.
Ajax
For some reason it is customary to write Ajax rather than AJAX.
A year ago, Ajax was pretty new, with Google maps and Gmail being the main users. Today no framework will succeed without Ajax support. Tarek Ziade of Nuxeo gave a talk on the state of the art. He covered a number of approaches.
Firstly, CrackAjax is a tool to convert python into Ajax. I guess the number of constructs is quite limited, unlike an Ajax backend for pypy, which was apparently demonstrated at the conference. CrackAjax has not been developed for about 8 months.
Azax allows you to describe the JS behaviour in an XML file, though the number of possible behaviours is currently quite limited.
Tarek's recommended solution is a client-side framework, the two examples he suggested being Scriptaculous with Prototype, and MochiKit.
Tarek has been good and uploaded his slides; they're certainly worth a look if you need to do Ajax in python web apps.
WSGI
The Weally Simple Gateway Interface, defined in PEP 333, is gathering pace rapidly as people realise just how powerful it is. James Gardner's talk on WSGI covers the topic nicely, and he has also uploaded slides, so I refer the reader to them. Also look at Pylons, a web framework where most of the components are integrated using WSGI.
CPSSkins
This is a very impressive technology allowing through-the-web development of web designs using a very cunning bit of Ajax on top of Zope 3. Just drag and drop your portlets and data widgets, and you're done. How cool is that going to be? There are demos at www.z3lab.org http://www.z3lab.org/sections/front-page/design-features/news-portlet-widget
Language
Python 3000
Guido's keynote on Python 3000 was, for me, very encouraging for the future of the language. At present there are not too many new and complex features coming into the language; the major changes will be getting rid of bug-inducing features.
The announcement that print will become a function instead of a statement prompted a synchronised intake of breath through the whole lecture theatre. The argument is that, well, it should be, and it makes it much easier to replace printing with logging during application development.
Relative imports will have to be explicit. Hooray!
PyPy
Unfortunately I didn't get to attend many of the pypy talks. However, the lightning talk on the features of pypy 3000 was encouraging; the core team are still highly enthusiastic about the project and its potential. There are still plenty of ridiculous sounding ideas going rounds, such as being able to change the interpreter being used at runtime, which gives me great faith in the project. It hasn't got too serious yet, and is continuing to be the research project it set out to be.
Of course, pypy isn't really a python compiler, it's really a generic compiler. Obviously the first thing you want to compile is the compiler itself, just because it's cool, but now that's done, the options are pretty much unlimited. Whether pypy ever makes python interpretation as fast as C remains to be seen!
Libraries
zc.buildout
Jim Fulton introduced zc.buildout, a tool for setting up environments with many software packages. It's not Zope specific; it builds on eggs, adding the flexibility required to be able to guarantee the correct versions of different packages and databases for development and deployment. Well worth a look, IMO. Jim has also been nice and uploaded slides.
Applications
Mercurial
Mercurial is yet another source control management tool. But it's written in python, and applies patches much faster than subversion, whilst using less memory and having a smaller repository. Apparently it doesn't deal with conflicts as well as darcs does. Mercurial has been selected as the SCM tool for OpenSolaris; it's ideally suited to this, as it's an environment where people may regularly want to apply hundreds of patches to customise a standard source tree.
MailManager
MailManager has been around for quite a long while. LogicalWare were one of the first Open Source companies in the UK to secure funding. MailManager allows companies to distribute mail to enquiry and support helplines internally, and to track messages.
LogicalWare sell this software as a service, or as support, or as pre-installed boxes. Sadly they do not receive that many contributions to the project from the open source community. It is always hard for a company to get such contributions, and it is interesting seeing which companies succeed. I think Zope corporation does better because other people are using the software for their own projects continually, so are more likely to improve it that a system like MailManager which is installed and left, probably not by a developer.
Business
A discussion on the last afternoon concluded that to support customers, a good model may be to have a connected issue tracker, for customer use, and a bug tracker for developer use. This is the model used by Launchpad. It seems a good model, so I'd be surprised if there isn't already a product which offers this solution.
Summary
Well, thanks for reading. I'd be keen to hear other opinions!
Posted in Software development, Zope3, Python, Conferences | no comments
Posted by Graham Stratton
Wed, 18 Jan 2006 22:56:00 GMT
In the first talk at the Cambridge group of the SPA, Steve Cook, a Software Architect in the Enterprise Frameworks and Tools group in Microsoft Visual Studio, talked about Software Factories: Assembling Applications with Patterns, Models, Frameworks and Tools.
He started with some figures showing how poorly most projects match up to their requirements. 20% of the American GDP now goes on computing, that’s $250bn. Only 18% of software projects are delivered on time and on budget, and these on average implement 42% of their requirements. Shocking figures? Maybe.
Steve went on to compare the software industry to other types of industry, saying that at the moment it is still really a cottage industry, and the days of mass-production are still to come. Then he said that manufacture was trivial with software (buring CDs), and that it was design that is hard. He said that the software industry doesn’t compare well with other manufacturing industries. I would have thought that The Machine that Changed the World showed that the car industry can’t (or certainly couldn’t) do design efficiently either. Interestingly, the Japanese approach to car design is to get all the stakeholders together at the beginning, so that the design happens without ignoring anyone’s interest. This seems to tie in to some degree with the XP ideas.
The finale of Steve’s talk was the introduction of a facility in Visual Studio to enable simple creation of domain specific languages for certain tasks. Whether these tasks are likely to be performed by different people to those implement the DSL was not clear.
I have to admit being rather dubious as to whether this is often likely to be a useful feature. It is interesting that the Ruby community seem feel strongly that everything should be done in Ruby, and creating extra languages is the wrong way to tackle problems. They argue, that, as in LISP, you should build up the language to your problem, rather than building down to the base level below.
What the talk did not address, which I would have been interested in, is the increasing use of frameworks for developing certain types of application. To me this makes a lot of sense, as generally the outisde bit of applications is the same, and it’s the middle bit which varies. I can’t help feeling that automatic code generation (other than to create a skeleton which you then flesh out) is fundamentally evil, but that’s another issue.
What I consider to be a more interesting way of looking at things is to ask what the minimum amount of information is to describe what we want a program to do. Clearly the programmer is going to have to input that information, but no more (except unexpected exception handling). If we want to say ‘like application x, but with these changes’, then we either need a framework or a library containing the essential bits of application x.
To me, the most interesting thing about the Zope3 and twisted frameworks is that they allow you to do simple things simple and complex things, well, with no more effort than necessary. With low level languages I always got the impression that that was an impossible goal, but fortunately it seems not. Just how clever you need to be in order to create such a framework is another question altogether.
Posted in Software development | no comments
