Posted by Graham Stratton
Sat, 30 Jun 2007 12:16:04 GMT
Whilst trying to prepare my EuroPython presentation, I encountered the severe lack of a presentation program for the task. I certainly don’t have time to create such a program at the moment, but I thought I’d write down the requirements anyway. I suspect most of the necessary components are available, so it would just be a matter of integrating them.
What I’d like is a program that understands that certain sections are code snippets, allowing doctest to be run on them and providing syntax highlighting. That’s probably about it, really. I guess the obvious solution would be an extension to ReST doctests, in order to indicate presentation-specific information such as slide boundaries (is that it?). That would probably be enough to be useful, and would make it very easy to make a document out of a slideshow and vice versa.
Any advice on components to create such a system would be gratefully received!
Posted in Python, Conferences | 3 comments
Posted by Graham Stratton
Wed, 27 Jun 2007 22:22:00 GMT
Recently I wanted to try some modifications to a Pylons project against the latest Pylons trunk without upgrading the system-installed Pylons, so I decided it was about time to try Buildout. Jim Fulton’s buildout (or zc.buildout, since Jim likes to put things in namespaces), is a package to create development and deployment versions of software. It’s not in any way Zope-specific, and whilst it is written in Python primarily for installing Python programs, it’s not really Python-specific either.
It is quite complex. Maybe very complex. But I think that this is only because deployment is a complex business. I can imagine using nearly all the features in the buildout docs, and a few more besides. But I did find a lack of really simple examples for us mortals.
The example I’m going to give here is this: suppose you have a project in a repository. You can check it out, but when you’ve done so you want a way to install all the dependencies, without affecting what’s already installed on your system.
Buildout makes this easy. You need to add two files to your repository:
- bootstrap.py which installs the latest buildout and setuptools
- buildout.cfg which contains the configuration for your install
You can get bootstrap.py from the Zope Corp subversion server at http://svn.zope.org/zc.buildout/trunk/bootstrap/
Here’s my buildout.cfg:
[buildout]
parts = foxtrot
[foxtrot]
recipe = zc.recipe.egg
interpreter = python2.5
eggs = python-cjson
nose
twill
pastescript
pylons == 0.9.5
sqlalchemy
# pymssql
find-links = http://python.cx.hu/python-cjson/python-cjson-1.0.3x5.tar.gz
http://www.mirrorservice.org/sites/download.sourceforge.net/pub/sourceforge/p/py/pymssql/
So my buildout is simple. It only has one part, and the recipe for that is the simple egg recipe zc.recipe.egg. By specifying the interpreter option, I will get a python executable in my bin directory. When I run it, the environment will contain all the specified packages. I have commented out pymssql because it has too many system dependencies.
Packages which are explicitly specified (not installed as dependencies of other packages) and which define scripts to be installed will have those scripts written to ./bin/ instead of the system directory, and will work in the new environment. So I’ve explicitly specified nose and pastescript, so that I end up with ./bin/nosetests and ./bin/paster working with this new environment. Cool, eh?
The ‘find-links’ option is just a list of locations other than the cheeseshop in which to search for packages to install.
Once you have your configuration, setting up the environment is simple, if a little slow (mainly due to pypi being rather lethargic at the moment). First, run
python bootstrap.py
, to install buildout itself. Then run ./bin/buildout, and everything should magically install. If you want to run it again, it will be
much faster with the -N option, which tells buildout not to look for new versions of packages if it already has a version matching the requirements. You can specify version for packages as you would with easy_install.
Well, I thought it all worked. But actually buildout doesn’t seem to use the setuptools that it installs, despite it modifying the path:
$ ./bin/buildout -N
Uninstalling foxtrot.
Installing foxtrot.
Getting distribution for 'Routes>=1.6.3'.
The required version of setuptools (>=0.6c6) is not available, and
can't be installed while this script is running. Please install
a more recent version first.
(Currently using setuptools 0.6c5 (/usr/local/lib/python2.5/site-packages/setuptools-0.6c5-py2.5.egg))
error: Setup script exited with 2
Which is really odd, given that bin/buildout begins with:
#!/usr/local/bin/python2.5
import sys
sys.path[0:0] = [
'/Users/graham/development/pylons/foxtrot/eggs/zc.buildout-1.0.0b27-py2.5.egg',
'/Users/graham/development/pylons/foxtrot/eggs/setuptools-0.6c6-py2.5.egg',
]
And I had another problem. I wanted to install the development version of pylons into my buildout. So I specified pylons==dev. But buildout gave an error since the version which got installed (0.9.6dev-r2373) wasn’t the version requested. But then specifying pylons >= 0.9.6dev-r2373 worked.
Once you have everything running nicely in your buildout environment, you probably want to then deploy it somewhere. Creating consistent environments between systems in buildout’s main purpose, so how do we do it?
It’s described in detail in the buildout docs in the section controlling eggs used.
The configuration file buildout.cfg can contain sections specifying the versions of packages required. To find out what versions are being used, buildout can be run in verbose mode, using the -v option. Strangely, it seems that you then need to manually write these versions into the config file.
Hopefully this simple example will be enough to show the potential of buildout. The complete docs are very thorough and describe all the possible options.
Posted in Python, Software development, Pylons | 1 comment
Posted by Graham Stratton
Sun, 13 May 2007 13:47:00 GMT
Sadly, this isn’t quite as simple as it might be. First, set up /usr/local/src/ to contain the source of the packages to be installed:
mkdir /usr/local/src
sudo mkdir /usr/local/src
sudo chgrp admin /usr/local/src
sudo chmod -R 775 /usr/local/src
cd /usr/local/src
Custom packages are generally installed to /usr/local/, hence /usr/local/src is a sensible place to put the source.
Now install readline:
url -O ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
tar -xzvf readline-5.2.tar.gz
cd readline-5.2
./configure --prefix=/usr/local
make
sudo make install
cd ..
One of the exciting new batteries included with Python 2.5 is bindings for Sqlite. So, we’d better install that, too:
curl -O http://www.sqlite.org/sqlite-3.3.17.tar.gz
./configure --prefix=/usr/local --with-readline-dir=/usr/local
tar xzvf sqlite-3.3.17.tar.gz
cd sqlite-3.3.17
./configure --prefix=/usr/local --with-readline-dir=/usr/local
make
sudo make install
cd ..
And finally we can actually build Python:
wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tar.bz2
tar xjvf Python-2.5.1.tar.bz2
cd Python-2.5.1/
./configure --prefix=/usr/local --with-readline-dir=/usr/local --with-sqlite3=/usr/local
make
sudo make install
cd ..
Finally, enable tab completion in the interactive interpreter. Set ~/.pythonstartup to contain
import readline, rlcompleter
readline.parse_and_bind("tab: complete")
and set the environment variable PYTHONSTARTUP to point to this file,for example by adding
export PYTHONSTARTUP=~/.pythonstartup
to ~/.bashrc
Phew. Now I can actually do something useful! Maybe my next post will be able to contain some more significant substance!
A quick trick I came across today. I had code working in the interactive interpreter, but I wanted to copy into an editor. So I copied the block into TextMate, and needed to strip out the >>>s, the …s and the output. Text -> Filter Through Command, setting the command to be
egrep ">>>|..." | cut -c5-
and all sorted. Another instance of a nice blend of
GUI and command line in
OS X.
Posted in Python, Mac | no comments
Posted by Graham Stratton
Wed, 22 Nov 2006 21:54:00 GMT
It sounded like a trivial task: find a library which will produce pie charts for use on the web which has a python interface. Then install it and use it.
Well, it’s taken me most of a day, and I haven’t got there yet. There are a number of small tools which might do the job, but none of them looked quite right, so I thought I’d go the whole way and install matplotlib, which builds on scipy.
I thought I’d see whether these packages were available from the cheese shop. I think they all are, but the install of scipy failed. The first problem was that scientific stuff is written in fortran, so I had to install (from source, no darwinport) gfortran, the GNU F77 compiler.
Numpy installed fine, but scipy required fftw (which also had to be compiled from source, and some other things.
Compilation then stalled with:
/usr/bin/ld: can’t locate file for: -ldfftpack
which was solved by running
python setup.py build_clib
which creates libdfftpack.a, before continuing before the standard build. Poor dependency handling between numpy.distutils commands is apparently to blame.
Then I ran the scipy tests and some of them failed. There’s a point where it says it would use the Atlas libraries if they were available. I spent a while trying to get them to install, but failed.
I moved on to attempting to install matplotlib. I found a suitable egg, and tried to easy_install it. A linking error for libpng.
sudo port install libpng
didn’t produce an error, but didn’t fix things. Compiling from source manually did. Then I got errors about not being able to find libfreetype. I though a sym-link to the library deep in /Developer/SDKs/MacOSX10.4u.sdk/ would do the trick, but it wasn’t enough. So I depaired, and returned to Google.
I found some pre-built packages for OS X, but they used the system python 2.4, where I was happy with my compiled version, and didn’t want a new one, and figured that installing a new system python would confuse things.
Darwinports also contained a port of matplotlib, so I thought that I would try that. That also insisted on installing its own python, but I was despairing, so I tried it. The full install succeeded, but
import pylab
yielded
ImportError: No module named wx
For some reason I concluded that this was more than the simple dependency problem it appears to be, and decided to try to pre-built packages instead. They installed successfully, I just need to work out how to run the python interpreter I want.
The new system python is at /usr/bin/python – but none of the tons of stuff I had already installed is available. So I’m not sure that I’ve gained anything. Especially since
import pylab
now yields
ImportError: No module named numpy
and fixing that returns me to
No module named wx
Installing the wx binaries left me with a working import. Inspired by this, I installed the darwinports binaries with
sudo port install py-wxpython
I’m hoping this will solve my problems!
But no. Now I’ve even got two terminals which report the same path for ‘which python’, but actually start different versions of the interpreter.
In summary, it’s all a mess, and I ought to learn a bit more about how compiled languages work (in particular, how the linker looks for libraries). I’m going to have to install this on a debian system too at some point. I’m anticipating it being rather more trivial.
Posted in Python, Mac | no comments
Posted by Graham Stratton
Fri, 13 Oct 2006 09:23:00 GMT
Until recently I hadn’t realised that there is an issue with file uploads, but now I have and there definitely is.
The problem is that it’s very difficult to give the user information about the progress of their upload, and if she is uploading a large number of large files this might be an issue.
Photobox have gotten round the problem by writing a java applet which is responsible for uploads, but this is overkill for many projects. (But it does also allow users to efficiently upload large numbers of images). Facebook also have a java uploader for bulk uploads. Single file uploads use the iframe trick (more on this later) to display a static message whilst the file is uploading.
Client-side javascript can’t control file uploads, since files are OS-level objects and JavaScript shouldn’t be allowed anywhere near them. Any information about the progress of an upload must come from the server. There are a couple of commercial projects which offer software to solve this problem. Both are Perl scripts. There are no PHP solutions since PHP does not provide any way to monitor the progress on an upload.
It should really be the job of the browser to keep the user informed about the status of uploads, but no browsers seem to do so.
Problems on Both Sides
So we need to implement something on the client side to display status information, and something on the server side to monitor the status of the upload. Finally we need to get them to talk to each other. Strangely enough, this bit is probably the easiest bit, thanks to AJAX libraries.
Client side
The problem on the client side is that when a form is submitted, the frame from which it is submitted blocks until the request is completed, so there is no way that it can display status information. A common trick here is to use a hidden iframe for the upload fields. There are also slightly more complex examples using multiple iframes. (An iframe, by the way, is an inline frame, such as is often used to produce a scrollable box within a page.)
Server side
Here things are even more tricky, as we need to measure how much of a file we have received, and then provide a function that the client can call to return this information. This is very complex, but not as restrictive as I may have once claimed.
Posted in Python, Javascript | no comments
Posted by Graham Stratton
Thu, 31 Aug 2006 21:33:47 GMT
There is growing interest in scripting languages amongst SAP developers. As a python enthusiast, I naturally wanted to use python to interface to SAP.
Scripting languages have to connect to SAP via RFC calls. saprfc is a library to do this in python created by Piers Harding. It’s available on the cheese shop.
For web interfaces there are a number of python web frameworks available. Django and Turbogears are possibly the best known of these. Like Rails, Django is very much centred on database-driven applications. Since interfacing to SAP is not what your average web framework is designed for, I wanted a framework where all the components could be easily replaced. Pylons was suggested, and I have to say I’m really enjoying using it.
Pylons is well documented, flexible (for example, you can easily change templating engines or even mix templates in a controller), and generally feels very clean. For database apps you can use SQLAlchemy or SQLObject for ORM; SQLAlchemy is highly recommended for its flexibility. Form validation is easy using FormEncode, without requiring automated form generation, leaving you in full control of page design.
Posted in Python, Pylons | no comments
Posted by Graham Stratton
Tue, 29 Aug 2006 15:19:31 GMT
Having got my account set up so that I can send text messages from software, I wondered what I could do with it!!
I thought that I could provide a form on my website so that people could text me. But I wouldn’t want any robots to be able to use it. Now I don’t imagine there are any robots which go around looking for text message forms, but this got me thinking about ways to avoid such problems.
The standard solution is to require the user to identify text in an image (and if the developer is feeling like making things accessible, providing an audio alternative). Such tests are known as CAPTCHAs. There is a python project pycaptcha, which generates images using PIL. I installed it, but it didn’t work due to the PIL being built without FREETYPE2 support. If I can actually think of a use for it, I’ll try it sometime.
I recommend fastsms.co.uk if you want to send text messages from software to recipients within the UK; it feels like they do most things right.
Posted in Python | no comments
Posted by Graham Stratton
Thu, 10 Aug 2006 00:27:11 GMT
I decided that I wanted to put a Pylons application behind Apache. After the grief I had with Rails, I wasn’t keen to use fcgi. So I decided to try the mod_python approach, which is generally significantly faster anyway.
Since a Pylons app is a WSGI app, this should be trivial. The first thing to do was to install mod_python, a standard download, configure, compile and install. Then the WSGI app must be made suitable for deployment via mod_python. There is a project to create a mod_python gateway, but for some reason this wasn’t included in the latest version of mod_python. So I downloaded it, and saved it as mod_python/wsgi.py.
Then I created a two-liner python program startup.py:
from paste.deploy import loadapp
app = loadapp("config:/home/graham/flightsearch/deployment.ini")
Then I had to configure apache.
<Location /flightsearch>
SetHandler mod_python
PythonHandler mod_python.wsgi
PythonPath "['/home/graham/flightsearch'] + sys.path"
PythonOption wsgi.application startup::app
</Location>
I did all this and, erm, it didn’t work. In fact, I had segmentation faults in my Apache error log. A lot of Googling suggested a lot of possible reasons. It turned out to be that Apache uses the version of libexpat installed on the system, whereas python 2.4 ships with its own (later) version. This could just be an issue because the system is old (Fedora Core 1). Anyway, for some reason replacing the libraries on the system with the same version as python was using, and sym-linking them appropriately was enough to get things working.
Finally, Apache needs to be given access to some data and lock files associated with the application.
Posted in Python, Apache, Pylons | no comments
Posted by Graham Stratton
Thu, 13 Jul 2006 21:56:00 GMT
I wrote most of this pretty much the day after I returned from Europython, but I haven't gotten round to posting it until now.
<sidenote>
I don't like markdown. Apparently it's meant to be as much like plain text as possible, but which do you do:
Heading
=======
or
###Heading###
?
Hashes being so much effort to type on Mac doesn't make them any more endearing, either. We need a Zope 3 blog application!
</sidenote>
Introduction
Notes from Europython 2006, at CERN in Switzerland. As last year, the conference was great fun, really interesting and left my brain crammed with ideas. I hope I got most of them written down!
I think that there were slightly fewer people there this year, but the enthusiam and number of interesting projects seemed to still be very high. I'm happy python's still en route to taking over the world. Not quickly, but it's getting there. Ironically the IT industry seems to be one of the slowest moving out there.
Web Frameworks
In a stark contrast to last year, when there were oodles of Zope talks and no other Python web frameworks, this year most of the talk was about the three new Python web frameworks: Django, TurboGears and Pylons.
Zope
Europython definitely left me with the impression that Zope is losing momentum. This doesn't necessarily the end for Zope; firstly, it could just be an illusion, as people were too busy being enthusiastic about other frameworks. But more importantly, there never have been that many sites built with raw Zope; Zope's success has been with the use of Plone and other CMSs built on it.
However, I do feel that the Zope 3 project is losing momentum. Most of the ideas have been experimented with, and what remains is the hard slog to make existing applications reasonably compatible with it. There's a limit to how many more major architectural changes people will put up with once major projects start to be built with it.
Zope 3 in general feels too complex to me at the moment. There were few major changes in the last release; most of them were removals of 'convenience ZCML' directives, which, whilst I agree with Philipp that the changes make Zope 3 a bit easier to understand, don't really make life any easier. The component architecture is wonderful, but there's a big step between that and a working web application.
I think that either Zope 3 is too complex, and most people will never understand it well enough to use it, or there's just far too big a gap from the books to what you need to know and understand to write an application. Maybe not on the implementation side, but in terms of data structuring and things like that. There have been a lot of discussions on the mailing lists about how to do many to many relations, whether one should do a direct object reference, and things like that. Most users could probably manage to do one way or another, but get stuck on trying to work out which one to do.
Tarek's talk on zope-cookbook.org was quite interesting for me. I had assumed that it was just a place people could upload random docs, but it's actually very carefully planned and structured. If it gets a lot more recipes in it, it should become a very valuable resource. I think it probably needs a lot more design guides. Working out how to do things is hard, but nowhere near as hard as working out what should be done in the first place.
The new boys
I attended talks on TurboGears, Django and Pylons, but didn't really take in too much, and I can't entirely remember which is which. Maybe someone will draw up a comparison table showing which ORM, templating engine, etc, each of them use. I was impressed by the number of things Pylons does using WSGI; adding functionality using WSGI demonstrates that components are genuinely easily pluggable.
Ajax
For some reason it is customary to write Ajax rather than AJAX.
A year ago, Ajax was pretty new, with Google maps and Gmail being the main users. Today no framework will succeed without Ajax support. Tarek Ziade of Nuxeo gave a talk on the state of the art. He covered a number of approaches.
Firstly, CrackAjax is a tool to convert python into Ajax. I guess the number of constructs is quite limited, unlike an Ajax backend for pypy, which was apparently demonstrated at the conference. CrackAjax has not been developed for about 8 months.
Azax allows you to describe the JS behaviour in an XML file, though the number of possible behaviours is currently quite limited.
Tarek's recommended solution is a client-side framework, the two examples he suggested being Scriptaculous with Prototype, and MochiKit.
Tarek has been good and uploaded his slides; they're certainly worth a look if you need to do Ajax in python web apps.
WSGI
The Weally Simple Gateway Interface, defined in PEP 333, is gathering pace rapidly as people realise just how powerful it is. James Gardner's talk on WSGI covers the topic nicely, and he has also uploaded slides, so I refer the reader to them. Also look at Pylons, a web framework where most of the components are integrated using WSGI.
CPSSkins
This is a very impressive technology allowing through-the-web development of web designs using a very cunning bit of Ajax on top of Zope 3. Just drag and drop your portlets and data widgets, and you're done. How cool is that going to be? There are demos at www.z3lab.org http://www.z3lab.org/sections/front-page/design-features/news-portlet-widget
Language
Python 3000
Guido's keynote on Python 3000 was, for me, very encouraging for the future of the language. At present there are not too many new and complex features coming into the language; the major changes will be getting rid of bug-inducing features.
The announcement that print will become a function instead of a statement prompted a synchronised intake of breath through the whole lecture theatre. The argument is that, well, it should be, and it makes it much easier to replace printing with logging during application development.
Relative imports will have to be explicit. Hooray!
PyPy
Unfortunately I didn't get to attend many of the pypy talks. However, the lightning talk on the features of pypy 3000 was encouraging; the core team are still highly enthusiastic about the project and its potential. There are still plenty of ridiculous sounding ideas going rounds, such as being able to change the interpreter being used at runtime, which gives me great faith in the project. It hasn't got too serious yet, and is continuing to be the research project it set out to be.
Of course, pypy isn't really a python compiler, it's really a generic compiler. Obviously the first thing you want to compile is the compiler itself, just because it's cool, but now that's done, the options are pretty much unlimited. Whether pypy ever makes python interpretation as fast as C remains to be seen!
Libraries
zc.buildout
Jim Fulton introduced zc.buildout, a tool for setting up environments with many software packages. It's not Zope specific; it builds on eggs, adding the flexibility required to be able to guarantee the correct versions of different packages and databases for development and deployment. Well worth a look, IMO. Jim has also been nice and uploaded slides.
Applications
Mercurial
Mercurial is yet another source control management tool. But it's written in python, and applies patches much faster than subversion, whilst using less memory and having a smaller repository. Apparently it doesn't deal with conflicts as well as darcs does. Mercurial has been selected as the SCM tool for OpenSolaris; it's ideally suited to this, as it's an environment where people may regularly want to apply hundreds of patches to customise a standard source tree.
MailManager
MailManager has been around for quite a long while. LogicalWare were one of the first Open Source companies in the UK to secure funding. MailManager allows companies to distribute mail to enquiry and support helplines internally, and to track messages.
LogicalWare sell this software as a service, or as support, or as pre-installed boxes. Sadly they do not receive that many contributions to the project from the open source community. It is always hard for a company to get such contributions, and it is interesting seeing which companies succeed. I think Zope corporation does better because other people are using the software for their own projects continually, so are more likely to improve it that a system like MailManager which is installed and left, probably not by a developer.
Business
A discussion on the last afternoon concluded that to support customers, a good model may be to have a connected issue tracker, for customer use, and a bug tracker for developer use. This is the model used by Launchpad. It seems a good model, so I'd be surprised if there isn't already a product which offers this solution.
Summary
Well, thanks for reading. I'd be keen to hear other opinions!
Posted in Software development, Zope3, Python, Conferences | no comments
Posted by Graham Stratton
Tue, 17 Jan 2006 09:46:00 GMT
I need a more interesting name for my blog. Last night I thought ‘import this’ would be good. Oh well. Not only has it been done, it’s been done by someone whose primary interests are python and ruby web frameworks.
However, from that blog I did come across a small conference in the states called Snakes and Rubies, and the website has some interesting information from it.
I haven’t heard on any dedicated UK python conference, uk ruby conference, or uk rails conference. I wonder how many people would be interested if such an event happened. I’d guess there are enough enthusiasts in the UK. Anyone else out there interested?
Posted in Conferences, Rails, Ruby, Python | no comments
