FormEncode

14 June 2008 10:27

This article was written in order to help prepare my Europython talk on FormEncode. It isn't complete, but I thought it might be useful to publish it as is.

FormEncode is one of my favourite Python libraries. However, I have struggled quite a bit with it at times, and from reading the Pylons mailing list it seems that other people also have issues with it. Here I'm going to try to document how to perform some common tasks in FormEncode with Pylons. I'm far from an expert on FormEncode, and this article is as much for my own education as anyone else's; still, I hope it's useful to someone. I'm not going to try to cover "FormEncode's philosophy", that's covered on the FormEncode site itself.

Prerequisites

For reference, this article refers to FormEncode 0.7.1, tested with Python 2.5. There are significant bugs in the integration between Pylons 0.9.5 and FormEncode (notably with multiple inputs); you should use at least 0.9.6.

The first half of the article deals with FormEncode separately, the second half looks at the integration with Pylons.

Example

If you want to use FormEncode with Pylons, chances are you want to validate some form input against some Schema. In FormEncode, a schema is itself a validator, so it's possible to nest Schema into arbitrarily complex data structures.

Let's start by defining a simple schema at the interpreter. We will define a schema which expects a Unicode string, a list of integers and a date. The schema's to_python() method will make a Python structure out of the data passed in; the from_python method attempts to reverse that process.

>>> import formencode
>>> class DemoSchema(formencode.schema.Schema):
...  n = formencode.foreach.ForEach(formencode.validators.Int())
...  s = formencode.validators.UnicodeString()
...  d = formencode.validators.DateConverter(month_style="dd/mm/yyyy")

>>> validated = DemoSchema.to_python({'n':['1','2'], 's':'ABC', 'd':'31/1/2007'})
>>> validated
{'s': u'ABC', 'd': datetime.date(2007, 1, 31), 'n': [1, 2]}

>>> DemoSchema.from_python(validated)
{'s': 'ABC', 'd': '01/31/2007', 'n': [1, 2]}

If we make mistakes in our input, FormEncode will tell us about them:

>>> DemoSchema.to_python({'n':['one','two'], 'd':'31/6/2007'})
...
Invalid: d: That month only has 30 days
n: Errors:
Please enter an integer value
Please enter an integer value
s: Missing value

Above we have passed in a list as the value for n. But in Pylons, a form which returns multiple values will have those values stored in a MultiDict. Let's create a MultiDict and decode it with our Schema:

>>> from paste.util.multidict import MultiDict
>>> m = MultiDict([('n','1'), ('n','2'), ('s','ABC'), ('d','1/1/2008')])
>>> DemoSchema.to_python(m)
...
Invalid: The input field 'n' was not expected.

Hmm, that didn't work. Schema doesn't expect multiple keys. What we need to do is to map our multiple values to a list, leaving single items in place. There are a couple of ways of achieving this. Firstly, MultiDicts have a mixed() method, which does exactly what we want:

>>> from paste.util.multidict import MultiDict
>>> m = MultiDict([('n','1'), ('n','2'), ('s','ABC')])
>>> m.mixed()
{'s': 'ABC', 'n': ['1', '2']}

Secondly, FormEncode.variabledecode.variable_decode() will do this amongst other cleverer things for mapping flat fields to structures.

>>> from formencode.variabledecode import variable_decode
>>> m = MultiDict([('s-1','one'),('s-2',2),('s-3','Buckle my'),
...                 ('s-3','shoe'),('o.i','first'),('o.j','second')])
>>> variable_decode(m)
{'s': ['one', 2, ['Buckle my', 'shoe']], 'o': {'i': 'first', 'j': 'second'}}

As you can see, variable_decode maps numbered items (split with a '-') into a list, and also makes identically-named items into a list. It uses a dot notation to indicate dictionary items. This is how to use FormEncode with complex data structures.

Note: FormEncode is growing support for MultiDicts, so in future it may not be necessary to call the .mixed() method before passing schema data back to htmlfill.

Let's customise our validation slightly. Suppose we want to ensure that the list has either one or two members. We can do that like this:

>>> from formencode.compound import All
>>> from formencode.foreach import ForEach
>>> from formencode import validators
>>> OneOrTwoInts = All(ForEach(validators.Int()),
                       validators.MaxLength(2),
                       validators.MinLength(1))
>>> OneOrTwoInts.to_python([])
...
Invalid: Enter a value at least 1 characters long

So how does that work? How does FormEncode use all those validators at once? How will it decide what value to use if the validation is successful? Well, there are validators and converters. Validators raise an Exception if the data isn't valid. Converters do this too, but also return a converted value. MaxLength is a validator, whereas Int is a converter. So All() only gets passed one converter.

Now, we'd probably like a better exception message, since we're not dealing with characters. This isn't too hard:

>>> maxItems = validators.MaxLength(2, messages={'tooLong':
                   "Please select at most %(maxLength)i items"})
>>> OneOrTwoInts = All(ForEach(validators.Int()),
                       maxItems,
                       validators.MinLength(1))
>>> OneOrTwoInts.to_python(['1','2','3'])
...
Invalid: Please select at most 2 items

We might even choose to go one stage further and create our own validator which does both checks at once, and produces a nice error message:

>>> from formencode.api import FancyValidator, Invalid
>>> class OneOrTwoItems(FancyValidator):
...    messages = {
...             'wrongLength':"Please select one or two items",
...             'invalid': "Invalid value (value with length expected)"
...               }
...    def validate_python(self, value, state):
...        try:
...            if len(value) not in (1,2):
...                raise Invalid(self.message('wrongLength', state), value, state)
...        except ValueError:
...            raise Invalid(self.message('invalid', state), value, state)

>>> OneOrTwoItems.to_python([])
...
Invalid: Please select one or two items

Of course, the exciting bit of FormEncode is the integration with HTML. Here's a brief demo with a multiple select:

>>> from formencode import htmlfill
>>> selecthtml="""<select name="n">
...   <option value="1">
...   <option value="2">
...   <option value="3">
... </select>
... """

>>> print htmlfill.render(selecthtml, defaults = {'n':[1,3]})
<select name="n">
  <option value="1" selected="selected">
  <option value="2">
  <option value="3" selected="selected">
</select>

>>> print htmlfill.render(selecthtml, defaults = {'n':2})
<select name="n">
  <option value="1">
  <option value="2" selected="selected">
  <option value="3">
</select>

htmlfill.render() takes input in the format generated by the mixed() method of the original MultiDict of form data. You cannot necessarily pass validated form data back into a form and get the same data rendered back. Note that FormEncode is sneakily replacing the integers in our values with their string representations. This can lead to problems, since what we should generically do is to call the .from_python() method on our validator with our data to map it back into a suitable format for html_fill. When all items are strings or values whose str() representation is the required string, all is fine. But look what happens with a date:

>>> dc = validators.DateConverter(month_style='dd/mm/yyyy')
>>> datewidget = '<input type="text" name="date" />\n'
>>> dateinput = '20/10/1984'
>>> date = dc.to_python(dateinput)
>>> date
datetime.date(1984, 10, 20)
>>> htmlfill.render(datewidget, {'date':date},{})
'<input type="text" name="date" value="1984-10-20" />'

Now we've rendered the widget with the date in an invalid input format! So we need to remember to call .from_python() on the validator or schema before passing the value to htmlfill:

>>> htmlfill.render(datewidget, {'date':dc.from_python(date)},{})
'<input type="text" name="date" value="10/20/1984" />'

I can't think of a case where htmlfill's automatic calling of str() with non-string-type objects is useful.

Those familiar with Python will recognise that this article has been written in doctest format. In order to test the file, I cobbled together a little script. I don't know whether there's a simpler way of doing this, but this works for me:

#!/usr/local/bin/python2.5
import unittest, doctest
import sys, os
path = os.path.join(os.path.abspath(os.path.curdir), sys.argv[1])
d = doctest.DocFileSuite(path, module_relative=False)
unittest.TextTestRunner().run(d)

What would be really handy would be a doctest runner that dropped you into the interactive interpreter on test failure.

It would be even more handy if doctest didn't consider dictionaries printed in different orders to be distinct! I'm pretty sure I've seen an easy way to deal with this problem, but I've now forgotten it.

Pylons integration

The integration of FormEncode into Pylons in mainly courtesy of the @validate decorator, to be found in pylons.decorators.__init__.py. By decorating a method of your controller, you can ensure that the form data passed into that function is valid and mapped to your desired python objects. If the data is not valid, the specified form is re-rendered using FormEncode's htmlfill with the errors indicated.

If validators in the Pylons decorator is not a dict then it is silently ignored.

Comments

Mark Huang wrote on 2 March 2010:

Wow, this is THE article that I've been searching for the past couple of months.

The Formencode website documentation is not helpful at all. The explanations are so vague and examples are weak.

Thank you so much! Sadly, I figured all of this out before finding this article. I will also do a short write up of what I learnt from the endless hours of meddling of Formencode.

Great stuff!

Leave a comment