Archive for the ‘Python’ Category

A warning to anyone using Twisted 10.2 on FreeBSD

Monday, February 14th, 2011

Read this ticket first!

Basically, it comes down to running:

mount -t fdescfs null /dev/fd

and putting the equivalent in your /etc/fstab — otherwise, Twisted will have mysterious problems related to file descriptors and child processes. A fix is due for review and should be included in trunk at some point.

Thanks to David (big thanks for hosting), Glyph, JP and Itamar — I had a great time at the Twisted sprint last night :-)

Announcing txMySQL – native async Twisted MySQL protocol

Tuesday, February 8th, 2011

I am pleased to announce the release of txMySQL, a native Twisted MySQL protocol implementation at — and my first ever open source release.

The bulk of this code is courtesy of _habnabit (thank you!), I just added authentication support and fixed a couple of bugs which were stopping the MySQL protocol parser working.

This works well enough to .fetchall() basic results sets and .query() any other MySQL statements you care to run. See

Feel free to fork, tweak, fix, use, report issues, etc.


Sunday, July 11th, 2010

One of my favourite Twisted patterns is the combination of the powerful twisted.internet.defer.Deferred with collections.defaultdict.

Start with the right imports:

from collections import defaultdict
from twisted.internet.defer import Deferred

Now, a defaultdict allows you to say:

a = defaultdict(lambda: 0)
a[10] += 1
print a[10] # => 1

That is, you pass it a function (lambda: 0 is just the function that returns zero every time you call it) which returns the default value for the values of the mapping, and then you can access the keys of the mapping and get the default value for a key which doesn’t yet exist in the mapping, instead of a KeyError.

Combining this with deferreds allows an extraordinarily powerful mechanism to attach event handlers to arbitrary events which may not have happened yet. Here’s a contrived example (recall, in Python a class constructor is just a callable which returns an instance of that class, which is why we don’t wrap Deferred in a lambda):

on_mount = defaultdict(Deferred)

# ... I want to do something if filesystem abc ever gets mounted...

def hooray(arg):
	print "My favourite filesystem got mounted - %s" % str(arg)


Now, we can do this without ever knowing that ‘abc’ might get mounted (the filesystem name might come from the network), or pre-populating the dictionary at all.

Later on in the code, we might actually mount ‘abc':

def mount_handler(filesystem):
	on_mount[filesystem] = Deferred() # reset the deferred

The beauty of it is that any number of event handlers (zero or more) can be attached to on_mount, with no additional house-keeping. All the callbacks attached to the deferred will get fired when the filesystem gets mounted.

The only thing to notice is that we have to reset the deferred if this event might be triggered again, as deferreds are one-shot. If it’s a one-off event, you can omit this, and then you get the additional nice behaviour that if you addCallback to a deferred after it’s been fired, you get an immediate callback with None as the argument, to indicate that the deferred has already run (from which you can deduce that the filesystem is already mounted).