http://twistedmatrix.com/trac/ticket/4881

Basically, it comes down to running:

mount -t fdescfs null /dev/fd

and putting the equivalent in your /etc/fstab — otherwise, Twisted will have mysterious problems related to file descriptors and child processes. A fix is due for review and should be included in trunk at some point.

Thanks to David (big thanks for hosting), Glyph, JP and Itamar — I had a great time at the Twisted sprint last night

The bulk of this code is courtesy of _habnabit (thank you!), I just added authentication support and fixed a couple of bugs which were stopping the MySQL protocol parser working.

This works well enough to .fetchall() basic results sets and .query() any other MySQL statements you care to run. See example.py.

Feel free to fork, tweak, fix, use, report issues, etc.

Just before the kernel boots (which you have to be quick to catch with Flexiant’s VNC client) hit F6 on the bootloader and type:

set hw.clflush_disable=1
boot

This will allow you to boot even a GENERIC kernel. Once you’re booted, chuck hw.clflush_disable="1" into /boot/loader.conf to make this permanent.

You’ll then want to build your own kernel for paravirtualised network and disk drivers. Edit the XENHVM kernel config (see the FreeBSD handbook on compiling your own kernel) – comment out the MODULES_OVERRIDE line which disables building all the modules (assuming you want ZFS support) and also comment out the whole section about WITNESS and INVARIANT, as having this enabled will slow down your kernel quite significantly.

Then you’ll need to patch the network driver as per this post (manually, since the code has changed a bit), else you get a lot of dropped packets:

http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00598.html

Then just (as root):

cd /usr/src
make buildkernel KERNCONF=XENHVM
make installkernel KERNCONF=XENHVM
shutdown -r now

And enjoy your speedy FreeBSD 8.1 VM in the cloud!

Oh and check out our cloud web hosting platform, Hybrid Web Cluster which takes advantage of this

A couple of Fridays ago, after a lovely mini-holiday in Dublin with Kate (in the days after she was talking at the Aristotelian society conference), I demonstrated my web cluster to ElasticHosts, and they loved it! We’ve provisionally agreed a referral agreement as their “cloud sites provider of choice”. My favourite bit was when Richard, their CEO, described our platform as “clearly the next generation of shared hosting”. I’ve only been working for three years to hear that…

The following Wednesday I jumped straight in the deep end with public speaking, explaining the technology I’ve spent three years developing in front of a terrifyingly large room full of people at the London OpenSolaris User Group. It went well! After getting over the initial nerves I settled in nicely to the talk. Getting some laughs for “here’s some Dtrace, I haven’t got any idea how it works!” really boosted my confidence and the rest was easy. I can imagine talks could easily go the other way though. I was lucky.

The same Thursday, hours after my talk, I went to the TweetDeck offices in Old Street, London. They offered me a job! I’m really excited about this. Reza, their sole backend engineer at present, and my new boss, looks set to be a real mentor for me. Here’s a short list of just some of the truly exciting technologies I’ll be working with there: Twisted (of course), RabbitMQ, Memcache, Django, Boto, Celery, Fabric, Node.js, Redis, Nginx. As technologies for building web-scale distributed systems with an emphasis on Python, this couldn’t be a better fit! The fact that they’re flexible enough to offer me a position with two days in London and one day working from home makes it totally awesome. Thanks TweetDeck!

I should say at this point that I’m really indebted to Simpleweb back in Bristol. They’ve been really flexible with me and I’ve really valued the fact that my job there has chilled me out, rather than stressing me out. It was exactly what I needed to get away from the freelance lifestyle which was causing me serious headaches about 10 months ago. On the flipside, I am also glad that I won’t be building websites any more. Even better that I can say that I’ll hopefully never have to write another line of PHP. Systems development makes more sense for me.

So on Friday I handed in my notice, which was a relief to get out in the open. I really don’t like keeping secrets.

Then I had Saturday off! Actually OFF! It was awesome. Sol was around, so we spent the whole day drinking ale, having multiple barbecues and playing with the chickens in our garden. Got to see Lewis too. Really nice day off.

So that brings us pretty much up to today… I’m on the train back to Bristol at 12:49am (work tomorrow at 9 *sigh*) after a great FreeBSD users meeting where I was talking again. I had a really great evening. It was a much smaller event, about 20 people in a small-ish conference room with free beer and crisps. Really friendly and very technical vibe, seemingly quite a lot of core FreeBSD and OpenBSD contributors. Exciting to get to know some of the community.

Got a warm reception to my talk once again and enjoyed it more this time. As it was a much more informal event there was interruptions and dialogue, which was fun as it allowed us to go off on interesting tangents. Went to the pub with everyone afterwards and over a few drinks had several rather interesting conversations. Sean (whose rack Digital Crocus is hosted in) came along to the pub afterwards as well. I was surprised by just how many techies working in banks there were there, pretty much 100%!

One such guy I spoke with was talking about his background in synchronous distributed filesystems for a banking operation (for data centres 20 miles apart, failover via BGP). His broad impression of our product seemed good, he said he’d be interested in how we deal with the problems we’ll find when running clusters of hundreds of nodes across three data centres! I can’t wait to get to the point of solving those problems

One other bit of news — we had our first genuine unsolicited sales enquiry today from a Belgian ISP who want a 12-node cluster. Woohoo!

So I’m tired but happy — looking forward to my holiday in August. I feel like I’m pretty well sorted here in England. I think working in London is going to present many exciting opportunities. I can’t believe how well everything’s going!

Start with the right imports:

from collections import defaultdict
from twisted.internet.defer import Deferred

Now, a defaultdict allows you to say:

a = defaultdict(lambda: 0)
a[10] += 1
print a[10] # => 1

That is, you pass it a function (lambda: 0 is just the function that returns zero every time you call it) which returns the default value for the values of the mapping, and then you can access the keys of the mapping and get the default value for a key which doesn’t yet exist in the mapping, instead of a KeyError.

Combining this with deferreds allows an extraordinarily powerful mechanism to attach event handlers to arbitrary events which may not have happened yet. Here’s a contrived example (recall, in Python a class constructor is just a callable which returns an instance of that class, which is why we don’t wrap Deferred in a lambda):

on_mount = defaultdict(Deferred)

# ... I want to do something if filesystem abc ever gets mounted...

def hooray(arg):
	print "My favourite filesystem got mounted - %s" % str(arg)

on_mount['abc'].addCallback(hooray)

Now, we can do this without ever knowing that ‘abc’ might get mounted (the filesystem name might come from the network), or pre-populating the dictionary at all.

Later on in the code, we might actually mount ‘abc’:

def mount_handler(filesystem):
	on_mount[filesystem].callback(True)
	on_mount[filesystem] = Deferred() # reset the deferred

The beauty of it is that any number of event handlers (zero or more) can be attached to on_mount, with no additional house-keeping. All the callbacks attached to the deferred will get fired when the filesystem gets mounted.

The only thing to notice is that we have to reset the deferred if this event might be triggered again, as deferreds are one-shot. If it’s a one-off event, you can omit this, and then you get the additional nice behaviour that if you addCallback to a deferred after it’s been fired, you get an immediate callback with None as the argument, to indicate that the deferred has already run (from which you can deduce that the filesystem is already mounted).

Gprof2Dot outputs pretty awesome graphs. This is what you get when you run AwesomeProxy for three minutes at 10 requests per second (each request taking 3 seconds and doing a database update). I really like how you can see the structure of the code

It’s from a rare 12″ vinyl, and it’s totally beautiful.

Thanks to Kate for finding it