Here are just a few of the images from our road-trip to Boom Festival 2010:
Boom Festival 2010 road-trip
September 3rd, 2010
Last three weeks
July 27th, 2010The last three weeks have been pretty big for me.
A couple of Fridays ago, after a lovely mini-holiday in Dublin with Kate (in the days after she was talking at the Aristotelian society conference), I demonstrated my web cluster to ElasticHosts, and they loved it! We’ve provisionally agreed a referral agreement as their “cloud sites provider of choice”. My favourite bit was when Richard, their CEO, described our platform as “clearly the next generation of shared hosting”. I’ve only been working for three years to hear that… 
The following Wednesday I jumped straight in the deep end with public speaking, explaining the technology I’ve spent three years developing in front of a terrifyingly large room full of people at the London OpenSolaris User Group. It went well! After getting over the initial nerves I settled in nicely to the talk. Getting some laughs for “here’s some Dtrace, I haven’t got any idea how it works!” really boosted my confidence and the rest was easy. I can imagine talks could easily go the other way though. I was lucky.
The same Thursday, hours after my talk, I went to the TweetDeck offices in Old Street, London. They offered me a job! I’m really excited about this. Reza, their sole backend engineer at present, and my new boss, looks set to be a real mentor for me. Here’s a short list of just some of the truly exciting technologies I’ll be working with there: Twisted (of course), RabbitMQ, Memcache, Django, Boto, Celery, Fabric, Node.js, Redis, Nginx. As technologies for building web-scale distributed systems with an emphasis on Python, this couldn’t be a better fit! The fact that they’re flexible enough to offer me a position with two days in London and one day working from home makes it totally awesome. Thanks TweetDeck!
I should say at this point that I’m really indebted to Simpleweb back in Bristol. They’ve been really flexible with me and I’ve really valued the fact that my job there has chilled me out, rather than stressing me out. It was exactly what I needed to get away from the freelance lifestyle which was causing me serious headaches about 10 months ago. On the flipside, I am also glad that I won’t be building websites any more. Even better that I can say that I’ll hopefully never have to write another line of PHP. Systems development makes more sense for me.
So on Friday I handed in my notice, which was a relief to get out in the open. I really don’t like keeping secrets.
Then I had Saturday off! Actually OFF! It was awesome. Sol was around, so we spent the whole day drinking ale, having multiple barbecues and playing with the chickens in our garden. Got to see Lewis too. Really nice day off.
So that brings us pretty much up to today… I’m on the train back to Bristol at 12:49am (work tomorrow at 9 *sigh*) after a great FreeBSD users meeting where I was talking again. I had a really great evening. It was a much smaller event, about 20 people in a small-ish conference room with free beer and crisps. Really friendly and very technical vibe, seemingly quite a lot of core FreeBSD and OpenBSD contributors. Exciting to get to know some of the community.
Got a warm reception to my talk once again and enjoyed it more this time. As it was a much more informal event there was interruptions and dialogue, which was fun as it allowed us to go off on interesting tangents. Went to the pub with everyone afterwards and over a few drinks had several rather interesting conversations. Sean (whose rack Digital Crocus is hosted in) came along to the pub afterwards as well. I was surprised by just how many techies working in banks there were there, pretty much 100%!
One such guy I spoke with was talking about his background in synchronous distributed filesystems for a banking operation (for data centres 20 miles apart, failover via BGP). His broad impression of our product seemed good, he said he’d be interested in how we deal with the problems we’ll find when running clusters of hundreds of nodes across three data centres! I can’t wait to get to the point of solving those problems
One other bit of news — we had our first genuine unsolicited sales enquiry today from a Belgian ISP who want a 12-node cluster. Woohoo!
So I’m tired but happy — looking forward to my holiday in August. I feel like I’m pretty well sorted here in England. I think working in London is going to present many exciting opportunities. I can’t believe how well everything’s going!
defaultdict(Deferred)
July 11th, 2010One of my favourite Twisted patterns is the combination of the powerful twisted.internet.defer.Deferred with collections.defaultdict.
Start with the right imports:
from collections import defaultdict from twisted.internet.defer import Deferred
Now, a defaultdict allows you to say:
a = defaultdict(lambda: 0) a[10] += 1 print a[10] # => 1
That is, you pass it a function (lambda: 0 is just the function that returns zero every time you call it) which returns the default value for the values of the mapping, and then you can access the keys of the mapping and get the default value for a key which doesn’t yet exist in the mapping, instead of a KeyError.
Combining this with deferreds allows an extraordinarily powerful mechanism to attach event handlers to arbitrary events which may not have happened yet. Here’s a contrived example (recall, in Python a class constructor is just a callable which returns an instance of that class, which is why we don’t wrap Deferred in a lambda):
on_mount = defaultdict(Deferred) # ... I want to do something if filesystem abc ever gets mounted... def hooray(arg): print "My favourite filesystem got mounted - %s" % str(arg) on_mount['abc'].addCallback(hooray)
Now, we can do this without ever knowing that ‘abc’ might get mounted (the filesystem name might come from the network), or pre-populating the dictionary at all.
Later on in the code, we might actually mount ‘abc’:
def mount_handler(filesystem): on_mount[filesystem].callback(True) on_mount[filesystem] = Deferred() # reset the deferred
The beauty of it is that any number of event handlers (zero or more) can be attached to on_mount, with no additional house-keeping. All the callbacks attached to the deferred will get fired when the filesystem gets mounted.
The only thing to notice is that we have to reset the deferred if this event might be triggered again, as deferreds are one-shot. If it’s a one-off event, you can omit this, and then you get the additional nice behaviour that if you addCallback to a deferred after it’s been fired, you get an immediate callback with None as the argument, to indicate that the deferred has already run (from which you can deduce that the filesystem is already mounted).
Awesome profile visualisation
June 26th, 2010This is call profile graph of my latest invention, AwesomeProxy. This lets us move sites and databases between servers without a single failed HTTP request. Working on some optimisations, I wanted to see how much time was spent in each function call.
Gprof2Dot outputs pretty awesome graphs. This is what you get when you run AwesomeProxy for three minutes at 10 requests per second (each request taking 3 seconds and doing a database update). I really like how you can see the structure of the code
Burial & Four Tet
April 28th, 2010This is what I’m listening to at the moment: Burial & Four Tet – Moth
It’s from a rare 12″ vinyl, and it’s totally beautiful.
Thanks to Kate for finding it 
Hybrid Cluster website
March 10th, 2010I give you:
Clifton Suspension Bridge
February 16th, 2010
Can you tell that dusk is my favourite time of day for photography?
Solution to “No such file or directory: Couldn’t initialize cross-process lock in child” in Apache 2.2.13 on FreeBSD 8.0
February 12th, 2010This issue in the Apache error log was causing the web server to “partially crash” during a graceful restart with the following errors:
[Tue Feb 09 18:24:52 2010] [emerg] (2)No such file or directory: Couldn't initialize cross-process lock in child (/etc/opt/HybridCluster/apache2/logs/accept.lock.82760) (5)
[Tue Feb 09 18:24:52 2010] [notice] Apache/2.2.13 (FreeBSD) PHP/5.2.11 with Suhosin-Patch mod_ssl/2.2.13 OpenSSL/0.9.8k configured -- resuming normal operations
[Tue Feb 09 18:24:52 2010] [alert] Child 85206 returned a Fatal error... Apache is exiting!
Previously reported here, this issue occurs when hammering Apache simultaneously with requests and graceful restarts. Sometimes it would take several hours for the problem to manifest. The consequence is that Apache gets stuck with only two worker processes, meaning that requests succeed but very slowly — as they get queued — when under any load. Insidious! The solution was to change the type of locking Apache was using in httpd.conf:
AcceptMutex fcntl
This is a problem on FreeBSD 8.0 with Apache 2.2.13 using the prefork MPM, with the default UFS2 filesystem for the locks directory.
I hope someone finds this useful!
Virgin Media respond to Twitter!
December 14th, 2009Update: Virgin were crap, nothing ever got fixed, and they sent a 17 year old “engineer” who didn’t know what a Traceroute was. We’re now happily on an ADSL line, soon-to-be with the nice folk at Bethere.co.uk.
This is my latest post on the Virgin Media support newsgroup (virginmedia.support.broadband.cable on news.virginmedia.com):
I just had a very interesting 20 minute phone conversation with Jason Golik (see http://www.linkedin.com/in/bungieboy) a senior Network Operation Centre shift manager at Virgin Media.
He got in touch with me via Twitter after I started tweeting loudly about how poor the quality of service and support at Virgin was.
Now, the bad news is that Virgin aren’t going to fix this particular problem with upstream capacity in Bristol until January 4th at the earliest since they have a freeze on engineering work over the holidays!
The really bad news is that the customer support staff on the phones and on the newsgroup would never have told us this! They’d just keep us hanging on with more of this “we have no update on this yet” or “it will be fixed on December 9th” or “Kam will not contact you today” bullshit.
The good news is that there’s someone with an ounce of clue and (potentially) enough power to get problems sorted, and that he’s happy to phone you personally if you make enough noise about the problem on Twitter (because it’s bad PR obviously). So, power to the consumer through social networking!
Unfortunately that’s not enough to keep me with Virgin Media — unless, and here’s a challenge for you Jason — you can get the problem sorted before the morning of December 17th — this Thursday — because that’s the last date I can cancel the installation of the BT line before I’m tied into an 18 month contract with BT.
And I don’t mean just get the problem sorted for ME — I mean for everyone experiencing serious packet loss in Bristol.
Anyone who wants to have a similar conversation can contact Jason by tweeting with the text “@bungieboy” in your tweet. Adding the hashtag “#virginmedia” may also help get you heard.
In a way it’s completely ridiculous that it took 3 weeks to get nothing from VM via the “regular” support channels (even though the NNTP server is hardly your average user’s cup of tea), but that simply by broadcasting my discontent on Twitter, I got an instant and satisfactory response.
Well, satisfactory in that it was at least honest about the timescales. Jason wasn’t able to confirm whether I’d get compensation or free contract termination as a result of these problems, but he did suggest that having documented communications with support over the last 2-3 weeks would help my case.
Let me know your experiences, this will be an interesting one.
Cheers,
Luke
CRU hack — taking a step back
December 12th, 2009The CRU hack really made me stop and think about my views on global warming, in particular this article at The Register was interesting to me as a programmer because it points to some analysis of the data which is a pretty damning read. Specifically this analysis of the “Harry Readme file” is very fierce in its criticism — and rightly so, it looks like the source code and comments that we got to see were pretty awful.
But I think it helps to take a step back: We have no idea who the hackers were, but consider the possible range of their political motives. If they were lefties, it’s fair to say they wouldn’t have done it. It’s possible that they had no political motive but just wanted to get “the truth” out there. But it seems much more likely to me — especially given the timing just before Copenhagen and the impact they must have known it would have, that the hackers were climate-change deniers intent on causing as much disruption to the public impression of scientific consensus as possible. And it worked — it made me question what I had hitherto considered to be scientific fact.
Indeed their political motives seem to be reflected in the code and text the hackers chose to expose. Since they hacked the CRU’s fileserver, they would have had access to enormous amounts of data and code. And they managed to find some pretty nasty stuff in there. The problem is, any large project has a fair amount of bad code and confusing documentation, often not currently used but stored away in case bits of it might be useful later. So we almost certainly didn’t get a balanced view of the work that the CRU do, we just saw the darkest corners of their fileservers. If it is the current status quo, then the CRU has a big apology to make, and some serious data management and software engineering to learn, fast. But one must be very careful in making assumptions based on incomplete information.
So here’s an idea: why don’t the CRU balance out the bad code by releasing all their code, warts and all, under an open-source license? Then we’d know whether the source code that we got to see was indicative of a real problem — also we’d get to see whether the bad code released is still being used for modern climate modeling, or whether it’s been superseded by something of higher quality.
It helps to balance the arguments from the likes of The Register with this editorial from Nature. Coming from an academic background at Oxford University, it’s fair to say that in my experience academics are truth-seekers, and peer-reviewed journals are still the best way we’ve got to be rigorous about our science.
