Archive for the ‘FreeBSD’ Category

A warning to anyone using Twisted 10.2 on FreeBSD

Monday, February 14th, 2011

Read this ticket first!

http://twistedmatrix.com/trac/ticket/4881

Basically, it comes down to running:

mount -t fdescfs null /dev/fd

and putting the equivalent in your /etc/fstab — otherwise, Twisted will have mysterious problems related to file descriptors and child processes. A fix is due for review and should be included in trunk at some point.

Thanks to David (big thanks for hosting), Glyph, JP and Itamar — I had a great time at the Twisted sprint last night :-)

Running FreeBSD 8.1 as a Xen HVM DomU on Flexiant

Friday, November 26th, 2010

Just thought I’d share the incantations which were necessary to get FreeBSD 8.1 XENHVM kernel to work well on Flexiant, with paravirtualised network and disk:

Just before the kernel boots (which you have to be quick to catch with Flexiant’s VNC client) hit F6 on the bootloader and type:

set hw.clflush_disable=1
boot

This will allow you to boot even a GENERIC kernel. Once you’re booted, chuck hw.clflush_disable="1" into /boot/loader.conf to make this permanent.

You’ll then want to build your own kernel for paravirtualised network and disk drivers. Edit the XENHVM kernel config (see the FreeBSD handbook on compiling your own kernel) – comment out the MODULES_OVERRIDE line which disables building all the modules (assuming you want ZFS support) and also comment out the whole section about WITNESS and INVARIANT, as having this enabled will slow down your kernel quite significantly.

Then you’ll need to patch the network driver as per this post (manually, since the code has changed a bit), else you get a lot of dropped packets:

http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00598.html

Then just (as root):

cd /usr/src
make buildkernel KERNCONF=XENHVM
make installkernel KERNCONF=XENHVM
shutdown -r now

And enjoy your speedy FreeBSD 8.1 VM in the cloud!

Oh and check out our cloud web hosting platform, Hybrid Web Cluster which takes advantage of this ;-)

Last three weeks

Tuesday, July 27th, 2010

The last three weeks have been pretty big for me.

A couple of Fridays ago, after a lovely mini-holiday in Dublin with Kate (in the days after she was talking at the Aristotelian society conference), I demonstrated my web cluster to ElasticHosts, and they loved it! We’ve provisionally agreed a referral agreement as their “cloud sites provider of choice”. My favourite bit was when Richard, their CEO, described our platform as “clearly the next generation of shared hosting”. I’ve only been working for three years to hear that… :)

The following Wednesday I jumped straight in the deep end with public speaking, explaining the technology I’ve spent three years developing in front of a terrifyingly large room full of people at the London OpenSolaris User Group. It went well! After getting over the initial nerves I settled in nicely to the talk. Getting some laughs for “here’s some Dtrace, I haven’t got any idea how it works!” really boosted my confidence and the rest was easy. I can imagine talks could easily go the other way though. I was lucky.

The same Thursday, hours after my talk, I went to the TweetDeck offices in Old Street, London. They offered me a job! I’m really excited about this. Reza, their sole backend engineer at present, and my new boss, looks set to be a real mentor for me. Here’s a short list of just some of the truly exciting technologies I’ll be working with there: Twisted (of course), RabbitMQ, Memcache, Django, Boto, Celery, Fabric, Node.js, Redis, Nginx. As technologies for building web-scale distributed systems with an emphasis on Python, this couldn’t be a better fit! The fact that they’re flexible enough to offer me a position with two days in London and one day working from home makes it totally awesome. Thanks TweetDeck!

I should say at this point that I’m really indebted to Simpleweb back in Bristol. They’ve been really flexible with me and I’ve really valued the fact that my job there has chilled me out, rather than stressing me out. It was exactly what I needed to get away from the freelance lifestyle which was causing me serious headaches about 10 months ago. On the flipside, I am also glad that I won’t be building websites any more. Even better that I can say that I’ll hopefully never have to write another line of PHP. Systems development makes more sense for me.

So on Friday I handed in my notice, which was a relief to get out in the open. I really don’t like keeping secrets.

Then I had Saturday off! Actually OFF! It was awesome. Sol was around, so we spent the whole day drinking ale, having multiple barbecues and playing with the chickens in our garden. Got to see Lewis too. Really nice day off.

So that brings us pretty much up to today… I’m on the train back to Bristol at 12:49am (work tomorrow at 9 *sigh*) after a great FreeBSD users meeting where I was talking again. I had a really great evening. It was a much smaller event, about 20 people in a small-ish conference room with free beer and crisps. Really friendly and very technical vibe, seemingly quite a lot of core FreeBSD and OpenBSD contributors. Exciting to get to know some of the community.

Got a warm reception to my talk once again and enjoyed it more this time. As it was a much more informal event there was interruptions and dialogue, which was fun as it allowed us to go off on interesting tangents. Went to the pub with everyone afterwards and over a few drinks had several rather interesting conversations. Sean (whose rack Digital Crocus is hosted in) came along to the pub afterwards as well. I was surprised by just how many techies working in banks there were there, pretty much 100%!

One such guy I spoke with was talking about his background in synchronous distributed filesystems for a banking operation (for data centres 20 miles apart, failover via BGP). His broad impression of our product seemed good, he said he’d be interested in how we deal with the problems we’ll find when running clusters of hundreds of nodes across three data centres! I can’t wait to get to the point of solving those problems :)

One other bit of news — we had our first genuine unsolicited sales enquiry today from a Belgian ISP who want a 12-node cluster. Woohoo!

So I’m tired but happy — looking forward to my holiday in August. I feel like I’m pretty well sorted here in England. I think working in London is going to present many exciting opportunities. I can’t believe how well everything’s going!

Solution to “No such file or directory: Couldn’t initialize cross-process lock in child” in Apache 2.2.13 on FreeBSD 8.0

Friday, February 12th, 2010

This issue in the Apache error log was causing the web server to “partially crash” during a graceful restart with the following errors:

[Tue Feb 09 18:24:52 2010] [emerg] (2)No such file or directory: Couldn't initialize cross-process lock in child (/etc/opt/HybridCluster/apache2/logs/accept.lock.82760) (5)
[Tue Feb 09 18:24:52 2010] [notice] Apache/2.2.13 (FreeBSD) PHP/5.2.11 with Suhosin-Patch mod_ssl/2.2.13 OpenSSL/0.9.8k configured -- resuming normal operations
[Tue Feb 09 18:24:52 2010] [alert] Child 85206 returned a Fatal error... Apache is exiting!

Previously reported here, this issue occurs when hammering Apache simultaneously with requests and graceful restarts. Sometimes it would take several hours for the problem to manifest. The consequence is that Apache gets stuck with only two worker processes, meaning that requests succeed but very slowly — as they get queued — when under any load. Insidious! The solution was to change the type of locking Apache was using in httpd.conf:

AcceptMutex fcntl

This is a problem on FreeBSD 8.0 with Apache 2.2.13 using the prefork MPM, with the default UFS2 filesystem for the locks directory.

I hope someone finds this useful!