About Archive Tags RSS Feed


It's no use pretending it hasn't happened cause it has

14 July 2008 21:50

Yesterday I was forced to test my backup system in anger, on a large scale, for the first time in months.

A broken package upgrade meant that my anti-spam system lost the contents of all its MySQL databases.

That was a little traumatic, to say the least. But happily I have a good scheme of backups in place, and only a single MX machine was affected.

So, whilst there was approximately an hour of downtime on the primary MX the service as a whole continued to run, and the secondary (+ trial tertiary) MX machines managed to handle the load between them.

I'm almost pleased I had to suffer this downtime, because it did convince me that my split-architecture is stable - and that the loss of the primary MX machine isn't a catastrophic failure.

The main reason for panicing was that I was late for a night in the pub. Thankfully the people I were due to meet believe in flexible approaches to start times - something I personally don't really believe in.

Anyway the mail service is running well, and I've setup "instant activation now", combined with a full month of free service which is helping attract more users.

Apart from that I've continued my plan of migrating away from Xen, and toward KVM. That is going well.

I've got a few guests up and running, and I'm impressed at how stable, fast, and simple the whole process is. :)

ObQuote: Brief Encounter

(That is a great film; and a true classic. Recommended.)



Comments on this entry

icon Saint Aardvark at 19:20 on 14 July 2008
"I've continued my plan of migrating away from Xen, and toward KVM." I'm curious why you're migrating. What's good about KVM? Is there something bad you've found about Xen?
icon Anonymous at 19:53 on 14 July 2008
I look forward to seeing kvm-tools. :)
icon Alex at 22:53 on 14 July 2008
I'm curious what happened and interrupted your night in the pub; what package is broken and fscked things up?
icon cstamas at 01:04 on 15 July 2008
I think PostgreSQL is a better choice. I always use postgres if I can.
icon Steve Kemp at 09:56 on 15 July 2008

Alex: I'm suspiscious of DSA-1608-1 - although we've had no reports of breakage elsewhere, or even on any of my boxes.

I just know I upgraded to that and suddenly simple queries started giving "file not found" errors from MySQL. I dropped the database and restored from the backup that was only a few minutes old, and all was well. (Took a while to import 3Gb of backup!)

Saint: KVM? Well it is just "cleaner". It doesn't require a whole load of python, or a special kernel, so it runs on more hardware. (Albeit hardware that has built in support for virtualisation.)

The downside is that you can't do migrations, and you can't save running machines via your init scripts. But I made a couple of wrappers to run my guests inside GNU Screen and I'm happy enough.

Anonymous: KVM-tools could be fun. But I just installed via a netboot image, and copied the disk images to get started - I only need about 6-8 guests to make my desktop happy, so I'll probably not bother...)