Entries tagged kvm

Yesterday I was forced to test my backup system in anger, on a large scale, for the first time in months.

A broken package upgrade meant that my anti-spam system lost the contents of all its MySQL databases.

That was a little traumatic, to say the least. But happily I have a good scheme of backups in place, and only a single MX machine was affected.

So, whilst there was approximately an hour of downtime on the primary MX the service as a whole continued to run, and the secondary (+ trial tertiary) MX machines managed to handle the load between them.

I'm almost pleased I had to suffer this downtime, because it did convince me that my split-architecture is stable - and that the loss of the primary MX machine isn't a catastrophic failure.

The main reason for panicing was that I was late for a night in the pub. Thankfully the people I were due to meet believe in flexible approaches to start times - something I personally don't really believe in.

Anyway the mail service is running well, and I've setup "instant activation now", combined with a full month of free service which is helping attract more users.

Apart from that I've continued my plan of migrating away from Xen, and toward KVM. That is going well.

I've got a few guests up and running, and I'm impressed at how stable, fast, and simple the whole process is. :)

ObQuote: Brief Encounter

(That is a great film; and a true classic. Recommended.)

Tags: kvm, mail-scanning, xen | 5 comments

KVM Utility

Gunnar Wolf made an interesting post about KVM today which is timely.

He points to a simple shell script for managing running instances of KVM which was a big improvement on mine - and so is worth a look if you're doing that stuff yourself.

Once I find time I will document my reasons for changing from Xen to KVM, but barring a few irritations I'm liking it a lot.

Chronicle Theme Update

I made a new release of the chronicle blog compiler yesterday, mostly to update one of the themes.

That was for purely selfish reasons as I've taken the time to update the antispam protection site I'm maintaining. There have been some nice changes to make it scale more and now it is time for me to make it look prettier.

(A common theme - I'm very bad at doing website design.)

So now the site blog matches the real site.

ObQuote: Resident Evil

Tags: chronicle, kvm, mail-scanning, xen | No comments

I think I've made the decision that at some point in the next few months the xen-hosting.org setup I maintain will be going away, and will be replaced with kvm-hosting(.org).

The only issue I need to ponder is handling the migration with the minimum downtime.

The plan would probably involve upgrading the host machine to Lenny, then installing KVM and fiddling with filesystems until the guests boot. I suspect it wouldn't be a huge job, but there are a few issues that will need to be planned.

Most notably I expect that most of the current guests don't have grub installed, etc, so we'd be in the position to use an external kernel + initrd. That's not an insurmountable problem, but I know that externally supplied kernels have caused me problem in the past with KVM.

Perhaps the actual plan would be to wait until September at which point I could order a new machine and cancel the current one. That would mean another increase in spec and the migration process would be a lot simpler - instead of everybody being offline for a few hours I could migrate guests individually from the old host to the new.

Anyway decisions decisions ..

ObFilm: Buffy - But we'll pretend the TV series counts as a film, kthxbye?

Tags: kvm, xen, xen-hosting | 13 comments

Since I already shared it elsewhere here is my KVM-launcher, and the mercurial repository it lives in.

I'll add my kvm-shell program later - the tools I've written so far is mostly standalone, rather than a package.

This is almost a content-free post, but I can pretend it isn't because I'm testing a new theme on my blog. The theme is included in the new release of my chronicle blog compiler which was released yesterday.

ObFilm: Heathers

Tags: blogging, chronicle, kvm | No comments

This weekend I mostly fiddled around migrating machines from Xen hosting to KVM hosting. Ultimately it was largely a waste of time, due to various other factors. Still with a bit of luck it will be possible to move the machiens next week.

That aside I spent a while updating my blogspam detection site. As a brief recap this site offers a simple XML-RPC service which allows you to test whether incoming blog comments are spam or not.

Originally this was put together to fight an invasion of comments submited to the Debian Administration website: The site currently shows:

Site	Spam	Non-Spam	% spam
debian-administration.org	238	372	60.98% spam

Depressing. But not as depressing as the real live stats which show since I last reset the counters 36,995 spam comments vs. 1,206 non-spam comments. (live updating counters here)

Anyway I updated the service today to add two new plugins, both of which are a little reactionary.

The first new plugin is called "multilink" and is based upon the observation that spammers rarely know the markup of the site they are submitting comments to. This means you can frequently see submitted comments like this:

 <a href="http://spam.com">buy viagra</a>
 [url=http://spam.com]buy viagra[/url]
 [link=http://spam.com]buy me[/link]

Here we have three different styles of links - "a href", "link=", and "url=". I figure this is a clear indicator of a confused mind, or more likely a spammer.

The second new plugin is designed to stop people who enter "<strong>" words. It is a little coarse but actuall zero false positives in the real world so I'm going to leave it live to see how it works out.

In happier news I'm just back from a trip to the beach. Sand rocks. Even if it wasn't windy enough for my kite ..

ObFilm: Dracula ("Bram Stoker's Dracula" - 1992)

Tags: blogspam, kvm, xen | 2 comments

Recently I've been spidering the internet, merrily downloading content for the past few days.

The intention behind the spidering is to record, in a database, the following pieces of information for each image it stumbles across:

• The page that contained the link to this image. (i.e. the image parent)
• The image URL.
• The MD5sum of the image itself.
• The dimensions of the image.

I was motivated by seeing an image upon a website and thinking "Hang on I've seen that before - but where?".

Thus far I've got details of about 30,000 images and I can now find duplicates or answer the question "Does this image appear on the internet and if so where?".

Obviously this is going to be foiled trivially via rotations, cropping, or even resizing. But I'm going to let the spider run for the next few days at least to see what interesting things the data can be used for.

In other news I'm a little behind schedule but I'm going to be moving from Xen to KVM over the next week or ten days.

My current plan is to setup the new host on Monday, move myself there that same day. Once that's been demonstrated to work I can move the other users over one by one, probably one a day. That will allow a little bit of freedom for people to choose their downtime window, and will ensure that its not an all-or-nothing thing.

The new management system is pretty good, but I have the advantage here in that I've worked upon about four systems for driving KVM hosting. The system allows people to enable/disable VNC access, use the serial console, and either use one of a number of pre-cooked kernels or upload their own. (Hmmm security you say?)

ObFilm: Chasing Amy

Tags: images, kvm, kvm-hosting, projects, searching, xen, xen-hosting | 15 comments

This week I've been mostly migrating guests from Xen to KVM. This has been a a pretty painless process, and I'm happy with the progress.

The migration process is basically:

• Stop the Xen guest (domU).
• Mount the filesystem (LVM-based) upon the Xen host (dom0).
• Copy those mounted contents over to a new LVM location upon the KVM host using rsync.
• Patch the filesystem once the rsync has been moved:
  • Create /dev nodes for the new root & swap devices.
  • Update /etc/fstab to use those devices.
• Fiddle with routing to ensure traffic for the guest arrives at the KVM host, rather than the Xen host.
• Hardwire static routes on the dom0 so that cross-guest traffic works correctly.
• Boot up the new guest, and hope for the best.

The main delay in the migration comes from the rsync step which can take a while when there are a lot of small files involved. In the future I guess I should ask users to do this themselves in advance, or investigate the patches to rsync that let block devices be transferred - rather than filesystem contents.

Thankfully all of the guests I've moved thus far have worked successfully post-migration, and performance is good. (The KVM host is going to be saturated with I/O when the rsyncing of a new guest is carried out - so I expect performance to dip while that happens, but once everybody is moved it should otherwise perform well.)

So Xen vs. KVM? Its swings and roundabouts really. In terms of what I'm offering to users there isn't too much difference between them. The only significant change this time round is that I'll let users upload their own kernel and one brave soul has already done that!

ObFilm: Pitch Black

Tags: kvm, xen | 13 comments

Good people steal ideas, right? On that basis I setup a static domain to host the javascript and icons I use upon a few different sites & projects. This was preempted by the release of a new version of the excellent jQuery library.

I also managed to put together a tremendous hack to solve a pretty annoying problem running multiple distributions from a single external kernel under KVM.

Ubuntu users, in particular, will be well aware of dmesg SPAM coming from the use of CONFIG_SYSFS_DEPRECATED.

In short the way that the kernel presents information beneath the /sys tree has changed over the life of the kernel - and this has a knock-on effect to the userspace supplied by different distributions and releases of GNU/Linux.

Some distributions need an "old" kernel and an "old" udev with "old" udev rules in order to create the appropriate device nodes such that the kernel will boot & mount its filesystems. (i.e. These need CONFIG_SYSFS_DEPRECATED to be set.)

Conversely some distributions mandate a "new" minimum kernel version, and supply a "new" version of udev with "new" udev rules and they absolutely will not function when presented with an "old" kernel. (i.e. They must have kernels without CONFIG_SYSFS_DEPRECATED set.)

I've solved this problem via a kernel patch which is both evil and genius. The details are a little me-specific, but in short:

• devtmpfs is used to setup and mount an initial /dev tree before /sbin/init is launched..
• udev launches later and mounts a tmpfs over /dev such that it can start creating its own nodes.
• At this point evil begins: I've patched the kernel such that any attempt to mount a tmpfs filesystem at /dev is silently changed to mount a devtmpfss filesystem instead.
  • The alternative is that udev creates many nodes, but manages to fail to create the root & swap nodes such that the KVM guests fail to boot.

Ultimately udev doesn't get an empty /dev tree to play with, instead it finds one already pre-populated, such that any devices it cannot create are there regardless - because the devtmpfs implementation has already created them.

Genius. And evil. So very evil.

Meh.

Steal that idea. I dare you .. (I'm impressed at how well devtmpfs works, and how easy I was able to make my "patch of evil"^tm. Just a few lines in fs/namespace.c.)

ObSubject: The Last House On The Left

Tags: evil, jquery, kernels, kvm, linux kernel, stealing, udev | 2 comments

I've been a little quieter than usual recently, having spent more time outdoors putting cute people in front of the camera. However that said I've still been doing some things. Most interestingly I've given away my first ever project.

The collection of small scripts known as xen-tools (which was initially a sleazy hack to go with a small introduction to Xen article) has now got new developers, and a new home where development is continuing.

This isn't the first time I've stopped working on something, but it is the first time I've explicitly "given away" a project. (Mostly on the basis that if I didn't care nobody else did either, or people cared but were too busy/unable to actually do soemthing useful.)

I'll be following new updates with interest, even though these days I'm 100% Xen-free. No need to go into huge details about why, but I'm enjoying KVM.

Having said that I recently got into a huge mess with a combination of LVM, KVM, and ext3. I've written up the details on ServerFault in the optimistic hope that somebody will report having experienced a similar problem. If you have seen something similar I'd love to hear from you.

Otherwise I'm genuinely at a loss to understand what went wrong, and why things failed. I could suspect hardware issues, but that feels like a cop-out, albeit one that has a potential solution (Mad Hatter: All Change!) rather than my current answer and explaination "It broke. I don't know why. It might happen again. It might not. Trust me?".

ObFilm: Alice In Wonderland (1951 version.)

Tags: kvm, lvm, xen, xen-tools | 2 comments

This evening I've mostly been using Sinatra to build a little file storage service which uses a REST API.

That means I can upload a file:

skx@birthday:~/hg/sinatra$ curl -X PUT -F file=@/etc/fstab http://localhost:4567/
{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","path":"/"}

Download that same file:

skx@birthday:~/hg/sinatra$ curl -X GET -F id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a  \
   http://localhost:4567/
# /etc/fstab: static file system information.
..
/dev/cdrom        /media/cdrom0   udf,iso9660 user,noauto     0       0

Get an index of files:

skx@birthday:~/hg/sinatra$ curl http://localhost:4567/
[{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","type":"file"}]

And finally we can delete a file:

skx@birthday:~/hg/sinatra$ curl -X DELETE -F "id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a" \
  http://localhost:4567/
Removed

We can also upload to different paths so we can replicate a file-system if we wanted to. (I added in "type" to hold either "file" or "directory", though I guess if we were to code up a FUSE client we'd want to store things like ctime, UID, GID, etc. THe list operation will show both files and sub-directories)

The code was trivial once I got the hang of Sinatra, and I'm pretty pleased with it so far. I don't yet need to use it for anything, but I'm thinking of unifying the way that I store images on a couple of sites - and fetching them via JSON and Javascript might be an option this was an experiment in that direction. (Though I'd probably want to hook in rsync so we replicated the eventual upload location for safety.)

In other news I've been all organized and upgraded the kernel on my guest:

steve@steve:~$ uptime
 22:00:28 up  4:18,  1 user,  load average: 0.14, 0.05, 0.05
steve@steve:~$ uname -r
3.2.0-kvm-hosting.org-i386-20120106

So for once I'm up to date with a cutting edge kernel. Happy times.

ObQuote: "How you expect to run with the wolves come night when you spend all day sparring with the puppies? " - The Wire (Omar)

Tags: json, kvm, kvm-hosting, ruby | 2 comments

This evening I sat down and migrated my personal virtual machine from a 32-bit installation of Debian GNU/Linux to a 64-bit installation.

I've been meaning to make this change for a good few months, but it took me until this evening until I decided it was as good a time as any.

Mostly the process is painless:

• Ensure you have a 64-bit kernel, with support for 32-bit binaries too.
• Install the 32-bit compatibility libraries, such that your old binaries work.
• Overwrite your binaries and libraries in-place so you have a 64-bit base system.
• Patch it up afterwards.

I overwrote a lot of the libraries and binaries on the system such that I had a working 64-bit apt-get, dpkg, sash, etc, and associated libraries. Then once I had that I could use those tools to pull the resto of the system up to date.

One thing I hadn't counted on is that I needed to have a 64-bit version of bzip such that "apt-get update" didn't complain about errors. I suspect I could have fixed that by re-configuring my system to disable compression. Still it was easily solved.

Along the way I also shot myself in the foot by having a local caching DNS resolver, listening on 127.0.0.1, which broke. With no DNS I couldn't use apt-get - but once the problem was identified it was trivial to fix.

Anyway all seems OK now. My websites are up, email is flowing and I guess anything else can wait until the morning.

ObQuote: "Somebody's coming up. Somebody serious." - Leon

Tags: amd64, debian, kvm, squeeze | 7 comments

Tonight I upgraded my personal machine to run the recently released 3.5[.0] kernel.

On my personal machine(s) I'm usually loathe to change a running kernel, but this one was a good step forward because it allows me to experiment with seccomp filters.

I've tested the trivial "no new privileges" pctl and I followed along with the nice seccomp tutorial which gave me simple working code which I married to my javascript interpreter.

On top of that I upgraded node.js, which meant I had to clean up a little depreciated code in my node reverse proxy - which is the public face of the websites I run upon my box. (The proxy tunnels to about 10 different thttpd instances, each running upon 127.0.0.1:xx).

Happily however my weekend was not full of code, it was brightened by the opportunity to take pictures of Aurora and her long hair - more to come as I've still got about 350 images to wade through..

ObQuote: "Don't you think I make a remarkable queen? " - St. Trinian's (2007)

Tags: kernels, kvm, kvm-hosting, seccomp | No comments

If this post is visible I should have migrated the following virtual machines to a new home:

• mail.steve.org.uk - SMTP, IMAP, & etc.
• www.steve.org.uk - And N other hosts.
• rsync.io - Offsite backups for local people.

These previously existed on a machine at Bytemark, running under screen and KVM. Now they exist upon a different Bytemark-rented host.

TODO: Move 4096.io, configure an auto-builder guest (I have a slaughter policy for that), and allocate a /48 so that I regain IPv6 support (/56 would do, I guess. I want a /64 for each guest.).

Tags: hosting, kvm | 2 comments

I've now completed all my KVM migrations. Moving my personal virtual machines from one host to another.

There were a few niggles, for example I didn't have a working IPv6 allocation at the time I moved things so I had to set that up post-migration.

I've also joined each of the hosts into a VPN which makes cross-guest communication secure and simple.

Finally I've overhauled my firewalls and service lists.

I installed a couple of extra guests, using libvirt and booting from the Debian ISO. The Debian installer continues to impress, though it did make me think I should overhaul my PXE setup at home.

It wouldn't be hard to have a Raspberry PI running as a TFTP + DHCP server. You could plug it into a network, reboot your desktop, and then have it boot into the imager. At the moment I run DHCP + TFTPD + etc on my main desktop, and that allows me to reimage any of the hosts in the flat easily, except itself obviously.

The last time I reinstalled this system I had to reconfigure DHCP + PXE + TFTP on another host. I think the next time I need to reinstall any system I'll "waste" an SD-card on an image-server host.

Finally I've recently read the Rick Cook Wizardy Series:

• Geeky developer gets transferred to a typical fantasy land:
  • Where magic works/exists.
  • There are dragons.
  • He writes a magic-compiler using FORTH to build primitives into bigger spells.

Fun idea. Horrible puns. Some of the books were too long, or left plot elements dangling, but on average they were more good than bad. Albeit a little predictable and "simple".

Tags: kvm, pi, random | No comments

The other week I was toying with using GNU stow to build an NFS-share, which would allow remote machines to boot from it.

It worked. It worked well. (Standard stuff, PXE booting with an NFS-root.)

Then I started wondering about distributions, since in one sense what I'd built was a minimal distribution.

On that basis yesterday I started hacking something more minimal:

• I compiled a monolithic GNU/Linux kernel.
• I created a minimal initrd image, using busybox.
• I built a static version of the tcc compiler.
• I got the thing booting, via KVM.

Unfortunately here is where I ran out of patience. Using tcc and the static C library I can compile code. But I can't link it.

$ cat > t.c <>EOF
int main ( int argc, char *argv[] )
{
        printf("OK\n" );
        return 1;
}
EOF
$ /opt/tcc/bin/tcc t.c
tcc: error: file 'crt1.o' not found
tcc: error: file 'crti.o' not found
..

Attempting to fix this up resulted in nothing much better:

$ /opt/tcc/bin/tcc t.c -I/opt/musl/include -L/opt/musl/lib/

And because I don't have a full system I cannot compile t.c to t.o and use ld to link (because I have no ld.)

I had a brief flirt with the portable c-compiler, pcc, but didn't get any further with that.

I suspect the real solution here is to install gcc onto my host system, with something like --prefix=/opt/gcc, and then rsync that into my (suddenly huge) intramfs image. Then I have all the toys.

Tags: initramfs, kvm, musl, tcc | 2 comments

Today I took down my KVM-host machine, rebooting it and restarting all of my guests. It has been a while since I'd done so and I was a little nerveous, as it turned out this nerveousness was prophetic.

I'd forgotten to hardwire the use of proxy_arp so my guests were all broken when the systems came back online.

If you're curious this is what my incoming graph of email SPAM looks like:

I think it is obvious where the downtime occurred, right?

In other news I'm awaiting news from the system administration job I applied for here in Edinburgh, if that doesn't work out I'll need to hunt for another position..

Finally I've started hacking on my console based mail-client some more. It is a modal client which means you're always in one of three states/modes:

• maildir - Viewing a list of maildir folders.
• index - Viewing a list of messages.
• message - Viewing a single message.

As a result of a lot of hacking there is now a fourth mode/state "text-mode". Which allows you to view arbitrary text, for example scrolling up and down a file on-disk, to read the manual, or viewing messages in interesting ways.

Support is still basic at the moment, but both of these work:

  --
  -- Show a single file
  --
  show_file_contents( "/etc/passwd" )
  global_mode( "text" )

Or:

function x()
   txt = { "${colour:red}Steve",
           "${colour:blue}Kemp",
           "${bold}Has",
           "${underline}Definitely",
           "Made this work" }
   show_text( txt )
   global_mode( "text")
end

x()

There will be a new release within the week, I guess, I just need to wire up a few more primitives, write more of a manual, and close some more bugs.

Happy Thursday, or as we say in this house, Hyvää torstai!

Tags: jobs, kvm, lumail, misc, spam | 1 comment

Much has already been written about the recent bash security problem, allocated the CVE identifier CVE-2014-6271, so I'm not even going to touch it.

It did remind me to double-check my systems to make sure that I didn't have any packages installed that I didn't need though, because obviously having fewer packages installed and fewer services running reduces the potential attack surface.

I had noticed in the past I had python installed and just though "Oh, yeah, I must have python utilities running". It turns out though that on 16 out of 19 servers I control I had python installed solely for the lsb_release script!

So I hacked up a horrible replacement for `lsb_release in pure shell, and then became cruel:

~ # dpkg --purge python python-minimal python2.7 python2.7-minimal lsb-release

That horrible replacement is horrible because it defers detection of all the names/numbers to the /etc/os-release which wasn't present in earlier versions of Debian. Happily all my Debian GNU/Linux hosts run Wheezy or later, so it all works out.

So that left three hosts that had a legitimate use for Python:

• My mail-host runs offlineimap
  • So I purged it.
  • I replaced it with isync.
• My host-machine runs KVM guests, via qemu-kvm.
  • qemu-kvm depends on Python solely for the script /usr/bin/kvm_stat.
  • I'm not pleased about that but will tolerate it for now.
• The final host was my ex-mercurial host.
  • Since I've switched to git I just removed tha package.

So now 1/19 hosts has Python installed. I'm not averse to the language, but given that I don't personally develop in it very often (read "once or twice in the past year") and by accident I had no python-scripts installed I see no reason to keep it on the off-chance.

My biggest surprise of the day was that now that we can use dash as our default shell we still can't purge bash. Since it is marked as Essential. Perhaps in the future.

Tags: kvm, offlineimap, python | 11 comments

I've spent the past thirty minutes installing FreeBSD as a KVM guest. This mostly involved fetching the ISO (I chose the latest stable release 10.0), and accepting all the defaults. A pleasant experience.

As I'm running KVM inside screen I wanted to see the boot prompt, etc, via the serial console, which took two distinct steps:

• Enabling the serial console - which lets boot stuff show up
• Enabling a login prompt on the serial console in case I screw up the networking.

To configure boot messages to display via the serial console, issue the following command as the superuser:

 # echo 'console="comconsole"' >> /boot/loader.conf

To get a login: prompt you'll want to edit /etc/ttys and change "off" to "on" and "dialup" to "vt100" for the ttyu0 entry. Once you've done that reload init via:

 # kill -HUP 1

Enable remote root logins, if you're brave, or disable PAM and password authentication if you're sensible:

 vi /etc/ssh/sshd_config
 /etc/rc.d/sshd restart

Configure the system to allow binary package-installation - to be honest I was hazy on why this was required, but I ran the two command and it all worked out:

 pkg
 pkg2ng

Now you may install a package via a simple command such as:

 pkg add screen

Removing packages you no longer want is as simple as using the delete option:

 pkg delete curl

You can see installed packages via "pkg info", and there are more options to be found via "pkg help". In the future you can apply updates via:

 pkg update && pkg upgrade

Finally I've installed 10.0-RELEASE which can be upgraded in the future via "freebsd-update" - This seems to boil down to "freebsd-update fetch" and "freebsd-update install" but I'm hazy on that just yet. For the moment you can see your installed version via:

 uname -a ; freebsd-version

Expect my future CPAN releases, etc, to be tested on FreeBSD too now :)

Tags: exploration, freebsd, kvm | 2 comments