Entries tagged xen-tools

My Christmas day was primarily spent at home watching Red Dwarf.

Apart from that I made new releases of xen-shell and xen-tools.

I was suprised to recieve a sudden flurry of bug reports. No less than ten bugs reported by two users in one day! Still I fixed most of them and made a new release.

It was nice to notice Isaac Clerencia writing about using xen-tools for automated testing. The idea has occurred to me before: Create a fresh Xen installation of Debian then run a test suite and trash the system afterwards. I even setup a test Yawns builder for a while.

Are people finding interesting users for xen-tools? I'd love to have some use-cases, and more user-feedback is always appreciated. Feel free to get in touch or comment here.

Personally I use a few Xen guests on my home LAN and I regularly create new Sarge images for building backported Xen packages. Only recently did I move my home cfengine setup onto a domU.

In other news I recently received a copy of Programming Ruby from my Amazon wishlist, I'm learning a lot about Ruby and I think I'm in love. The only downside is the mental confusion I experience when I see "@variable" and "$variable" used. I am too wedded to Perl to stop thinking "array" and "string" respectively.

I should find a toy project to write in Ruby to get used to it. Right now nothing occurs, but I'm sure it is just a matter of time.

Tags: cfengine, red dwarf, xen-shell, xen-tools | No comments

These are my personal goals for the debcamp week:

• Develop xen-tools to fix bugs reported by Henning.
• Unify debootstrap.
• Get involved in discussions of Debian init-systems.

The second item is the one that I'm most interersted in right now, but I'll leave it a day to see if there is any useful feedback or massive objections right now.

The debootstrap problem

Right now there are two versions of debootstrap - the "Debian one" and the "Ubuntu one".

The Ubuntu debootstrap package has support for installing additional suites; dapper, feisty, etc.. (My current understanding is that the Ubuntu debootstrap can still install Etch, Sarge, etc.)

Right now if you want to bootstrap an Ubuntu system on a Debian host you're out of luck unless you install the Ubuntu debootstrap package.

There is no real reason why this should be the case. Either distribution should be able to install the other.

Why I care

I wrote/maintain xen-tools. This package allows you to create Xen guests of different distributions. Most of the time the tool will install a distribution by invoking "debootstrap ...".

It would be nice if this could work upon a plain Debian system such that installing Dapper, Fiesty, etc, was supported.

What We Can't Solve

There are problems with debootstrap which I'm not going to attempt to solve. The most obvious one is that Sarge's debootstrap cannot install Etch.

There are a few open bugs in the Debian BTS which I will triage though, since some of them are safe to close.

The Approach

There are three approaches for this:

The hack

Download the source to Ubuntu's package. Test it can install Sarge, Etch & Lenny. Then upload to Debian.

What I'll do

I'll do three things:

• Triage open bugs on debootstrap.
• I'll compare the two scripts + manpages, to see if they've diverged.
• I'll move the Ubuntu scripts into the Debian package.

The neat way forward

Split debootstrap into :

• debootstrap
  • Compatability package, to pull in the next three:
• debootstrap-common
  • The tool.
• debootstrap-debian
  • Contains the scripts/support for installing Sarge, Etch & Lenny.
• debootstrap-ubuntu
  • Contains the scripts/support for installing Dapper, Feisty, etc.

This second approach should be straightforward. The first thing to do is to test that actual debootstrap script differ little between the two distributions. That should be simple enough.

The next thing is to create the additional packages.

Acceptance?

For this work to be useful it needs support from the maintainers of the Debian & Ubuntu packages. (I guess mostly the Debian maintainers actually.)

Progress

The good news is that between the version of debootstrap in Ubuntu right now, debootstrap-0.3.3.3ubuntu4, and that in Debian sid, debootstrap-0.3.3.3, the code has only one minor change. There is only a minor typo-fix in the manpage too.

So I can move the scripts into the package trivially...

Tags: debconf7, debootstrap, init, todo, xen-tools | No comments

If you use xen-tools and you've ever wanted to be able to setup arbitrary partitions within your new guests you can do so now.

skx@vain:~$ sudo xen-create-image  --boot --hostname=pt.my.flat --partitions=sample-server
...
...
General Information
--------------------
Hostname       :  pt.my.flat
Distribution   :  etch
Partitions     :  swap            2G    (swap)
                  /               1G    (ext3)
                  /var            4G    (xfs)
                  /opt            1.5G  (xfs)
                  /usr            4G    (xfs)
                  /tmp            0.5G  (xfs)
                  /home           1G    (xfs)
                  /var/tmp        1.5G  (xfs)
...

How cool is that? Many thanks to Sascha Kettler for the patch.

Tags: patches, user-greatness, xen-tools | No comments

Bootstrapping non-Debian distributions suck.

The only available tool appears to be rpmstrap which quite frankly fails more often than it works.

Since my xen-tools project needs to do carry out this kind of operation I've been pondering the idea of writing a tool which will install CentOS/Fedora/SuSE into a directory, in a similar fashion to debootstrap.

If there's nothing out there that you can point to, then I think that will be my next project.

Tags: projects, software which should exist, todo, xen-tools | No comments

I now have a working system for creating minimal installations of Fedora Core (6), and CentOS (4.5 & 5).

I've updated xen-tools to make use of this, via --install-method=rinse - and now I can create new Xen guests :)

There were several false starts:

Have a static list of .rpm names + versions

This is precisely what rpmstrap does, and it is a broken approach. I'm happy to have a static list of packages, but restricting the packages to specific versions is fragile and broken.

Use Yum

yum is used by many RPM based distributions, but unfortunately each distro uses slightly different versions to the one in Debian - so whilst I could run:

mkdir /tmp/fc6
yum -c /tmp/yum.conf --installroot /tmp/fc6 install yum

This results in a system which cannot be updated:

chroot /tmp/fc6 /usr/bin/yum update
[errors about yum database formats]

I suspect this is because the versions of the Berkeley database are different .. but I'm not sure.

Parse Dependencies

This would be a neat approach, however it is difficult. Too difficult.

Abandoned after several fruitless hours.

Cheating

I cheated in the end.

• Download a list of packages which are known to be required to produce a working yum, and unpack them.
• Copy those downloaded packages into the yum cache tree.
• Run yum install yum - to make the local Yum + RPM database match what we fetched manually
  • Unfortunately "rpm rebuilddb" couldn't do that for us.

Worst case? I download each package twice. Once for the initial download, and once again when the Yum process runs. I can live with that.

I'll talk about something else next time. Promise.

Tags: rinse, xen-tools | No comments

According to popcon I have just under 1000 users of xen-tools.

That was quite a suprise to discover via a random google search, although I guess there have been a lot of bugs filed against the package during its lifetime.

Funny how some things which start as random hacks (this was originally a quick and dirty hack for a Xen introduction article) become quite useful/popular, whereas other tools which were planned and designed go virtually unnoticed...

Tags: random, xen-tools | No comments

This week has consisted of fighting registrars and doing a bit of hacking on xen-shell, xen-tools, and the chronicle blog compiler.

CJ has done some good work trying to get the code modularised, and I expect between the pair of us we can make things neater and better generally.

I've also fixed a couple of bugs relating to the hard-wiring of device names (/dev/sda, /dev/tty1, etc). These devices are replaced in newer versions of Xen which wants to use /dev/xvc0 and /dev/xvd[a-z] instead.

There's nothing else happening at the moment; I'm just having a lot of fun laughing at our new kitten sliding around on our polished wooden floor!

Chronicle seems to be getting pretty popular which is ironic because it was a quick hack to allow me to post blog entries on a couple of hosted sites - which I've not yet done. Oops.

In other news I'm loving the Nintendo DS at the moment, Megan brought me one back from America on her recent trip and I think a day hasn't passed where one or both of us has played less than 30 minutes each.

I'm annoyed that Sim City DS only allows us to play with one city - right now it is her turn, and I'll have to wait until she's finished with her creation before I can have a go - because otherwise I'll wipe her city out.. :(

Tags: chronicle, nintendo ds, patches, wedding, xen-shell, xen-tools | No comments

The following companies, or people, are using xen-shell with their Xen hosting product(s):

• xen kibbutz
• xen hosting (me!)
• bit folk
• Datarealm internet services
• Chris Lamb
• Cernio

I'm sure there are more, but I've been lax at keeping track. I do want a list, if only for advertising purposes. I'll update the list here if/when people point out omissions.

In other news yesterday I managed to release security updates for KDM & fetchmail which were missing i386 builds. Not good.

The problem was fixed fairly quickly once it was actually noticed, with the help of Noah. But I did get several angry emails during the wait. *sigh*

It looks like people are starting the notice the downside to having broken security buildd machines which frequently cause these kind of problems for us - so hopefully these kind of errors will diminish in time. (Who am I kidding?)

Tags: buildd, dsa, irritations, security advisories, xen-tools | No comments

After years of on-again-off-again infatuation I now own my very own steam engine.

This makes me far far happier than I was expecting.

In other news a new release of xen-tools was made to fix some stupid bugs, a release of xen-shell is pending for the same reason, and after declaring I'd not make another release of GNUMP3d again I'm just about to do that too.

I am finding myself rather bored working on the same projects for so long. Though thankfully each of those three is essentially complete. If I died tomorrow there would be no real need for future development. I guess I need a new project to work upon.

The mail-scanning was going to be that project, but that has been sidetracked as I've been stripping out all the pretty and functional user-interface code with the aim of selling the code to $company as a one-off deal. (Not confirmed. Nice idea though.)

So I need a new project. People like Joey Hess who can almost instantly start fun projects (such as moreutils, ikiwiki, pristine-tar, mr) which just seem so obvious, necessary and useful on a whim make me jealous!

All in all life is good and steamy.

I just need inspiration. And more hours in the day too.

Tags: gnump3d, steam, xen-shell, xen-tools | No comments

xen-tools has just got a new command:

xen-resize-guest --hostname=foo.my.flat --increase=5Gb

That will take the existing guest foo.my.flat and magically resize the main disk image to be 5Gb larger. This works for LVM & loopback images, but not yet for users of EVMS. Whilst doing this job manually isn't terribly difficult it can be troublesome to perform all the steps in order without screwing up. Hence the new command.

In other news I've managed to fix my broken greylisting implementation - so all the mails which were previously being queued upon klecker.debian.org should now be delivered/bounced.

I believe the blame here was 50/50 me and exim's back-off behaviour, but I'll know better in the future.

Remember me when you come to choose your next anti-spam service. It copes beautifully with a sustained delivery rate of 300-700 messages a minute when queues suddenly restart delivery ;)

TODO: Catchup on mail. Implement message tagging for mutt to better keep track of items which are pending/claimed by me. Until we get RT.

Tags: anti-spam, xen-tools | No comments

Whilst I'm very pleased with my new segmented network setup, and the new machine, I'm extremely annoyed that I cannot get a couple of (graphical) Xen guest desktop guests up and running.

The initial idea was that I would setup a 64-bit installation of Etch and then communicate with it via VNC - xen-tools will do the necessary magic if you create your guest with "--role=gdm". Unfortunately it doesn't work.

When vncserver attempts to start upon an AMD64 host it dies with a segfault - meaning that I cannot create a scratch desktop environment to play with.

All of this works perfectly with a 32-bit guest, and that actually is pretty neat. It lets me create a fully virtualised, restorable, environment for working with flash/java/etc.

The bug was filed over three years ago as #276948, but there doesn't appear to be a solution.

Also, only on the amd64 guest, I'm seeing errors when I try to start X which mention things like "no such file or directory /dev/tty0". I've no idea whats going on there - though it could be a vt (virtual terminal) thing?.

The upshot of all this is that I currenly have fewer guests than I was expecting:

skx@gold:~/blog/data$ xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     3114     2 r-----   1180.6
cfmaster.services.xen                      1      256     1 -b----      1.0
etch32.desktop.xen                         2      256     1 -b----      1.4
etch32.security-build.xen                  3      128     1 -b----      1.4
etch64.security-build.xen                  4      128     1 -b----      1.4
sarge32.security-build.xen                 5      128     1 -b----      1.0

Tags: lazyweb, woe is me, xen, xen-tools | 5 comments

Recently I bought a new desktop PC and rotated my existing machines around, leaving me with one spare. Last night I donated that PC to a local friend, (co-incidentally the same woman who had been out cat-sitter when I went to stay at Megan's parents for Christmas).

I said "Its got Debian on it, that funny operating system that you used when you were looking after our kitten, remember?".

She said "Linux ssems to be getting popular these days I might as well try it out".

But seriously she used it. It had GNOME, it had firefox, she seemed happy. It isn't rocket science these days to point a technicalish user at a Debian Desktop and expect them to know what to do with it. The only minor complication was the lack of flash/java since it was etch/amd64 - but thats probably a plus with the amount of flash adverts!

In other news I made a new release of asql last night. This now copes with both Apache's common and combined logformats by default. (Yet another tool of mine which is now being used at work, which motivated the change.)

I've also started pushing commits to xen-tools last night, so there'll be a new release soon. Mostly I'm bored with that code though, it doesn't need significant updates to work any longer. All the interesting things have been done, so it's probably only a matter of time until I drift off. (To the extent that I've unsubscribed myself from xen-users & xen-devel mailing lists)

Virtualisation is increasingly a commodity these days. If xen dies people probably won't care, there are enough other projects out there doing similar things. Being tied to one particular platform will probably come to be regarded as a mistake.

(Talking of which I should try out this lguest thing; I just find it hard to be motivated...)

Tags: asql, chronicle, xen-tools | 4 comments

If you try using some of my software, or any software come to think of it, and it doesn't work, or causes you problems then there is a simple solution.

Tell me. I might not be able to fix it immediately, I might not ever be able to fix it. But chances are I can, and if there is a record it'll help others out in the future regardless.

I've bumped into this in the past "Oh yes I tried to use that tool you wrote but it didn't work, so I ended up with something else."

Tags: bug-reports, random, xen-tools | 2 comments

Well I've had a busy weekend, but I'm sober now.

I made a new release of xen-tools, which has a couple of minor bugfixes and not much else. I also released a new update for Debian of the xen-shell which fixes a couple of bashisms.

Finally I've managed to sign up two new users to my anti-spam proxy. Hopefully they're very happy.

In real news I painted about 1 square meter of my flat, (we're now into week three of painting a single room...), and replaced five light bulbs:

My eyes! The goggles do nothing!

Now I need to install a request tracking system (otrs2) and catch up on significantly outstanding RT status updates.

I'm getting hopeless again.

Maybe I should just give it all up and become a plumber. Plumbing is easy: Water goes downhill. The rest is just regulations and a willingness to get dirty...

Tags: diy, flat, xen-shell, xen-tools | No comments

I've been a little quieter than usual recently, having spent more time outdoors putting cute people in front of the camera. However that said I've still been doing some things. Most interestingly I've given away my first ever project.

The collection of small scripts known as xen-tools (which was initially a sleazy hack to go with a small introduction to Xen article) has now got new developers, and a new home where development is continuing.

This isn't the first time I've stopped working on something, but it is the first time I've explicitly "given away" a project. (Mostly on the basis that if I didn't care nobody else did either, or people cared but were too busy/unable to actually do soemthing useful.)

I'll be following new updates with interest, even though these days I'm 100% Xen-free. No need to go into huge details about why, but I'm enjoying KVM.

Having said that I recently got into a huge mess with a combination of LVM, KVM, and ext3. I've written up the details on ServerFault in the optimistic hope that somebody will report having experienced a similar problem. If you have seen something similar I'd love to hear from you.

Otherwise I'm genuinely at a loss to understand what went wrong, and why things failed. I could suspect hardware issues, but that feels like a cop-out, albeit one that has a potential solution (Mad Hatter: All Change!) rather than my current answer and explaination "It broke. I don't know why. It might happen again. It might not. Trust me?".

ObFilm: Alice In Wonderland (1951 version.)

Tags: kvm, lvm, xen, xen-tools | 2 comments