Entries tagged kvm-hosting

Recently I've been spidering the internet, merrily downloading content for the past few days.

The intention behind the spidering is to record, in a database, the following pieces of information for each image it stumbles across:

• The page that contained the link to this image. (i.e. the image parent)
• The image URL.
• The MD5sum of the image itself.
• The dimensions of the image.

I was motivated by seeing an image upon a website and thinking "Hang on I've seen that before - but where?".

Thus far I've got details of about 30,000 images and I can now find duplicates or answer the question "Does this image appear on the internet and if so where?".

Obviously this is going to be foiled trivially via rotations, cropping, or even resizing. But I'm going to let the spider run for the next few days at least to see what interesting things the data can be used for.

In other news I'm a little behind schedule but I'm going to be moving from Xen to KVM over the next week or ten days.

My current plan is to setup the new host on Monday, move myself there that same day. Once that's been demonstrated to work I can move the other users over one by one, probably one a day. That will allow a little bit of freedom for people to choose their downtime window, and will ensure that its not an all-or-nothing thing.

The new management system is pretty good, but I have the advantage here in that I've worked upon about four systems for driving KVM hosting. The system allows people to enable/disable VNC access, use the serial console, and either use one of a number of pre-cooked kernels or upload their own. (Hmmm security you say?)

ObFilm: Chasing Amy

Tags: images, kvm, kvm-hosting, projects, searching, xen, xen-hosting | 15 comments

I've now gotten around to deploying IPv6 for my personal site(s) and machines. So potentially people visiting this blog are using IPv6 right now!

The Debian Administration website has been IPv6-connected and aware for a year or two now, although over 98% of users continue to connect via by IPv4.

Handling IPv6 & KVM wasn't too difficult at least, although I've done the simplest possible thing I could do, route a /64 to each guest and don't worry too much about it.

(The host machine has a /64 allocation of its own, and we've been granted an /48 allocation so we're not liable to run out. Ever.)

ObQuote: "You're not seriously gonna believe this man, are you? Are you? HE ISN'T EVEN FROM 'ROUND HERE! " - Hot Fuzz

Tags: ipv6, kvm-hosting | 3 comments

I've been informed by a couple of people that the Debian Administration site is down. Sadly it is; at the moment the host isn't showing anything on the serial console and remotely power-cycling it isn't showing any signs of life.

At this time of year I don't want to drag anybody in to take care of it, so ETA on recovery/replacement hardware is Monday/Tuesday.

In other news I've made it to year five of the KVM hosting sub-project/thing. Originally started as a Xen host its been running happily for quite some time. I suspect next year, or the year after that the price/specification ratio will end up losing out and we'll cancel the whole thing - but there are no immediate reasons to make any change.

Finally I knocked up a simple tool to validate my TinyDNS records prior to uploading them. It is simplistic, but adequate to catch the kind of mistakes I make:

• http://tinydns-lint.repository.steve.org.uk/file

Honestly it probably wants to be rationalised a little more - and check records more carefully. e.g. Ensure that the host a CNAME refers to itself exists, and making sure that the nameservers specified are valid.

I just wanted to make something quick after accidentally uploading a zonefile where I'd managed to fat-finger several important records. le sigh.

Oddly enough asking on serverfault.com showed no real suggestions - other than actually running tinydns locally and doing a zone-xfer to validate records. Overkill and harder than I'd like.

Happy New year if you care about such things..

"I finished growing up, Léon. I just get older. " - Leon

Tags: debian-administration, kvm-hosting, tinydns | No comments

This evening I've mostly been using Sinatra to build a little file storage service which uses a REST API.

That means I can upload a file:

skx@birthday:~/hg/sinatra$ curl -X PUT -F file=@/etc/fstab http://localhost:4567/
{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","path":"/"}

Download that same file:

skx@birthday:~/hg/sinatra$ curl -X GET -F id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a  \
   http://localhost:4567/
# /etc/fstab: static file system information.
..
/dev/cdrom        /media/cdrom0   udf,iso9660 user,noauto     0       0

Get an index of files:

skx@birthday:~/hg/sinatra$ curl http://localhost:4567/
[{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","type":"file"}]

And finally we can delete a file:

skx@birthday:~/hg/sinatra$ curl -X DELETE -F "id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a" \
  http://localhost:4567/
Removed

We can also upload to different paths so we can replicate a file-system if we wanted to. (I added in "type" to hold either "file" or "directory", though I guess if we were to code up a FUSE client we'd want to store things like ctime, UID, GID, etc. THe list operation will show both files and sub-directories)

The code was trivial once I got the hang of Sinatra, and I'm pretty pleased with it so far. I don't yet need to use it for anything, but I'm thinking of unifying the way that I store images on a couple of sites - and fetching them via JSON and Javascript might be an option this was an experiment in that direction. (Though I'd probably want to hook in rsync so we replicated the eventual upload location for safety.)

In other news I've been all organized and upgraded the kernel on my guest:

steve@steve:~$ uptime
 22:00:28 up  4:18,  1 user,  load average: 0.14, 0.05, 0.05
steve@steve:~$ uname -r
3.2.0-kvm-hosting.org-i386-20120106

So for once I'm up to date with a cutting edge kernel. Happy times.

ObQuote: "How you expect to run with the wolves come night when you spend all day sparring with the puppies? " - The Wire (Omar)

Tags: json, kvm, kvm-hosting, ruby | 2 comments

Tonight I upgraded my personal machine to run the recently released 3.5[.0] kernel.

On my personal machine(s) I'm usually loathe to change a running kernel, but this one was a good step forward because it allows me to experiment with seccomp filters.

I've tested the trivial "no new privileges" pctl and I followed along with the nice seccomp tutorial which gave me simple working code which I married to my javascript interpreter.

On top of that I upgraded node.js, which meant I had to clean up a little depreciated code in my node reverse proxy - which is the public face of the websites I run upon my box. (The proxy tunnels to about 10 different thttpd instances, each running upon 127.0.0.1:xx).

Happily however my weekend was not full of code, it was brightened by the opportunity to take pictures of Aurora and her long hair - more to come as I've still got about 350 images to wade through..

ObQuote: "Don't you think I make a remarkable queen? " - St. Trinian's (2007)

Tags: kernels, kvm, kvm-hosting, seccomp | No comments

Yesterday I received an automated alert from my kvm-hosting host-machine, informing me that one of the drives in the RAID-pair had failed.

This particular machine has been up and running since 2009, and according to my outage log this is the first downtime in three years. (The uptime was over 1000 days, which seems to confirm that pretty nicely.)

I like reliable systems, and sometimes it's worth remembering just how well they can work.

In other news I'm currently continuing to chase a new job. The companies I've approached, or which have approached me, are being a little slow in replying which is a shame, but I'm not hugely concerned .. yet.

I'm going to give things another week, or so, and then add a banner to the Debian-Administration website, and see if that results in anything interesting.

In the meantime I've got some wood, and a new mitre saw, and I will be spending the remainder of today working on my new desk. Doing physical things is always fun, and right now especially.

Tags: jobs, kvm-hosting | 9 comments

Seven years ago I wanted to move on from the small virtual machine I had to a larger one. Looking at the the options available it seemed the best approach was to rent a big host, and divide it up into virtual machines myself.

Renting a machine with 8Gb of RAM and 500Gb of disk-space, then dividing that into eights would give a decent spec and assuming that I found enough users to pay for the other slots/shares it would be economically viable too.

After a few weeks I took the plunge, advertised here, and found users.

I had six users:

• 1/8th for me.
• 1/8th left empty/idle for the host machine.
• 6/8th for other users.

There were some niggles, one user seemed to suffer from connectivity problems more than the others, but on the whole the experiment worked out well.

These days, thanks to BigV, Digital Ocean, and all the new-comers there is less need for this kind of thing so last December I announced that the service would cease - and gave all current users 1 year of free service to give them time to migrate away.

The service was due to terminate in December, but triggered by some upcoming downtime where our host would have been moved, in the back of a van, from Manchester to York, I've taken the decision to stop it early.

It was a fun experiment, it provided me with low cost hosting (subsidized by the other paying users), and provided some other people with hosting of their own that was setup nicely.

The only outstanding question is what to do with the domain-names? I could let them expire, I could try to sell them, or I could donate them to other people running hosting setups.

If anybody reading this has a use for kvm-hosting.org, kvm-hosting.net, or kvm-hosting.com, then do feel free to get in touch. No promises, obviously, but it'd be a shame for them to end up hosting adverts in a year or twos time..

Tags: kvm-hosting | 4 comments