Entries tagged custodian

The following companies are amongst those showing Christmas Adverts on television before the start of December:

• Tesco
• Homebase.
• M&S
• Waitrose.
• John Lewis.

I will boycott these companies until next year.

In happier news I've spent the past week or two replacing the monitoring system that we use at work.

Our previous monitoring system had been struggling to keep up with the sheer number of tests it was being asked to process. This was partly because we carry out many ping-tests, ssh-tests, http-tests, dns-tests, etc. The other reason was that our monitoring system was a behemoth of threaded-ruby, which all ran upon a single host. This made adding another monitoring host a complex undertaking.

The new solution uses a work-queue:

• Tests to apply are parsed and inserted into a single, global, beanstalkd queue.
• Workers continuously poll the queue for tests to execute. They then execute them, and alert on failures as appropriate.

The code is open-source, written in Ruby, and available here:

• https://projects.bytemark.co.uk/projects/custodian/wiki

I've completed the process of tidying up the code to the extent I'm happy with it, and I believe I've also abstracted away the work-specific pieces of the code.

That said I'd not be surprised if it needs a few minor tweaks before it it useful for other people.

Tags: bytemark, christmas, custodian | 2 comments

Some brief software updates:

Custodian

This is the monitoring tool I wrote for Bytemark. It still rocks, and has run over 10 million tests without failure.

I'd love more outside feedback, even if just to say "documentation needs work".

Slaughter

This is my sysadmin tool for multiple hosts - consider it cfengine-lite, or cfengine-trivial more likely.

The 2.x release is finally out, and features pluggable transports. That means your central server can be running HTTP, RSYNC, FTP, or anything you like.

90% of the changes came from or were inspired by Csillag Tamas, to whom I owe a debt of thanks.

Templer

A static-site generator, written in Perl.

I use this to generate blogspam.net, and other sites from simple layouts. Tutorial available online.

redis-document-store

A trivial hack which allows using Redis as a schema-less document storage system.

Assuming you never delete documents it is simple, transparent, and already in live use for Debian Administration

Random Comment on Templer:

Although I've made extensive notes on common static site generators, and they will be discussed at length in the near future, I do want to highlight one problem common to 90% of them: Symbolic links.

For example webgen fails my simple test:

~/hg/websites$ webgen create test.example.com
~/hg/websites$ cd test.example.com/src/

~/hg/websites/test.example.com/src$ mkdir jquery-1.2.3
~/hg/websites/test.example.com/src$ touch jquery-1.2.3/jquery.js
~/hg/websites/test.example.com/src$ ln -s jquery-1.2.3 jquery

~/hg/websites/test.example.com$ webgen
Starting webgen...
...
Finished
~/hg/websites/test.example.com$ ls out/  | grep jq
jquery-1.2.3

Here we see creating a symlink to a directory has not produced a matching symlink in the output. Something I use frequently. for example.

Some tools mangled symlinked directories, or files, some ignore them completely. Neither is acceptible.

Tags: custodian, slaughter, static, templer | 2 comments

The tail end of this week was mostly spoiled by the discovery that libbeanstalkclient-ruby was not included in Wheezy.

Apparently it was removed because the maintainer had no time, and there were no reverse dependencies - #650308.

Debian maintainers really need to appreciate that no official dependencies doesn't mean a package is unused.

Last year I needed to redesign our companies monitoring software, because we ran out of options that scaled well. I came up with the (obvious) solution:

• Have a central queue containing jobs to process.
  • e.g. Run a ping-test on host1.example.com
  • e.g. Run an SSH-probe on host99.example.com
  • e.g. Fetch a web-page from https://example3.net/ and test it has some text or a given HTTP status code.
  • (About 15 different test-types are available).
• Have N workers each pull one job from the queue, execute it, and send the results somewhere.

I chose beanstalkd for my central queue precisely because it was packaged for Debian, had a client library I could use, and seemed to be a good fit. It was a good fit, a year on and we're still running around 5000 tests every minute with 10 workers.

The monitoring tool is called Custodian Custodian, and I think I've mentioned it before here and on the company blog.

It looks like we'll need to re-package the Ruby beanstalk client, and distribute it alongside our project now. That's not ideal, but also not a huge amount of work.

In summary? Debian you're awesome. But libraries shouldn't be removed unless it can't be helped, because you have more users than you know.

Tags: custodian, debian, ruby | 4 comments