About Archive Tags RSS Feed

 

It is an army bred for a single purpose

9 February 2009 21:50

It is funny the way things work out when you're looking for help.

Recently I was working on a Ruby + FUSE based filesystem and as part of the development I added simple diagnostic output via trivial code such as this:

@debug && puts "called foo(#{param});"

That was adequate for minimal interactive use, but not so good for real live use. In real live use I started outputing messages to a dedicated logfile, but in practise became overwhelmed by thousands of lines of output describing everything ever applied to the filesystem.

I figured the natural solution was to have a ring-buffer. (Everybody knows what a ringbuffer is, right?) It could keep the last 500 messages and newer debug information would just replace older entreis. That'd be just enough to be useful if I had a problem, but not so overwhelming it would get ignored.

In Perl I found a nice ringbuffer library, but for Ruby nothing. Locking a region of shared memory via shmget, shmset and keeping an array of a few hunded strings would be simple, but it seems odd I have to code this myself.

I started searching around and I accidentally stumbled upon the unrelated IPC::DirQueue perl module. Not useful for my ringbuffer logging problem, but beautifully useful.

There is no package for Debian but that was easily created:

dh-make-perl --build --cpan IPC::DirQueue

Already I have a million and one uses for it - not least to solve my problem of maintaining a centralised quarantine for all the spam mail rejected by N MX machines. (Which currently uses a combination of rsync and lockfiles.)

This is the reason why sites like Perl Advent Calendar are useful - they introduce a useful module every day or two, and introduce you to thinks that you can use in the future.

Of course keeping a sustainable site like that up and running is hard which is why sites like debaday struggle to attract contributors, for example.

Anyway random happyness.

ObFilm: Lord of the rings: Two Towers

Tags: fuse, perl, random, ruby | 3 comments

 

Comments on this entry

icon Matt Sayler at 17:51 on 10 February 2009
DirQueue looks really nice. One related trick I have up my sleeve is using (GNU) xargs' -P option to keep several sub-commands busy. This tends to be even more useful in this brave new multicore everywhere world.
Something like
find . -size 100M -a \! -name \*z | xargs -P 4 -n 1 gzip
will keep 4 gzips busy as long as there are files to be found.
This can also be handy for things like recursive greps, where you are IO-bound by metadata lookups on lots of small files:
find . | xargs -P 10 -n 10 grep pattern


icon Matthew Bloch at 22:33 on 10 February 2009
Like I said in the office, you are overengineering :-) If you want a few recent log lines, just do 'log = Logger.new("my.log", 10, 1000000)' That'll give you my.log.0, my.log.1 etc, to .9 each a megabyte in size, in order of age, represented in the filesystem. Perfectly bounded resource usage, no external libraries.
icon Steve at 22:36 on 10 February 2009

Using parallel xargs is a cute trick if you're not IO-bound and things can be done in a parallel manner :)

Using Loger and constraining the size is an adequate solution, but it just seems a little annoying. Worse still you cannot constrain on size without allowing rotation.

e.g. The following fails Logger.new("foo.log",0,100); - you have to allow at least two logfiles ("foo.log" "foo.log.0").

Still its a good compromise in the absence of a Log4Ruby package.

 

Created by ephemeris.

© Steve Kemp