Entries posted in August 2008

It is nice when you work for a company where you can say:

"Ice-lolly break..."

The response?

"Me too!"

Tonight has been a productive evening, I guess the ice-lolly helped!

I managed to optimize the storage of rejected SPAM mail for my commercial service. That is something I've been obsessing over recently since the volume of SPAM is currently hovering around 2.5 million messages.

Still I suspect it is only a matter of weeks before I need to expand. The current setup has me using three machines:

• Primary machine runs:
  • Web Application
  • SMTP processing/filtering/delivery
• Secondary machine runs:
  • SMTP processing/filtering/delivery
• Offsite machine:
  • Runs the blog.
  • Runs the support system.
  • Runs the service status page

Ideally I'd like to split that up further so that I have a single machine running the web application (the part the user interacts with), a pair of MX machines, and the offsite machine doing the minimal work it does.

That way the incoming mail will not affect the application at all directly.

Thankfully the split should be trivial. The only hard part is finding a fast webhost that can offer me ~1Gb of RAM, ~1000Gb of disk space, and won't charge much. Ideally around £15/$30 a month. (hahaha! hahaha! ha!)

ObQuote: Léon

Tags: hosting, mail-scanning, random, work | 4 comments

Well a semi-productive week during which I submitted another patch to GNU Screen - this is a trivial one, and the bug itself probably doesn't require an "important" severity.

However I'm a little disappointed to see that a bug which I submitted late last year in the exaile package (#451303) is still not fixed - and worse still we're going to be stuck with it in Lenny.

Still who knows, the recent activity suggests there might be a fix. But with the words "DNS cache poisoning" still ringing in our ears packages which automatically download and execute code from remote HTTP servers should ring alarm bells. Loudly.

(Don't forget exaile.org has already been hacked once.)

Next week I'll try to fix a couple of bugs. Publicly. Again. Unless that is dull to watch, or I find my time eaten by a .. grue.

ObFilm: The Princess Bride

Tags: exaile, screen | 3 comments

This has been a rather random few days.

• I bought a steam-boat (not really; but close enough.)
• I fell in love with mod_perl

For those of you that don't know it mod_perl is an Apache module which embeds perl into your webserver.

You can use this to write extensions, handlers, and all kinds of fun things in pure perl.

Me? I just changed the beefy CGI script that I use to power a couple of my sites from being plain-CGI to being mod_perl-CGI - that means:

• The same perl engine & copy of my script stays in memory.
• I don't need the fork()/exec() overhead for each incoming rquest.

The downside is that I have to "/etc/init.d/apache2 reload" if I change my script, or any of my custom modules it uses. (I suspect this is something I can fix; I just don't know how yet :)

All that was possible, with zero changes to my applications as I use the CGI::Application framework - lucky? or planned? I'll let you decide ...

In terms of speedup I can now process about 100 requests a second, compared to 10. As reported by the Apache-benchmark tool. Cool.

ObFilm: The Mummy: Tomb of the Dragon Emperor

Tags: cgi::application, mod_perl, perl, perl, steam | 4 comments

One of the sites that I no longer use, but have fond memories of is dotfiles.com.

It had some pretty coarse catagories and allowed you to view other peoples configuration files. (I have no idea how the upload worked. Probably email submission I'd guess.)

I know that my my own dotfiles have benefited from seeing other peoples snippets.

Sadly it seems that the last upload to their site was back in 2006.

With all the Web2.0 lust around it would seem to be a perfect candidate for reinvention.

We need:

• The ability to create/delete an account.
• The ability to upload a file (<100k say)
• The ability to tag all files with multiple arbitrary labels.
• Possibly the ability to comment / rate / vote on submissions.
• The ability to flag uploads as being "spam"

Somebody competant could probably knock up a reasonable hack in a day or two. I guess we have some sites out there already like DZone snippets, snipplr, & swik, but none of those are exactly the same thing.

Consider it my challenge to the world - just don't tempt me. I've got enough to do as it is.

ObQuote: Hellboy

Tags: dotfiles, free-time, todo | 7 comments

This week has been a little hectic, as I've been struggling with testing different versions of the GNU/Linux kernel.

Specifically I've been trying to solve a problem where a Phenom processor, when coupled with 8Gb, would kernel panic under heavy load.

After testing various patches, kernel versions, and random things I believe the problem is fixed in the kernel version 2.6.27RC4 - however nothing in the changelog appears relevant, so I guess only time will tell.

Now we need to solve the problem of Atom processors panicing when attempting to boot 64-bit kernels. That is still present in the 2.6.27RC4 kernel.

(ObRandom: If there are any interested parties I can provide remote serial console access to such a system.)

Finally I've also been playing with PAM, the plugabble authentication module. Again specific use-case here. At work we want to allow people to ssh to some systems (to access serial consoles, etc), and we wish their connections to be tested against our internal single-sign-on mechanism.

That could have meant a whole new PAM module, which would do XML-RPC-fu. Instead it meant packaging libpam-external - which is a neat PAM module allowing you to specify a shellscript to validate users & passwords.

(libpam-external is very similar to mod_authnz_external which is a similar pluggable Apache2 module)

So, this week "kernel hacking", & "pam hacking". Does that make me a real developer now?

ObQuote: Time Bandits

Tags: apache2, kernels, pam | 3 comments

Steven Brust is a big tease.

His most recent Vlad Taltos novel is full of tease for two reasons:

• He jumps back in the timeline so that we hear nothing of Lady Teldra.
• The acknowledgements of the novel mention the use of some "emacs macros" with no hint of what they are, or why he uses them.

It was a fun read though, and didn't make me as hungry as the previous volume did. (Mmmmmm pies food.)

I always liked him as an author, and he rocks for publishing Dzur around the time I was telling local people "Too many people seem to write novels in which nobody really eats. Forget all that action, dialog, and exposition. Lets have a bunch of folk sit down and eat an exceptionally well described meal."

(Many things that people do are never described in books. We all know why. Still on the same subject I love the scene in Terry Pratchetts Pyramids where Teppic puts his outfit on. "And slowly falls over". Nice)

ObFilm: Blade

Tags: authors:brust, authors:pratchett, books, emacs | 1 comment

So I got bored tonight and figured I'd write a game...

I'm genuinely not sure whether I've seen this concept before, or came up with it myself. I suspect the former. I know that I sat down with the intention of coding this game and knew how it would play and what the mechanics would be.

Having said that though I cannot think of a similar game I've played - though parts are obviously derivitive.

Anyway the aim of the game:

• A (single currently) ball bounces around the screen.
• You draw lines upon the screen, using the mouse, to influence the movement of the ball.
• The level (game) is over when the ball lands in the "exit box".

Thus far the game exists only in the skelital form with the minimum required functionality. There are two modes currently: "easy" & "hard". The hard mode was primarily added to prove to myself that the "leveling" system could work in a fun way.

Feedback welcome. Especially if it can tell me where I'm going wrong with the collision detection - but even if it is to critique my hacked-up SDL coding.

(The only other SDL coding I've done was in C, and was the mousetrap game.)

Obviously the game is written in perl, and I admit nasty perl at that. To play it you'll only need:

apt-get install libsdl-perl

Code:

• http://www.steve.org.uk/Software/tmp/skx.pl

ObQuote: The Chronicles of Riddick

Tags: bouncy bouncy, game, mousetrap, perl, sdl | 12 comments

I've updated my simple Simple SDL based perl game, so that:

• It has a name.
• It has an intro screen which uses that name!
• It has multiple balls.
• It gets harder, ie. more balls are added.
• The collision detection is much improved.

I still need to work on the rebound-angle but otherwise it is as complete as it will probably ever become. It would also be nice if the balls could collide with each other, and be different colours..

Regardless it was a fun diversion for a few hours, and probably tells me that I shouldn't attempt to waste more time doing gamy things, and that maths is too hard for me these days.

ObQuote: Interview with the vampire

Tags: bouncy bouncy, game, sdl | 3 comments

When volume becomes high enough you start to observe patterns in SPAM pretty easily. I think that this is primarily because people like to see patterns, whether they are present or not.

The trick is determining whether they are real patterns or not, and then to a lesser extent whether they are useful patterns.

For example I host mail for a business domain. That means that incoming messages come primarily from existing customers, and very rarely from potential new ones.

In practise that means that email is expected to arrive from 9am til 6pm (+/-2hours) Email received at 2AM? Either it is somebody working remotely, a foreign contact, or much more likely it is SPAM.

Now clearly you cannot dump all messages received at unusual times of the day, but it is a surprisingly robust SPAM indicator for that particular domain.

All heuristics are fallable, but some are useful regardless..

I'd love to know what people can learn from their SPAM. This week I'm handling approximately 80,000 messages a day, per MX, which isn't huge (ie. 2-3 million a month).

ObQuote: Highlander

Tags: spam | 5 comments