Entries tagged random

Related tags: acon, alsa, asql, audio, boo-boo, books, bug-reports, bugs, bytemark, camera-toys, canon, charity, children, chronicle, cloud, computers, content-free, cv, debian, discontinuation, diy, dns, docker, doctors, edinburgh, emacs, email, facebook, fantasy, filesystems, films, flat, fuse, github, gpg, hacks, health, history, holiday, holidays, hosting, html, html-tool, images, imap, irony, javascript, kettle, kindle, kite, kvm, languages, life, lighting, lighttpd, links, linkti.me, logs, lua, lumail, mail-scanning, markdown, meh, meta, misc, morbid, mutt, mybin, mysql, namecheck, names, nhs, nintendo ds, node, node.js, nodejs, not-job-hunting-mr.boss, openid, orange, paging, palm pre, pass, pastebin, perl, personal, photography, pi, picshare, planet-debian, planet-search, politics, pop3, privoxy, procmail, proxies, pubtokens, pwsafe, rants, redis, redisfs, resume, rhetorical, ruby, security, seo, serial numbers, setuid, sexism, shopping, simple-vm, skxstats, slaughter, sms, software, soul-stealing, spam, sql injection, squeeze, squeezebox, streaming, sword of truth, symbiosis, sysadmin, syslog, tef, television, terry goodkind, thttpd, todo, tools, travel, twitter, usb, utilities, webstats, work, xen-tools, xkcd, xml, xss.

Reducing, or redirecting at least, charitable donations.

Tuesday, 30 December 2014

This is the time of year when there are lots of adverts shown on TV solicating donations for charities, which frequently end with the two words "thank you".

I've always felt there were too many charities in the world, and that it was hard to half-heartedly give money to one charity this month, one the next, and still another next year. On that basis I decided long ago to give my money solely to three charities. If I had money that was spare, or I felt generous that month, I would give it to one of "my" charities. Any other appeals I just ignored (with minor exceptions for one-off events like tsunamis, etc).

I won't claim credit for this idea, it came directly from my mom who does the same thing. I've given money to the same three charities for twenty years now. Maybe not thousands, but hopefully enough to be useful. Certainly more than I'd have given if my donation were split between more recipients.

Now I'm changing. As of next year only one charitable organization will get my pennies. The other two haven't done anything bad, wrong, or failed/succeeded (sadly), but it feels better for me to stick to a single recipient.

Happy Christmas.

(Details shouldn't matter, but to answer the obvious question the charity I've kept is the RNLI.)

| 2 comments.

 

On the names we use in email

Saturday, 18 October 2014

Yesterday I received a small rush of SPAM mails, all of which were 419 scams, and all of them sent by "Mrs Elizabeth PETERSEN".

It struck me that I can't think of ever receiving a legitimate mail from a "Mrs XXX [YYY]", but I was too busy to check.

Today I've done so. Of the 38,553 emails I've received during the month of October 2014 I've got a hell of a lot of mails with a From address including a "Mrs" prefix:

"Mrs.Clanzo Amaki" <marilobouabre14@yahoo.co.jp>
"Mrs Sarah Mamadou"<investment@payment.com>
"Mrs Abia Abrahim" <missfatimajinnah@yahoo.co.jp>
"Mrs. Josie Wilson" <linn3_2008@yahoo.co.jp>
"Mrs. Theresa Luis"<tomaslima@jorgelima.com>

There are thousands more. Not a single one of them was legitimate.

I have one false-positive when repeating the search for a Mr-prefix. I have one friend who has set his sender-address to "Mr Bob Smith", which always reads weirdly to me, but every single other email with a Mr-prefix was SPAM.

I'm not going to use this in any way, since I'm happy with my mail-filtering setup, but it was interesting observation.

Names are funny. My wife changed her surname post-marriage, but that was done largely on the basis that introducing herself as "Doctor Kemp" was simpler than "Doctor Foreign-Name", she'd certainly never introduce herself ever as Mrs Kemp.

Trivia: In Finnish the word for "Man" and "Husband" is the same (mies), but the word for "Woman" (nainen) is different than the word for "Wife" (vaimo).

| 3 comments.

 

Writing your own e-books is useful

Wednesday, 8 October 2014

Before our recent trip to Poland I took the time to create my own e-book, containing the names/addresses of people to whom we wanted to send postcards.

Authoring ebooks is simple, and this was a useful use. (Ordinarily I'd have my contacts on my phone, but I deliberately left it at home ..)

I did mean to copy and paste some notes from wikipedia about transport, tourist destinations, etc, into a brief guide. But I forgot.

In other news the toy virtual machine I hacked together got a decent series of updates, allowing you to embed it and add your own custom opcode(s) easily. That was neat, and fell out naturely from the switch to using function-pointers for the opcode implementation.

| 3 comments.

 

Before I forget, a simple virtual machine

Sunday, 5 October 2014

Before I forget I had meant to write about a toy virtual machine which I'ce been playing with.

It is register-based with ten registers, each of which can hold either a string or int, and there are enough instructions to make it fun to use.

I didn't go overboard and write a complete grammer, or a real compiler, but I did do enough that you can compile and execute obvious programs.

First compile from the source to the bytecodes:

$ ./compiler examples/loop.in

Mmm bytecodes are fun:

$ xxd  ./examples/loop.raw
0000000: 3001 1943 6f75 6e74 696e 6720 6672 6f6d  0..Counting from
0000010: 2074 656e 2074 6f20 7a65 726f 3101 0101   ten to zero1...
0000020: 0a00 0102 0100 2201 0102 0201 1226 0030  ......"......&.0
0000030: 0104 446f 6e65 3101 00                   ..Done1..

Now the compiled program can be executed:

$ ./simple-vm ./examples/loop.raw
[stdout] register R01 = Counting from ten to zero
[stdout] register R01 = 9 [Hex:0009]
[stdout] register R01 = 8 [Hex:0008]
[stdout] register R01 = 7 [Hex:0007]
[stdout] register R01 = 6 [Hex:0006]
[stdout] register R01 = 5 [Hex:0005]
[stdout] register R01 = 4 [Hex:0004]
[stdout] register R01 = 3 [Hex:0003]
[stdout] register R01 = 2 [Hex:0002]
[stdout] register R01 = 1 [Hex:0001]
[stdout] register R01 = 0 [Hex:0000]
[stdout] register R01 = Done

There could be more operations added, but I'm pleased with the general behaviour, and embedding is trivial. The only two things that make this even remotely interesting are:

  • Most toy virtual machines don't cope with labels and jumps. This does.
    • Even though it was a real pain to go patching up the offsets.
    • Having labels be callable before they're defined is pretty mandatory in practice.
  • Most toy virtual machines don't allow integers and strings to be stored in registers.
    • Now I've done that I'm not 100% sure its a good idea.

Anyway that concludes todays computer-fun.

| 4 comments.

 

If you signed my old key, please consider repeating the process

Thursday, 4 September 2014

I'm in the process of rejoining the Debian project. When I was previously a member I had a 1024-bit key, which is considered to be a poor size these days.

Happily I've already generated a new key, which is much bigger.

If you've signed my old key, and thus trust my identity was confirmed at some point in time, then please do consider repeating the process with the new one.

As I've signed the new with the old there should be no concern that it is random/spurious/malicious.

Obviously the ideal scenario is that I meet local-people to perform signing rites, in exchange for cake, beer, or other bribery.

Old key:

pub   1024D/CD4C0D9D 2002-05-29
      Key fingerprint = DB1F F3FB 1D08 FC01 ED22  2243 C0CF C6B3 CD4C 0D9D
uid                  Steve Kemp <steve@steve.org.uk>
sub   2048g/AC995563 2002-05-29

New key:

pub   4096R/0C626242 2014-03-24
      Key fingerprint = D516 C42B 1D0E 3F85 4CAB  9723 1909 D408 0C62 6242
uid                  Steve Kemp (Edinburgh, Scotland) <steve@steve.org.uk>
sub   4096R/229A4066 2014-03-24

| 3 comments.

 

A diversion - The National Health Service

Sunday, 31 August 2014

Today we have a little diversion to talk about the National Health Service. The NHS is the publicly funded healthcare system in the UK.

Actually there are four such services in the UK, only one of which has this name:

  • The national health service (England)
  • Health and Social Care in Northern Ireland.
  • NHS Scotland.
  • NHS Wales.

In theory this doesn't matter, if you're in the UK and you break your leg you get carried to a hospital and you get treated. There are differences in policies because different rules apply, but the basic stuff "free health care" applies to all locations.

(Differences? In Scotland you get eye-tests for free, in England you pay.)

My wife works as an accident & emergency doctor, and has recently changed jobs. Hearing her talk about her work is fascinating.

The hospitals she's worked in (Dundee, Perth, Kirkcaldy, Edinburgh, Livingstone) are interesting places. During the week things are usually reasonably quiet, and during the weekend things get significantly more busy. (This might mean there are 20 doctors to hand, versus three at quieter times.)

Weekends are busy largely because people fall down hills, get drunk and fight, and are at home rather than at work - where 90% of accidents occur.

Of course even a "quiet" week can be busy, because folk will have heart-attacks round the clock, and somebody somewhere will always be playing with a power tool, a ladder, or both!

So what was the point of this post? Well she's recently transferred to working for a childrens hospital (still in A&E) and the patiences are so very different.

I expected the injuries/patients she'd see to differ. Few 10 year olds will arrive drunk (though it does happen), and few adults fall out of trees, or eat washing machine detergent, but talking to her about her day when she returns home is fascinating how many things are completely different from how I expected.

Adults come to hospital mostly because they're sick, injured, or drunk.

Children come to hospital mostly because their parents are paranoid.

A child has a rash? Doctors are closed? Lets go to the emergency ward!

A child has fallen out of a tree and has a bruise, a lump, or complains of pain? Doctors are closed? Lets go to the emergency ward!

I've not kept statistics, though I wish I could, but it seems that she can go 3-5 days between seeing an actually injured or chronicly-sick child. It's the first-time-parents who bring kids in when they don't need to.

Understandable, completely understandable, but at the same time I'm sure it is more than a little frustrating for all involved.

Finally one thing I've learned, which seems completely stupid, is the NHS-Scotland approach to recruitment. You apply for a role, such as "A&E doctor" and after an interview, etc, you get told "You've been accepted - you will now work in Glasgow".

In short you apply for a post, and then get told where it will be based afterward. There's no ability to say "I'd like to be a Doctor in city X - where I live", you apply, and get told where it is post-acceptance. If it is 100+ miles away you either choose to commute, or decline and go through the process again.

This has lead to Kirsi working in hospitals with a radius of about 100km from the city we live in, and has meant she's had to turn down several posts.

And that is all I have to say about the NHS for the moment, except for the implicit pity for people who have to pay (inflated and life-changing) prices for things in other countries.

| 3 comments.

 

A tale of two products

Friday, 15 August 2014

This is a random post inspired by recent purchases. Some things we buy are practical, others are a little arbitrary.

I tend to avoid buying things for the sake of it, and have explicitly started decluttering our house over the past few years. That said sometimes things just seem sufficiently "cool" that they get bought without too much thought.

This entry is about two things.

A couple of years ago my bathroom was ripped apart and refitted. Gone was the old and nasty room, and in its place was a glorious space. There was only one downside to the new bathroom - you turn on the light and the fan comes on too.

When your wife works funny shifts at the hospital you can find that the (quiet) fan sounds very loud in the middle of the night and wakes you up..

So I figured we could buy a couple of LED lights and scatter them around the place - when it is dark the movement sensors turn on the lights.

These things are amazing. We have one sat on a shelf, one velcroed to the bottom of the sink, and one on the floor, just hidden underneath the toilet.

Due to the shiny-white walls of the room they're all you need in the dark.

By contrast my second purchase was a mistake - The Logitech Harmony 650 Universal Remote Control should be great. It clearly has the features I want - Able to power:

  • Our TV.
  • Our Sky-box.
  • OUr DVD player.

The problem is solely due to the horrific software. You program the device via an application/website which works only under Windows.

I had to resort to installing Windows in a virtual machine to make it run:

# Get the Bus/ID for the USB device
bus=$(lsusb |grep -i Harmony | awk '{print $2}' | tr -d 0)
id=$(lsusb |grep -i Harmony | awk '{print $4}' | tr -d 0:)

# pass to kvm
kvm -localtime ..  -usb -device usb-host,hostbus=$bus,hostaddr=$id ..

That allows the device to be passed through to windows, though you'll later have to jump onto the Qemu console to re-add the device as the software disconnects and reconnects it at random times, and the bus changes. Sigh.

I guess I can pretend it works, and has cut down on the number of remotes sat on our table, but .. The overwhelmingly negative setup and configuration process has really soured me on it.

There is a linux application which will take a configuration file and squirt it onto the device, when attached via a USB cable. This software, which I found during research prior to buying it, is useful but not as much as I'd expected. Why? Well the software lets you upload the config file, but to get a config file you must fully complete the setup on Windows. It is impossible to configure/use this device solely using GNU/Linux.

(Apparently there is MacOS software too, I don't use macs. *shrugs*)

In conclusion - Motion-activated LED lights, more useful than expected, but Harmony causes Discord.

| 3 comments.

 

Free (orange) SMS alerts

Tuesday, 5 August 2014

In the past I used to pay for an email->SMS gateway, which was used to alert me about some urgent things. That was nice because it was bi-directional, and at one point I could restart particular services via sending SMS messages.

These days I get it for free, and for my own reference here is how you get to receive free SMS alerts via Orange, which is my mobile phone company. If you don't use Orange/EE this will probably not help you.

The first step is to register an Orange email-account, which can be done here:

Once you've done that you'll have an email address of the form example@orange.net, which is kinda-sorta linked to your mobile number. You'll sign in and be shown something that looks like webmail from the early 90s.

The thing that makes this interesting is that you can look in the left-hand menu and see a link called "SMS Alerts". Visit it. That will let you do things like set the number of SMSs you wish to receive a month (I chose "1000"), and the hours during which delivery will be made (I chose "All the time").

Anyway if you go through this dance you'll end up with an email address example@orange.net, and when an email arrives at that destination an SMS will be sent to your phone.

The content of the SMS will be the subject of the mail, truncated if necessary, so you can send a hello message to yourself like this:

echo "nop" | mail -s "Hello, urgent message is present" username@orange.net

Delivery seems pretty reliable, and I've scheduled the mailbox to be purged every week, to avoid it getting full:

Hostnamepop.orange.net
UsernameYour mobile number
PasswordYour password

If you wished to send mail from this you can use smtp.orange.net, but I pity the fool who used their mobile phone company for their primary email address.

| 8 comments.

 

The selfish programmer

Friday, 25 July 2014

Once upon a time I wrote a piece of software for scheduling the classes available to a college.

There was a bug in the scheduler: Students who happened to be named 'Steve Kemp' had a significantly higher chance (>=80% IIRC) of being placed in lessons where the class makeup was more than 50% female.

This bug was never fixed. Which was nice, because I spent several hours both implementing and disguising this feature.

I'm was a bad coder when I was a teenager.

These days I'm still a bad coder, but in different ways.

| 5 comments.

 

So I bought some new hardware, for audio purposes.

Thursday, 6 March 2014

This week I received a logitech squeezebox radio, which is basically an expensive toy that allows you to listen to either "internet radio", or music streamed from your own PC via a portable device that accesses the network wirelessly.

The main goal of this purchase was to allow us to listen to media stored on a local computer in the bedroom, or living-room.

The hardware scans your network looking for a media server, so the first step is to install that:

The media-server has a couple of open ports; one for streaming the media, and one for a user-browsable HTML interface. Interestingly the radio-device shows up in the web-interface, so you can mess around with the currently loaded playlist from your office, while your wife is casually listening to music in the bedroom. (I'm not sure if that's a feature or not yet ;)

Although I didn't find any alternative server-implementations I did find a software-client which you can use to play music from the central server - slimp3slave - and again you can push playlists, media, etc, to this.

My impressions are pretty positive; the device was too expensive, certainly I wouldn't buy two, but it is functional. The user-interface is decent, and the software being available and open is a big win.

Downsides? No remote-control for the player, because paying an additional £70 is never going to happen, but otherwise I can't think of anything.

(Shame the squeezebox product line seems to have been cancelled (?))

Procmail Alternatives?

Although I did start hacking a C & Lua alternative, it looks like there are enough implementations out there that I don't feel so strongly any more.

I'm working in a different way to most people, rather than sort mails at delivery time I'm going to write a trivial daemon that will just watch ~/Maildir/.Incoming, and move mails out of there. That means that no errors will cause mail to be lost at SMTP/delivery time.

I'm going to base my work on Email::Filter since it offers 90% of the primitives I want. The only missing thing is the ability to filter mails via external commands which has now been reported as a bug/omission.

| 10 comments.

 

Some direction, some distraction

Thursday, 27 February 2014

It seems that several people replied to the effect that they would pay people to take care of applying security updates, or even configuring adhoc things such as wikis, graphite, and MySQL.

Not enough people to rely upon, but perhaps there is scope for remote stuff being done in exchange for folding-money. (Of course some of those that replied are in foreign countries which makes receiving payment an annoyance, that's a separate problem though.)

Food for thought.

In the meantime I've settled into my use of lighttpd, which I've recently migrated to.

One interesting thing is that you can set your own "Server Name" directive:

# Set server name/version
server.tag = "lighttpd/(steve)"

This value is used by mod_dirlisting, so for example if you examine a directory which doesn't contain an index.html file you see the server-name. Cute.

Well cute unless, or until, somebody sets:

# Set server name/version
server.tag = "<script>alert(3)</script>"

That does indeed show javascript to all your visitors. Not a security problem itself, as you need to be root on the remote site. If you're root in the remote server you could just modify the actual HTML pages being served to include your javascript. That said it's a little icky.

The following patch avoids the issue:

--- mod_dirlisting.c.org	2014-02-26 00:14:43.296373275 +0000
+++ mod_dirlisting.c	2014-02-26 00:16:28.332371547 +0000
@@ -618,7 +618,7 @@
 		} else if (buffer_is_empty(con->conf.server_tag)) {
 			buffer_append_string_len(out, CONST_STR_LEN(PACKAGE_DESC));
 		} else {
-			buffer_append_string_buffer(out, con->conf.server_tag);
+                        buffer_append_string_encoded(out, CONST_BUF_LEN(con->conf.server_tag), ENCODING_HTML);
 		}

 		buffer_append_string_len(out, CONST_STR_LEN(

| 2 comments.

 

What do you pay for, and what would you pay for?

Tuesday, 25 February 2014

There are times when I consider launching my own company again, most often when it is late at night and the inpetitude of so many other companies gets me too worked up. Then I sit back and think about details and write it off.

I've worked for myself in the past a couple of times, and each time it was both more fun and more difficult than expected. Getting a couple of clients is usually easy, getting a ten more is common, but getting "many" is hard and getting "lots" is something I've never done - lots of users for free sites though, along with the associated support burdon!

So the though dies away once I sit down and work out the net profit I'd need to live. My expenses are low, so let us pretend I can easily live on £1000 a month. So the "company" has to make more than that, to cover costs, but perhaps not much.

Pretend you were offering DNS hosting you'd probably be able to implement that easily on, say 10, virtual machines, net of £150 a month. Imagine clients pay £5 for an unlimited number of domains that means you need to have 1000+150/5 = 230 clients. Not impossible, but also not easy.

Pretend instead you're offering backup space, and the numbers get bigger because disk is expensive. Again getting some users would be easy, but getting lots would be hard because your competition is dropbox, skydrive, etc, etc.

Once you start thinking of "ideas" they come easily, but the hard part is being realistic about what people would pay for. As always the idea is the easy part, the execution is the hardest part. Realistically if I were to be desperate to work for myself at short notic I'd do the obvious thing - I'd buy a pair of ladders, a bucket, and clean windows. Low overheads, reasonable demand, and I'd be both "fit" and "outdoors".

When it comes to paying for online services off the top of my head I personally pay for maybe two things, both of them niche (although profitable for their providers I'm sure), and I know many people who live on the internet but pay for nothing.

For example I'm a VIP member of an online modeling community, which in theory allows me a higher chance of persuading interesting people to pose for me.

In practice the turnover on those sites is immense. Lots of cute boys and girls hear constantly "You're so pretty, you should be a model", which is true in perhaps 1% of cases, and the net result is you have a few hard working people who do good things day in day out, and many flighty teenagers who'll pose for two-three people, and then never do it again because they realise it is neither glamourous nor easy money.

Two things I've semi-serously considered recently where hosted "status pages", and hosted "domain parking", but both have many competitors and both I can see a) some people would pay for but b) not very many.

I suspect there is no universal "I'd pay for this" online service hwich is both competition free and genuinely trivial to setup, but I'd be curious to see what people are missing, and even more curious to see what people do pay for.

| 5 comments.

 

My pastebin will now run under docker.

Monday, 17 February 2014

I've updated my markdown-pastebin site, to be a little cleaner, and to avoid spidering issues.

Previously every piece of uploaded text received an incrementing integer to describe it - which meant it was trivially easy for others to see how many pieces of text had been uploaded, and to spider all past uploads (unless the user deleted them).

Now each fresh paste receives a random UUID to describe it, and this means spidering is no longer feasible.

I've also posted the source code to Gitub so folk can report bugs, fork, etc:

That source code now includes a Dockerfile which allows you to quickly and easily build your own container running this wonderful service, and launch it without worrying about trashing your server ;)

Anyway other than the user-interface overhaul it is still as functional, or not, as it used to be!

| No comments

 

Pastebin site with markdown support

Sunday, 16 February 2014

Today I setup a new website:

Something I want, something I'll use, and something that might be useful to others?

| 4 comments.

 

So I found a job.

Friday, 17 January 2014

Just to recap my life since December:

I had worked with Bytemark for seven years and left for reasons which made sense. I started working for "big corp" with a job that on-paper sounded good, but ultimately turned out to be a poor fit for my tastes.

I spent a month trying to decide "Is this bad, or is this just not what I'm used to?", because I was aware that there would obviously be big differences as well as little ones.

At the point I realized some of the niggles could be fixed but most couldn't then I resigned, rather than prolong the initial probationary training period - because I knew I wouldn't stay, and it seemed unfair and misleading to stay for the full duration of the probationary period knowing full well I'd leave the moment it concluded - and the notice period switched from seven days to one month.

A couple of people were kind enough to get in touch and discuss potential offers, both locally, remotely in the UK, and from abroad (the latter surprised me, but pleased me too).

I spent a couple of days "contracting", by which I really mean doing a few favours for friends, some of whom paid me in Amazon vouchers, and some of whom paid me in beer.

e.g. I tweaked the upcoming death Knight site to handle 3000 simultaneous HTTP connections, then I upgraded some servers from Squeeze to Wheezy for some other folk.

That aside I've largely been idle for about 10 days and have now picked the company to work for - so I'm going to be a contractor with a day-rate for an American firm for the next couple of months. If that goes well then I'll become a full-time employee, hopefully.

| 2 comments.

 

A good week?

Sunday, 29 December 2013

This week my small collection of sysadmin tools received a lot of attention; I've no idea what triggered it, but it ended up on the front-page of github as a "trending repository".

Otherwise I've recently spent some time "playing about" with some security stuff. My first recent report wasn't deemed worthy of a security update, but it was still a fun one. From the package description rush is described as:

GNU Rush is a restricted shell designed for sites providing only limited access to resources for remote users. The main binary executable is configurable as a user login shell, intended for users that only are allowed remote login to the system at hand.

As the description says this is primarily intended for use by remote users, but if it is installed locally you can read "any file" on the local system.

How? Well the program is setuid(root) and allows you to specify an arbitrary configuration file as input. The very very first thing I tried to do with this program was feed it an invalid and unreadable-to-me configuration file.

Helpfully there is a debugging option you can add --lint to help you setup the software. Using it is as simple as:

shelob ~ $ rush --lint /etc/shadow
rush: Info: /etc/shadow:1: unknown statement: root:$6$zwJQWKVo$ofoV2xwfsff...Mxo/:15884:0:99999:7:::
rush: Info: /etc/shadow:2: unknown statement: daemon:*:15884:0:99999:7:::
rush: Info: /etc/shadow:3: unknown statement: bin:*:15884:0:99999:7:::
rush: Info: /etc/shadow:4: unknown statement: sys:*:15884:0:99999:7:::
..

How nice?

The only mitigating factor here is that only the first token on the line is reported - In this case we've exposed /etc/shadow which doesn't contain whitespace for the interesting users, so it's enough to start cracking those password hashes.

If you maintain a setuid binary you must be trying things like this.

If you maintain a setuid binary you must be confident in the codebase.

People will be happy to stress-test, audit, examine, and help you - just ask.

Simple security issues like this are frankly embarassing.

Anyway that's enough: #733505 / CVE-2013-6889.

| No comments

 

So PaaS

Friday, 6 December 2013

I just realised a lot of my projects are deployed in the same way:

  • They run under runit.
  • They operate directly from git clones.

This includes both Apache-based projects, and node.js projects.

I'm sure I could generalize this, and do clever things with git-hooks. Right now for example I have run-scripts which look like this:

#!/bin/sh
#
# /etc/service/blogspam.js/run - Runs the blogspam.net API.
#


# update the repository.
git pull --update --quiet

# install dependencies, if appropriate.
npm install

# launche
exec node server.js

It seems the only thing that differs is the name of the directory and the remote git clone URL.

With a bit of scripting magic I'm sure you could push applications to a virgin Debian installation and have it do the right thing.

I think the only obvious thing I'm missing is a list of Debian dependencies. Perhaps adding soemthing like the packages.json file I could add an extra step:

apt-get update -qq
apt-get install --yes --force-yes $(cat packages.apt)

Making deployments easy is a good thing, and consistency helps..

| 2 comments.

 

Things have settled down nicely

Saturday, 23 November 2013

I've now completed all my KVM migrations. Moving my personal virtual machines from one host to another.

There were a few niggles, for example I didn't have a working IPv6 allocation at the time I moved things so I had to set that up post-migration.

I've also joined each of the hosts into a VPN which makes cross-guest communication secure and simple.

Finally I've overhauled my firewalls and service lists.

I installed a couple of extra guests, using libvirt and booting from the Debian ISO. The Debian installer continues to impress, though it did make me think I should overhaul my PXE setup at home.

It wouldn't be hard to have a Raspberry PI running as a TFTP + DHCP server. You could plug it into a network, reboot your desktop, and then have it boot into the imager. At the moment I run DHCP + TFTPD + etc on my main desktop, and that allows me to reimage any of the hosts in the flat easily, except itself obviously.

The last time I reinstalled this system I had to reconfigure DHCP + PXE + TFTP on another host. I think the next time I need to reinstall any system I'll "waste" an SD-card on an image-server host.

Finally I've recently read the Rick Cook Wizardy Series:

  • Geeky developer gets transferred to a typical fantasy land:
    • Where magic works/exists.
    • There are dragons.
    • He writes a magic-compiler using FORTH to build primitives into bigger spells.

Fun idea. Horrible puns. Some of the books were too long, or left plot elements dangling, but on average they were more good than bad. Albeit a little predictable and "simple".

| No comments

 

Some days you just want to do nothing

Sunday, 22 September 2013

Today I finally pushed out a new binary release of my slaughter server-automation tool. (Think "CFEngine-lite", with perl. full documetnation is available. Though nobody ever reads it.)

Otherwise the weekend is being quiet; we spent last night mostly drinking vodka, until midnight rolled over, and along with some messing around with a camera ("Wow, your arms are getting bigger!")

Today has consisted of a Turkish breakfast, an Indonesian dinner, and an ice-cream based tea.

I could write more, but I'm hung-over. A rare thing for me.

| 2 comments.

 

So mid-summer is over.

Wednesday, 26 June 2013

So my partner and I spent last week in Helsinki, visiting for Midsummer.

The place was lovely, albeit "too damn hot"tm. (Despite my regularly shaved head I'm a pale-skinned redhead covered in freckles. In sustained sun I can be burned in the space of 10-20 minutes. Ouch.)

I ate cake, went to a sauna, and all the usual things. I also had personal interviews with all the mosquitos.

The only downside to the holiday was the timing:

  • Friday Worked.
  • Saturday woke up early and flew.
  • Time passes, Thorin sits down and starts singing about gold.
  • Sunday flew back.
  • Monday .. worked.
  • Tuesday worked ..

I'm a sysadmin and sometimes that involves out of hours work, so you can shuffle virtual machines around, upgrade processors, and do all the kind of "disruptive" work outside core business hours.

Yesterday I was awake for 22 hours and I was working for 14 of them. Today I'm doing nothing that involves a computer other than a token check-in or three to make sure everything is fine.

99% of the time I don't mind working late, starting early, or scheduling an event outside working hours (e.g. last night I worked from 10PM-3AM ). But with the holiday I've definitely been feeling burned. My sleeping is screwed up, and I'm just getting grumpy and stupid.

Still there were some highlights and I took some nice photos, met some good people, and learned some more Finnish.

I actually tried to find some Finnish instructor(s) here in Edinburgh, upon my return, and was amused to discover that there is an Edinburgh-based Finnish Society. Amused? They cancelled their midsummer event due to poor weather.

For the rest of the week I'm going to be very careful to count working hours and do nothing excessive. I've also got to get back into my gym-routine.

Still there is good news on the horizon. I get a new computer next Monday.

The new machine will run Wheezy, and Awesome. With 8GB RAM I'll stop hitting OOM conditions once I process many photographs, and I'm going to be very dedicated in using revision control for everything.

(I've noticed I've gotten lazy and have started storing bookmarks locally again, instead of under revision control which is a bad sign.)

Rambling. I woke up early (10:30AM) because "Microsoft Technical Support" called me and told me my computer had a virus ..

| 1 comment.

 

How do people deal with email?

Wednesday, 5 June 2013

As part of writing a new mail client I'm wondering about how to change my email-life, and how other people process/handle their incoming email.

I sort my incoming email into folders at delivery-time using procmail. Mail is generally filtered into mailboxes on the basis of the company that sent it, the person that sent it, or the machine which generated it.

Because I manage a lot of machines personally I've split things up so that I have a folder per host. So on a morning I might have unread mail in the following folders:

machines.steve.org.uk/
machines.da-db1/
machines.da-db2/
machines.da-web1/
machines.da-web2/
machines.da-web3/
machines.da-web4/

The per-machine mailboxes usually contain a single mail every day from LogWatch, along with output from any cron-jobs. For example today I received the mail:

From: Cron Daemon
To: steve@steve.org.uk
Subject: Cron steve@steve.org.uk /home/steve/bin/download-check

URL http://nodejs.org/ - no longer matches v0.10.9

Generally speaking I don't need to read the per-machine messages. I'll keep the most recent 100 for reference, but only need to look if something seems "off" on a machine. But if I don't look I'd not see the node upgrade notice, so find that I do read them after all.

This suggest to me that email isn't the right way to handle this kind of thing. Instead I should use a notification system - at work we have a central service called MauveAlert (yes, Red Dwarf reference). Mauve receives "alerts" of various kinds, via UDP. The alerts are then fanned out to appropriate people via XMPP, Email, or SMSs.

I have a similarly-inspired system I use on my Debian Administration cluster. A (node) service runs non-stop collecting UDP messages and showing them on a dashboard. I look at it throughout the day to see when slaughter runs, etc.

Anyway in conslusion I get a lot of mail. Some of it is related to random projects, and all ends up in the steve.org.uk/ mailbox, some of it relates to machines, and gets filed away, and I have regular conversions with folk so I have a .people.kirsi/ folder which receives a lot of attention, for example.

ObRandom:daily() - Mark ~/Maildir/.machines.*, etc, read.

| 8 comments.

 

A mixed week with minor tweaks

Saturday, 13 April 2013

As previously mentioned I was looking to package pwsafe for Wheezy, as this is one of the few tools that I rely upon which isn't present.

There are now packages available, with the source on github.

I've also been doing some minor scripting because I've run into a few common problems recently:

run-parts

run-parts is a simple utility which will run every executable in a directory, more or less.

In Debian-land run-parts is the mechanism for /etc/cron.daily and /etc/cron.hourly - and that is where I've had problems recently.

Imagine you run a backup via cron.daily. Further imagine that you run a post-backup rsync and that this might take many many hours. If your backup takes >=24 hours you're screwed.

To that end I've patched my run-parts tool to alert and exit if a prior invocation is still running.

silent-run

I think everybody has this script - hide all output when running a command, unless the command fails. Looking today I see chronic from Joey's excellent moreutils does this. D'oh.

I think I've done more, but I cannot remember. In conclusion software is both easy and hard - easy because these two trivial changes were within my reach, but hard because years after encountering GNU/Linux we still have to add in the missing pieces.

Still could be worse, I spent four/five hours yesterday evening fighting with MS-SQL server, and that is time I'm never going to get back.

| 8 comments.

 

Artificially inflation of facebook-likes

Thursday, 4 October 2012

Facebook Like-Inflation

If you have a website, with a "Facebook Like" box on it, it probably shows something like this:

  • 400 People Like this

Did you know that number is not just the total number of people who clicked "Like" on your page? Did you know you can artificially inflate that number?

Interesting stuff.

Send a message to yourself with the URL in the body, such that it becomes an "attachment". Watch as the like-counter increases by 1 or even 2. Lather. Rinse. Repeat.

Sending messages to other people probably does the same thing. But sending to yourself is sufficient.

| 1 comment.

 

Sometimes the important things are those you don't do

Saturday, 16 June 2012

Usually in life we focus upon the things we do, and not the things we don't do, or avoid.

I don't have many "rules" for the way I behave, the things I do, but there are some.

  • I refuse, point-blank, to purchase any food which includes the phrase "low fat" upon the label.
  • I eat, fry, and cook with real butter.
    • I refuse to use any of the alternatives. Especially the low-fat alternatives.
  • I have yet to view a film in 3D.
    • I suspect this is something I'm semi-seriously avoiding, but I've not yet explicitly decided to skip them.
  • I refuse to watch any film which is "supernateral".
  • I refuse to buy anything from a pound-land, pound-stretcher, or similarly low-end store which is labeled either "deluxe" or "luxury".
    • I made this mistake when I was a young student.

I could go on, but the "rules" are surprisingly hard to remember, having been in place for many years, they're just part of the way that I do (or do not do) things!

Finally: As of this year I've now lived in Edinburgh for over half my life.

ObQuote: "I don't know. We should go and like, stalk him or something. " - Megan Is Missing

| 7 comments.

 

So I have a new camera. Again.

Sunday, 25 March 2012

Until recently I've had a Canon EOS 1000D, my starter-camera, and a Canon EOS 40D which is my real-camera.

The 40D is older, but it probably counts as a "semi-pro" body, albeit an old mid-range one. From an image size point of view there isn't too much to tell them apart - both produce 10MP images. But from a hardware and ease of use sense the 40D has several key features which made it a compelling upgrade:

  • Dual controls. So we can use one wheel for shutter speed, and one for aperture size.
  • Better feeling body, which is slightly larger and more solid.
  • Top display for instantly obvious settings.

Anyway both these cameras have been my friends for the past year or two, although I did buy a toy camera for those times when I didn't want to carry the DLSR around.

I've made sure I only bought "posh" lens, including the fabulous and horrifically expensive 70-200 f/2.8 MK 2 lens (Just short of £2000) and I'd been wanting to use those on a full-frame camera.

Now it is upgrade time once more and I've just bought the EOS 5D MK II - a full-frame camera which means I don't have to worry about crop-factors any more.

So far I've only had it a couple of days but I'm in love. The output images are 21MP so I get far fewer to a (CF) card. But the detail is sublime.

Future portraits and photos of people will be wonderful - although I hope they already are to a large degree!

This upgrade was a hard choice. The 5D is a full-frame, but a little slow. (Faster than my 40D by a hairs-bredth) The alternative would have been a 7D which is fast, and wonderful, but still uses a cropped sensor. Given that I have fast lenses and don't do sports (often) the 5D seemed like the sanest approach.

For my reference - my Canon serial numbers:

EOS 1000D1780312242
EOS 40D1230734041
EOS 5D MK II4131916951

ObQuote: "Courage is only required when facing that which you fear. " -Stargate: The Ark Of Truth

| No comments

 

Happy birthday to me

Saturday, 10 March 2012

Recently I accidentally flooded Planet Debian with my blog feed. This was an accident caused by some of my older blog entries not having valid "Date:" headers. (I use chronicle which parses simple text files to build a blog, and if there is no Date: header present in entries it uses the CTIME of the file(s).)

So why did my CTIMEs get lost? Short version I had a drive failure and a PSU failure which lead to me rebuilding a few things and cloning a fresh copy of my blog to ~/hg/blog/.

My host is now once again OK, but during the pain the on-board sound started to die. Horribly crackly and sounding bad. I figure the PSU might have caused some collateral damage, but so far thats the only sign I see.

I disabled the on-board sound and ordered a cheap USB sound device which now provides me with perfect sound under the Squeeze release of Debian GNU/Linux.

In the past I've ranted about GNU/Linux sound. So I think it is only fair to say this time things worked perfectly - I plugged in the device, it was visible in the output of dmesg, and /proc/asound/cards and suddenly everything just worked. Playing music (mpd + sonata) worked immediately, and when I decided to try playing a movie with xine just for fun sound was mixed appropriately - such that I could hear both "song" + "movie" at the same time. Woo.

(I'm not sure if I should try using pulse-audio, or similar black magic. Right now I've just got ALSA running.)

Anyway as part of the re-deployment of my desktop I generated and pass-phrased a new SSH key, and then deployed that with my slaughter tool. My various websites all run under their own UID on my remote host, and a reverse-proxy redirects connections. So far example I have a Unix "s-stolen" user for the site stolen-souls.com, a s-tasteful user for the site tasteful.xxx, etc. (Right now I cannot remember why I gave each "webserver user" an "s-" prefix, but it made sense at the time!)

Anyway once I'd fixed up SSH keys I went on a spree of tidying up and built a bunch of meta-packages to make it a little more straightforward to re-deploy hosts in the future. I'm quite pleased with the way those turned out to be useful.

Finally I decided to do something radical. I installed the bluetile window manager, which allows you to toggle between "tiling" and "normal" modes. This is my first foray into tiling window managers, but it seems to be going well. I've got the hang of resizing via the keyboard and tweaked a couple of virtual desktops so I can work well both at home and on my work machine. (I suspect I will eventually migrate to awesome, or similar, this is very much a deliberate "ease myself into it" step.)

ObQuote: "Being Swedish, the walk from the bathroom to her room didn't need to be a modest one. " - Cashback.

| 4 comments.

 

Symbiosis is wonderful

Sunday, 4 March 2012

Symbiosis

Symbiosis is the collective name given to a group of Debian GNU/Linux packages which implement simple virtual hosting. It is developed by my employers Bytemark.

Symbiosis is basically a collection of configuration snippets, code, and libraries which works to offer virtual hosting in a reliable consistent and easy to understand fashion.

You implement hosting for a new domain by merely creating a directory tree. So for example you might configure the hosting for the domain example.com by running:

mkdir -p /srv/example.com/public/htdocs
echo "hello, world" >> /srv/example.com/public/htdocs/index.html

mkdir -p /srv/example.com/mailboxes/webmaster
echo "super-secret" > /srv/example.com/mailboxes/webmaster/password

mkdir -p /srv/example.com/config
echo "3l33t" > /srv/example.com/mailboxes/config/ftp-password

There you are, now http://www.example.com/ and http://example.com/ will work, and you may login to check mail with the email address webmaster@example.com via POP3, IMAP, IMAPS, or POP3S. Finally you can FTP with username example.com and be dropped into the public directory.

The mail handling is very flexible, and the webhosting supports wonderful things.

I don't generally talk about work-stuff explicitly, but we've just made a major new release of the Symbiosis system such that it works upon Squeeze and has lots of IPv6 support out of the box. (Email, DNS, HTTP, Firewalling, FTP etc.)

All in all it is simple, well-documented, and open-source with a reasonably large user-base. More external testers, users, and developers would be a wonderful thing..

Mutt Mailboxes & Idle Hooks?

Mutt is wonderful but I'm starting to get annoyed by its lack of auto-mailbox discovery.

Assuming you use procmail you might deliver mail to ~/Maildir/.foo/ and mutt won't notice that if the directory is created once it starts.

(This is because generally mailboxes are defined via "mailboxes =one =two ..", even if you use a shell snippet it won't get updated unless you re-read configuration, or re-exec mutt).

I wish it were possible to use inotify/dnotify/something magic such that everything beneath ~/Maildir would just work.

(Re-reading mailboxes manually is one solution but it is .. nasty?)

I'm thinking that of all the possible solutions one of the most potentially interesting would be to define a new hook: "idle-hook command .."

That way "command" would be executed every time the client is idle. (This is a distinct state unrelated to IMAP IDLE times.)

Nopte: There is already "mail_check" & "timeout" options. Even running a defined command immediately following the code for mail_check would be reasonable.

Reverse Proxy

I continue to use, love, and enjoy my node.js-based reverse HTTP proxy, and pub discussions seemed to suggest it is a great idea (due to flexibility) but it will never take on because people don't trust node.

I'm almost tempted to re-code it in LUA & C. But I can't help but think that would be a waste of time which would not increase adoption - after all most people use "simple" reverse proxies, and they are well suited by Apache, nginx, or even varnish.

Still no rush I suppose.

In more personal news after living in this flat for 7 years, or so, I'm getting a new bathroom designed and deployed. Good times.

In the meantime I've been steadily watching Stargate SG-1 having recently purchased a box-set of series 1-10. I've just started series six this evening, and I'm enjoying it a lot.

ObQuote: "You have been recruited by the Star League to defend the frontier against Xur and the Ko-Dan armada. " - The Last Starfighter (1984). First film I ever saw at a cinema as a child.

| 9 comments.

 

So I removed some more software from my host

Saturday, 3 December 2011

Today I was idly performing some maintainence upon one of my hosts, and it crossed my mind to look beneath /etc in there I found:

/etc/python
/etc/python2.4
/etc/python2.5
/etc/python2.6

That made me look more closely at the contents of /etc - the following command output was surprising:

steve@steve:~$ ls /etc | wc -l
187

Is that average? Heavy? Light? I have no idea, but I purged a hell of a lot of software today. Now I have only python v2.6 although for some reason I still have:

python
python-apt
python-apt-common
python-central
python-minimal
python-support
python2.6
python2.6-minimal

I suspect I could drop the pything2.6-minimal package, but for the moment I'm done. I have to make pretty people look exceptional with my magical camera.

Anyway as part of this cleanup I ran a quick sanity-check on which processes are running and I think, short of kernel processes, I'm as minimal as I can be. I understand the purpose and reason for every running service:

UID        PID  CMD
root         1  init [2]
pdnsd    14091  /usr/sbin/pdnsd --daemon -p /var/run/pdnsd.pid
root     14199  /usr/sbin/monit -c /etc/monit/monitrc -s /var/lib/monit/monit.state
root     14206  /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
root     14234  /usr/sbin/cron
102      14595  /usr/sbin/exim4 -bd -q30m
redis    14627  /usr/bin/redis-server /etc/redis/redis.conf
root     14637  /usr/sbin/sshd

These are basic services; I use monit to ensure those essential daemons keep running. The only oddity there is probably the local DNS cache, but it is useful if you run any kind of DNS blacklist-using service, for example.

root     14794  /sbin/getty -L ttyS0 9600 vt100

I need a serial console login for emergencies.

root     14796  runsv node-reverse-proxy
root     14797  /bin/sh ./run
root     14799  /opt/node/bin/node node-reverse-proxy.js --config ./rewrites.js

These three processes combine to run my reverse proxy which routes incoming HTTP requests to a number of local thttpd instances.

qpsmtpd  27309    /usr/bin/perl -Tw /usr/bin/qpsmtpd-prefork --port 25 --user qpsmtpd --pid-file /var/run/qpsmtpd/qpsmtpd.pid --detach
..

The perl SMTP daemon which runs my incoming mail, passing it to exim4 which listens upon 127.0.0.1:2525. You can read about my setup in the out-of-date writeup Chris & I put together.

 /usr/bin/memcached -m 64 -p 11211 -u root -l 127.0.0.1

Memory cache for transient items.

s-blog    thttpd -C /etc/thttpd/sites.enabled/blog.steve.org.uk
1030      thttpd -C /etc/thttpd/sites.enabled/edinburgh-portraits.com
s-hg      thttpd -C /etc/thttpd/sites.enabled/hg.steve.org.uk
s-ipv4    thttpd -C /etc/thttpd/sites.enabled/ipv4.steve.org.uk
s-ipv6    thttpd -C /etc/thttpd/sites.enabled/ipv6.steve.org.uk
s-kvm     thttpd -C /etc/thttpd/sites.enabled/kvm-hosting.org
...

One thttpd instance is launched for each distinct HTTP site my server runs. Each site runs under its own UID, with its own chrooted directory tree. This is important for security.

Each local instance listens upon 127.0.0.1 - and the reverse proxy previously mentioned rewrites connections to the appropriate one.

1016     28812     /usr/bin/perl -I./lib/ -I./ /usr/local/bin/blogspam

My anti-spam filter for blog comments.

Here is my christmas challenge. Can you identify each service upon your host? Do you know why you're running what you're running?

Me? I had no idea I had a dbus deamon running. Now I've purged it. Ha!

ObQuote - "I owe everything to George Bailey. Help him, dear Father." - It's a wonderful life.

| 6 comments.

 

Goodbye mysql ..

Tuesday, 22 November 2011

Yesterday evening I updated my server to remove MySQL:

steve:~# dpkg --purge mysql-client-5.1 \
                      mysql-common     \
                      mysql-server-5.1 \
                      mysql-server-core-5.1 \
                      python-mysqldb        \
                      libdbd-mysql-perl     \
                      libdatetime-format-mysql-perl

Until last month I had two database in use, one each for a pair of web-applications. As of now one is using redis - which I'm already using for my image hosting - and the other application is using SQLite.

Until recently I had a high opinion of SQLite, although that has now been downgraded a little, it is still a thoroughly excellent piece of software. I was just surprised at little things it was missing, to the extent I had to rewrite my applications SQL.

Still one less service is a good thing, and the migration wasn't so painful..

In more productive news I recently acquired a nice external flash - the Yongnuo YN-460 II is (very) cheap and cheerful, it can be fired remotely with my triggers so I've had a lot of fun with opportunistically taking pictures and experimenting with lighting.

Most of the results are NSFW, but there are some other examples lurking around including the first time I managed to successfully capture a falling water-drop. (Not the best picture, not the most explicit effect, but fun regardless. I both can and will do better next time!)

Somebody recently asked me to write about "camera stuff under linux" and happily I declined.

Why decline? Because there are so many good tools, applications, and utilities. (I use local tools for organisation and duplicate detection, rawtherapee for RAW conversion and GIMP for touchups). Having many available options is fantastic though, and something hard to appreciate for "newcomers" to Linux.

(Yeah I waited 90 seconds - if I remembered to add -nojava - for Netscape Navigator to start, under X10, with 8Mb of RAM. Happier days are here. Sure DRM is bad, secure boot .. an open question, but damn we have it good compared to almost any previous point in time!)

ObQuote: "Yeah, obviously it is only a tactical party. I'm only having a party to eventually get sex." - Peep Show

| 2 comments.

 

Scriptable email clients

Monday, 5 September 2011

This is just a quick post to remind myself in the morning, as soon as I've made it I intend to turn my computer off and leave it off until I can re-organize my office.

I've been using mutt for my email for the past few years. Nothing compares to the flexibility of procmail/sieve for organizing server-side mail, and then mutt is ideal for reading them.

With the addition of the mutt-patched sidebar mode you can even go for a few days before realizing you're not in a graphical environment. But one thing I do long for is the ability to execute scripts at various times.

Thus far I've not actually planned what I'd like to do, but as a starting point imagine being able to execute a hook when new mail arrives? Or when you send a message matching a pattern in some fashion?

There are some things out there, such as the various hacks which are designed to abort sending a message if you mention "See attachment" in a message body but fail to add one before sendign the message. These hacks generally abuse the sendmail configuration such that they're extremely ad-hoc and hard to chain/nest.

I've mellowed out over the years and I have no interest in attempting to write a mail-client (though at the same time how hard can it be? Just restrict yourself to using inotify on ~/Maildir and offload delivery to exim and you're almost done? I guess the hard part is the UI, though I do like the mutt + sidebar layout. Write the whole thing in some scripty language?)

I'll re-examine notmuch and gnus over the next week or two, but I suspect both will continue to disappoint in various ways.

Anyway, for the moment I'm just pondering. But threading is an obvious concern. Most current mutt hooks relate to the local folder, or the local message. If I were viewing a message in one directory and a new mail notification fired for a delivery to both ~/Maildir and ~/Maildir/.people.foo I'd need to either serialise them or thread them.

Ponder ponder.

In other news I've been doing more photography recently. Nothing cohesive except for my recent experiment with shooting a "street-girl" outdoors in falling light, but that was an interesting challenge and the results were sufficient to make me want to try shooting outdoors in an organized fashion again. (Some random images have been linked to from my wee twitter page.)

ObFilm: "She doesn't get eaten by the eels at this time " - The Princess Bride

| 5 comments.

 

There is a reason why I test sites

Saturday, 16 July 2011

Recently I was at a pub and there was an advert for pub tokens displayed on the window. Seemed like a cute idea:

  • Buy & donate tokens which can be spent (only) on beer.

Perfect for friends, family, remote hackers/developers & similar.

When I got home I checked out their site. Seemed simple and nice enough, with good coverage (in terms of local drinking establishments that would accept their tokens).

I decided to sign up, with the intention of gifting my sister with some delicious beer. Unfortunately that's where it all went wrong.

I tend to act the same on all new sites. Partly to amuse myself, partly to get a feel for how safe/secure/good the site is, I'll try to login with a few different values.

You know you're in trouble when you see responses like this:

SELECT * FROM cms_module_pubtokens_users where email = '"'' AND
   password =''"' LIMIT 1

Fatal error: Call to a member function FetchRow() on a non-object in
  /home/pubtokens/U79P18WQ/htdocs/includes/functions.php on line 291

Suffice it to say I sent them an email, then poked them on twitter, but to no avail.

In conclusion they don't get my money, and I couldn't recommend them to anybody else at this point either. As I'm not a customer at least I can rest easy knowing my details haven't been compromised at any point over the past few months.

ObQuote: "It can't rain all the time" - The Crow

| 4 comments.

 

So I put together another picture sharing site

Tuesday, 10 May 2011

Almost a year ago I put together a simple picture sharing site, which was designed to accept uploaded images via OpenID logins and then share them. It was created because none of the existing image sharing sites allowed OpenID logins - and because frankly I rarely trust sites I don't control. (Especially when they're free sites, with no obvious revenue stream. We can reference recent compromises if we wish to be topical, but even without them it is all about trust.)

I opened up the site to allow others to use it, but assumed it would remain Steve-only - After all why would somebody choose to use my site rather than imgur, or similar?

Over the year I've spotted several minor problems with the site. The built-in support for comments was never used, primarily because I didn't have any other users, and some of the tagging aspects were hard to use.

Since I've started using twitter I again found myself wanting a picture-hosting site, one with "short links". I started using twitpic.com, but wasn't happy with it.

So the plan was formed; build another picture sharing site. But this time with short URLS. Sadly my first choice of http://picc.ie was denied to me - I couldn't register it as I don't live in Ireland. But my second choice http://linkti.me was free.

As a challenge to myself this site uses Redis for storing all user-details, all image details, and all session data. It hasn't been difficult to implemenet a multi-user site, with logins, file uploads, and tagging. I guess I cheated as a lot of the code is borrowed from the previous site, but this time it looks prettier and there have been many cleanups due to my previous experience.

This time round I was more focussed upon short links, so each image is allowed an incrementing number, which is then encoded using base 36 math - which is the alphabet you get if you use [0-9A-Z]. I designed three shareable links for each image to keep it simple:

Why "/r" for the thumbnail? Well because it is a random thumbnail. To cope with difference sizes, and allow the recent image display to look regular, I wanted each thumbnail to be 150x150 pixels. My thumbnail generation simply picks a random X,Y co-ordinate from the image and then cuts out a 150x150 square block starting at that offset.

The results of a random segment of an uploaded image are sometimes very cool, altough equally often they are misleading. If I upload naked people there is the possability that we'll just get a nipple or a cock - but I'm actually quite smug about the way it turned out. Simpler to code than to resize, scale, or similar and surprisingly random when you see 9x9 random images (as you do on the front page).

Anyway that's all for now. I was tempted to talk about implementation or migration, but I've written too much for an early morning already. Some other time?

ObQuote: "He's just trying to freak us out. " - 30 Days of Night.

| 4 comments.

 

So I finally succumbed and joined twitter.

Wednesday, 20 April 2011

So I've succumbed and joined twitter. I'm pretending I'm a photographer, rather than a techie/geeky/random person, whether that persists is anybodys guess.

At the very least I don't feel any real need to write/post/ask about Linux, development, or similar. I can do that here or elsewhere. (Not least because length-restrictions are annoying; yes it is true. Size is important.)

So far I've been posting semi-random pictures, I guess that is one way to pass the time, while I play with my camera.

ObQuote: "Please, please, please drive faster!" - Wanted.

| 1 comment.

 

IPv6 and thttpd

Friday, 8 April 2011

thttpd is a simple, small, portable, fast, and secure HTTP server which supports both IPv4 & IPv6.

However one noticable omission in the handling of requests for thttpd is support for the X-Forwarded-For header - which is even noted upo nthe thttpd wikipedia entry.

There is a simple patch floating around which claims to fix this; but as I belatedly noticed tonight it only works for IPv4.

If you look at libhttpd.h of the thttpd source you'll see this:

typedef union {
    struct sockaddr sa;
    struct sockaddr_in sa_in;
#ifdef USE_IPV6
    struct sockaddr_in6 sa_in6;
    struct sockaddr_storage sa_stor;
#endif /* USE_IPV6 */
    } httpd_sockaddr;

As a quick hack I updated this structure to add the following member:

    char real_ip[200];

Now I could update that member when a client connects and later update it as a result of any X-Forwarded-For: headers which might be present in incoming requests. Finally I updated the logging to use this field rather than anything else and the job was complete.

Without this work if you're running as a proxy and you receive an IPv6 connection you'll see it reported as 127.0.0.1.

I'm sure my approach isn't as clean as it could be - due to the extra member- but it will suffice for now.

ObQuote: "This gun you're holding belonged to your father; he could conduct a symphony orchestra with it. " - Wanted

| 3 comments.

 

nodejs is fun

Friday, 18 March 2011

A while back I was working on a mod_rewrite compatible proxy server written in C. The reason for this is that my current webhost uses Apache2 in front ofa number of thttpd processes, and I'd like to remove apache and use something smaller/faster/neater.

Dividing things up I'm running about ten domains, and only around half of them use mod_rewrite rules - small enough perhaps to port the rules, large enough to make it annoying.

Upon reflection I think the thing to do is to replace apache with javascript - via node.js. Writing proxies with node.js is almost ridiculously simple - in fact doing anything HTTP-like is very very simple thanks to the bundled libraries.

Time will tell whether this is a waste of time or not, but I'm confident I could listen upon *:80 and route requests to localhost:N for a few domains.

As a trivial toy I wrote a simple transforming proxy last night which performed simple rewrites as it passed traffic about.

Another fun, and possibly useful, thing I put together was a node.js port of the simple httpstat.us website. That server would have been easy to write in any number of languages, but in node it was almost too easy.

Update - here is my rewriting node.js-based reverse-proxy.

ObQuote: "I just need you to stop being nice to me unless you're gonna marry me. " - He's just not that into you.

| No comments

 

I updated my redis-based filesystem

Wednesday, 2 March 2011

In July last year I made a brief post about a simple filesystem I'd put together which used Redis for the storage.

At that time I thought it was a cute hack, and didn't spend too much time with it. But recently I found a use for it so I cleaned it up, synced up the C client for Redis which I used and generally started to care again.

If it is useful you can now find it online:

The basic idea is the same as it was before, except I did eventually move to an INODE-like system. Each file/directory entry receives a unique identifier (integer) - and then I store the meta-data in a key based off that name.

This means for a file I might have keys, and values,like this:

KeyValue
INODE:1:NAMEThe name of the file (e.g. "passwd").
INODE:1:SIZEThe size of the file (e.g. "1661" )
INODE:1:GIDThe group ID of the file's owner (e.g. "0")
INODE:1:UIDThe user ID of the file's owner (e.g. "0")
INODE:1:MODEThe mode of the file (e.g. 0755)

To store these things I use a Redis "SET" which allows me to easily iterate over all the entries in each directory.

ObQuote: "They fuck up, they get beat. We fuck up, they give us pensions. " - The Wire

| 3 comments.

 

I like languages

Thursday, 24 February 2011

One of the reasons I like Scotland is the fun that Scottish people have with language. I'm going to use two examples to illustrate my point:

  • "Mind" is often used as "Remember"
  • "How" is often used as "why".

The last one is particularly fun when you use questions such as "How no?" - meaning roughly "Why not?".

Languages, and idioms, vary wildly in different parts of the world, even when you restrict yourself to English-speaking languages. I'll not even get started on Accents. The UK is tiny compared to many other countries, yet we have a wide array of accents - Australia, by contrast is huge, but I can think of only two accents across the country. (Rationally I expect that there are many accents in different parts of Australia, and I'm merely ignorant.)

In conclusion languages are fun, and some places this is more evident than in others. I will most likely contintue to say "The shop is open from 9 while 4" rather than the more typical "From 9 til 4" - I'm allowed to do that, having grown up in Yorkshire!

(PS. PHP still sucks - Even if you post it upon a PHP-powered blog. ;)

ObQuote: "People take you for granted, you know. We gotta make people miss you." - Hancock

| 3 comments.

 

You seem uncomfortable.

Saturday, 6 February 2010

I've been trying to remember to post the pictures I like online for the past few months. So this is a reminder to myself.

This image below didn't turn out quite how I wanted it to:

  • I was hoping for a nicer sihouet upon the lady's face.
  • The tree-branch on the left irritates me.

But that said I keep on coming back to look at it. I like the lighting, and I love the way that the brick wall on the right hand side angles towards the building on the horizon.

Enjoy. Or not.

Sunset

A similarly "not perfect" image is this outdoor shot. I have only one irritation with this shot - and that is that the trees are clipped at the top. Meh, such is life.

(I have two styles of photography; semi-random where I snap what is in front of me, and staged where I try to construct a particular picture - the two images above? One of each.)

ObFilm: Bound

| No comments

 

For the record, that's a question you never have to ask.

Friday, 8 January 2010

Five years ago I spent an hour wandering around a large department store looking to buy a kettle & a set of bathroom scales. Much to the amusement of the woman I was shopping with I spent a very long time trying to find the cheapest available set of scales. (We're talking at least 20 minutes, due to the nature of the store and the crowds.)

Once I'd selected the cheapest possible set of bathroom scales we walked over to the kitchen section of the store. I glanced over all the available kettles and picked up the one that looked the nicest (in terms of size, shape, and handle design) with no regard for the price at all.

Why? A set of bathroom scales I use maybe twice a year. A kettle I use in excess of ten times a day. Something you use that often should be right. Even if over time you take it for granted and forget about it. (FWIW the scales were £6.50 and the kettle cost me £39.95 - John Lewis 15/03/2005 - I kept the reciept!)

I'll haggle and quibble over prices for a lot of things, trying to ensure that I don't pay too much. But there are items which are worth paying for (and I don't just mean that "expensive == good" idea some people seem to have). On that basis I'll think nothing of paying £150 for a pair of shoes for example, even though I'll go out of my way to save £5-£10 on a DVD player. Because shoes are important, used very very often, and DVD players just aren't.

(ObReference: I have one pair of shoes. I have five pairs of boots. I might pretend I don't but I also have a pair of sandals. Sshhh it'll be our little secret. ;)

Anyway today my kettle broke. I had to buy a new one at short notice. I did so and the replacement is obviously more advanced. It boils quickly and quietly which is technically an advangtage but in practise is actually a drawback.

Generally speaking I'll fill the kettle, turn it on, then wander away. I'll only return to the kitchen to make my delicous beverage when I hear the "click" signaling that the kettle's job is done. This new one? From outside the kitchen I cannot hear it at all...

In conclusion: Technology and progress is all around us. Sometimes a technical step forward "being quiet" is a bad thing.

In other news I'm fighting with IPv6 & a head cold. Both suck.

ObTitle: Alias

| 3 comments.

 

I work alone like you. We always work alone.

Friday, 27 November 2009

A couple of days ago I was lamenting the state of webstats, although I was a little vague as to my purpose. Specifically I was wanting to find out about the screen resolutions and user-agents viewing a couple of sites.

To get screen resolutions you really need to inject javascript into your pages, which is icky. Still its a small price to pay, and chances are most people won't notice.

Of course there are drawbacks:

Javascript dependency.

If the visitors don't use/enable javascript you see nothing.

You cannot capture everything.

e.g. HTTP status code isn't available.

To solve this problem completely you therefore need to have access to both your apache logs and your javascript-captured information. Probably.

As a proof of concept I've injected the following javascript into most pages of three sites. This code:

  • Finds the screen resolution.
  • Finds the HTTP referer.
  • Finds the current page's title.
  • Then submits that to a server-side collection script, via a one-by-one pixel IMG

The script that receives the data writes out the data to a small per-domain SQLite database, which I can then use to generate prettyness. However I suck at being pretty, in most ways, so I've only got functional:

All of this is dynamic and most of the data is anchored to "today", as thats proof of concept enough. Were piwik not written in vile PHP I'd use that - I don't see anything similar out there which is Perl..

The big decision is now "Keep it dynamic" vs. "Output static pages". (vs. call off the experiment now I know that I'm safe to assume "big resolutions").

(Naming software is hard. Recent stuff I've done has had an skx prefix primarily for google-juice. e.g. Randomly I notice that if you search for my personal site on Google's UK engine I come top. Cool.)

ObSubject: The Bourne Identity

| 2 comments.

 

Do I look like your travel agent?

Sunday, 1 November 2009

This entry is primarily composed of "random".

Palm Pre

Several people have persuaded me that I need to change my phone. I've elected to purchase a Palm Pre. Rooting them, and installing Debian is trivial, though I think its missing an openssh client - so I can read my mail in mutt via the device.

(I've seen mentions of the "scary black window" as a terminal; it isn't obvious how well that works.)

I went shopping yesterday to purchase one, but because my mobile phone contract doesn't end for another 9 days I'd have to pay an extra fee. Instead I'm going to wait 9 days and get it for free.

Getting Bigger

Having randomly remembered the idea that people shrink over the course of a day due to gravity affecting the spine I decided to test this.

For a few days in a row I measured my height before going to bed, and then again in the morning. It certainly appears to be true, average difference in height is about 9mm for me.

SEO - Is it hard?
I was involved with the setup of a new site last weekend. Today it is top-5 when searched for by two pretty broad keywords in google.

This does not seem unusual for me - though I appreciate there is a difference between "site being popular/succeeding" and "site being findable".

(I remember once attending an interview for a hotel portal site. The interview wasn't that interesting, but I remember they perked up a lot when I said "Search google for Steve Kemp - I come top".)

ObFilm: Mortal Kombat

| 8 comments.

 

Looks like me an Vincent caught you boys at breakfast

Sunday, 27 September 2009

It is interesting that François Marier recently posted a brief "howto" document on debugging problems caused by overly-agressive filtering with privoxy, as I've recently been having problems with that tool.

My home network frequently changes configuration depending on what I'm concentrating upon, but every few months I'll start/cease using the following tools:

  • squid - The caching proxy server.
  • tor - The onion router.
  • privoxy - The filtering cache.

Recently I was experimenting with XSS attacks against various browsers, which meant using them for real. As not all browsers have the same anti-advert setups I was running privoxy to filter out web-annoyances, and I spotted a major flaw with it.

Unfortunately I can only describe the problem, not reproduce it, or track it down. I'm 80% certain the bug is in privoxy, but the stack is suitably high that determining that for sure is problematic.

In short the issue is that HTTP requests would end up being sent to the wrong host:

  • I load my start page in one tab: http://www.steve.org.uk/start/
  • I click to open the following URL in another tab: http://www.perlmonks.org/?node=Newest Nodes.
  • The request gets sent to http://steve.org.uk/?node=...

After that clicking around consistently sends requests to the first HTTP host which was accessed succesfully. So, for example, attempting to visit http://foo.com/bar/ will send the request to http://steve.org.uk/bar - which then gives a 404.

In terms of setup I use a dnsmasq DNS cache, privoxy and iceweasel from Debian unstable. From the symptoms I'm not sure if iceweasel's "KeepAlive" system is to blame, or if privoxy has a bad cache of hosts. Perhaps it is dnsmasq returning bogus DNS data, or my cable connection itself having DNS issues.

Anyway once the symptoms present themselves closing the browser and restarting the cache fixes it. Until the next time which might be hours or days later.

I'd report it as a bug - but I don't know where it should be. Privoxy caching things it shouldn't? iceweasel having keepalive issues? dnsmasq returning wrong DNS entries?

I'd ask "Have you seen this before, internet world?" but I guess if you have tracked it down it'd be fixed by now, and it clearly isn't!

Anyway for the moment I've uninstalled privoxy.

ObFilm: Pulp Fiction

| 4 comments.

 

Oh, this should be stunning.

Saturday, 8 August 2009

Recently I've been writing some documentation using the docbook toolset.

"Helpfully" the docbook tools produce a nice table of contents for your documentation. For example it will produce an index.html file containing a list of chapters, list of figures, list of tables, and finally a list of examples.

For my specific use I only wanted a table of contents listing chapters, all the other lists were just noise.

Unfortunately I've produced my documentation using the naive docbook2html tool, and all the details I can find online about customising the table of contents to remove specific items refer to using xslt and other more low-level tools.

So I thought I'd cheat. Looking at the generated index.html file I notice that the contents I wish to remove have got class attributes of TOC.

Is there a tool to parse HTML removing items with particular ID attributes? Or removing items having a particular CLASS?

I couldn't find one. So I knocked one up, using HTML::TreeBuilder::XPath, perhaps it will be useful to ohters:

html-tool --file=index.html --cut-class=foo --indent

The file index.html will be read, parsed, and all items with "class='foo'" will be removed. The output will be indented in a pretty fashion and written to STDOUT.

This example does a similar thing:

html-tool --url=http://www.steve.org.uk/ --output=x.html \
  --cut-id=top --cut-class=mbox --indent

I dabbled with allowing you to just dump HTML sections, so you could run:

html-tool --show-class=foo --file=index.html

But that didn't seem as obviously useful, so I dyked it out. Other similar operations could make it more generally useful though - right now it's more of a html-cut than a html-tool!

ObFilm: The Breakfast Club

| 7 comments.

 

You tortured me? You tortured me!

Thursday, 30 July 2009

DNS is hard, let's go shopping.

<irony>

CNAME & MX records do not mix.

</irony>

ObFilm: V for Vendetta.

| No comments

 

I'm getting married, I'm not joining a convent!

Tuesday, 21 July 2009

(This post was accidentally made live before it was completed; it is now complet.)

I'll keep this brief and to the point.

syslog indexing and searching

Jason Hedden suggested using swish-e to index and then search syslog files which are stored on disk - rather than inserting the log entries into mysql.

I have 120+ machines writing to a central server, and running a search of 'sshd.*refused' takes less than a second to complete now.

(To be fair using php-syslog-ng was fast, it was just ugly, hard to manage, and the mysql database got overloaded)

Cloud Storage .. but on my machines

I've become increasingly interested in both centralised hosting, and reliable backups.

Cloud storage, where I contrl all the nodes, allows good backups.

So far I've experimented with both mogilefs and peerfuse. Neither setup is entirely appropriate for me, but I love the idea of seamless replication.

ice-creams

Many ice-creams bought in supermarkets come in packs of three. Annoying:

  • One for madam x.
  • one for me.
  • Who gets the spare? (Me, when she's gone ;)

It happens too often to be a coincidence: my cynicism wonders if it is designed to ensure people buy two boxes..?

new software releases

skxlist, the simple mailing list manager, got a couple of new options after user-submitted suggestions.

asql got a bugfix.

My todolist code is now running on at least one other site!

Nothing else much to say mostly because I'm suffering from poor sleep at the moment. In part because I've got a new clock on my bedroom windowsill and the ticking is distracting me (not to mention the on-the-hour chime!)

Still I'm sure it will pass. I grew up in York in a house that had the back yard abutting the local convent. Every hour, on the hour, they'd have bell ring! We moved house when I was about 11, but for months after the move I'd still wake up at midnight confused that the bells hadn't rung...

ObFilm: Mamma Mia!

| 5 comments.

 

Not even if you let me video tape it.

Friday, 10 July 2009

The online todo list seems popular, or rather a lot of people logged in with the posted details and created/resolved tasks at least.

It is kinda cute to watch multiple people all using a site with only one set of login credentials - I guess this is a special case because you cannot easily delete things. Still I wonder how other sites would work like that? Would visitors self-organise, some trashing things, and others repairing damage? (I guess we could consider wikipedia a good example.)

Anyway I've spent a little while this morning and this lunchtime adding in the missing features and doing things suggested by users.

So now:

  • "Duration" is shown for both open & completed tasks.
  • The "home" page is removed as it added no value.
  • Tasks may be flagged as urgent.
  • *Tasks which have titles beginning with "*" are urgent by default).
  • Searching works across tags, notes, and titles.
  • Tag name completion is pending.

I think in terms of features that I'm done. I just need to wire up creation of accounts, and the submission of tasks via email. Then un-fuck my actual code.

I guess as a final thing I need to consider email notices. I deliberately do not support or mandate "due dates" for tasks. I think I prefer the idea of an email alert beign sent if a task is marked as urgent and has had no activity in the past 24 hours. (Where activity means "new note". e.g. you'd add "still working on this", or similar to cancel the pending alert)

Sending alerts via twitter could also be an option, although I still mostly abhor it.

I've had a brief look at both tadalist.com and rememberthemilk.com both seem nice .. but I'm still not sure on a winner yet.

ObFilm: Chasing Amy

| 4 comments.

 

I bet you're a real tiger in disguise.

Sunday, 5 July 2009

I've not been online much for the past week, for two main reasons:

My cat was injured

Usually my cat lives outdoors for just over half the day, but last week he came back with a bad limp and since then he's been an indoor kitty.

He stopped limping so badly pretty much the next day, but has managed to scratch/bite a fair amount of fur from his leg worrying at it.

Still that seems to have stopped and I'm sure he'll be fine. Once the fur has grown back I'll throw him back outside and hope he's more careful in the future!

(No idea whether it was a bad fall, a collision, or a fight with a cat/dog/fox/squirrel that caused it .. no obvious injuries when he was at the vets such as bites, scratches, or things embedded in the limb.)

Geomtery Wars Galaxies

I might be a bit slow, but this nintendo DS game rocks. Hard.

End Communiqué

ObFilm: Faster, Pussycat! Kill! Kill!

| 5 comments.

 

The doctors say you're going to live, that's the bad news.

Thursday, 11 June 2009

It is annoying that some protocols and systems are more complex than you might expect them to be.

Jabber is a protocol that is notionally simple: XML Messages pass back and forth between server(s) and client(s). But if you look at the contents of XML which is passed around you'll soon discover that even logging in is a complex operation and that Jabber is not implemented in a pleasant fashion.

By contrast many other protocols are lovely. I'm sure I'm not alone in using and debugging many common protocols with nothing more than telnet. SMTP, HTTP, POP3, etc, are all pretty easy to drive interactively.

I think 90% of programmers at some point in their lives implement a HTTP server. But I draw the line at that kind of thing these days, client-side applications are useful and simple enough with the right libraries. (e.g. my sift client-side IMAP scripter has replaced procmail on a couple of machines. Watching to see if I get a reply from somebody specific and sending me an SMS on a match..)

But recently I've been flirting with the development of an IMAP server.

Dovecot appears to be the canonincal IMAP/POP3 server these days and it is pretty close to meeting my needs, but it isn't close enough unless I jump through and change the way my mailboxes are organised. (ie. The maildir mailboxes are arranged in such a fashion that dovecot cannot easily handle them, unless I mess about with symlink farms and make them all read-only.)

I guess in conclusion it would be nice if there were a basic IMAP server framework which you could just subclass "login" and "mailbox" sections and then instantiate.

I wrote a quick inetd-driven hack which supports only the bare essentials ("NOOP", "CAPABILITY", "LOGIN", "FETCH", "SELECT" and "LIST") That allows me to connect via IMAP in both mutt and thunderbird, view folders and download messages.

Still I'm strongly suspecting that there are better uses of my time, even if I could use it in several ways..

ObFilm: La Femme Nikita

| 9 comments.

 

You'd better get yourself a garlic T-shirt, buddy, or it's your funeral

Friday, 6 March 2009

There are times when I hate xkcd. Mostly these are:

1. When reading a discussion on /. and you just know a particular image will be posted.

2. When you spend hours searching for a specific comic that you're certain exists.

The latter is what bit me tonight - I'm certain there exists a cartoon which has a plot of:

Woman says hi.

Guy says hi.

Woman looks confused.

Guy realises she was talking to her phone, not him.

Cannot find the image for the life of me - only phone-related image I could find was tones.

I thought I might get lucky if I knocked up a quick hack to search the alt-text on all the images, but sadly not.

Still it was a fun project. To be uber-useful we'd need to persuade people to imput the text in each cartoon, along with the number.

Given that there are only 550ish cartoons published thus far creating a database would take a person a day, or a group of people a couple of hours.

Tempting .. very tempting ..

ObFilm: The Lost Boys. Yay!

| 7 comments.

 

It is an army bred for a single purpose

Monday, 9 February 2009

It is funny the way things work out when you're looking for help.

Recently I was working on a Ruby + FUSE based filesystem and as part of the development I added simple diagnostic output via trivial code such as this:

@debug && puts "called foo(#{param});"

That was adequate for minimal interactive use, but not so good for real live use. In real live use I started outputing messages to a dedicated logfile, but in practise became overwhelmed by thousands of lines of output describing everything ever applied to the filesystem.

I figured the natural solution was to have a ring-buffer. (Everybody knows what a ringbuffer is, right?) It could keep the last 500 messages and newer debug information would just replace older entreis. That'd be just enough to be useful if I had a problem, but not so overwhelming it would get ignored.

In Perl I found a nice ringbuffer library, but for Ruby nothing. Locking a region of shared memory via shmget, shmset and keeping an array of a few hunded strings would be simple, but it seems odd I have to code this myself.

I started searching around and I accidentally stumbled upon the unrelated IPC::DirQueue perl module. Not useful for my ringbuffer logging problem, but beautifully useful.

There is no package for Debian but that was easily created:

dh-make-perl --build --cpan IPC::DirQueue

Already I have a million and one uses for it - not least to solve my problem of maintaining a centralised quarantine for all the spam mail rejected by N MX machines. (Which currently uses a combination of rsync and lockfiles.)

This is the reason why sites like Perl Advent Calendar are useful - they introduce a useful module every day or two, and introduce you to thinks that you can use in the future.

Of course keeping a sustainable site like that up and running is hard which is why sites like debaday struggle to attract contributors, for example.

Anyway random happyness.

ObFilm: Lord of the rings: Two Towers

| 3 comments.

 

I saw green fields and flowers. I could smell the grass.

Tuesday, 20 January 2009

Fabio Tranchitella recently posted about his new filesystem which really reminded me of an outstanding problem I have.

I do some email filtering, and that is setup in a nice distributed fashion. I have a web/db machine, and then I have a number of MX machines which process incoming mail rejecting spam and queuing good mail for delivery.

I try not to talk about it very often, because that just smells of marketting. More users would be good, but I find explicit promotion & advertising distasteful. (It helps to genuinly consider users as users, and not customers even though money changes hands.)

Anyway I handle mail for just over 150 domains (some domains will receive 40,000 emails a day others will receive 10 emails a week) and each of these domains has different settings, such as "is virus scanning enabled?" and "which are the valid localparts at this domain?", then there are whitelists, blacklists, all that good stuff.

The user is encouraged to fiddle with their settings via the web/db/master machine - but ultimately any settings actually applied and used upon the MX boxes. This was initially achieved by having MySQL database slaves, but eventually I settled upon a simpler and more robust scheme: Using the filesystem. (Many reasons why, but perhaps the simplest justification is that this way things continue to work even if the master machine goes offline, or there are network routing issues. Each MX machine is essentially standalone and doesn't need to be always talking to the master host. This is good.)

On the master each domain has settings beneath /srv. Changes are applied to the files there, and to make the settings live on the slave MX boxes I can merely rsync the contents over.

Here's an anonymized example of a settings hierarchy:

/srv/foo.com/
|-- basics
|   `-- enabled
|-- dnsbl
|   |-- action
|   `-- zones
|       |-- foo.example.com
|       `-- bar.spam-house.com
|-- language
|   `-- english-only
|-- mx
|-- quarantine
|   `-- admin_._admin
|-- spam
|   |-- action
|   |-- enabled
|   `-- text
|-- spamtraps
|   |-- anonymous
|   `-- bobby
|-- uribl
|   |-- action
|   |-- enabled
|   `-- text
|-- users
|   |-- bob
|   |-- root
|   |-- simon
|   |-- smith
|   |-- steve
|   `-- wildcard
|-- virus
|   |-- action
|   |-- enabled
|   `-- text
`-- whitelisted
    |-- enabled
    |-- hosts
    |-- subjects
    |   `-- [blah]
    |-- recipients
    |   `-- simon
    `-- senders
        |-- root@steve.orgy
        |-- @someisp.com
        `-- foo@bar.com

So a user makes a change on the web machine. That updates /srv on the master machine immediately - and then every fifteen minutes, or so, the settigngs are pushed accross to the MX boxes where the incoming mail is actually processed.

Now ideally I want the updates to be applied immediately. That means I should look at using sshfs or similar. But also as a matter of policy I want to keep things reliable. If the main box dies I don't want the machines to suddenly cease working. So that rules out remotely mounting via sshfs, nfs or similar.

Thus far I've not really looked at the possabilities, but I'm leaning towards having each MX machine look for settings in two places:

  • Look for "live" copies in /srv/
  • If that isn't available then fall back to reading settings from /backup/

That way I can rsync to /backup on a fixed schedule, but expect that in everyday operation I'll get current/live settings from /srv via NFS, sshfs, or something similar.

My job for the weekend is to look around and see what filesystems are available and look at testing them.

Obmovie:Alive

| 9 comments.

 

There's something out there waiting for us, and it ain't no man.

Sunday, 11 January 2009

Things have turned a little morbid here.

I imagine that if I were to cease to be alive things would mostly keep ticking over for a while. But for how long exactly?

Assuming that you've got your hosting paid for, supported, or otherwise managed that would continue to exist. But after a while domain names would start to expire, and manual intervention would be required (that is assuming that manual intervention were not required in advance.)

So when I die, I'd have to assume everything I maintained myself would disappear within two years.

Is that depressing, or realistic? I'm not sure. But definitely morbid.

ObFilm: Predator

| 10 comments.

 

They look like big, good, strong hands, don't they?

Monday, 22 December 2008

Russ Allbery recently commented that it is really nice to receive patches for trivial scripts posted online.

I agree.

More than once I've posted a trivial script and had it be improved by people, or later included elsewhere.

So in the spirit of sharing here is my latest toy script:

This is a trivial script which searches a Maildir hierarchy and outputs a list of each email address which you've ever sent mail to.

Why would you want that? In my case my (personal) spam filtering makes use of whitelisting, and the assumption is that if I've ever mailed you in the past then I want to see your replies, and you get a break.

These days my (personal) mail filtering has a couple of broad rules:

  • If your mail is HTML it is junk. Unless I'm bored.
  • If your mail is GPG signed/encrypted I will see it.
  • If your mail address is on my whitelist then I want to see it.

After that then I see your message only if CRM119 decides I should.

#
# remove potentially spoofed header
#
:0 fhw
* ^X-whitelist:
| $FORMAIL -I "X-whitelist"

#
#  GPG-signed messages are OK and will be whitelisted
#
:0fW
* < 1024000
|/home/steve/bin/isgpged

:0e
| $FORMAIL -A "X-whitelist: yes" -A "X-GPG-Signed: Yes"

#
#  Get the sender of the message.
#
FROM=`formail -x From:| sed 's/^\([^@]*[ <]\)//' | sed 's/\([ >]\).*$//'`

#
# Add a whitelist tag if appropriate
#
:0 fhw
* !^X-whitelist: yes
* ? test -s $HOME/.procmail_whitelist
* ? echo $FROM| fgrep -qisf $HOME/.procmail_whitelist
| $FORMAIL -A "X-whitelist: yes" -A "X-Whitelist-Test: $FROM"

The net result of these tests is that I can now run the spam filter on non-whitelisted mails:

#
# Run CRM114 mailreaver
#
:0fw: .msgid.lock
* !^X-whitelist: yes
| /usr/bin/crm -u /home/steve/.crm /usr/share/crm114/mailreaver.crm

#
#  Spam.
#
:0:
* ^X-CRM114-Status: SPAM.*
* !^X-whitelist: yes
.CRM.Spam/

#
#  Unsure.
#
:0
* ^X-CRM114-Status:.*UNSURE
* !^X-whitelist: yes
.CRM.Unsure/

There is more to my setup than that, but that's the minimum you'd need to see.

Of course this is a reminder, once more, that the kind of filtering that you carry out for yourself is different from that that other people will do.

ObFilm: The NeverEnding Story

| 3 comments.

 

They look uncannily like something you should be very, very afraid of

Thursday, 4 December 2008

"I've got chills
They're multiplying
"

I guess technically I could have used that as a subject, but ugh.

If ever you're a bit shiverry, and a bit unwell, don't shave your head. It'll take three times as long, and you'll cut yourself.

ObQuote: Red Dwarf.

| 2 comments.

 

Have you been following that man?

Thursday, 27 November 2008

meta-hacking

I've had a lot of fun over the past few years detecting and fixing XSS attacks - a few months ago compromising several thousand user-accounts belonging to a particular niche social networking site and then more recently experimenting with XSS issues upon a popular software developer's advocate blog.

One thing I've been wondering about recently is meta-XSS attacks.

Consider the LKML (linux kernel mailing list). This list receives lots of long patches, submitted by email, which are copied verbatum to various sites. For example if I mailed an interesting patch to LKML chances are it would get posted to:

(Obviously the challenge here is to make a patch sufficiently interesting that it received more than usual coverage.)

Do each of those sites HTML-encode patches? In general they do, certainly the ones I looked at had code like this:

#include &lt;linux.h&gt;
...
...

But I'm certain that not all sites do so. I'm also pretty sure there are interesting avenues to explore here, and the general idea of indirectly attacking a specific target is ripe for exploration.

Anyway I'm probably not the person to go playing in the field these days; I don't have the time. But it is certainly interesting to think about.

ObFilm: Dirty Harry

| No comments

 

It's in your nature to destroy yourselves.

Thursday, 6 November 2008

Elections

I've said this elsewhere, but it bears repeating:

Anybody who expects a nation to turnaround overnight, due to a changing government, hasn't watched/read enough documentaries.

Television

Who is going to make documentaries when David Attenborough dies?

ObFilm: Terminator 2

| 1 comment.

 

Then don't knock it, it's got it's own key.

Monday, 13 October 2008

ObRandom:

Any blog post, comment thread, question, or email which starts "Hi guys" is bad, wrong, and probably not worth reading.

ObTitle: Dawn of the dead - the original and best version - 1978

| 4 comments.

 

So cunning you could brush your teeth with it.

Wednesday, 8 October 2008

Lets take a look at a new tool available to Lenny & Sid:

apt-get source acon
int main(int argc,char **argv)
{
        int i,tty,useunicode=0;
        char *fontf=0,*translationf=0,*keymapf=0;

        get_ids();
        set_user_id();

        /*Read configure file if no input options*/
        if(argc<2)
        {
                char *env;
                FILE *fp;
                char font[300],translation[300],keymap[300];
                char tmp[300];

                font[0]=translation[0]=keymap[0]=0;
                if((env=getenv("HOME")))
                        sprintf(tmp,"%s/.acon.conf",env);
                else
                        strcpy(tmp,"/etc/acon.conf");

Hmmm. Nice use of the environment there. I wonder what permissions the binary has:

skx@gold:~$ ls -l /usr/bin/acon
-rwsr-xr-x 1 root root 48672 2008-06-09 10:50 /usr/bin/acon

setuid(0) - just say no.

ObTitle: Blackadder II

| 13 comments.

 

You like playing rough, huh?

Monday, 29 September 2008

According to my small business advisor it is possible to advertise your company, service, or product on the internet.

Who knows what gem of advice they'll offer next?

In unrelated news all mail delivered to me personally in HTML-only format(s) will be dropped. I've given up being patient.

Finally OpenID - what a pain it is to implement! I've fought with it over the weekend, in amongst rewiring my lighting. Setting up a Perl script to authenticate to an OpenID server is just gnarly. (I now have motion-sensitive lighting in my bathroom, which my kitten loves, and radio controlled lighting in the bedroom. Lazyness is ..)

ObQuote: Resident Evil Extinction

| 1 comment.

 

Hey, Ash, where are we?

Wednesday, 17 September 2008

I'm currently fighting with MySQL. The following takes too long:

mysql> SELECT COUNT(id) FROM q_archive;
+-----------+
| COUNT(id) |
+-----------+
|   2738048 |
+-----------+
1 row in set (17.95 sec)

I would like it to take significantly less time, even with memcached being in use it gets hit too often. I've added an index to the table - but I didn't expect that to help, and I wasn't disappointed.

Ho hum.

Maybe another case where flat-files are best. Sure counting them would take a while, but once I've counted them I can just `cat sum`.

This is probably a case where tweaking memory of MySQL would help. But I'm fairly certain if I start messing with that I'll get into trouble with other parts of my site.

ObFilm: The Evil Dead

| 16 comments.

 

Five grand a head

Tuesday, 5 August 2008

It is nice when you work for a company where you can say:

"Ice-lolly break..."

The response?

"Me too!"

Tonight has been a productive evening, I guess the ice-lolly helped!

I managed to optimize the storage of rejected SPAM mail for my commercial service. That is something I've been obsessing over recently since the volume of SPAM is currently hovering around 2.5 million messages.

Still I suspect it is only a matter of weeks before I need to expand. The current setup has me using three machines:

  • Primary machine runs:
    • Web Application
    • SMTP processing/filtering/delivery
  • Secondary machine runs:
    • SMTP processing/filtering/delivery
  • Offsite machine:

Ideally I'd like to split that up further so that I have a single machine running the web application (the part the user interacts with), a pair of MX machines, and the offsite machine doing the minimal work it does.

That way the incoming mail will not affect the application at all directly.

Thankfully the split should be trivial. The only hard part is finding a fast webhost that can offer me ~1Gb of RAM, ~1000Gb of disk space, and won't charge much. Ideally around £15/$30 a month. (hahaha! hahaha! ha!)

ObQuote: Léon

| 4 comments.

 

I spent my life trying not to be careless

Monday, 28 July 2008

There should be a word for those silly little ways you can fool your body & brain. For example recently I've been having trouble with my boiler - so getting hot water is a challenge.

I find myself doing the crazy thing:

  • Turn on hot tap(s)
  • Stick my hands under them to see if the water is hot.
  • Think to myself "Hey it is getting warmer..".
  • Realise actually I just imagined it.

Lather, rinse, repeat.

Similarly there are times when you can imaging all kinds of bodily sensations. More than once I've been walking out, or sat at home, convinced that my mobile phone vibrated in my pocket. And it hadn't at all.

I remember, random, conversations with people who agreed they sometimes believe their phones are vibrating when they are not. Seems to be a common thing.

Which begs the question, is this a modern thing? Ten years ago if you had something vibrating against your body you damn well knew about it ... because you were doing it deliberately!

It is only recently that it was possible to have something semi-randomly vibrating against you, without your explicit control. Right?

(OK that sounds rude. It'll be our little secret.)

ObQuote: Godfather (Pt.1)

| 4 comments.

 

Alcohol's illegal this month

Saturday, 31 May 2008

Busy times, despite being on holiday.

Mostly this has been doing "business" work, and fiddling with self-promotion. But despite this I managed to find time to write some extremely useful new Lisp:

Anyway very little time over the coming week will be spent online. All being well. Still enjoying playing with my (loaned) Nokia 770 - maybe I'll get another one of my own eventually.

ObQuote: 30 Days Of Night

| No comments

 

I'm the only one qualified to remote-pilot the ship anyway.

Friday, 11 April 2008

http://10.print.debian.rocks.twentygototen.org/

ObQuote: Aliens

| No comments

 

This is the Voice of Doom calling

Tuesday, 25 March 2008

My biscuits keep breaking up and falling into my coffee.

Help!

ObQuote: The Philadelphia Story

| 1 comment.

 

I wish I could tie you up in chains

Sunday, 10 February 2008

Today I've been mostly unwell. Although I have managed to write some minor new code, and watch a little bit of Doctor Who on DVD.

Recently several people have been ranting about Ruby on Rails. I like it, but I wouldn't use it for personal development in a hurry. Deployment is fiddly, and upgrades are annoying.

But one thing that I utterly condemn Rails for is helping to spread bad paging throughout the online world.

So, what is "bad paging" and why is it important? Well cool URLs don't change, right? "Bad paging" is any user-interface which presents you with a limited view upon a changing list of items which is non-bookmarkable.

Consider the following "list". Assume it represents your view of a collection of items numbering 100+. You may only view ten items at a time; clicking "next", or "previous", to navigate your viewport:

1.  first item
2.  second item
..
10. tenth item

[see next: /start/1] [see prev]

Whats wrong with this picture? It is subtle, but this list is broken. The issue is that when the list grows new items are prepended to the front, yet the navigation is linked to the starting page number.

If that description wasn't clear consider what happens if you want to bookmark the page containing item 11. How can you?

Right now it is at /start/1. If a ten new items are appended to the head of the list then it will instead become /start/2 - as items that are currently numbered 1-10 will be shifted forward to become items 11-20, and and they will be on page /start/1 instead.

The solution is simple enough once you consider what you want to happen:

  • Either append items to the end of the list.
    • Such that /start/1 always gives the items 11-20.
  • Number the links in the reverse order.

So why does nobody do that? (As a counter example look at my website: Rather than the 'Show previous' items linking you to the changing link /start/1, it instead links you to /start/569 (for example).

| 2 comments.

 

Never Say Goodbye

Thursday, 31 January 2008

If you try using some of my software, or any software come to think of it, and it doesn't work, or causes you problems then there is a simple solution.

Tell me. I might not be able to fix it immediately, I might not ever be able to fix it. But chances are I can, and if there is a record it'll help others out in the future regardless.

I've bumped into this in the past "Oh yes I tried to use that tool you wrote but it didn't work, so I ended up with something else."

| 2 comments.

 

There's a hell of a lot more to me

Sunday, 27 January 2008

This weekend has been an interesting mix of activities. Mostly I've been tweaking my mail filtering service now that it has more users it is more interesting to do that.

The basic process of mail-scanning is pretty simple, but there are some fun things in the mix which make it slightly more fiddly than I'd like.

The basic recipe goes something like this:

  • Accept mail.
  • Validate the mail is addressed to a domain hosted upon the machine.
  • Do the spam filtering / magic (many steps missing here)
    • If the mail should be rejected archive it to a local Maildir folder and bounce it.
    • If the mail should be accepted then forward it to the destination machine.

The archiving of all rejected messages is a big win. It means that if there is a mistake in the handling of any mail we could undo it, retraining the spam database etc. It also provides, via a web page/rss feed, a way for a user to see what a good job the filtering system is doing - by saying "Here's what you would have had ..".

Today I switched the way that the archived mail is displayed via the Web GUI. Previously I used some nasty Maildir parsing code, but now I'm running IMAP upon localhost - so the viewing of messages is a lot more straightforward. (via Net::IMAP::Simple.)

More interestingly, to most readers I'm sure, today I managed to take a new Kite out for flying. A cold and windy day, but lots of fun. There was beer, pies, and near-death!

This was also the second weekend I carried out some painting of my front room. At this rate I'll have painted all four walls of the room in less than two months! (The last time I painted a room it took approximately six months to complete. Move furnuture & paint one wall. Wait several weeks, then repeat until all walls are complete!)

| 5 comments.

 

You can't hide the knives

Thursday, 6 December 2007

After recently intending to drop the Planet Debian search and recieving complaints that it was/is still useful it looks like there is a good solution.

The code will be made live and official upon the planet debian in the near future.

The DSA team promptly installed the SQLite3 package for me, and I've ported the code to work with it. Once Apache us updated to allow me to execute CGI scripts it'll be moved over, and I'll export the current data to the new database.

In other news I'm going to file an ITP bug against asql as I find myself using it more and more...

| No comments

 

don't go breaking my heart

Tuesday, 4 December 2007

If you're interested in working upon your CV/Resume, as Otavio Salvador was recently, then I'd highly recommend the xml-resume-library.

It allows you to write your address, previous jobs, and skills as XML then generate PDF, HTML, and plain text format documents via a simple Makefile.

It won't help with clueless agencies that mandate the use of Microsoft Word Documents for submission, so they can butcher your submission and "earn" their fee(s), but otherwise it rocks.

| No comments

 

Let the bells ring out for Christmas

Sunday, 2 December 2007

In the next week I intend to drop the search engine which archives content posted to Planet Debian.

It appears to have very little use, except for myself, and I'm significantly better at bookmarking posts of interest these days.

If you'd like to run your own copy the code is available and pretty trivial to reimplement regardless. There are only two parts:

  • Poll and archive content from the planet RSS feed - taking care of duplicates.
  • Scanning for /robots.txt upon the source-host, to avoid archiving content which should be "private".

Once you've done that you'll have a database populated with blog entries, and you just need to write a little search script.

ObRandom: In the time it has been running it has archived 15,464 posts!

| No comments

 

We are the champions my friend

Friday, 30 November 2007

My tool to query apache logfiles via SQL seems suprisingly popular.

Just as a recap the process goes like this:

  • Start the shell.
  • A temporary SQLite database is created.
  • You load any number of apache logfiles into it.
  • Then queries may be executed against those records until you exit.
  • The temporary database is dropped.

Now it is possible to save and load the SQLite database, so that you don't need to reparse the apache logs each time, that gives a nice speed increase for non-changing files.

By tonight I'll have aliases working for queries so you can bookmark them:

alias refers SELECT distinct(referer) FROM logs

Then in the future the 'refers' command will be available and will run the named query. Neat.

Now that I'm comfortable with SQL queries it just seems so natural, easy, and right to query logfiles this way. I guess that makes me strange.

| No comments

 

You're not going to end up like your mum and dad

Sunday, 18 November 2007

I've been working on updating my online film list since Thursday evening.

I have some code which will convert static data-files containing film entries into a browsable HTML site.

The next job is to actually go through all our DVDs and make sure the lists are correct.

I've updated all our TV shows, and I've made an initial pass at making sure all our films are present but it'll take me a few more days to ensure the lists are completely correct.

In the past I used to browse my list of films via my mobile phone to make sure I didn't buy duplicate films (more than once in the past I had managed to do that!) These days I don't seem to need to, but it is nice for organizing and it appeals to my love of lists..

I'm not sure which is worse, me doing it or Megan taking one look and saying "That's so cool!".

| No comments

 

Since you've been gone

Saturday, 10 November 2007

Confessor - Terry Goodkind's last novel in the Sword of Truth series.

Brilliant.

Exceptionally Brilliant.

Well worth waiting for, and the annoyance of 'Chainfire' itself which seemed to go nowhere despite its length.

| No comments

 

walking on the moon

Friday, 10 August 2007

According to popcon I have just under 1000 users of xen-tools.

That was quite a suprise to discover via a random google search, although I guess there have been a lot of bugs filed against the package during its lifetime.

Funny how some things which start as random hacks (this was originally a quick and dirty hack for a Xen introduction article) become quite useful/popular, whereas other tools which were planned and designed go virtually unnoticed...

| No comments

 

Recent Posts

Recent Tags