Entries posted in March 2008

I've been re-reading RFC 2822 again over the weekend, for obvious reasons, and I'm amused I've not noticed this section in the past:

3.6.5. Informational fields

The informational fields are all optional. The "Keywords:" field contains a comma-separated list of one or more words or quoted-strings. The "Subject:" and "Comments:" fields are unstructured fields as defined in section 2.2.1, and therefore may contain text or folding white space.

subject  = "Subject:" unstructured CRLF

comments = "Comments:" unstructured CRLF

keywords = "Keywords:" phrase *("," phrase) CRLF

Now we all know that emails have subjects, but how many people have ever used the Keywords: header, or the Comments: one?

It'd be nice if we could use these fields in mails - I can immediately think of "keywords" as tags, and I'm sure I'm not alone.

I've looked at multiple "tags for mutt" systems, but all of them fall down for the same reason. I can add tags to a mail, and limit a folder to those mails that contain a given tag. But I cannot do that for multiple folders and that makes them useless :(

Has anybody worked on a multi-folder tag system for Mutt? If so pointers welcome. If not I'd be tempted to create one.

I guess implementation would be very simple. There are three caeses:

• Adding a tag
• Deleting a tag
• Finding all messages with agiven tag.

The first two are easy. The second could be done by writing a cronjob to scan messages for Keyword: headers, and writing a simple index. That could then be used to populate an "~/Maildir/.tag-results" folder, via hardlinks, of all matching messages.

Better yet you could pre-populate ~/Maildir/.tag-$foo containing each message with a given tag. Then theres no searching required! (Although your cronjob would need to run often enough when the tag were added to a message it would appear there within a reasonable timeframe.

Update: I've written the indexer now. It works pretty quickly after the initial run, and is quite neat! tagging messages with mutt.

Tags: mail-scanning, mutt, rfc, tags | 6 comments

So, as I previously mentioned I want to be able to tag messages in Mutt.

There exist folder-based solutions already, using the X-Label header. There doesn't appear to be any existing solution allowing you to view all messages with a given tag across mailboxes.

So I wrote a simple shell script to create virtual mailboxes, such as ~/Maildir/tags-debian for all messages with a debian tag, using hardlinks.

My conclusion is that this solution will not work properly in practise, primarily because of deficiencies in mutt.

The simple case works just fine. I add a tag to a message, and later when the indexing job runs the virtual folder is created. I can open it and work on it just fine.

So where's the problem? Well in my case I tend to tag messages with a label such as "todo". Once I've done whatever I was supposed to I can remove the tag.

Using this hardlinking scheme I cannot remove the tag(s) in the virtual folder - I have to remove it in the original message which is a real pain.

Why? Well quite simply mutt will not let me work on my virtual message without destroying the hardlink.. If I use the edit function, for example, I am presented with a copy of the mail for editing - and the hardlink is replaced when that copy is saved.

Even the edit-label patch which allows you to edit the X-Label header from within mutt ends up replacing the hardlink with a new file!

So whats the solution? Well I guess I want to be able to run an external command against a message in mutt - passing the filename of the Maildir message as an argument. That way I can edit the live file.

Right now I don't believe that is possible, but I'd love to be told different.

If anybody has any solutions of editing, or even just deleting, a header from a message within mutt - in such a way that the hardlink isn't destroyed please do let me know.

Simple reproducer:

mkdir -p ~/Maildir/.foo/cur
mkdir    ~/Maildir/.foo/new
mkdir    ~/Maildir/.foo/tmp
cp 'validmessage' ~/Maildir/
ln validmessage ~/Maildir/.foo/cur

Now edit the message - start mutt open the message in the index and press 'e' - the hardlink is now gone. Replaced by a new file with the contents, so the original mail message is unchanged.

Update: I've got an "edit-inplace" primitive working, via the very hacky header-fu patch. It is not complete, but it demonstrates that it can be done. My world is now complete.

Tags: mutt, rants, tags | 7 comments

One final post then I'm done discussing mutt. (Primarily because too many people failed to understand the problem, I guess I wasn't being as good at explaining as I could have been.)

So the goal was to be able to tag individual messages within mutt, such that a short while later a new folder would be created containing all messages of that tag, regardless of which folder the message(s) are in.

Actually finding the tags and creating the virtual folders is easy to do. I wrote a simple indexer which scans for tags in messages and creates hardlinks as necessary.

The problem with this is that many operations on those hardlinked messags would create a new copy of the message and operate on that - trashing the hardlink. For some operations that's just fine. But I did specifically want to be able to untag a tagged message and not have to hunt around for the original. (e.g. remove the "todo" tag from messages where I'd done the relevant action.)

So after a bit of trial and error I came up with a patch which allows the editing of a message in-place, in the hardlinked folder, without trashing the hardlink. This patch just invokes "$EDITINPLACE <filename>" where "filename" is the name actual file the message I'm working on is stored on disk. (i.e. ~/Maildir/.people.thewomanmeg/cur/1.2.3.txt).

By setting the EDITINPLACE variable to point to my ~/bin/editlabel script I may easily remove any tags from the hardlinked-message and have it apply to the original. Job done!

There are some limitations with this approach that are worth mentioning though:

• IMAP? Ha! Nope. We only work on Maildirs.
• The header-cache facility of Mutt breaks my "find the filename of the Maildir message" function. Not sure why, so I just disabled it.
• Many operations on the virtual mailbox will still break the symlink; because they don't edit in place.

I've documented things in a rather random fashion, and I've made a backported package of mutt with my patch available online too. Primarily so I dont lose the source like I did with my mutt-ng backport.

As many of you might not know I love steam. Steam power, specifically.

Steam engines represent one of the most interesting engineering developments in history as far as I'm concerned, (closely followed by the accurate mechanical clock).

Why?

When I was a kid I had a computer. Each part of it was labeled and described in a manual. I knew what it was. I understood how it worked. I could build one myself if I had the time and re$ources.

Nowadays? Hell no. Sure I can assemble the different components, but I can no longer visualize the whole entity, and I couldn't build one from scratch.

I'm amazed at youngsters (~18 yrs old) who can understand a modern PC. Me? I do, but only barely, and that is primarily because I've watched the transition: Hercules Graphics -> CGA -> VGA -> SVGA -> & etc.

Coming in cold? Right now? I'd be lost. I'm impressed (saddened? That this stuff isn't more accessible?) that people can pick up technology and understand it, without having watched the developments along the way. "Southbridge"? "Northbridge"? "APIC"?

It's the same with everything. Fixing a car engine? These days? With electronics? Hell no. My (non-existant) car breaks, I'd be fucked.

I can strip a simple engine. I could repair it if I had to. I could build a steam engine. I could, given a suitable template and collection of eye-glasses, build a mechanical clock. But computers? Modern Cars? Aeroplanes? More complex than a simple person like me can understand. And somehow I think that's a bad idea. It almost reminds me of the old Asimov stories where only computers could design new machines. People just didn't undestand them anymore.

So, in conclusion. I have new steam car. It works, as you can see in this adhoc video. (Randomly captured by my partner whilst I was too busy squee'ing. Excuse the quality and lack of editting. Notice too that I've removed the seats for the test drive, just to give me a safe place to pick it up without burning my fingers!)

Lets hope the rise of the machines is soon, otherwise mankind will have collectively lost all knowledge of things less advanced than hot & cold electricity and the microprocessor.

(As you can tell it is late and I've just had many beers and a beautiful curry. All is well! Thanks for reading!)

Tags: engines, mamod, steam | 2 comments

Tomorrow I turn 32, so I've got to be all mature and responsible and stuff now. Maybe.

This means no more song lyrics will be used for blog titles. Instead I shall switch to film quotes. Films are more mature than songs, right?

There are three candidates standing for the DPL this year. I'm glad I didn't stand, but I came pretty close to doing so. I just think I'd be unlikely to receive the votes, and busy enough to fail even if I did get picked.

It seems to me that every year people promise to do too many things; fixing NEW, fixing the NM process & etc..

Me? I'd have only one goal: Open up the keyring-handling. Nothing more. Nothing less.

(Sure I'd blow all the Debian money on new toys for people but that'll be our little secret ;)

Anyway since there was a bit of interest I've uploaded a new steam engine video and I've started to document some of my collection:

• My steam engine page.

More updates later. Really I need to sit down, clean my toys, and then get some good pictures taken.

Maybe next month I'll find the time. (Ha!)

Tags: birthday, dpl, mamod, steam | 3 comments

Since my last example of fixing a bug received some interesting feedback (although I notice no upload of the package in question ..) we'll have another go.

Looking over my ~/.bash_history file one command I use multiple times a day is make. Happily GNU make has at least one interesting bug open:

• #419190 - make: double free or corruption

I verified this bug by saving the Makefile in the report and running make:

skx@gold:~$ make
make: file.c:84: lookup_file: Assertion `*name != '\0'' failed.
Aborted

(OK so this isn't a segfault; but an assertion failure is just as bad. Honest!)

So I downloaded the source to make, and rebuilt it. This left me with a binary with debugging symbols. The execution was much more interesting this time round:

skx@gold:~$ ./make
*** glibc detected ***
  /home/skx/./make: double free or corruption (fasttop): 0x00000000006327b0 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b273dbdd8a8]
/lib/libc.so.6(cfree+0x76)[0x2b273dbdf9b6]
/home/skx/./make[0x4120a5]
/home/skx/./make[0x4068ee]
/home/skx/./make[0x406fb2]
...
[snip mucho texto]

And once I'd allowed core-file creation ("ulimit -c 9999999") I found I had a core file to help debugging.

Running the unstripped version under gdb showed this:

(gdb) up
#5  0x00000000004120a5 in multi_glob (chain=0x1c, size=40) at read.c:3106
3106			    free (memname);

So it seems likely that this free is causing the abort. There are two simple things to do here:

• Comment out the free() call - to see if the crash goes away (!)
• Understand the code to see why this pointer might be causing us pain.

To get started I did the first of these: Commenting out the free() call did indeed fix the problem, or at least mask it (at the cost of a memory leak):

skx@gold:~$ ./make
make: *** No rule to make target `Erreur_Lexicale.o', needed by `compilateur'.  Stop.

So, now we need to go back to read.c and see why that free was causing problems.

The function containing the free() is "multi_glob". It has scary pointer magic in it, and it took me a lot of tracing to determine the source of the bug. In short we need to change this:

free (memname);

To this:

free (memname);
memname = 0;

Otherwise the memory is freed multiple times, (once each time through the loop in that function. See the source for details).

Patch mailed.

Tags: bugs, debian, make, public bugfixing | 5 comments

So early this evening I looked for more bugs to fix in the Debian packages I use the most, instead of doing security work. (Because I'm waiting for buildds..)

Anyway I figured I'd take a peak at the mutt-patched package - since I have my own patched backport for Etch these days. I'd like to get into the habit of making sure it stays current, but honestly I'll probably forget in a few months.

One bug #464189 caught my eye. It is asking for a couple of patches, neither of which I'd heard of. One of them is obviously extremely useful though - it is designed to change the sidebar in such a way that it only displays mailboxes with unread messages in them.

I hunted high and low for the patch and had to admit defeat.

So I wrote my own, and now my backported package contains a suitable patch.

In other news I made a new release of the chronicle blog compiler, which I'm now using in a few more places.

All in all it has been a busy day and I got a fair amount of hacking done!

The other thing, that I can speak about, which I did today was package the Perl Lingua::Identify module so that my spam filtering service can offer users the opportunity to reject mails written in, say, Russian, or Italian.

I'm tempted to give them up to Sarah for adoption, as she was making noises at the weekend about joining Debian (then again so was Jacob Appelbaum). The only issue is that I think she should use the Debian Perl group to get involved, and I've not seen any sign of activity there from her since she first mentioned joining it. Sarah: Consider this a reminder!

Update: Feel free to nominate any *crash* bugs you'd like me to attempt to fix. Providing they're not in video or audio playback code! Could be a fun challenge..

ObQuote: Cruel Intentions.

Tags: mutt, mutt-patched, sidebar | 1 comment

I've uploaded a new make package to fix the memory corruption bug which I recently tracked down, with kind permission from the maintainer.

I've also been working on a Debian Planet filtering/exclusion system. I've put together a (working) online demo, and I think I could probably inject it via greasemonkey without too many problems. (I'm a little reluctant to install that addon, because I suspect the security implications are severe).

Still it was a nice hack, and actually reminds me that I like javascript these days. The demo will probably disappear in a week or two, but otherwise works as expected - just a couple of GUI issues to solve.

(As with the Debian Planet search this isn't tied to our install, and could work on any PlanetPlanet installation.)

Maybe it isn't the friendliest of ideas, but I think it is a good one regardless.

ObQuote: Last King of Scotland

Tags: make, planet-debian, planet-planet | 1 comment

Planet Debian now has Javascript to facilitate the persistent hiding of particular feeds.

The code is still in flux, but appears to work.

Comments welcome, especially from people better at Javascript than myself!

The way it works was chosen to minimise the changes requred:

• Each entry is wrapped within a new <div>
• The DIV has an ID & a CLASS attribute added.

At load-time the function hideHosts() is called. If a comma-separated cookie called "excludes" is present this is split and iterated over.

For each domain-name in the list the current document is searched for elements having a class prefixed with that hostname - if any match and they have an ID defined (regardless of what that might be) they are hidden.

And thats it.

The changes to the template were minimal; each index entry already has a "link" attribute, so I just had to add this:

 <div class="<!-- tmpl_var name='link' escape='html' -->"
         id="<!-- tmpl_var name='link' escape='html' -->" >
 ....
 </div>

Because the ID and LINK attributes are URLs there is a little mangling, and inefficiency. But I didn't want to change the core of the PlanetPlanet to define a "hostname" attribute for each feed member...

In an ideal world I'd add "class='feed, $link'" and then iterate over that at load-time to attach handlers, and an ID appropriately. But that is a little scary.. If it works in IE great. I've tested Firefox & Ephinay

ObQuote: Terminator II

Update: Moved javascript into external file, and removed the image toggling, it seemed to fail and I'm not sure why. I will investigate.

Tags: planet-debian, planet-planet | 2 comments

I've just finished reading "Don't You Have Time to Think", a collection of letters written to and from Richard P. Feynman. A birthday present from my wishlist.

Previously I've read the collection of letters to/from Tolkien. (Several times actually. Very nice collection!)

It suddenly struck me that over my lifetime I've probably written <200 letters to people.

When I was young I had a couple of pen pals, and when I was entering university I was involved with a couple of play by mail games which involved writing random letters involving strategy & etc.

Personal letters though? I've written very few, and I think they've mostly consisted of letters to my partner/partners of the time.

(For example Megan went home for a few months at the end of a university year about two months after I initially met her. So there were many letters back and forth. Recently she spent two months working in the USA; counting eggs and avoiding alligators so again there was a flurry of written letters, maybe 20 total during the duration of her trip.)

I guess that most of my (hand)written messages to people have been in the form of postcards whilst on holiday.

A long time ago I offered to mail postcards to Debian developers. I know I sent at least two, and I received at least one back - but the thing I remember most was exchanging addresses with Amayita and getting into character set issues. Her emails, containing her Spanish address, were difficult to understand as my mutt/console refused to display the foreign character set properly.

I can't recall whether she did ultimately receive a card from me, but I'm sure she'll remind me if she did.

Anyway I have no resolution, intention, or expectation that I will suddenly start writing more physical mails to people. But I think it almost counts as something we do less of these days. The telephone and internet have become the norm.

In some ways this is fantastic. In others it less good.

On the left my handwriting is so bad that maybe this isn't necessarily a problem.

ObQuote: E.T.

Tags: books, debian, letters, postcards | 5 comments

I spent a while yesterday playing with an OpenGear serial console device. It presents an interface to a number of devices via TCP.

Unfortunately it is a little flawed for our purposes. We'd like to be able to telnet directly to a serial console of a connected device, like so:

telnet $ip 2000+N

That would give us the serial console of the device in port N.

However rather than dropping us straight into the serial console it instead presents a login: + password: prompt. We'd like that removed.

The solution according to the manual is to enable the use of RFC 2217 - but that will be a pain.

Apparently downloading the CDK will allow a new firmware to be built, and I could fix this problem. Unfortunately the firmware produced by this built fails to flash - failing with an error "Error: Firmware is for a different product".

Sigh.

Still I have got a partial solution. / is mounted read-only. But /etc/config is writable and /etc/config/rc.local is executed at boot-time.

My file contains this:

#!/bin/sh
if [ ! -d /tmp/bin ]; then
  mkdir /tmp/bin
fi

# fake login which won't prompt for username / password
echo '#!/bin/sh' > /tmp/bin/login
echo 'pmshell -l $4' >> /tmp/bin/login
chmod 755 /tmp/bin/login

# fake inetd which exists with the modified PATH so our
# fake login will be invoked.
echo '#!/bin/sh' > /tmp/bin/inetd
echo 'PATH=/tmp/bin:$PATH /bin/inetd $*' >> /tmp/bin/inetd
chmod 755 /tmp/bin/inetd

# use our inetd
kill -9  $(ps -f | grep [i]net| awk '{print $1}')
/tmp/bin/inetd &

Suprisingly this solution works, but it is nasty.

If anybody has any success rebuilding firmware for a CM4148 (running stock 2.3.1u3 firmware currently) I'd appreciate pointers. Unfortunately the source contained upon the opengearweb.org website also fail to flash, when rebuilt.

ObQuote: Shooting Fish

Tags: busybox, embedded, opengear | No comments

Well this has been a busy week, and no mistake.

Still I've advocated a new individual who wishes to become a Debian Developer. I guess now I get to watch second-hand to see how long the process takes!

(I messed up though; the first sponsored upload for her has the wrong mail address. It'll get REJECTed, and then we'll try again. D'oh.)

In more optimistic news this weekend I'm going to attempt to finish painting my front room. The painting of this room was started n the 3rd of February. So we're coming close to two months. A new record!

Also this weekend I must write some letters ...

Tonight will involve some Balvenie and a copy of The Godfather (part 1).

ObQuote: Eight Legged Freaks.

Tags: debian, diy, new-maintainer | No comments

Today I spent a while fixing some more segfault bugs. I guess that this work qualifies as either fixing RC bugs, or potential security bugs.

Anyway I did an NMU of libpam-tmpdir a while back to fix all but one of the open bugs against it.

I provided a patch for #461625 yelp: segfault while loading info documentation, which fixes the symptoms of bad info-parsing, and avoids the segfault.

I also looked into the #466771 busybox cpio: double free or corruption during cpio extraction of hardlinks - but it turns out that was already fixed in Sid.

Finally I found a segfault bug open against ftp:

• ftp: GET segfaults. Buffer overflow?
• Buffer overflow: comman buffer size is not checked (Almost certainly a duplication bug.)

To reproduce this bug run:

skx@gold:~$ ftp ftp.debian.org
220 saens.debian.org FTP server (vsftpd)
Name (ftp.debian.org:skx): anonymous
331 Please specify the password
Password: foo@bar.com
ftp> cd debian/doc
250 Directory successfully changed.
ftp> get dedication-2.2.cn.txt dedication-2.2.de.txt dedication-2.2.es.txt ..
local: dedication-2.2.de.txt remote: dedication-2.2.cn.txt
Segmentation fault

You need to repeat the arguments about 50 times. But keep adding more and more copies of the three files to the line until you get the crash.

It isn't interesting as a security issue as it is client side only; but as a trivially reproducable issue it becomes fun to solve.

Core was generated by `./ftp/ftp ftp.debian.org'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002b85ad77f1cf in fwrite () from /lib/libc.so.6
(gdb) up
#1  0x0000000000408c3e in command (fmt=0x40dd15 "TYPE %s") at ftp.c:366
366		fputs("\r\n", cout);
(gdb) up
#2  0x0000000000402c3e in changetype (newtype=3, show=)
    at cmds.c:348
348			comret = command("TYPE %s", p->t_mode);
(gdb) up
#3  0x000000000040a569 in recvrequest (cmd=,
    local=0x623d10 "dedication-2.2.de.txt",
    remote=0x6238d4 "dedication-2.2.cn.txt", lmode=0x40e310 "w",
    printnames=) at ftp.c:935
935			changetype(type, 0);

char **
makeargv(int *pargc, char **parg)
{
	static char *rargv[20];
	int rargc = 0;
	char **argp;

I mailed the maintainer of FTP and said unless I heard differently I'd NMU and cleanup the package in a week.

All being well this entry will be nicely truncated in the RSS feeds as support for the <cut> tag was the main new feature in my previous upload of chronicle - the blog compiler I use/wrote/maintain.

ObQuote: Razor Blade Smile

Tags: chronicle, debian, ftp, public bugfixing | No comments

I've spent a lot of hours over the weekend stripping out markup on my personal website, so that I can regenerate it from a templating system.

My template-compiler is horribly hacky, but it produces reasonably good results from simple input files. The only motivation for this was to finally transition my site to a consistant layout, and allow me to easily change the layout again in the future.

I looked over WML, htmlpp, and a few other static web-compilers before writing my own tiny processor. I guess most of them are more powerful than mine, but similarly more complex to get started with.

I'll share the code if there is any interest. (Primarily it is template expansion of a body, optional per-page menus, and per-page HTML header inclusion.)

The only significant thing that has changed in my website is a tiny RSS feed of updates (also compiled!) and the removal of many sections from the top-level menu. (For example my body piercing section has been obsolete for years. It is still present, but I see no point in advertising it. Better resources exist..

The plus side is that I've managed to do all this whilst watching "Dead Like Me" over the Easter weekend. Both series from start to finish. Good show..

Obquote: 10 Things I Hate About You

Tags: css, templating, tv, websites | 4 comments

My biscuits keep breaking up and falling into my coffee.

Help!

ObQuote: The Philadelphia Story

Tags: random | 1 comment