Entries posted in March 2008

I don't shut up, I grow up

Tuesday, 1 April 2008

Recently I took stock of my Javascript programming efforts, with the intention of simplifying things. To date I've put together three sites which make use of Javascript:

debian-administration.org

There isn't very much javascript in use upon this site, and you might be forgiven for thinking there was none at all if you don't have an account, or don't login.

There's a tagging system which is starting to creak under the sheer number of different tags, and several back-end parts of the site make use of AJAX calls.

Most of the script lives in a single file common.js which I cobbled via a process of trial and error, augmented with a little copy & paste coding.

It works. But I knew I could do better ..

ctrl-alt-date

This was my first attempt to make a site be truely dynamic and "pretty". It has succeeded in that respect, although the lack of members makes the site itself essentially a failure.

This time round I decided that I'd utterly refuse to write my own Javascript. So instead I used script.aculo.us (damn I hate those ugly URLS).

This library made it almost too easy to add flash. I liked it a lot.

Having said that though the sheer scope of the library and the way it didn't fit in the way that I coded made it painful to use at times.

It works, and it works well. Like it? Yes. Love it no?

mail-scanning.com

This site has a fair amount of javascript upon it, but that isn't obvious unless you're actually a user. The only piece that I can recall is the real-time count of spam caught upon the front-page. (Realtime stats are cool!)

Most of the code here is the simple kind, reverting back to the way I worked on the Debian Administration site; we're talking about basic effects such as:

  • show/hide a div
  • make an AJAX request every now and again.
  • Do a bit of auto-completion.

To get more of a feel for whats out there I wrote this initially with my own code, then later migrated it to jQuery.

Quite frankly jQuery rocks. The way it works is a little strange at first, but it is so natural after a while. As an example:

// find the div called "foo" - hide it.
$("#foo").hide()

I'm liking this library a lot recently, but only time will tell if I use it more.

In conclusion I filed #473125: ITP jQuery failing to see the existing ITP already present.

Once the package makes it through the NEW queue I will update it to follow the skeletal javascript-policy.

I disagree about the naming scheme suggested by that policy primarily because we already have packages of several of the "big" javascript libraries, such as scriptalicious, and see no gain in renaming them. But otherwise the policy is sane enough; following it will cause no harm at the very least.

ObQuote: Stand By Me

| No comments

 

This is the Voice of Doom calling

Tuesday, 25 March 2008

My biscuits keep breaking up and falling into my coffee.

Help!

ObQuote: The Philadelphia Story

| 1 comment.

 

Make anyone cry today?

Sunday, 23 March 2008

I've spent a lot of hours over the weekend stripping out markup on my personal website, so that I can regenerate it from a templating system.

My template-compiler is horribly hacky, but it produces reasonably good results from simple input files. The only motivation for this was to finally transition my site to a consistant layout, and allow me to easily change the layout again in the future.

I looked over WML, htmlpp, and a few other static web-compilers before writing my own tiny processor. I guess most of them are more powerful than mine, but similarly more complex to get started with.

I'll share the code if there is any interest. (Primarily it is template expansion of a body, optional per-page menus, and per-page HTML header inclusion.)

The only significant thing that has changed in my website is a tiny RSS feed of updates (also compiled!) and the removal of many sections from the top-level menu. (For example my body piercing section has been obsolete for years. It is still present, but I see no point in advertising it. Better resources exist..

The plus side is that I've managed to do all this whilst watching "Dead Like Me" over the Easter weekend. Both series from start to finish. Good show..

Obquote: 10 Things I Hate About You

| 4 comments.

 

Don't you just hate loose ends?

Friday, 21 March 2008

Today I spent a while fixing some more segfault bugs. I guess that this work qualifies as either fixing RC bugs, or potential security bugs.

Anyway I did an NMU of libpam-tmpdir a while back to fix all but one of the open bugs against it.

I provided a patch for #461625 yelp: segfault while loading info documentation, which fixes the symptoms of bad info-parsing, and avoids the segfault.

I also looked into the #466771 busybox cpio: double free or corruption during cpio extraction of hardlinks - but it turns out that was already fixed in Sid.

Finally I found a segfault bug open against ftp:

To reproduce this bug run:

skx@gold:~$ ftp ftp.debian.org
220 saens.debian.org FTP server (vsftpd)
Name (ftp.debian.org:skx): anonymous
331 Please specify the password
Password: foo@bar.com
ftp> cd debian/doc
250 Directory successfully changed.
ftp> get dedication-2.2.cn.txt dedication-2.2.de.txt dedication-2.2.es.txt ..
local: dedication-2.2.de.txt remote: dedication-2.2.cn.txt
Segmentation fault

You need to repeat the arguments about 50 times. But keep adding more and more copies of the three files to the line until you get the crash.

It isn't interesting as a security issue as it is client side only; but as a trivially reproducable issue it becomes fun to solve.

Lets build it with debugging information, and run it again. Here is what we see:

Core was generated by `./ftp/ftp ftp.debian.org'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002b85ad77f1cf in fwrite () from /lib/libc.so.6
(gdb) up
#1  0x0000000000408c3e in command (fmt=0x40dd15 "TYPE %s") at ftp.c:366
366		fputs("\r\n", cout);
(gdb) up
#2  0x0000000000402c3e in changetype (newtype=3, show=)
    at cmds.c:348
348			comret = command("TYPE %s", p->t_mode);
(gdb) up
#3  0x000000000040a569 in recvrequest (cmd=,
    local=0x623d10 "dedication-2.2.de.txt",
    remote=0x6238d4 "dedication-2.2.cn.txt", lmode=0x40e310 "w",
    printnames=) at ftp.c:935
935			changetype(type, 0);

OK so things look trashed, and not in the middle of a copy/sprintf/similar. i.e. there is no obvious problem.

Lets take a step back from things. We know that the crash occurs when we send a long command line. Looking over the code we see the fucntion main.c:makeargv(). This attempts to split the input command line string into an array of tokens.

Interestingly we see this:

char **
makeargv(int *pargc, char **parg)
{
	static char *rargv[20];
	int rargc = 0;
	char **argp;

I wonder what happens if we set the 20 to 2048? Good guess. The crash no longer occurs. (Though I'm sure it would if you entered enough tokens...)

So we know that the crash is relating to the number of space-separated tokens upon the command line. If we increase the limit we'll be fine. But of course we want to fix it properly. There are two ways forward:

  • Abort handling a line if there are >15 "space" characters on the line.
  • Recode the makeargv function to work properly.

I did eventually submit a patch to the bug report which uses dynamic memory allocation, and should always work. Job done.

I mailed the maintainer of FTP and said unless I heard differently I'd NMU and cleanup the package in a week.

All being well this entry will be nicely truncated in the RSS feeds as support for the <cut> tag was the main new feature in my previous upload of chronicle - the blog compiler I use/wrote/maintain.

ObQuote: Razor Blade Smile

| No comments

 

Thanks for the flashback

Thursday, 20 March 2008

Well this has been a busy week, and no mistake.

Still I've advocated a new individual who wishes to become a Debian Developer. I guess now I get to watch second-hand to see how long the process takes!

(I messed up though; the first sponsored upload for her has the wrong mail address. It'll get REJECTed, and then we'll try again. D'oh.)

In more optimistic news this weekend I'm going to attempt to finish painting my front room. The painting of this room was started n the 3rd of February. So we're coming close to two months. A new record!

Also this weekend I must write some letters ...

Tonight will involve some Balvenie and a copy of The Godfather (part 1).

ObQuote: Eight Legged Freaks.

| No comments

 

I feel like Wedding-Day Barbie

Tuesday, 18 March 2008

I spent a while yesterday playing with an OpenGear serial console device. It presents an interface to a number of devices via TCP.

Unfortunately it is a little flawed for our purposes. We'd like to be able to telnet directly to a serial console of a connected device, like so:

telnet $ip 2000+N

That would give us the serial console of the device in port N.

However rather than dropping us straight into the serial console it instead presents a login: + password: prompt. We'd like that removed.

The solution according to the manual is to enable the use of RFC 2217 - but that will be a pain.

Apparently downloading the CDK will allow a new firmware to be built, and I could fix this problem. Unfortunately the firmware produced by this built fails to flash - failing with an error "Error: Firmware is for a different product".

Sigh.

Still I have got a partial solution. / is mounted read-only. But /etc/config is writable and /etc/config/rc.local is executed at boot-time.

My file contains this:

#!/bin/sh
if [ ! -d /tmp/bin ]; then
  mkdir /tmp/bin
fi

# fake login which won't prompt for username / password
echo '#!/bin/sh' > /tmp/bin/login
echo 'pmshell -l $4' >> /tmp/bin/login
chmod 755 /tmp/bin/login

# fake inetd which exists with the modified PATH so our
# fake login will be invoked.
echo '#!/bin/sh' > /tmp/bin/inetd
echo 'PATH=/tmp/bin:$PATH /bin/inetd $*' >> /tmp/bin/inetd
chmod 755 /tmp/bin/inetd

# use our inetd
kill -9  $(ps -f | grep [i]net| awk '{print $1}')
/tmp/bin/inetd &

Suprisingly this solution works, but it is nasty.

If anybody has any success rebuilding firmware for a CM4148 (running stock 2.3.1u3 firmware currently) I'd appreciate pointers. Unfortunately the source contained upon the opengearweb.org website also fail to flash, when rebuilt.

ObQuote: Shooting Fish

| No comments

 

E.T. phone home

Sunday, 16 March 2008

I've just finished reading "Don't You Have Time to Think", a collection of letters written to and from Richard P. Feynman. A birthday present from my wishlist.

Previously I've read the collection of letters to/from Tolkien. (Several times actually. Very nice collection!)

It suddenly struck me that over my lifetime I've probably written <200 letters to people.

When I was young I had a couple of pen pals, and when I was entering university I was involved with a couple of play by mail games which involved writing random letters involving strategy & etc.

Personal letters though? I've written very few, and I think they've mostly consisted of letters to my partner/partners of the time.

(For example Megan went home for a few months at the end of a university year about two months after I initially met her. So there were many letters back and forth. Recently she spent two months working in the USA; counting eggs and avoiding alligators so again there was a flurry of written letters, maybe 20 total during the duration of her trip.)

I guess that most of my (hand)written messages to people have been in the form of postcards whilst on holiday.

A long time ago I offered to mail postcards to Debian developers. I know I sent at least two, and I received at least one back - but the thing I remember most was exchanging addresses with Amayita and getting into character set issues. Her emails, containing her Spanish address, were difficult to understand as my mutt/console refused to display the foreign character set properly.

I can't recall whether she did ultimately receive a card from me, but I'm sure she'll remind me if she did.

Anyway I have no resolution, intention, or expectation that I will suddenly start writing more physical mails to people. But I think it almost counts as something we do less of these days. The telephone and internet have become the norm.

In some ways this is fantastic. In others it less good.

On the left my handwriting is so bad that maybe this isn't necessarily a problem.

ObQuote: E.T.

| 5 comments.

 

I promise that I will not kill anyone

Saturday, 15 March 2008

Planet Debian now has Javascript to facilitate the persistent hiding of particular feeds.

The code is still in flux, but appears to work.

Comments welcome, especially from people better at Javascript than myself!

The way it works was chosen to minimise the changes requred:

  • Each entry is wrapped within a new <div>
  • The DIV has an ID & a CLASS attribute added.

At load-time the function hideHosts() is called. If a comma-separated cookie called "excludes" is present this is split and iterated over.

For each domain-name in the list the current document is searched for elements having a class prefixed with that hostname - if any match and they have an ID defined (regardless of what that might be) they are hidden.

And thats it.

The changes to the template were minimal; each index entry already has a "link" attribute, so I just had to add this:

 <div class="<!-- tmpl_var name='link' escape='html' -->"
         id="<!-- tmpl_var name='link' escape='html' -->" >
 ....
 </div>

Because the ID and LINK attributes are URLs there is a little mangling, and inefficiency. But I didn't want to change the core of the PlanetPlanet to define a "hostname" attribute for each feed member...

In an ideal world I'd add "class='feed, $link'" and then iterate over that at load-time to attach handlers, and an ID appropriately. But that is a little scary.. If it works in IE great. I've tested Firefox & Ephinay

ObQuote: Terminator II

Update: Moved javascript into external file, and removed the image toggling, it seemed to fail and I'm not sure why. I will investigate.

| 2 comments.

 

Do you have monkeys in Scotland?

Thursday, 13 March 2008

I've uploaded a new make package to fix the memory corruption bug which I recently tracked down, with kind permission from the maintainer.

I've also been working on a Debian Planet filtering/exclusion system. I've put together a (working) online demo, and I think I could probably inject it via greasemonkey without too many problems. (I'm a little reluctant to install that addon, because I suspect the security implications are severe).

Still it was a nice hack, and actually reminds me that I like javascript these days. The demo will probably disappear in a week or two, but otherwise works as expected - just a couple of GUI issues to solve.

(As with the Debian Planet search this isn't tied to our install, and could work on any PlanetPlanet installation.)

Maybe it isn't the friendliest of ideas, but I think it is a good one regardless.

ObQuote: Last King of Scotland

| 1 comment.

 

Cecile, this is what I like to call "quiet time"

Tuesday, 11 March 2008

So early this evening I looked for more bugs to fix in the Debian packages I use the most, instead of doing security work. (Because I'm waiting for buildds..)

Anyway I figured I'd take a peak at the mutt-patched package - since I have my own patched backport for Etch these days. I'd like to get into the habit of making sure it stays current, but honestly I'll probably forget in a few months.

One bug #464189 caught my eye. It is asking for a couple of patches, neither of which I'd heard of. One of them is obviously extremely useful though - it is designed to change the sidebar in such a way that it only displays mailboxes with unread messages in them.

I hunted high and low for the patch and had to admit defeat.

So I wrote my own, and now my backported package contains a suitable patch.

In other news I made a new release of the chronicle blog compiler, which I'm now using in a few more places.

All in all it has been a busy day and I got a fair amount of hacking done!

The other thing, that I can speak about, which I did today was package the Perl Lingua::Identify module so that my spam filtering service can offer users the opportunity to reject mails written in, say, Russian, or Italian.

I'm tempted to give them up to Sarah for adoption, as she was making noises at the weekend about joining Debian (then again so was Jacob Appelbaum). The only issue is that I think she should use the Debian Perl group to get involved, and I've not seen any sign of activity there from her since she first mentioned joining it. Sarah: Consider this a reminder!

Update: Feel free to nominate any *crash* bugs you'd like me to attempt to fix. Providing they're not in video or audio playback code! Could be a fun challenge..

ObQuote: Cruel Intentions.

| No comments

 

Some people get by with a little understanding

Sunday, 9 March 2008

Since my last example of fixing a bug received some interesting feedback (although I notice no upload of the package in question ..) we'll have another go.

Looking over my ~/.bash_history file one command I use multiple times a day is make. Happily GNU make has at least one interesting bug open:

I verified this bug by saving the Makefile in the report and running make:

skx@gold:~$ make
make: file.c:84: lookup_file: Assertion `*name != '\0'' failed.
Aborted

(OK so this isn't a segfault; but an assertion failure is just as bad. Honest!)

So I downloaded the source to make, and rebuilt it. This left me with a binary with debugging symbols. The execution was much more interesting this time round:

skx@gold:~$ ./make
*** glibc detected ***
  /home/skx/./make: double free or corruption (fasttop): 0x00000000006327b0 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b273dbdd8a8]
/lib/libc.so.6(cfree+0x76)[0x2b273dbdf9b6]
/home/skx/./make[0x4120a5]
/home/skx/./make[0x4068ee]
/home/skx/./make[0x406fb2]
...
[snip mucho texto]

And once I'd allowed core-file creation ("ulimit -c 9999999") I found I had a core file to help debugging.

Running the unstripped version under gdb showed this:

(gdb) up
#5  0x00000000004120a5 in multi_glob (chain=0x1c, size=40) at read.c:3106
3106			    free (memname);

So it seems likely that this free is causing the abort. There are two simple things to do here:

  • Comment out the free() call - to see if the crash goes away (!)
  • Understand the code to see why this pointer might be causing us pain.

To get started I did the first of these: Commenting out the free() call did indeed fix the problem, or at least mask it (at the cost of a memory leak):

skx@gold:~$ ./make
make: *** No rule to make target `Erreur_Lexicale.o', needed by `compilateur'.  Stop.

So, now we need to go back to read.c and see why that free was causing problems.

The function containing the free() is "multi_glob". It has scary pointer magic in it, and it took me a lot of tracing to determine the source of the bug. In short we need to change this:

free (memname);

To this:

free (memname);
memname = 0;

Otherwise the memory is freed multiple times, (once each time through the loop in that function. See the source for details).

Patch mailed.

| 5 comments.

 

Book Us A Ticket On The Next Space Shuttle

Sunday, 9 March 2008

Tomorrow I turn 32, so I've got to be all mature and responsible and stuff now. Maybe.

This means no more song lyrics will be used for blog titles. Instead I shall switch to film quotes. Films are more mature than songs, right?

There are three candidates standing for the DPL this year. I'm glad I didn't stand, but I came pretty close to doing so. I just think I'd be unlikely to receive the votes, and busy enough to fail even if I did get picked.

It seems to me that every year people promise to do too many things; fixing NEW, fixing the NM process & etc..

Me? I'd have only one goal: Open up the keyring-handling. Nothing more. Nothing less.

(Sure I'd blow all the Debian money on new toys for people but that'll be our little secret ;)

Anyway since there was a bit of interest I've uploaded a new steam engine video and I've started to document some of my collection:

More updates later. Really I need to sit down, clean my toys, and then get some good pictures taken.

Maybe next month I'll find the time. (Ha!)

| 3 comments.

 

One of the four beasts sang "come and see".

Wednesday, 5 March 2008

As many of you might not know I love steam. Steam power, specifically.

Steam engines represent one of the most interesting engineering developments in history as far as I'm concerned, (closely followed by the accurate mechanical clock).

Why?

When I was a kid I had a computer. Each part of it was labeled and described in a manual. I knew what it was. I understood how it worked. I could build one myself if I had the time and re$ources.

Nowadays? Hell no. Sure I can assemble the different components, but I can no longer visualize the whole entity, and I couldn't build one from scratch.

I'm amazed at youngsters (~18 yrs old) who can understand a modern PC. Me? I do, but only barely, and that is primarily because I've watched the transition: Hercules Graphics -> CGA -> VGA -> SVGA -> & etc.

Coming in cold? Right now? I'd be lost. I'm impressed (saddened? That this stuff isn't more accessible?) that people can pick up technology and understand it, without having watched the developments along the way. "Southbridge"? "Northbridge"? "APIC"?

It's the same with everything. Fixing a car engine? These days? With electronics? Hell no. My (non-existant) car breaks, I'd be fucked.

I can strip a simple engine. I could repair it if I had to. I could build a steam engine. I could, given a suitable template and collection of eye-glasses, build a mechanical clock. But computers? Modern Cars? Aeroplanes? More complex than a simple person like me can understand. And somehow I think that's a bad idea. It almost reminds me of the old Asimov stories where only computers could design new machines. People just didn't undestand them anymore.

 

So, in conclusion. I have new steam car. It works, as you can see in this adhoc video. (Randomly captured by my partner whilst I was too busy squee'ing. Excuse the quality and lack of editting. Notice too that I've removed the seats for the test drive, just to give me a safe place to pick it up without burning my fingers!)

Lets hope the rise of the machines is soon, otherwise mankind will have collectively lost all knowledge of things less advanced than hot & cold electricity and the microprocessor.

(As you can tell it is late and I've just had many beers and a beautiful curry. All is well! Thanks for reading!)

| No comments

 

Feeling with your skin

Monday, 3 March 2008

One final post then I'm done discussing mutt. (Primarily because too many people failed to understand the problem, I guess I wasn't being as good at explaining as I could have been.)

So the goal was to be able to tag individual messages within mutt, such that a short while later a new folder would be created containing all messages of that tag, regardless of which folder the message(s) are in.

Actually finding the tags and creating the virtual folders is easy to do. I wrote a simple indexer which scans for tags in messages and creates hardlinks as necessary.

The problem with this is that many operations on those hardlinked messags would create a new copy of the message and operate on that - trashing the hardlink. For some operations that's just fine. But I did specifically want to be able to untag a tagged message and not have to hunt around for the original. (e.g. remove the "todo" tag from messages where I'd done the relevant action.)

So after a bit of trial and error I came up with a patch which allows the editing of a message in-place, in the hardlinked folder, without trashing the hardlink. This patch just invokes "$EDITINPLACE <filename>" where "filename" is the name actual file the message I'm working on is stored on disk. (i.e. ~/Maildir/.people.thewomanmeg/cur/1.2.3.txt).

By setting the EDITINPLACE variable to point to my ~/bin/editlabel script I may easily remove any tags from the hardlinked-message and have it apply to the original. Job done!

There are some limitations with this approach that are worth mentioning though:

  • IMAP? Ha! Nope. We only work on Maildirs.
  • The header-cache facility of Mutt breaks my "find the filename of the Maildir message" function. Not sure why, so I just disabled it.
  • Many operations on the virtual mailbox will still break the symlink; because they don't edit in place.

I've documented things in a rather random fashion, and I've made a backported package of mutt with my patch available online too. Primarily so I dont lose the source like I did with my mutt-ng backport.

Pointless Work

I spent about 30 minutes rebuilding mutt to use a bubble-sort on the folder list which is displayed in the sidebar.

Only when I was just about to install this patched build did I realise that the mailboxes were displayed in the order they are listed in my .muttrc file. One quick edit with Emacs and the folder list was sorted properly.

Boy is my face red..

ObRandom: I'm loving my deaf-friendly alarm clock. It rocks.

Place it under your pillow and it vibrates to wake you up. Simple. Effective, and above all it doesn't disturb my partner up when I get up to catch an early morning train.

(Thought to be fair I usually wake her up anyway; can't leave without a goodbye kiss!)

I'm still waiting for the vibrating ring, but this is good enough in the meantime.

| 2 comments.

 

And time keeps dragging on

Sunday, 2 March 2008

So, as I previously mentioned I want to be able to tag messages in Mutt.

There exist folder-based solutions already, using the X-Label header. There doesn't appear to be any existing solution allowing you to view all messages with a given tag across mailboxes.

So I wrote a simple shell script to create virtual mailboxes, such as ~/Maildir/tags-debian for all messages with a debian tag, using hardlinks.

My conclusion is that this solution will not work properly in practise, primarily because of deficiencies in mutt.

The simple case works just fine. I add a tag to a message, and later when the indexing job runs the virtual folder is created. I can open it and work on it just fine.

So where's the problem? Well in my case I tend to tag messages with a label such as "todo". Once I've done whatever I was supposed to I can remove the tag.

Using this hardlinking scheme I cannot remove the tag(s) in the virtual folder - I have to remove it in the original message which is a real pain.

Why? Well quite simply mutt will not let me work on my virtual message without destroying the hardlink.. If I use the edit function, for example, I am presented with a copy of the mail for editing - and the hardlink is replaced when that copy is saved.

Even the edit-label patch which allows you to edit the X-Label header from within mutt ends up replacing the hardlink with a new file!

So whats the solution? Well I guess I want to be able to run an external command against a message in mutt - passing the filename of the Maildir message as an argument. That way I can edit the live file.

Right now I don't believe that is possible, but I'd love to be told different.

If anybody has any solutions of editing, or even just deleting, a header from a message within mutt - in such a way that the hardlink isn't destroyed please do let me know.

Simple reproducer:

mkdir -p ~/Maildir/.foo/cur
mkdir    ~/Maildir/.foo/new
mkdir    ~/Maildir/.foo/tmp
cp 'validmessage' ~/Maildir/
ln validmessage ~/Maildir/.foo/cur

Now edit the message - start mutt open the message in the index and press 'e' - the hardlink is now gone. Replaced by a new file with the contents, so the original mail message is unchanged.

Update: I've got an "edit-inplace" primitive working, via the very hacky header-fu patch. It is not complete, but it demonstrates that it can be done. My world is now complete.

| 7 comments.

 

Recent Posts

Recent Tags