Entries posted in December 2008

Named after a hot dog, you poor man, you poor, poor man.

Wednesday, 31 December 2008

I've updated the blogspam site a little. Now:

Still has a sucky site / layout...

ObFilm: Ghostbusters II

| 1 comment.

 

When the light is green, the trap is clean.

Tuesday, 30 December 2008

Recently the Debian Administration website has been besieged by spammers submitting bogus comments.

I also run a couple of blogs which are constantly having random gibberish submitted to them - although it is less of a problem there because I approve them manually due to the offline nature of my blog, and the blog compiler I use.

Regardless I figured it was time to do something about it, in part because Chris Searle suggested that I should be able to query arbitrary text for spammyness due to my experience with fighting email spam. (As it turns out that is not so useful experience. Comments and emails are not the same. The same header-trickery you see in bogus email, false EHLOs, etc, are not useful concepts to carry accross.)

Anyway we were saying, I was going to do something? So I did. My solution? An XML-RPC server which you submit your comments to. This will return either:

OKThe comment is clean.
SPAM:[reason]The comment is spam, and the optional reason explains further

The server itself is very simple, and it comes with a collection of simple plugins which do the job of testing different things. More plugins will almost certainly be added over time - so far I've seen the lotsaurls plugin capturing most of the SPAM, although the DNSRBL lookup is also working nicely having captured live spam without my touching it. Huzzah. & etc..

If there is any interest feel free to let me know, although as the server is open, and the client side usage is documented I imagine other people could run with it easily enough too - avoiding me becoming a single point of failure.

For reference my initial change to hook this up live on the site was only a few lines of code - although later I did need to make some additional changes to make it cope with failures, & etc.

ObFilm: Ghostbusters

| 3 comments.

 

He said he'd think about it.

Sunday, 28 December 2008

Back a while I created an updated mutt package which builds upon the previous work with mutt-ng.

The package which I had constructed had two new features, well if you're generous. I guess there was only technically one new feature:

  • The ability to cause the sidebar to only display folders with new mail in them.
  • + Updated navigation to make next/prev work correctly with only new mail-containing folders.

I still hate quilt with a passion, but I've synced my patches with the current unstable Mutt - and there are now source & binary packages (x86 + amd64) in my apt-get repository for Etch. I can rebuild for Lenny if there is any interest. But it might take me a while to setup lenny chroots.

As a graphical example this is what you will end up with:

OK I've tweaked the settings a little, but only a little:

# set up the sidebar, default to being visible
set sidebar_width=45
set sidebar_visible=yes
set sidebar_delim='|'
color sidebar_new red default

# ctrl-n, ctrl-p to select next, prev folder
# ctrl-o to open selected folder
bind index \CJ sidebar-scroll-down
bind index \CK sidebar-scroll-up
bind index \CP sidebar-prev
bind index \CN sidebar-next
bind pager \CO sidebar-open
bind index \CO sidebar-open

# Remap bounce-message function to "B"
bind index B bounce-message

#
#  Toggle visability of the sidebar.
#
macro index b '<enter-command>toggle sidebar_visible<enter><refresh>'
macro pager b '<enter-command>toggle sidebar_visible<enter><redraw-screen>'

#
#  Show folders with new mail only
#
macro index E '<enter-command>toggle sidebar_newmail_only<enter>'
macro pager E '<enter-command>toggle sidebar_newmail_only<enter>'

ObFilm: Joan of Arc

| No comments

 

Guess he wasn't too popular at the end, huh?

Tuesday, 23 December 2008

Previously I've posted a couple of running commentries describing how I've examined and gone about fixing a couple of bugs in Debian packages I use, enjoy, or have stumbled upon by accident.

Each of these commentries has resulted in a change or two to the affected software to make the bug vanish.

Fixing the bug is usually the hard part, but obviously it isn't the only thing that you need to do. While it is fun to have a locally fixed piece of software if you don't share it then the very next release will have the same bug again - and you'll spend your life doing nothing more than fixing the same bug again and again.

Generally the way that I report my fixes is by sending email to the Debian bug tracker - because I usually only try to fix bugs that I see reported already. Specificially I tend to only care about:

  • Bugs in packages that I maintain.
  • Bugs that affect me personally. I'm selfish.
  • Bugs in packages I use, even if they don't affect me.
  • Segfaults. Because segfaults and security issues often go hand in hand.

So my starting point is generally an existing bug report, such as the last bug I attempted to fix:

This bug was pretty simple to track down, and once I had added a couple of lines to the source to fix it creating the patch and reporting it was pretty simple.

The way I generally work is to download the source tree of the Debian package to my local system and work with it in-place until I think I've fixed the issue. I generally get the current sources to a package by running:

apt-get source "package"

Once I'm done fixing I'll want to create a patch. A patch is just a simple way of saying:

  • open file foo.c
  • Change "xxx" on line 12 to be "yyy".
  • Add "blah blah" after line 25

Assuming I made my changes in the local source in /tmp/gnomad2-2.9.1 I'll move that somewhere safe, and download another copy of the unmodified source, so I can create a diff and ensure that I've got the change recorded correctly:

skx@gold:/tmp$ mv gnomad2-2.9.1 gnomad2-2.9.1.new
skx@gold:/tmp$ apt-get source gnomad2

Now I have two trees:

  • /tmp/gnomad2-2.9.1.new - My modified, fixed, and updated directory.
  • /tmp/gnomad2-2.9.1 - The current source in Debian's unstable branch.

Creating the patch just means running:

 diff --recursive \
      --ignore-space-change \
      --context \
 gnomad2-2.9.1 gnomad2-2.9.1.new/

This will output the diff to the console. You can save it to a file too by rusing tee:

 diff --recursive \
      --ignore-space-change \
      --context \
 gnomad2-2.9.1 gnomad2-2.9.1.new/  | tee /tmp/patch.diff

If the new directory isn't clean your patch will include things like:

Only in gnomad2-2.9.1.new/src: player.o
Only in gnomad2-2.9.1.new/src: playlists.o
Only in gnomad2-2.9.1.new/src: prefs.o
Only in gnomad2-2.9.1.new/src: tagfile.o
Only in gnomad2-2.9.1.new/src: util.o
Only in gnomad2-2.9.1.new/src: wavfile.o

Just edit those lines out of the diff.

So the next step is to submit that to the bug report. Simply mail "bugnumber@bugs.debian.org" (509288@bugs.debian.org in our example) including your patch in the mail. If all goes well you'll receive an auto-reply after a while.

Finally to make things neat you can manipulate the bug tracker by email by sending a mail :

To: control@bugs.debian.org
Subject: updates

tags 12344 + patch
end
stop
bored now

Here is one I sent earlier.

This will ensure that the bug is reported as having a patch present in it.

Job done.

In an ideal world the next time the package is uploaded to Debian the bug will be fixed, marked as closed, and the world will be a little happier.

In a non-ideal world your patch will sit in the bug tracker for years with no further comment. If that happens there is not too much you can do, except send reminders by email, or distract yourself with a nice curry.

Happily Debian maintainers really do seem to appreciate bug fixes, and I'd say it is rare that my fixes have been ignored. It happens, but not often enough to make me give up.

ObFilm: Ghostbusters 2

| 7 comments.

 

What can you do? Sparta will need sons.

Tuesday, 23 December 2008

Since I've ranted a little recently lets do another public bugfix. The last few times people seemed to like them, and writing things down helps me keep track of where I am, what I'm doing, and how soon it will be "beer o'clock".

So I looked over the release critical bug list, looking for things that might be easy to fix.

One bug jumped out at me:

I installed the package:

skx@gold:~$ apt-get install gnomad2
..
The following NEW packages will be installed
  gnomad2 libmtp7 libnjb5 libtagc0
..

Once I copied an .mp3 file to my home directory, with the .ogg suffix I got a segfault on startup:

skx@gold:~$ cp /home/music/Audio/RedDwarf-back-to-reality.mp3 ~/foo.ogg
skx@gold:~$ gnomad2
..
LIBMTP_Get_First_Device: No Devices Attached
PDE device NULL.
TagLib: Ogg::File::packet() -- Could not find the requested packet.
TagLib: Vorbis::File::read() - Could not find the Vorbis comment header.
[segfault]
skx@gold:~$

So I downloaded the source ("apt-get source gnomad2"), and the dependencies for rebuilding ("apt-get build-dep gnomad2"). This allowed me to rebuild it locally:

skx@gold:/tmp/gnomad2-2.9.1$ ./configure --enable-debug && make
skx@gold:/tmp/gnomad2-2.9.1$ cd src/

And now it can be run under GDB.

skx@gold:/tmp/gnomad2-2.9.1/src$ gdb ./gnomad2
GNU gdb 6.8-debian
...
(gdb) run
Starting program: /tmp/gnomad2-2.9.1/src/gnomad2
...
LIBMTP_Get_First_Device: No Devices Attached
PDE device NULL.
[New Thread 0x41dcd950 (LWP 23593)]
TagLib: Ogg::File::packet() -- Could not find the requested packet.
TagLib: Vorbis::File::read() - Could not find the Vorbis comment header.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x41dcd950 (LWP 23593)]
0x00007fc89e0340b0 in taglib_tag_artist () from /usr/lib/libtag_c.so.0
(gdb)

Interestingly the crash comes from a library, libtag_c.so.0. So either:

  • The bug is realy in the libtag module.
  • The gnomad2 package doesn't handle a failure case it should.

Time to guess which that is. Charitably we'll assume any library that segfaults will be quickly fixed, because it will have more users than a single program..

Moving on we can look at the gnomad2 source for mentions of taglib. Several mentions are found, but src/tagfile.c is clearly the correct place to look. That file contains the following code:

void
get_tag_for_file (metadata_t *meta)
{
  gchar *tmppath = filename_fromutf8(meta->path);
  TagLib_Tag *tag;
  const TagLib_AudioProperties *properties;
  TagLib_File *file = taglib_file_new(tmppath);

  if(file == NULL) {
    g_printf("could not open file %s", tmppath);
    g_free(tmppath);
    return;
  }
  g_free(tmppath);

  tag = taglib_file_tag(file);
  properties = taglib_file_audioproperties(file);

  gchar* artist = taglib_tag_artist(tag);
  ..

This looks like a great place to explore because opening a file to read the tags is most likely where the crash is going to be coming from.

Interestingly we see the code:

  • Calls taglib_file_new to get a handle of some kind relating to the file.
  • Tests for errors with that operation.
  • Calls taglib_file_tag to fetch tag information using the handle, and indirectly the file.
  • But then uses taglib_tag_artist to fetch the artist (?I guess?) without testing the handle previously obtained was valid.

Let us guess that the file opening is succeeding but that the tag structure fetched via taglib_file_tag is NULL - and this causes the crash.

A quick edit:

  g_free(tmppath);

  tag = taglib_file_tag(file);
  if(tag == NULL) {
    g_printf("tag was null");
    return;
  }

  properties = taglib_file_audioproperties(file);
  gchar* artist = taglib_tag_artist(tag);

Rebuild, and the segfault is gone. We have a winner. Now we just need to file a patch...

ObFilm: 300

| 7 comments.

 

They look like big, good, strong hands, don't they?

Monday, 22 December 2008

Russ Allbery recently commented that it is really nice to receive patches for trivial scripts posted online.

I agree.

More than once I've posted a trivial script and had it be improved by people, or later included elsewhere.

So in the spirit of sharing here is my latest toy script:

This is a trivial script which searches a Maildir hierarchy and outputs a list of each email address which you've ever sent mail to.

Why would you want that? In my case my (personal) spam filtering makes use of whitelisting, and the assumption is that if I've ever mailed you in the past then I want to see your replies, and you get a break.

These days my (personal) mail filtering has a couple of broad rules:

  • If your mail is HTML it is junk. Unless I'm bored.
  • If your mail is GPG signed/encrypted I will see it.
  • If your mail address is on my whitelist then I want to see it.

After that then I see your message only if CRM119 decides I should.

#
# remove potentially spoofed header
#
:0 fhw
* ^X-whitelist:
| $FORMAIL -I "X-whitelist"

#
#  GPG-signed messages are OK and will be whitelisted
#
:0fW
* < 1024000
|/home/steve/bin/isgpged

:0e
| $FORMAIL -A "X-whitelist: yes" -A "X-GPG-Signed: Yes"

#
#  Get the sender of the message.
#
FROM=`formail -x From:| sed 's/^\([^@]*[ <]\)//' | sed 's/\([ >]\).*$//'`

#
# Add a whitelist tag if appropriate
#
:0 fhw
* !^X-whitelist: yes
* ? test -s $HOME/.procmail_whitelist
* ? echo $FROM| fgrep -qisf $HOME/.procmail_whitelist
| $FORMAIL -A "X-whitelist: yes" -A "X-Whitelist-Test: $FROM"

The net result of these tests is that I can now run the spam filter on non-whitelisted mails:

#
# Run CRM114 mailreaver
#
:0fw: .msgid.lock
* !^X-whitelist: yes
| /usr/bin/crm -u /home/steve/.crm /usr/share/crm114/mailreaver.crm

#
#  Spam.
#
:0:
* ^X-CRM114-Status: SPAM.*
* !^X-whitelist: yes
.CRM.Spam/

#
#  Unsure.
#
:0
* ^X-CRM114-Status:.*UNSURE
* !^X-whitelist: yes
.CRM.Unsure/

There is more to my setup than that, but that's the minimum you'd need to see.

Of course this is a reminder, once more, that the kind of filtering that you carry out for yourself is different from that that other people will do.

ObFilm: The NeverEnding Story

| 3 comments.

 

She must suffer to her last breath.

Sunday, 21 December 2008

Is Debian slowly tearing itself apart, or am I just unduly dramatic?

ObFilm: Kill Bill (volume two)

| 6 comments.

 

I am not going to sit on my ass as the events that affect me unfold to determine the course of my life

Wednesday, 17 December 2008

So I finally got round to vote on the Debian Lenny Firmware issue.

I can't help thinking that most of the discussion so far has been a waste of time, on the grounds that developers either

  • Care a lot about the subject and already have an opinion.
  • Don't care and get bored by the discussion.

Either way the discussion has two, or more, sides talking past each other rather than any convergence upon a consensus. (Me? I'm in the middle. On the fence. Getting bored. Hitting "Delete thread" a lot. Who knows, maybe I could have my vote influenced, but right now I'm just not caring enough either way.)

Anyway enough on that subject. Tonight I have mostly been looking around to come up with a simple way to give an "overview" of mail traffic. Something like:

  • Most prolific poster.
  • Most popular thread.
  • etc etc.

Individually these jobs are easy. Making it look pretty is hard. (Also I'd like to have archives of the lists in question, which are searchable.)

Seems like nothing out there does everything I want.

ObFilm: Ferris Bueller's Day Off.

| 1 comment.

 

They look uncannily like something you should be very, very afraid of

Thursday, 4 December 2008

"I've got chills
They're multiplying
"

I guess technically I could have used that as a subject, but ugh.

If ever you're a bit shiverry, and a bit unwell, don't shave your head. It'll take three times as long, and you'll cut yourself.

ObQuote: Red Dwarf.

| 2 comments.

 

Recent Posts

Recent Tags