Entries tagged maildir

For various reasons I've recently been thinking about forums.

Many technical users dislike forums, because they are things that are hard to follow. Even with RSS feeds & etc you need to keep a login and remember to return to see if your post(s) have been answered.

However non-technical users love forums, and from a community-building perspective they're very cheap and easy. Particularly if you manage to appoint moderators from within the comunity.

I currently find myself in a position where I'd kinda like to have a forum package. Something that I can integrate into existing site easily.

Unfortunately most of the "best" forum packages are PHP-based, and have their own complex login, group, and admin facilities. That makes it hard to update them to authenticate against my existing MySQL table(s). (We'll leave my PHP-allergy in the background)

So, once more, I've been contemplating the bad route; create my own forum software. I'm well aware that down that path lies badness madnesss.

Let us recap. What is a forum?

• A forum is an online site.
• With a coarse list of topics.
• Inside each topic is a list of threads.
• Each thread is comprised of a number of (threaded) messages.

Sound familiar? It should if you use email:

• ~/Maildir contains storage for a collection of mailboxes.
• Each mailbox is a course list of topic-specific discussion.
• Each topic is comprised of a number of (threaded) messages.

So, the unthinkable, could we convert (bi/uni-directinally?) from a Maildir hierarchy to an online forum?

Would that make sense? On the face of it. Yes.

There are implementation details - the forum index would be essentially a list of Maildir folders (perhaps "~/Maildir/topic1/.title" would be require to give it a pretty name).

Each thread topic would be a rendered display of the messages in the folder.

So, what are the drawbacks? Well reading Maildir folders gives us threading, and subjects, bodies, etc. But it does mean a fair bit of overhead parsing messages.

(Times like this I remember Hughe. Every time we've gotten together for beer & geekery the topic of an extensible perl-based IMAP server comes up. I'm sure it should be written ..)

I'll wrap this up now. I'm sure I've made the point. There are some details which have impact - Should the forum accept new posts online? Or only via gated email-delivery? Will it work? Should it be Maildir, or IMAP based? Still at least filtering your SPAM would be easy ;)

More questions. Some questions have no answers. Some answers we ignore because we don't like.

I need to sleep.

ObFilm: 007: Quantum of Solace

Bad film. Don't waste your pennies.

Tags: forums, maildir, perl, software which should exist | 13 comments

I made a new release of the chronicle blog compiler today, and learned to hate the freshmeat.net website a little more.

The only real change is that now each compiled blog will receive a generated sitemap.xml file containing links to every output page. This will be useful for those folk that use real titles for their posts.

Nothing too much to report upon, although I noted with interest Antti-Juhani Kaijanaho's recent forum installation.

I love the idea of having a forum be a mere wrapper around a real transport system, which supports threading natively - but as I said almost a year ago I'd have done it using Mailing lists and/or Maildir folders....

ObFilm: Brief Encounter.

Tags: forums, maildir, nntp, perl | No comments

Email. We get a lot of it. We filter out the SPAM. Then if we're well-organized we file it away into folders as it arrives. (To be fair some people use priority settings such that all mail stays in their INBOX until they're "done" with it. I've never had the patience for that kind of behaviour.)

One problem which I often encounter is wanting to have email be delivered, archived, and stored, but I don't want to read it. Yet when I see a folder in my mail client which has unread mail in it I cannot unsee.

In my case I deliver mail to folders as it arrives via either Exim's filter language, or procmail. Procmail blows goats on the whole, but it is common and available everywhere - I just don't trust my own DSL enough to rely upon it (which is a dangerous sign).

So, how do you deliver a new mail to a Maildir folder, and mark it read at the same time? Well you can be naïve and do what I did which is to invoke formail to add "Status: ro" to the header. Unfortunately that's insufficient to mark a mail as read when viewed in mutt.

When an email is stored in a Maildir folder its status is encoded in part of its filename - which is why you'll have files like:

new/1288039894.28406_3.steve.org.uk
cur/1283711971.11157_3.skx.xen-hosting.net:2,S

The latter file has been Seen. So to mark a message as not-new you need to do two things:

• Save it to ./cur/ not ./new/.
• Append the appropriate flags to the filename (generally :2,S).

I've seen some horrific shell + procmail code to do the job, but the simpler version is:

:0
*^(To:|From:).*root@
*^X-added-header:.*debian-administration.org
| ~/bin/read-to-maildir .machines.debian-administration.org/

Similarly you can use Exim's filter language to do the same job:

# Exim filter
if $h_to: contains "hostmaster@" then
    pipe "/home/skx/bin/read-to-maildir /home/steve/Maildir/.hostmaster/"
finish

Cute. Obvious too? Maybe not to me.

ObSubject: Why did you murder someone, Raymond? - In Bruges

Tags: maildir, procmail, sieve | 4 comments

Once upon a time I wrote an email client, in a combination of C++ and Lua.

Later I realized it was flawed, and because I hadn't realized that writing email clients is hard I decided to write it anew (again in C++ and Lua).

Nowadays I do realize how hard writing email clients is, so I'm not going to do that again. But still .. but still ..

I was doing some mail-searching recently and realized I wanted to write something that processed all the messages in a Maildir folder. Imagine I wanted to run:

 message-dump ~/Maildir/people-foo/ ~/Maildir/people-bar/  \
     --format '${flags} ${filename} ${subject}'

As this required access to (arbitrary) headers I had to read, parse, and process each message. It was slow, but it wasn't that slow. The second time I ran it, even after adjusting the format-string, it was nice and fast because buffer-caches rock.

Anyway after that I wanted to write a script to dump the list of folders (because I store them recursively so ls -1 ~/Maildir wasn't enough):

 maildir-dump --format '${unread}/${total} ${path}'

I guess you can see where this is going now! If you have the following three primitives, you have a mail-client (albeit read-only)

• List "folders"
• List "messages"
• List a single message.

So I hacked up a simple client that would have a sub-command for each one of these tasks. I figured somebody else could actually use that, be a little retro, be a little cool, pretend they were using MH. Of course I'd have to write something horrid as a bash-script to prove it worked - probably using dialog to drive it.

And then I got interested. The end result is a single golang binary that will either:

• List maildirs, with a cute format string.
• List messages, with a cute format string.
• List a single message, decoding the RFC2047 headers, showing text/plain, etc.
• AND ALSO USE ITSELF TO PROVIDE A GUI

And now I wonder, am I crazy? Is writing an email client hard? I can't remember

• Current user-interface:
  • https://asciinema.org/a/FXjgOsnwjVu0lB5znx8EwRVWF
• Current source code:
  • https://github.com/skx/maildir-utils

Probably best to forget the GUI exists. Probably best to keep it a couple of standalone sub-commands for "scripting email stuff".

But still .. but still ..

Tags: email, golang, maildir, maildir-utils | 4 comments

After 10+ years I'm in the process of retiring my mail-host. In the future I'll no longer be running exim4/dovecot/similar, and handling my own mail. Instead it'll all go to a (paid) Google account.

It feels like the end of an era, as it means a lot of my daily life will not be spent inside a single host no longer will I run:

ssh [email protected]

I'm still within my Gsuite trial, but I've mostly finished importing my vast mail archive, via mbsync.

The only outstanding thing I need is some scripting for the mail. Since my mail has been self-hosted I've evolved a large and complex procmail configuration file which sorted incoming messages into Maildir folders.

Having a quick look around last night I couldn't find anything similar for the brave new world of Google Mail. So I hacked up a quick script which will automatically add labels to new messages that don't have any.

Finding messages which are new/unread and which don't have labels is a matter of searching for:

is:unread -has:userlabels

From there adding labels is pretty simple, if you decide what you want. For the moment I'm keeping it simple:

• If a message comes from "Bob Smith" <[email protected]>
  • I add the label "bob.smith".
  • I add the label "example.com".

Both labels will be created if they don't already exist, and the actual coding part was pretty simple. To be more complex/flexible I would probably need to integrate a scripting language (oh, I have one of those), and let the user decide what to do for each message.

The biggest annoyance is setting up the Google project, and all the OAUTH magic. I've documented briefly what I did but I don't actually know if anybody else could run the damn thing - there's just too much "magic" involved in these APIs.

Anyway procmail-lite for gmail. Job done.

Tags: dovecot, exim, gmail, golang, gsuite, maildir, procmail | 5 comments