About Archive Tags RSS Feed

 

The doctors say you're going to live, that's the bad news.

11 June 2009 21:50

It is annoying that some protocols and systems are more complex than you might expect them to be.

Jabber is a protocol that is notionally simple: XML Messages pass back and forth between server(s) and client(s). But if you look at the contents of XML which is passed around you'll soon discover that even logging in is a complex operation and that Jabber is not implemented in a pleasant fashion.

By contrast many other protocols are lovely. I'm sure I'm not alone in using and debugging many common protocols with nothing more than telnet. SMTP, HTTP, POP3, etc, are all pretty easy to drive interactively.

I think 90% of programmers at some point in their lives implement a HTTP server. But I draw the line at that kind of thing these days, client-side applications are useful and simple enough with the right libraries. (e.g. my sift client-side IMAP scripter has replaced procmail on a couple of machines. Watching to see if I get a reply from somebody specific and sending me an SMS on a match..)

But recently I've been flirting with the development of an IMAP server.

Dovecot appears to be the canonincal IMAP/POP3 server these days and it is pretty close to meeting my needs, but it isn't close enough unless I jump through and change the way my mailboxes are organised. (ie. The maildir mailboxes are arranged in such a fashion that dovecot cannot easily handle them, unless I mess about with symlink farms and make them all read-only.)

I guess in conclusion it would be nice if there were a basic IMAP server framework which you could just subclass "login" and "mailbox" sections and then instantiate.

I wrote a quick inetd-driven hack which supports only the bare essentials ("NOOP", "CAPABILITY", "LOGIN", "FETCH", "SELECT" and "LIST") That allows me to connect via IMAP in both mutt and thunderbird, view folders and download messages.

Still I'm strongly suspecting that there are better uses of my time, even if I could use it in several ways..

ObFilm: La Femme Nikita

| 9 comments

 

Comments on this entry

icon Matt Sayler at 01:09 on 12 June 2009
I've spent some time hacking IMAP code and it's not particularly pleasant.
What makes IMAP great is that you can just download certain headers and/or individual MIME-chunks of messages (contrast with POP). Unfortunately, this means that a minimally useful IMAP client needs to be able to parse all manner of shitty abominations to RFC[2]822 and RFC204x.
Other thorny issues: how do you generate persistent UUIDs for messages? How do you handle IDLE and locking of mailboxes (both SMTP and IMAP side). How many of the various login methods do you want to support? What kinds of user<->mailbox mapping do you want to support? Do you want to handle TLS yourself, or just punt to stunnel? Is Sieve important?
There is room out there for a good extensible IMAP server, but IMO Dovecot is the closest thing there is.
icon Christian at 01:53 on 12 June 2009
Could you elaborate on the maildir arrangement? I'm curious because I perceived dovecot to be quite flexible in this regard.
icon . at 06:11 on 12 June 2009
Ultra cool. I thought that the twisted (python) IMAP demo was pretty neat, but I'm a bit weary how it scales. Having something that does would be great!
icon Steve Kemp at 08:59 on 12 June 2009

Matt the situation with dovecot is a little bit complex to explain, but in short a legacy system which I sometimes help out with has mailboxes which are stored all over the place. There are two different systems being used for the archives - one maildir, and one mbox. The paths for them look like this:

# maildir
/archive/10-03-1976/example.com/bob/{new cur tmp}
# mbox /home/chris/example.com/10-03-1976/chris

The intention is that you'd login with "bob@example.com" and you'd see all the files available to you - but because they are stored beneath "/archive/$date/" it is hard to select all of them. (Ditto for ~/chris/example.com)

Were it just mbox we'd use symlink farms to present /mail/chris@example.com/box[1-N] but that fails for Maildir, due to files being renamed as they are marked read, etc.

Again you're right to draw attention to the trick parts. Parsing this is easy:

11 UID fetch 1:* (FLAGS)

Parsing, and responding correctly to, this is hard:

a0005 FETCH 1:1 (UID FLAGS INTERNALDATE RFC822.SIZE BODY.PEEK[HEADER.FIELDS (DATE FROM SUBJECT TO CC MESSAGE-ID REFERENCES CONTENT-TYPE CONTENT-DESCRIPTION IN-REPLY-TO REPLY-TO LINES LIST-POST X-LABEL)])

Still from a client point of view its lovely to be able to say "Hey server, gimme just these bits of the message.

Mostly generating UIDs isn't so hard, but locking is something I've not even attempted to handle yet, and login is just plain with stunnel being an option for TLS.

For the part-time server I'd like a reasonably "full" server. For my own needs I'd be happy with something that just viewed messages stored in mysql/on disk. e.g my spam service has a quarantine, exporting that read-only via IMAP would be a nice thing. Perhaps:

quarantine/
quarantine/example.com/
quarantine/example.com/11-06-2009/
quarantine/example.com/12-06-2009/
icon Matthew Sayler at 13:55 on 12 June 2009
I was heavily involved in running a regional ISP for many years -- and it's given me a hatred for email.
;-)
icon Matthew Sayler at 14:01 on 12 June 2009
"For the part-time server I'd like a reasonably "full" server. For my own needs I'd be happy with something that just viewed messages stored in mysql/on disk. e.g my spam service has a quarantine, exporting that read-only via IMAP would be a nice thing."
If you're OK with it being stored in mysql or postgresql, you might look into DBMail. This is the project I hacked on back in the day.. They had a pretty good IMAP implementation. At the time they were higher performing than Dovecot, though it may very well be that Dovecot has advanced significantly since then.
http://www.dbmail.org/
The authors were very helpful in debugging issues that I saw and the code was pretty decent C.

icon Steve Kemp at 14:27 on 12 June 2009

Thanks for the pointer Matthew!

Right now my mail-scanning service uses a hybrid approach with all mails stored on disk, but much of the meta-information stored in the database.

So, I have a table:

Fromfoo@bar.com
Tolocalpart
SubjectThis is a mail subject
......

The intention is that I can pull most quick searches and indexes from the database without needing to hit the disk unless the user wishes a body-text search, or to download the actual mail.

icon Dave at 20:16 on 12 June 2009
Haven't tried it myself, but Perl has a Net::IMAP::Server module that looks like it should be easy to subclass.
http://search.cpan.org/dist/Net-IMAP-Server/
icon Steve Kemp at 09:38 on 13 June 2009

Thanks for the link Dave - it seems like a nice module and a good set of abstractions.

My own attempt is here and splits things up into :

  • skximap::core - For all the input parsing.
  • skximap::local - Which is the interface for presenting messages.

The actual handling of the local-level protocol is masked in a way that I don't like, but I think that Net::IMAP::Server might be the way to go...