Entries tagged procmail

I'm back from Vienna (Ask Alfie for picture(s) of me wearing my partner's hat ..!) with a new need to update my killfile.

I use procmail to filter my mail, and my current setup looks like this:

# If the sender is in the killfile then drop the mail into killfile
:0:
# grep for it in the killfile
* ? grep -i `formail -rtzxTo:` $HOME/.procmail_killfile
/dev/null

This invokes formmail to get the sender of each incoming message and then drops the mail into /dev/null if the message is from somebody whos email address is contained in ~/.procmail_killfile - there are two problems with this approach:

• It doesn't handle killing threads, only senders. I guess using a similar approach to grep for message-id's in reference would work, but I'm having a hard time thinking it through properly.
• It doesn't cope with people who use multiple addresses. Perhaps message-id would be useful here, although again if people use multiple sending hosts then they would have different components.

Any improved recipes are greatfully received. And that's all I'm going to say about Debian communications for today.

Tags: killfile, megan, procmail, vienna | No comments

Tonight I'm going to enjoy a nice long sleep after attending The Beltane Fire Festival yesterday evening.

I did manage to sort out an SSL certificate yesterday, before I went out. A lengthier process than expected because the SSL-registrar was annoying and mailed the admin address listed in whois for my domain; rather than an address upon the domain itself.

I guess they can't be blamed for that, and the registrar did forward on the request when begged, so it wasn't the end of the world. For reference I used godaddy.com; who sold me a 3 year SSL certificate for about £25.

Today I've been mostly catatonic because I had only two hours sleep last night. But one good piece of news was receiving a (postal) mail from Runa in response to the letter I had sent her some time ago.

ObQuote: 300

Tags: mail-scanning, procmail, ssl | 1 comment

Procmail

This will be the last time I talk about this, but here's my anti-rubbish procmail filter.

It correctly copes with:

• Foreign character sets that I can't read.
• Bounces from joe-jobs.
• Malformed emails.

Obviously edit to suit your tastes. Especially with regard to character sets - it is a wide brush which is tarring a group unfairly. That said in practise it works for me.

If you want real antispam filtering then you should probably be looking at externalising it, or having a layered approach.

• Procmail snippets

ObQuote: Citizen Kane

Tags: bounces, procmail, spam | 1 comment

Russ Allbery recently commented that it is really nice to receive patches for trivial scripts posted online.

I agree.

More than once I've posted a trivial script and had it be improved by people, or later included elsewhere.

So in the spirit of sharing here is my latest toy script:

• make-whitelist

This is a trivial script which searches a Maildir hierarchy and outputs a list of each email address which you've ever sent mail to.

Why would you want that? In my case my (personal) spam filtering makes use of whitelisting, and the assumption is that if I've ever mailed you in the past then I want to see your replies, and you get a break.

These days my (personal) mail filtering has a couple of broad rules:

• If your mail is HTML it is junk. Unless I'm bored.
• If your mail is GPG signed/encrypted I will see it.
• If your mail address is on my whitelist then I want to see it.

After that then I see your message only if CRM119 decides I should.

#
# remove potentially spoofed header
#
:0 fhw
* ^X-whitelist:
| $FORMAIL -I "X-whitelist"

#
#  GPG-signed messages are OK and will be whitelisted
#
:0fW
* < 1024000
|/home/steve/bin/isgpged

:0e
| $FORMAIL -A "X-whitelist: yes" -A "X-GPG-Signed: Yes"

#
#  Get the sender of the message.
#
FROM=`formail -x From:| sed 's/^\([^@]*[ <]\)//' | sed 's/\([ >]\).*$//'`

#
# Add a whitelist tag if appropriate
#
:0 fhw
* !^X-whitelist: yes
* ? test -s $HOME/.procmail_whitelist
* ? echo $FROM| fgrep -qisf $HOME/.procmail_whitelist
| $FORMAIL -A "X-whitelist: yes" -A "X-Whitelist-Test: $FROM"

The net result of these tests is that I can now run the spam filter on non-whitelisted mails:

#
# Run CRM114 mailreaver
#
:0fw: .msgid.lock
* !^X-whitelist: yes
| /usr/bin/crm -u /home/steve/.crm /usr/share/crm114/mailreaver.crm

#
#  Spam.
#
:0:
* ^X-CRM114-Status: SPAM.*
* !^X-whitelist: yes
.CRM.Spam/

#
#  Unsure.
#
:0
* ^X-CRM114-Status:.*UNSURE
* !^X-whitelist: yes
.CRM.Unsure/

There is more to my setup than that, but that's the minimum you'd need to see.

Of course this is a reminder, once more, that the kind of filtering that you carry out for yourself is different from that that other people will do.

ObFilm: The NeverEnding Story

Tags: mybin, namecheck, procmail, random | 3 comments

Sometimes I write things that are for myself, and later decide to release on the off-chance other people might be interested.

I've hated procmail for a long time, but it is extremely flexible, and for the longest time I figured since I'd got things working the way I wanted there was little point changing.

When it comes to procmail there are few alternatives:

• Exim filters.
• Maildrop.
• Email::Filter

Unfortunately both Exim and Email::Filter suffer from a lack of "pipe" support. To be more specific Exim filters and Email::Filter allow you to pipe an incoming message to an external program - but they regard that as the end of the delivery process.

So, for example, you cannot receive a message (on STDIN), pipe it through crm114, then process that updated message. (i.e. The output of crm114).

Maildrop does allow pipes, but suffers from other problems which makes me "not like it".

My own approach is to have a simple mail-sieve command which is configured thusly:

set maildir=/home/steve/Maildir
set logfile=/home/.trash.d/mail-sieve.log

#
#  Null-senders
#
Return-Path: /<>/        save .Automated.bounces/

#
#  Spam filter
#
filter /usr/bin/crm -u /home/steve/.crm /usr/share/crm114/mailreaver.crm

#
#  Spam?
#
X-CRM114-Status: /SPAM/   save .CRM.Spam/
X-CRM114-Status: /Unsure/ save .CRM.Unsure/


#
#  People / Lists
#
From: /[email protected]/  save .people.foo/
From: /[email protected]/  save .people.bar/
..
..

#
#  Domains
#
To: /steve.org.uk$/               save .steve.org.uk/
To: /debian-administration.org$/  save .debian-administration.org.personal/

#
#  All done.
#
save .inbox.unfiled/

On the one hand this is simple, readable, and complete enough for myself. On the other hand if I were going to make it releasable I think I'd probably want to add both conditionals and the ability to match upon multiple header values.

Getting there would probably involve something like this on the ~/.mail-filter side :

if ( ( From: /[email protected] ) ||
     ( From: /[email protected] ) )
{
   save .people.example.com/
   exit
}
# ps. remind me how much I hate parsers and lexers?

That starts to look very much like Exim's filter language, at which point I think "why should I bother". Pragmatically the simplest solution would be to add a "Filter" primitive to Email::Filter - and pretend I understood the nasty "Exit" settings.

ObQuote: Andre, we don't use profanity or double negatives here at True Directions. - "But I'm a Cheerleader".

Tags: maildrop, procmail, sieve | 5 comments

Email. We get a lot of it. We filter out the SPAM. Then if we're well-organized we file it away into folders as it arrives. (To be fair some people use priority settings such that all mail stays in their INBOX until they're "done" with it. I've never had the patience for that kind of behaviour.)

One problem which I often encounter is wanting to have email be delivered, archived, and stored, but I don't want to read it. Yet when I see a folder in my mail client which has unread mail in it I cannot unsee.

In my case I deliver mail to folders as it arrives via either Exim's filter language, or procmail. Procmail blows goats on the whole, but it is common and available everywhere - I just don't trust my own DSL enough to rely upon it (which is a dangerous sign).

So, how do you deliver a new mail to a Maildir folder, and mark it read at the same time? Well you can be naïve and do what I did which is to invoke formail to add "Status: ro" to the header. Unfortunately that's insufficient to mark a mail as read when viewed in mutt.

When an email is stored in a Maildir folder its status is encoded in part of its filename - which is why you'll have files like:

new/1288039894.28406_3.steve.org.uk
cur/1283711971.11157_3.skx.xen-hosting.net:2,S

The latter file has been Seen. So to mark a message as not-new you need to do two things:

• Save it to ./cur/ not ./new/.
• Append the appropriate flags to the filename (generally :2,S).

I've seen some horrific shell + procmail code to do the job, but the simpler version is:

:0
*^(To:|From:).*root@
*^X-added-header:.*debian-administration.org
| ~/bin/read-to-maildir .machines.debian-administration.org/

Similarly you can use Exim's filter language to do the same job:

# Exim filter
if $h_to: contains "hostmaster@" then
    pipe "/home/skx/bin/read-to-maildir /home/steve/Maildir/.hostmaster/"
finish

Cute. Obvious too? Maybe not to me.

ObSubject: Why did you murder someone, Raymond? - In Bruges

Tags: maildir, procmail, sieve | 4 comments

There are some tools that we use daily, whether we realize it or not, that are unduly ugly. Over time you learn to use them and you forget just how hard they are to learn, and you take it for granted.

Today I had to guide somebody through using procmail, and I'd forgotten how annoying it is.

In brief I use procmail in three ways, each of which I had to document:

• Run a command, given a new email, and replace the original email with the output of that command.
• Run a command, silently. Just for fun.
• Match a regular expression on a header-field, and file accordingly.
  • Later extended to matching regexps on multiple headers. ("AND" + "OR" )

There are some projects that are too entrenched to ever be replaced ("make", I'm looking at you), but procmail? I reckon there's a chance a replacement would be useful, quickly.

Then again, maybe I'm biased.

Tags: make, procmail | 11 comments

This week I received a logitech squeezebox radio, which is basically an expensive toy that allows you to listen to either "internet radio", or music streamed from your own PC via a portable device that accesses the network wirelessly.

The main goal of this purchase was to allow us to listen to media stored on a local computer in the bedroom, or living-room.

The hardware scans your network looking for a media server, so the first step is to install that:

• The slimdevices wiki has decent information.
  • Or you can find the code on github.com.

The media-server has a couple of open ports; one for streaming the media, and one for a user-browsable HTML interface. Interestingly the radio-device shows up in the web-interface, so you can mess around with the currently loaded playlist from your office, while your wife is casually listening to music in the bedroom. (I'm not sure if that's a feature or not yet ;)

Although I didn't find any alternative server-implementations I did find a software-client which you can use to play music from the central server - slimp3slave - and again you can push playlists, media, etc, to this.

My impressions are pretty positive; the device was too expensive, certainly I wouldn't buy two, but it is functional. The user-interface is decent, and the software being available and open is a big win.

Downsides? No remote-control for the player, because paying an additional £70 is never going to happen, but otherwise I can't think of anything.

(Shame the squeezebox product line seems to have been cancelled (?))

Although I did start hacking a C & Lua alternative, it looks like there are enough implementations out there that I don't feel so strongly any more.

I'm working in a different way to most people, rather than sort mails at delivery time I'm going to write a trivial daemon that will just watch ~/Maildir/.Incoming, and move mails out of there. That means that no errors will cause mail to be lost at SMTP/delivery time.

I'm going to base my work on Email::Filter since it offers 90% of the primitives I want. The only missing thing is the ability to filter mails via external commands which has now been reported as a bug/omission.

Tags: audio, github, perl, procmail, random, squeezebox, streaming | 10 comments

After 10+ years I'm in the process of retiring my mail-host. In the future I'll no longer be running exim4/dovecot/similar, and handling my own mail. Instead it'll all go to a (paid) Google account.

It feels like the end of an era, as it means a lot of my daily life will not be spent inside a single host no longer will I run:

ssh [email protected]

I'm still within my Gsuite trial, but I've mostly finished importing my vast mail archive, via mbsync.

The only outstanding thing I need is some scripting for the mail. Since my mail has been self-hosted I've evolved a large and complex procmail configuration file which sorted incoming messages into Maildir folders.

Having a quick look around last night I couldn't find anything similar for the brave new world of Google Mail. So I hacked up a quick script which will automatically add labels to new messages that don't have any.

Finding messages which are new/unread and which don't have labels is a matter of searching for:

is:unread -has:userlabels

From there adding labels is pretty simple, if you decide what you want. For the moment I'm keeping it simple:

• If a message comes from "Bob Smith" <[email protected]>
  • I add the label "bob.smith".
  • I add the label "example.com".

Both labels will be created if they don't already exist, and the actual coding part was pretty simple. To be more complex/flexible I would probably need to integrate a scripting language (oh, I have one of those), and let the user decide what to do for each message.

The biggest annoyance is setting up the Google project, and all the OAUTH magic. I've documented briefly what I did but I don't actually know if anybody else could run the damn thing - there's just too much "magic" involved in these APIs.

Anyway procmail-lite for gmail. Job done.

Tags: dovecot, exim, gmail, golang, gsuite, maildir, procmail | 5 comments