So recently I've been on-call, expected to react to events around the clock. Of course to make it more of a challenge alerts are usually raised via messages to a specific channel in slack which come from a variety of sources. Let's pretend I'm all retro/hip and I'm using IRC instead.
Knowing what I'm like I knew there was essentially zero chance a single beep on my phone, from the slack/irc app, would wake me up. So I spent a couple of hours writing a simple bot:
- Connect to the server.
- Listen for messages.
- When an alert is posted in the channel:
- Trigger a voice-call via the twilio API.
That actually worked out really, really, really well. Twilio would initiate a call to my mobile which absolutely would, could, and did wake me up. I did discover a problem pretty quickly though; too many phone-calls!
Imagine something is broken. Imagine a notice goes to your channel, and then people start replying to it:
Some Bot: Help! Stuff is broken! I'm on Fire!! :fire: :hot: :boom:
Colleague Bob: Is this real?
Colleague Ann: Can you poke Chris?
Colleage Chris: Oh dears, woe is me.
The first night I was on call I got a phone call. Then another. Then another. Even I replied to the thread/chat to say "Yeah I'm on it". So the next step was to refine my alerting:
- If there is a message in the channel
- Which is not from Bob
- Which is not from Steve
- Which is not from Ann
- Which is not from Chris
- Which doesn't contain the text "common false-positive"
- Which doesn't contain the text "backup completed"
- Then make a phone-call.
Of course the next problem was predictable enough, so the rules got refined:
- If the time is between 7PM and 7AM raise the alert.
- Unless it is the weekend in which case we alert regardless of the time of day.
So I had a growing set of rules. All encoded in my goloang notification application. I moved some of them to JSON (specificially a list of users/messages to ignore) but things like the time of day were harder to move.
I figured I shouldn't be hardwiring these things. So last night put together a simple filter-library, an evaluation engine, in golang to handle them. Now I can load a script and filter things out much more dynamically. For example assume I have the following struct:
type Message struct {
Author string
Channel string
Message string
..
}
And an instance of that struct named message, I can run a user-written script against that object:
// Create a new eval-filter
eval, er := evalfilter.New( "script goes here ..." )
// Run it against the "message" object
out, err := eval.Run( message )
The logic of reacting now goes inside that script, which is hopefully easy to read - but more importantly can be edited without recompiling the application:
//
// This is a filter script:
//
// return false means "do nothing".
// return true means initiate a phone-call.
//
//
// Ignore messages on channels that we don't care about
//
if ( Channel !~ "_alerts" ) { return false; }
//
// Ignore messages from humans who might otherwise write in our channels
// of interest.
//
if ( Sender == "USER1" ) { return false; } // Steve
if ( Sender == "USER2" ) { return true; } // Ann
if ( Sender == "USER3" ) { return false; } // Bob
//
// Is it a weekend? Always alert.
//
if ( IsWeekend() ) { return true ; }
//
// OK so it is not a weekend.
//
// We only alert if 7pm-7am
//
// The WorkingHours() function returns `true` during working hours.
//
if ( WorkingHours() ) { return false ; }
//
// OK by this point we should raise a call:
//
// * The message was NOT from a colleague we've filtered out.
// * The message is upon a channel with an `_alerts` suffix.
// * It is not currently during working hours.
// * And we already handled weekends by raising calls above.
//
return true ;
If the script returns true I initiate a phone-call. If the script returns false we ignore the message/event.
The alerting script itself is trivial, and probably non-portable, but the filtering engine is pretty neat. I can see a few more uses for it, even without it having nested blocks and a real grammar. So take a look, if you like:
Tags: evalfilter, golang, hack, slack, twilio No comments