So recently I've been on-call, expected to react to events around the clock. Of course to make it more of a challenge alerts are usually raised via messages to a specific channel in slack which come from a variety of sources. Let's pretend I'm all retro/hip and I'm using IRC instead.

Knowing what I'm like I knew there was essentially zero chance a single beep on my phone, from the slack/irc app, would wake me up. So I spent a couple of hours writing a simple bot:

• Connect to the server.
• Listen for messages.
• When an alert is posted in the channel:
  • Trigger a voice-call via the twilio API.

That actually worked out really, really, really well. Twilio would initiate a call to my mobile which absolutely would, could, and did wake me up. I did discover a problem pretty quickly though; too many phone-calls!

Imagine something is broken. Imagine a notice goes to your channel, and then people start replying to it:

  Some Bot: Help! Stuff is broken!  I'm on Fire!!  :fire: :hot: :boom:
  Colleague Bob: Is this real?
  Colleague Ann: Can you poke Chris?
  Colleage Chris: Oh dears, woe is me.

The first night I was on call I got a phone call. Then another. Then another. Even I replied to the thread/chat to say "Yeah I'm on it". So the next step was to refine my alerting:

• If there is a message in the channel
  • Which is not from Bob
  • Which is not from Steve
  • Which is not from Ann
  • Which is not from Chris
  • Which doesn't contain the text "common false-positive"
  • Which doesn't contain the text "backup completed"
• Then make a phone-call.

Of course the next problem was predictable enough, so the rules got refined:

• If the time is between 7PM and 7AM raise the alert.
• Unless it is the weekend in which case we alert regardless of the time of day.

So I had a growing set of rules. All encoded in my goloang notification application. I moved some of them to JSON (specificially a list of users/messages to ignore) but things like the time of day were harder to move.

I figured I shouldn't be hardwiring these things. So last night put together a simple filter-library, an evaluation engine, in golang to handle them. Now I can load a script and filter things out much more dynamically. For example assume I have the following struct:

type Message struct {
    Author  string
    Channel string
    Message string
    ..
}

And an instance of that struct named message, I can run a user-written script against that object:

 // Create a new eval-filter
 eval, er := evalfilter.New( "script goes here ..." )

 // Run it against the "message" object
 out, err := eval.Run( message )

The logic of reacting now goes inside that script, which is hopefully easy to read - but more importantly can be edited without recompiling the application:

//
// This is a filter script:
//
//   return false means "do nothing".
//   return true means initiate a phone-call.
//

//
// Ignore messages on channels that we don't care about
//
if ( Channel !~ "_alerts" ) { return false; }

//
// Ignore messages from humans who might otherwise write in our channels
// of interest.
//
if ( Sender == "USER1" ) { return false; }   // Steve
if ( Sender == "USER2" ) { return true; }    // Ann
if ( Sender == "USER3" ) { return false; }   // Bob


//
// Is it a weekend? Always alert.
//
if ( IsWeekend() ) { return true ; }

//
// OK so it is not a weekend.
//
// We only alert if 7pm-7am
//
// The WorkingHours() function returns `true` during working hours.
//
if ( WorkingHours() ) { return false ; }

//
// OK by this point we should raise a call:
//
// * The message was NOT from a colleague we've filtered out.
// * The message is upon a channel with an `_alerts` suffix.
// * It is not currently during working hours.
//   * And we already handled weekends by raising calls above.
//
return true ;

If the script returns true I initiate a phone-call. If the script returns false we ignore the message/event.

The alerting script itself is trivial, and probably non-portable, but the filtering engine is pretty neat. I can see a few more uses for it, even without it having nested blocks and a real grammar. So take a look, if you like:

• https://github.com/skx/evalfilter

Tags: evalfilter, golang, hack, slack, twilio | No comments