About Archive Tags RSS Feed

 

So I should document the purple server a little more

15 June 2016 21:50

I should probably document the purple server I hacked together in Perl and mentioned in my last post. In short it allows you to centralise notifications. Send "alerts" to it, and when they are triggered they will be routed from that central location. There is only a primitive notifier included, which sends data to the console, but there are sample stubs for sending by email/pushover, and escalation.

In brief you create alerts by sending a JSON object via HTTP-POST. These objects contain a bunch of fields, but the two most important are:

  • id
    • A human-name for the alert. e.g. "disk-space", "heartbeat", or "unread-mail".
  • raise
    • When to raise the alert. e.g. "now", "+5m", "1466006086".

When an update is received any existing alert has its values updated, which makes heartbeat alerts trivial. Send a message with:

{ "id": "heartbeat",

 "raise": "+5m",
 .. }

The existing alert will be updated each time such a new event is submitted, which means that the time at which that alert will raise will be pushed back by five minutes. If you send this every 60 seconds then you'll get informed of an outage five minutes after your server explodes (because the "+5m" will have been turned into an absolute time, and that time will eventually become in the past - triggering a notification).

Alerts are keyed on the source IP which sent the submission and the id field, meaning you can send the same update from multiple hosts without causing any problems.

Notifications can be viewed in a reasonably pretty Web UI, so you can clear raised-alerts, see the pending ones, and suppress further notifications on something that has been raised. (By default notifications are issued every sixty seconds, until the alert is cleared. There is support for only raising an alert once, which is useful for services you might deliver events via, such as pushover which will repeat themselves.)

Anyway this is a fun project, which is a significantly simplified and less scalable version of a project which is open-sourced already and used at Bytemark.

| 4 comments

 

Comments on this entry

icon Quabla at 19:36 on 15 June 2016

I would like to know if there is a reason not to use the systemd-journal as backend. I never tried it, but I think it supports pulling journals from other servers for centralization.

icon Steve Kemp at 02:40 on 16 June 2016
https://www.steve.org.uk/

Quablo I'm either doing a bad job explaining or you've misunderstood - but the alerts/events here are unrelated to syslog/journald events.

We need a database to store things like:

  • Alert ID
  • Alert subject.
  • Time to raise at.
  • Time of last notification.
  • State of alert - raised, pending, cleared, etc.


icon Quabla at 09:29 on 17 June 2016

I was thinking about the journal because I am extracting error notifications from there anyways but had to write my own script for that.

In principle you could make everything journal entries. The journal supports arbitrary key/value pairs and at least "disk-space" pretty much fits a syslog warning.

However, the state of alert notifications would have to be stored elsewhere. Also, systemd-journal-remote is currently only in testing and I have no idea how complicated this system is.

icon Steve Kemp at 09:32 on 17 June 2016
https://www.steve.org.uk/

If I understand you correclty I think you're talking about about piping errors from the journal into the alerting system? If so yes, that could be done.

If you're talking about using the journal to store alerts then that might be possible, as you can certainly inject arbitrary content into them. But that's not what purple is about.

purple is all about receiving updates from remote clients, and scheduling notifications to inform you of error-conditions, or status-changes.