I should probably document the purple server I hacked together in Perl and mentioned in my last post. In short it allows you to centralise notifications. Send "alerts" to it, and when they are triggered they will be routed from that central location. There is only a primitive notifier included, which sends data to the console, but there are sample stubs for sending by email/pushover, and escalation.
In brief you create alerts by sending a JSON object via HTTP-POST. These objects contain a bunch of fields, but the two most important are:
id- A human-name for the alert. e.g. "
disk-space", "heartbeat", or "unread-mail".
- A human-name for the alert. e.g. "
raise- When to raise the alert. e.g. "
now", "+5m", "1466006086".
- When to raise the alert. e.g. "
When an update is received any existing alert has its values updated, which makes heartbeat alerts trivial. Send a message with:
{ "id": "heartbeat",
"raise": "+5m",
.. }
The existing alert will be updated each time such a new event is submitted, which means that the time at which that alert will raise will be pushed back by five minutes. If you send this every 60 seconds then you'll get informed of an outage five minutes after your server explodes (because the "+5m" will have been turned into an absolute time, and that time will eventually become in the past - triggering a notification).
Alerts are keyed on the source IP which sent the submission and
the id field, meaning you can send the same update from multiple
hosts without causing any problems.
Notifications can be viewed in a reasonably pretty Web UI, so you can clear raised-alerts, see the pending ones, and suppress further notifications on something that has been raised. (By default notifications are issued every sixty seconds, until the alert is cleared. There is support for only raising an alert once, which is useful for services you might deliver events via, such as pushover which will repeat themselves.)
Anyway this is a fun project, which is a significantly simplified and less scalable version of a project which is open-sourced already and used at Bytemark.
Tags: purple 4 comments
I would like to know if there is a reason not to use the systemd-journal as backend. I never tried it, but I think it supports pulling journals from other servers for centralization.