Writing a status panel the modern way

Tuesday, 19 June 2012

So status displays are cool. Seeing what is happening in real time is cool.

As a proof of concept I put together a trivial load-graph:

This is broken down into three parts:

Load Client

The load client is a trivial script which reads /proc/loadavg, and sends the 1-minute entry to a remote server, via a single UDP packet.

Load Server

The load-server is a service which listens for UDP traffic, and when it receives a new integer records that in a redis data-store.

Load Display

This is a HTML page which has the values from the store in it, which is then plotted using javscript.

So the UDP-server which receives load will receive two things:

  • load:N - The load figure. The text "load:" is literal, and present in case I decide to extend the stats..
  • x.x.x.x - The IP address from which it received the message.

This is inserted into a Redis database as an array. This array could then be fetched via an AJAX script to update the HTML display in real-time, but at the moment I just have a shell script which updates it in near-real time.

The idea of having a UDP-server receive values from remote clients is interesting. We just need to define a mapping to redis. For me I've just done this:

receive a UDP packet with value "load:1.2" from source
append "1.2" to key "".
append the value "" to the global "known_hosts"

The values received can be truncated (i.e. keep only the most recent 60 entries) with ease, due to the available Redis primitives, and we can easily graph these using the qjplot library.

Adding more metrics just means updating the clients to send "memfree:400m", "disk-free:50%", "users:2", "uptime:12345s", or similar. The storage is wonderfully abstract - all you need to do is get the graph-drawing code to a) Know which source to display, and b) which metric.

For example, if we did extend the client to send that data I could draw a graph of the memory on host foo.example.com just by selecting "memfree" against the origin "".

ObQuote: "Come here, damn you, I want to touch you. " - Hellraiser



Comments On This Entry

[gravitar] Rul

Submitted at 21:41:46 on 19 june 2012

Hi Steve, thank you for this post. How do you solve the problem of getting UDP datagrams with forged IP addresses? Wouldn't this lead to an (useless) attack to fake the information displayed in your status panel?

[gravitar] Steve Kemp

Submitted at 21:42:57 on 19 june 2012

I do it the naive way:

  • I firewall the server such that it can only receive UDP from the known-good clients.
  • Then I ignore the problem.

[gravitar] John Cooper

Submitted at 10:45:35 on 20 june 2012

Hi Steve,

Sounds interesting stuff. Have you looked at etsy's statsd? I think they are doing similar UDP based submission for graphite.


[gravitar] Steve Kemp

Submitted at 10:51:44 on 20 june 2012

Yes I looked at a few different systems using a similar mechanism. Each had pros and cons, but this was done primarily because I liked the idea of using javsacript to draw graphs on-demand.

The alternative of storing them, caching them, etc just seemed like a pain.


Comments are closed on posts which are more than ten days old.

Recent Posts

Recent Tags