About Archive Tags RSS Feed

 

Writing a status panel the modern way

19 June 2012 21:50

So status displays are cool. Seeing what is happening in real time is cool.

As a proof of concept I put together a trivial load-graph:

This is broken down into three parts:

Load Client

The load client is a trivial script which reads /proc/loadavg, and sends the 1-minute entry to a remote server, via a single UDP packet.

Load Server

The load-server is a service which listens for UDP traffic, and when it receives a new integer records that in a redis data-store.

Load Display

This is a HTML page which has the values from the store in it, which is then plotted using javscript.

So the UDP-server which receives load will receive two things:

  • load:N - The load figure. The text "load:" is literal, and present in case I decide to extend the stats..
  • x.x.x.x - The IP address from which it received the message.

This is inserted into a Redis database as an array. This array could then be fetched via an AJAX script to update the HTML display in real-time, but at the moment I just have a shell script which updates it in near-real time.

The idea of having a UDP-server receive values from remote clients is interesting. We just need to define a mapping to redis. For me I've just done this:

receive a UDP packet with value "load:1.2" from source 1.2.3.4
append "1.2" to key "1.2.3.4-load".
append the value "1.2.3.4" to the global "known_hosts"

The values received can be truncated (i.e. keep only the most recent 60 entries) with ease, due to the available Redis primitives, and we can easily graph these using the qjplot library.

Adding more metrics just means updating the clients to send "memfree:400m", "disk-free:50%", "users:2", "uptime:12345s", or similar. The storage is wonderfully abstract - all you need to do is get the graph-drawing code to a) Know which source to display, and b) which metric.

For example, if we did extend the client to send that data I could draw a graph of the memory on host foo.example.com just by selecting "memfree" against the origin "1.2.3.4".

ObQuote: "Come here, damn you, I want to touch you. " - Hellraiser

| 4 comments

 

Comments on this entry

icon Rul at 21:41 on 19 June 2012
http://kalgan.cc/~rul/blog/

Hi Steve, thank you for this post. How do you solve the problem of getting UDP datagrams with forged IP addresses? Wouldn't this lead to an (useless) attack to fake the information displayed in your status panel?

icon Steve Kemp at 21:42 on 19 June 2012
http://steve.org.uk/

I do it the naive way:

  • I firewall the server such that it can only receive UDP from the known-good clients.
  • Then I ignore the problem.

icon John Cooper at 10:45 on 20 June 2012

Hi Steve,

Sounds interesting stuff. Have you looked at etsy's statsd? I think they are doing similar UDP based submission for graphite.

john

icon Steve Kemp at 10:51 on 20 June 2012
http://steve.org.uk/

Yes I looked at a few different systems using a similar mechanism. Each had pros and cons, but this was done primarily because I liked the idea of using javsacript to draw graphs on-demand.

The alternative of storing them, caching them, etc just seemed like a pain.