About Archive Tags RSS Feed


It is unfortunate that many companies need the same sysadmin jobs carried out

27 January 2014 21:50

It is unfortunate that I've been exposed to several companies that all need the same kind of problems solving, again and again:

  • Imaging systems. Quickly.
  • Configuring MySQL & Postgres replication and fail-over.
  • Configuring local users.
  • Solving the problem of distributing usernames/passwords to 500+ client-systems within a team.

A lot of these are solved problems, yet they seem to keep cropping up. I guess the fact I've done some of these things more than once means I'm the local-expert, so that's why I get asked. But still..

There was a time when I thought my Debian Administration site would help solve these kind of problems; that writing documentation would encourage people to do things properly. Certainly it has helped me, and some other people are greatful, but it didn't do enough to help.

I'm not sure if the problem is that my documentation is a little ideosyncratic, isn't good enough, or isn't reaching the right kind of people. I suspect a combination of that and scale - You can't walk somebody though the idea of setting up a ten-node database-cluster, they need to suffer, they need to break things, they need to sweat on Christmas Day, at 5AM, as everything goes to hell. Then the next time they'll do it properly.

I'd love to take the time to write out recipes in Salt, Ansible, Puppet, Chef, CFengine, Slaughter, whatever, and support them.

Remote. Automated. System management.

Throw in monitoring of metrics, security fixes, and reporting and there's probably a valuable service there.

It probably can't happen though, for three main reasons:

  • The people that need it don't know they need it. They're fighting fires, they know it is important and they will fix things "soon", but other work takes priority.
  • The people that are tempted will baulk at the idea of unknown code from an external source running on their system(s).
  • Companies managed by one sysadmin will wonder why they need more help, because "everything is working, right?".

I did recently write some policies for setting up a two-node Master-Slave MySQL setup, with reporting, monitoring, and custom SMS-based alerts. I guess I'm wondering if I can be cheeky and sell the same work twice. ;)



Comments on this entry

icon Steve Kemp at 16:27 on 29 January 2014

I do not have a recipe for Postgres at the moment, having no systems that use it.

(These days I manage a couple of toy MySQL deployments, but no psql.)

icon Maciej at 16:19 on 29 January 2014

I was using a guide for PostgreSQL replication (http://debian-administration.org/article/677/) but implementing it in chef for many separate sets it is not such easy. Do you have any interesting chef cookbooks which make it easier?

icon Steve Kemp at 16:01 on 28 January 2014

We did something like that for Symbiosis at Bytemark.

Symbiosis is a collection of packages that pulls in Apache, dovecot, and similar related services and configures them all to work in a consistent fashion.

i.e. You could create a directory tree like this, and everything would work as you'd expect:


That's enough to get steve@steve.org.uk, root@steve.org.uk, and http://steve.org.uk/ all working for IMAP, POP3, and HTTP.

We had to jump through a lot of hoops with diversions and post-install scripts to make it all work, but the end result was worth-it.

icon Federico at 15:54 on 28 January 2014

A good while ago I was thinking about ways to automate deploying services based on Debian e.g. using packages that would install and configure other packages.
Unfortunately there isn't a clear winner across many system management tools and their recipes aren't compatible across different tools.

icon Pete Foster at 23:35 on 27 January 2014

I think that it's maybe also a case of it being easier to just 'get the experts in'.

Where this plan falls over is when the solution needs attention. If you haven't set things up and got an understanding of how things work then you've no chance when it comes to fire fighting.

Maybe you should just keep using the above attitudes to your advantage.

icon Alex at 00:37 on 28 January 2014

do you have a tutorial written on your 2 node mysql setup? Sounds very useful.

icon Matt Zagrabelny at 00:38 on 28 January 2014


What do you use for "Solving the problem of distributing usernames/passwords to 500+ client-systems within a team." ?



icon Jimmy Kaplowitz at 02:04 on 28 January 2014

This idea is sorely needed, though it'd be a full-time business.

In fact, I've thought along similar lines and described it to friends as a startup idea, though framed more broadly than you've said. Most people I've mentioned it to think it's a good idea and can effectively get traction when expanded and targeted right. Glad to hear someone smart like you also sees the importance of this.

I'm not planning to change my current focus this year, but maybe in the future.

icon Steve Kemp at 10:13 on 28 January 2014

Pete: Agreed that often you can't flounce into a company and make sweeping changes on systems, but equally when you see a badly setup cluster you know you can replace it with another pair of hosts, for example.

Alex: No tutorial written recently on MySQL replication, no. I've done it in the past, but this time round I wrote a decently flexible set of recipes for doing the setup given two hosts, and I was pretty pleased with that. I might publish it if I can make it more generic.

Matt: In the past solutions have ranged from flat-files with Unix permissions, to password-managers, to GPG-files, to in-house written code that stores system credentials in a database and presents a JSON-API to allow querying and access remotely.

In my mind the last solution is the nicest, but the real solution is along the lines of pwsafe/pwkeeper,etc. A couple of back-ends for "support", "dev", "production", etc.

Jimmy I suspect if you got a client or two quickly that would validate the idea nicely. The compromise is the size of company you're going for - too small and they'll never pay, too large and bureaucracy and politics will get in the way. One thing I've learned is that some clients will eat all your time, so you absolutely must be ready for that, and either catch it early or agree to avoid them in the future.