About Archive Tags RSS Feed


And if someone gets upset you say, "chill out"!

25 December 2009 21:50

It was interesting to see Clint Adams describe love and dissatification with configuration management.

At work I've got control of 150(ish) machines which are managed via CFEngine. These machines are exclusively running Debian Lenny. In addition to these hosts we also have several machines running Solaris, OpenBSD, and various Ubuntu releases for different purposes.

Unfortunately I made a mistake when I setup the CFEngine infrastructure and when writing all the policies, files, etc, I essentially said "OK CFEngine controlled? Then it is Debian". (This has been slowly changing over time, but not very quickly.)

But in short this means that the machines running *BSD, Solaris, and non-Debian distributions haven't been managed as well via CFEngine as the rest, even though technically they could have been.

A while back I decided that it was time to deal with this situation. Looking around the various options it seemed Puppet was the way of the future and using that we could rewrite/port our policies and make sure they were both cleanly organised and made no assumptions.

So I setup a puppetmaster machine, then I installed the client on a range of client machines (openbsd, debian lenny, ubuntu, solaris) so that I could convince myself my approach was valid, and that the tool itself could do everything I wanted it to do.

Unfortunately using puppet soon became painful. It has primitives for doing various things such as maintaining local users, working with cronjobs, and similar. Unfortunately not all primitives work upon all platforms, which kinda makes me think "what's the point?". For example the puppet client running upon FreeBSD will let you add a local user, setup a ~/.ssh/authorized_keys file but will not let you setup a password. (Which means you can add users who can login, but then cannot use sudo. Subpar)

At this point I've taken a step back. As I think I've mentioned before I don't actually do too much with CFEngine. Just a few jobs:

  • Fetch a file from the master machine and copy into the local filesystem. (Making no changes.)
  • Fetch a file from the master machine, move it to the local system after applying a simple edit. (e.g "s/##HOSTNAME##/`hostname`/g")
  • Install a package.
  • Purge a package.
  • Setup local user accounts, with ~/.ssh handled properly.
  • Apply one-line sed-style edits to files. (e.g. "s/ENABLED=no/ENABLED=yes/" /etc/default/foo)

(i.e. I don't use cron facilities, I add files to cron directories. Similarly I don't use process monitoring, instead I install the monit package and drop /etc/monit/monitrc into place.)

There is a pretty big decision to make in the future with the alternatives being:

  • Look at Chef.
  • Stick with CFEngine but start again with a better layout, with more care and attention to portability things.
  • Replace the whole mess with in-house-fu.

If we ignore the handling of local users, and sudo setup, then the tasks that remain are almost trivial. Creating a simple parser for a "toy-language" which can let you define copies, edits, and package operations would be an afternoons work. Then add some openssl key authentication and you've got a cfengine-lite.

For the moment I'm punting the decision but I'm 90% certain that the choice is CFEngine vs. Chef vs. In-House-Fu - and that puppet is no longer under consideration.

Anyway despite having taken months to arrive at this point I'm going to continue to punt. Instead my plan is to move toward using LDAP for all user management, login stuff, and sudo management. That will be useful in its own right, and it will coincidentally mean that whatever management system we do end up using will have on less task to deal with. (Which can only be a good thing.)

ObFilm: Terminator II



Comments on this entry

icon h01ger at 02:16 on 25 December 2009

have you looked at fai (as in apt-cache show fai-client). you dont need a fai-server (any vcs will do), you dont need to install with fai, you can use cfengine with fai, you can script anything with fai. sure, you need to write some simple scripts and hack it to fit the way you want, but I think thats better then yet another in-house system very similar to fai :-) fai is written in perl and bash, but fai scripts can be written in any language. its gpl2+. cheers!

icon natxo asenjo at 11:57 on 25 December 2009

If you want to dig deeper in cfengine and stick with the one in debian lenny, then get yourself a copy of 'Automating Linux and Unix System Administration, second edition' by Campi and Bauer. Brilliant book. They teach you how to combine FAI, kikstart and jumpstart with cfengine 2.

At work I manage about 20 servers like this and provided I am the only unix admin, I'm glad I do it so.

Cfengine 3 is apparently much better than 2, but I still have not found anything I cannot do with 2, so ...

I know I will have to upgrade sometime in the future, but I am in no rush. Maybe those same authors will make a 3rd edition for cfengine 3 ...

icon Steve Kemp at 12:04 on 25 December 2009

Thanks for the feedback - FAI looks interesting but I'd always had it down as a more complex and capable solution.

(We already have in-house tools for reimaging machines so the idea of using FAI didn't really occur to me, I assumed it would want to be used for everything.)

CFEngine3 does look interesting, I did experiment with it briefly, but it doesn't yet appear to be recommended - nor is it included in the *BSD port trees (unless I missed it, or things have changed over the past few months).

icon Matt Palmer at 21:28 on 25 December 2009

FAI's problem, in my experience, is that it's too focused on initial deployment, and not enough on on-going maintenance. Unless you reinstall all your machines to effect a config change, FAI isn't particularly useful.

Out of interest, Steve, why is option 4: "Extend Puppet to do what I need" not on the list? Puppet's primitives only do what someone has sat down and made them do, so when there's differences between platforms it's usually because whoever wrote the support for a particular platform didn't need that part of the type for their case.

icon Anonymous at 00:16 on 26 December 2009

Of course your users can use sudo...wait, you make them enter a password? :)

I don't consider NOPASSWD a security problem. On the contrary, I consider PASSWD a security problem. It means users actually have a password, rather than key-only access. It also means that a keylogger or trojan horse of some kind can obtain their password, providing instant access to many other systems they have accounts on, as well as sudo on those machines. Unless you seriously think most users have multiple remote-system passwords. NOPASSWD and key-only SSH access, on the other hand, means that a compromise only affects one system.

icon natxo asenjo at 01:05 on 26 December 2009

I am no freebsd user, but cfengine 3 is ported: http://www.freebsd.org/cgi/cvsweb.cgi/ports/sysutils/cfengine3/

FAI is obviously for hands-off installations, the name says it ;) the nice thing about it is that it works beautifully with cfengine's classes, so you can rollout new machines and have them inmediately set up to your needs.

icon Steve Kemp at 01:56 on 26 December 2009

Option "4. Fixing the broken bits" could have been listed but the bits that were broken weren't fully known, and it seemed like the potential gain from dong that wasn't too hard.

i.e. The task of creating puppet repository with all the rules/actions/steps and fixing puppet is harder work than dropping the CFEngine rules and rewriting those from scratch.

icon Steve Kemp at 01:58 on 26 December 2009

sudo PASSWD vs NOPASSWD is interesting.

Given that I disable remote-root by default to do something really bad a remote attacker needs to steal both a key and a password to compromise a machine fully - in the absence of security holes.

That to me seems saner than having to just steal a key and get root on my machine...

icon Michael Goetze at 10:40 on 26 December 2009

> FAI's problem, in my experience, is that it's too focused
> on initial deployment, and not enough on on-going
> maintenance. Unless you reinstall all your machines to
> effect a config change, FAI isn't particularly useful.

This is wrong, FAI's softupdate feature does not require that anything be reinstalled for a config change.

icon Anonymous at 08:05 on 27 December 2009

A key alone proves sufficient either way; just trojan-horse the user's account. For instance, add a fake sudo somewhere in their path, which fakes the password dialog, forwards to the real sudo to avoid suspicion, and deletes itself. Or, if you want a more active approach, prompt them for a password when they log in, and most users will just assume a configuration issue and enter their password. Or for a highly passive approach, just wait for them to use sudo and take advantage of the timestamp to run whatever you like as root.

icon Thomas Koch at 16:36 on 28 December 2009

How do you think about using debian packages for configuration management? You're limited to debian based systems, but there's nothing you can't do with debian packages and there's already a very powerful toolchain.

icon Steve Kemp at 11:49 on 29 December 2009

Using Debian packages would fail upon non-Debian systems - which would spoil their use.

Also I expect many of the packages would want to mess with the configuration files belonging to other packages - e.g. replacing /etc/sudoers, or modifying files beneath /etc/pam.d - so the packages I did install would be non-policy-compliant and liable to break upon future upgrades...

icon Thomas Koch at 14:02 on 30 December 2009

I just found http://packages.debian.org/sid/dpsyco which may help with the problems you described. Will have a look at it now.

icon duritong at 14:43 on 6 January 2010

so your only point against puppet is that some options are on some systems not available? Why do you think that something like chef would have solved that problem? Chef has so far much less platform support than puppet. So why do you think chef is the better option if you complain about platform support?

The only problem is that somebody didn't yet implement the os-/distro-specific parts for this option in puppet. Why not implement it yourself? If nobody is going to do it, require it or even just paying for it it will never be there...