I am lightened, can we drop this?

16 February 2010 21:50

As part of some house-keeping I've been checking over my systems and ensuring they're all tickity-boo for the past couple of days.

One thing that I'm getting increasingly tempted by is converting my kvm guest to a 64-bit system.

I've not quite sold myself on the prospect of what will be a fair amount of downtime, but I'm 90% there.

I do think that a lot of my setup needs an overhaul, for example:

Running all my websites under www-data is beginning to worry me.
Running services as root is beginning to make me more and more paranoid.

One possible plan is to wipe my system, and then restore data from backups. A perhaps saner approach is divide my guest into two smaller ones, and migrate services over one by one (e.g. website1, website2, .. websiteN, email, etc).

For the moment I've taken a complete dump of my existing guest, and I'm running it with an IP in the 10.0.0.0/24 range on my desktop. That's at least given me a clear idea of the amount of work involved.

I'm still a little unclear on how best to manage running N websites with the intention they'll each run under their own UID. I guess it comes down to having a few instances of nginx/lighttpd/apache and then proxy from *:80 to the actual back-end. Precisely which mixture of services to use is a little overwhelming. Though at some point soon I need to start enabling IPv6 support, and that changes things a little.

(Not least because nginx has no IPv6 support present in the Lenny release - I've got a backported package which I run on the Debian Administration website.)

It's possible I could hack mod_vhost_alias to redirect/proxy to a local port based upon the virtual hostname present in the request - that's pretty trivial and I've already done something similar for work purposes. Though something like that should presumably already exist? I would expect a map of some form:

example.org: 127.0.0.1:8080
example.net: 127.0.0.1:9090

That has to be about the minimum necessary information to make the decision; a pair of vhost name & local destination.

/me googles some..

Update

OK quick update I've added local users for some of my sites, and now have them running under thttpd.
skx:/etc/thttpd# ls -ltr /home/www/ | tail -n 4
drwxr-sr-x  4 s-static   s-static   4096 Jan 15 01:41 static.steve.org.uk
drwxr-sr-x  5 s-openid   s-openid   4096 Feb 16 21:31 openid.steve.org.uk
drwxr-sr-x  6 s-images   s-images   4096 Feb 16 21:52 images.steve.org.uk
drwxr-sr-x  5 s-packages s-packages 4096 Feb 16 22:03 packages.steve.org.uk
That seems to work well, with a small wrapper script to start N instances of thttpd instead of a single one. Minor issues are that I'm using mod_proxy to forward requests to the thtpd instances running upon the loopback - and it was initially logging 127.0.0.1 as the source IP. A quick patch later all is well.

I'll leave it running a couple of the simple sites for the next few days and see if it kills children. If it does I'll convert the rest.

Probably will aim to have nginx in front of thttpd, instead of Apache, but this way I don't have to worry about mod_rewrite rules just yet.

ObFilm: Cruel Intentions

Tags: amd64, apache, arch, i386, ipv6, lenny, lighttpd, nginx | 11 comments

Comments on this entry

Simon McVittie at 19:26 on 16 February 2010
http://smcv.pseudorandom.co.uk/

Have you looked at http://mpm-itk.sesse.net/ for your multiple-uid websites?

Steve Kemp at 19:28 on 16 February 2010

I did briefly consider it, and although I've not seen an previous security issues it does conflict with the other point I made about having things run as root.

As I understand it that module means that things run as root until the incoming request can be parsed enough to determine the virtual host name - which is a significantly larger window than any other deamon which just does a bind() and then drops privileges.

Adrian Bridgett at 21:25 on 16 February 2010

I've found that 64-bit guests take a surprising amount more memory. For a host, sure, go 64-bit. For a guest I'd seriously look at if you need to.

Just picking a few processes and looking at VSZ:
- ps 4MB vs 8MB
-- RSS is 900KB vs 1000KB
- getty is 1.8MB vs 5.8MB
-- RSS is 500KB vs 600KB
- pickup (from postfix) 6M vs 33MB!
-- RSS is 1.6MB vs 2.2MB FWIW

Maybe the 64-bit stuff just wastes huge amounts of virtual memory since to be fair, there is oodles of it. Even so, the RSS sizes of many things mean that I have to allocate substantially more memory to 64-bit VMs.

Steve Kemp at 22:36 on 16 February 2010

Thanks for that Adrian - it was something I'd not considered. It would be interesting to compare performance as well as memory usage to see if I'm getting my money's worth with a 64-bit migration.

I'll spend a day or two benchmarking my system, under KVM, as an i368 installation and compare that to a 64-bit guest to see how it compares.

Sam Morris at 00:39 on 17 February 2010

SuEXEC allows you to set a user/group per virtual host that CGI scripts will be spawned as. mod_fcgid lets you avoid the performance hit in doing so. They work great for my own (personal and small business) servers.

http://httpd.apache.org/mod_fcgid/

Jeff Schroeder at 01:29 on 17 February 2010

Steve, have you considered LXC? If you are running very new kernels and the latest lxc libvirtd, you can manage it all through that. It is surprising how nice lxc is.

I'm waiting for you to re-do this howto with the latest debian a newer kernel and put it on debian-administration:
http://www.stgraber.org/2009/11/06/lxc-containers-or-extremely-fast-virtualization

henkjan at 07:52 on 17 February 2010

Maybe mod_blink is what you need for user/group id switching for each vhost.

http://svn.blinkenlights.nl/viewvc.cgi/apache/trunk/mod_blink/Readme?revision=21&view=markup

Steve Kemp at 10:29 on 17 February 2010

Thanks for the feedback everybody.

It seems that mod_blink does the job in a similar fashion to mpm-itk - so the similar fear applies to it.

I've gotten into the bad habit of running a number of virtual hosts all owned by "steve:users" and all running as "www-data:www-data", and my specific goals are:

Create a local user per-site to identify which site was to blame if compromised.
Avoid running each site as the same user.
Avoid running more daemons as root.

Now creating local users is just a management issue, its not related to security per se.

But running each site as a dedicated user, rather than www-data, is intended to both limit scope of mass-defacement and to allow tracking potential compromises. Now suexec would help in that sense for purely dynamic stuff - but it doesn't really limit things overwise so I didn't really consider it for long.

Most of my sites are static, so CGI isn't even involved, and those that are dynamic are all Perl based. (I don't use rails/php/hotness.)

I think containers are probably overkill for just webserving, but I'll look into them later as the idea could be useful for other things.

Asheesh Laroia at 12:33 on 18 February 2010

I use Apache2's MPM-ITK to separate my vhosts. It's very simple to configure; I just add

AssignUserID username groupname

to the VirtualHost stanza. Now each VirtualHost runs as the UID/GID of my choosing.

David Schmitt at 12:49 on 18 February 2010

I'm using the ITK mpm and am quite content with it, although I do see your concern about the running-as-root.

A different thing that is bothering me with this setup is that all scripts now run with enough privileges to write to their source. While this is not required by the ITK mpm it is a easy trap to fall in and should be avoided.

Steve Kemp at 13:02 on 18 February 2010

Yes I've definitely been careful about making sure that things aren't as writeable as they should be - as that is a common problem.

Still I'm less worried about that given that my sites are 80% static and only 20% dynamic, there isn't the possibility of users uploading "content" for example..

I am lightened, can we drop this?

Comments on this entry

Recent Posts