Entries tagged nginx

Over the past couple of months the machine which hosts the Debian Administration website has been struggling with two distinct problems:

The dreaded scheduler bug/issue

The machine would frequently hang with the messages of the form:

Task xxx blocked for more than 120 seconds

This would usually require the application of raised elephants to recover from.

OOM-issues

The system would exhaust the generous 2Gb of memory it possessed, and start killing random tasks until the memory usage fell - at which point the server itself stopped functioning in a useful manner.

Hopefully these problems are now over:

• Bytemark have generously increased the memory installed upon this (donated) machine from 2Gb to 4Gb.
• I've cleaned up some of the site code and switched to using a combination of nginx & apache 2 to serve the site. (I've documented this upon the site itself : Speeding up dynamic websites via an nginx proxy.)

The combination of these two changes should resolve the memory issues, and I've installed a home-made 2.6.31.4 kernel which appears to have corrected the task-blocking scheduler issue.

ObTitle: Bridget Jones: The Edge of Reason

Tags: apache2, bytemark, debian-administration, nginx, performance | 2 comments

As part of some house-keeping I've been checking over my systems and ensuring they're all tickity-boo for the past couple of days.

One thing that I'm getting increasingly tempted by is converting my kvm guest to a 64-bit system.

I've not quite sold myself on the prospect of what will be a fair amount of downtime, but I'm 90% there.

I do think that a lot of my setup needs an overhaul, for example:

• Running all my websites under www-data is beginning to worry me.
• Running services as root is beginning to make me more and more paranoid.

One possible plan is to wipe my system, and then restore data from backups. A perhaps saner approach is divide my guest into two smaller ones, and migrate services over one by one (e.g. website1, website2, .. websiteN, email, etc).

For the moment I've taken a complete dump of my existing guest, and I'm running it with an IP in the 10.0.0.0/24 range on my desktop. That's at least given me a clear idea of the amount of work involved.

I'm still a little unclear on how best to manage running N websites with the intention they'll each run under their own UID. I guess it comes down to having a few instances of nginx/lighttpd/apache and then proxy from *:80 to the actual back-end. Precisely which mixture of services to use is a little overwhelming. Though at some point soon I need to start enabling IPv6 support, and that changes things a little.

(Not least because nginx has no IPv6 support present in the Lenny release - I've got a backported package which I run on the Debian Administration website.)

It's possible I could hack mod_vhost_alias to redirect/proxy to a local port based upon the virtual hostname present in the request - that's pretty trivial and I've already done something similar for work purposes. Though something like that should presumably already exist? I would expect a map of some form:

example.org: 127.0.0.1:8080
example.net: 127.0.0.1:9090

That has to be about the minimum necessary information to make the decision; a pair of vhost name & local destination.

/me googles some..

Update

OK quick update I've added local users for some of my sites, and now have them running under thttpd.

skx:/etc/thttpd# ls -ltr /home/www/ | tail -n 4
drwxr-sr-x  4 s-static   s-static   4096 Jan 15 01:41 static.steve.org.uk
drwxr-sr-x  5 s-openid   s-openid   4096 Feb 16 21:31 openid.steve.org.uk
drwxr-sr-x  6 s-images   s-images   4096 Feb 16 21:52 images.steve.org.uk
drwxr-sr-x  5 s-packages s-packages 4096 Feb 16 22:03 packages.steve.org.uk

That seems to work well, with a small wrapper script to start N instances of thttpd instead of a single one. Minor issues are that I'm using mod_proxy to forward requests to the thtpd instances running upon the loopback - and it was initially logging 127.0.0.1 as the source IP. A quick patch later all is well.

I'll leave it running a couple of the simple sites for the next few days and see if it kills children. If it does I'll convert the rest.

Probably will aim to have nginx in front of thttpd, instead of Apache, but this way I don't have to worry about mod_rewrite rules just yet.

ObFilm: Cruel Intentions

Tags: amd64, apache, arch, i386, ipv6, lenny, lighttpd, nginx | 11 comments

Today I am mostly in my bed suffering from "the plague".

Between naps I've worked on a new site a little:

• http://tweaked.io
  • Simple guides to optimizing particular services and servers.

Hopefully this will become updated, contributions welcome, and be useful to the world.

(Source available on github.)

Tags: dns, kernel, nginx, tweaked.io | 2 comments

For the past few years I've hosted all my websites in a "special" way:

• Each website runs under its own UID.
• Each website runs a local thttpd / webserver.
• Each server binds to localhost, on a high-port.
• My recipe is that the port of the webserver for user "foo" is "$(id -u foo)".
• On the front-end I have a proxy to route connections to the appropriate back-end, based on the Host header.

The webserver I chose initially was thttpd, which gained points because it was small, auditable, and simple to launch. Something like this was my recipe:

#!/bin/sh
exec thttpd -D -C /srv/steve.org.uk/thttpd.conf

Unfortunately thttpd suffers from a few omissions, most notably it doesn't support either "Keep-Alive", or "Compression" (i.e. gzip/deflate), so it would always be slower than I wanted.

On the plus side it was simple to use, supported CGI scripts, and served me well once I'd patched it to support X-Forwarded-For for IPv6 connections.

Recently I setup a server optimization site and was a little disappointed that the site itself scored poorly on Google's page-speed test. So I removed thttpd for that site, and replacing it with nginx. The end result was that the site scored 98/100 on Google's page-speed test. Progress. Unfortunately I couldn't do that globally because nginx doesn't support old-school plain CGI scripts.

So last night I removed both nginx and thttpd, and now every site on my box is hosted using lighttpd.

There weren't too many differences in the setup, though I had to add some rules to add caching for *.css, etc, and some of my code needed updating.

Beyond that today I've setup a dedicated docker host - which allows me to easily spin up containers. Currently I've got graphite monitoring for my random hosts, and a wordpress guest for plugin development/testing.

Now to go back to reading Off to be the wizard .. - not as good as Rick Cook's wizardry series (which got less good as time went on, but started off strongly), but still entertaining.

Tags: docker, graphite, lighttpd, nginx, thttpd | 2 comments