Might I recommend Warp as a starting point? Quite a nice framework for building anything HTTP based, and absurdly fast.

Might I ask what has proven Nginx to be insufficient for your needs? I'm really happy with it as a caching proxy. Very interested in your outcome though - I've considered similar or related projects. Python's gevent seemed a better fit for me though.

I guess it's a good opportunity to learn Go language ;)

The rules in nginx are basic, and you cannot do things like expire by prefix unless you pay for the commercial fork.

It should be possible to have flexible rules on caching. But not so flexible that you tie yourself in knots (e.g. the utter freedom and pain of using varnish)

This really sounds like a perfect match for golang.org … since it's a language to write servers – as long as one isn't afraid of pointers ;).

I assumed the configuration would be in a DSL, or some other scripting language, so that made me ignore Go.

That said it isn't an unreasonable idea, because I've been interested in exploring Go, and it is threaded and "cool".

I'll take a look at Warp later. For the moment I'm putting the finishing touches to a Key-Value store, which uses an append-only logfile, and is written in .. perl.

Is tengine any better than nginx wrt configurability? Apparently it's a free fork with many of the features found in the commercial offering.

tengine is something I've not explored. Perhaps I should, but I guess that the configuration will be lacking.

I assume I want to specify caching rules based on:

• Client-request.
• Server-response.
• Conditional on cookies, http-code, etc, etc.

To me that implies some "scripting language", or DSL, rather than a static nginx config. (Which might allow some flexibility, but not enough.)

I was going to comment with either Nginx or Varnish but I suppose a better thing would be to ask you ... which ones have you looked at and why have you ruled them all out? (I ask that, understaind the answer might be "it sounds like a fun problem to try to tackle" which is legitimate too.)

nginx is too simple, as mentioned above there are invalidation problems, but equally there is little flexibility.

varnish is something that works really really well, but it is a pain to configure because it is too complex.

I've seen people struggle with it for years, and I don't like it.

So this is a fun problem to have.

Have you considered using a more exotic language like Erlang or Haskell?

I'd not - just because I want some "scripting" configuration - I'm not sure if Haskell/Erlang would allow that.

Although I wish to avoid using C/C++ it might be that they're the most sane, since I could embed javascript/lua/similar to configure things.