About Archive Tags RSS Feed


That is it, I'm going to do it

22 March 2014 21:50

That's it, I'm going to do it: I have now committed myself to writing a scalable, caching, reverse HTTP proxy.

The biggest question right now is implementation language; obviously "threading" of some kind is required so it is a choice between Perl's anyevent, Python's twisted, Rubys event machine, or node.js.

I'm absolutely, definitely, not going to use C, or C++.

Writing a a reverse proxy in node.js is almost trivial, the hard part will be working out which language to express the caching behaviour, on a per type, and per-resource basis.

I will ponder.



Comments on this entry

icon Anonymous at 15:53 on 22 March 2014

Might I recommend Warp as a starting point? Quite a nice framework for building anything HTTP based, and absurdly fast.

icon Sam at 16:16 on 22 March 2014

Might I ask what has proven Nginx to be insufficient for your needs? I'm really happy with it as a caching proxy. Very interested in your outcome though - I've considered similar or related projects. Python's gevent seemed a better fit for me though.

icon Charles Darke at 17:01 on 22 March 2014

I guess it's a good opportunity to learn Go language ;)

icon Steve Kemp at 17:09 on 22 March 2014

The rules in nginx are basic, and you cannot do things like expire by prefix unless you pay for the commercial fork.

It should be possible to have flexible rules on caching. But not so flexible that you tie yourself in knots (e.g. the utter freedom and pain of using varnish)

icon Gopher at 17:26 on 22 March 2014

This really sounds like a perfect match for golang.org … since it's a language to write servers – as long as one isn't afraid of pointers ;).

icon Steve Kemp at 17:45 on 22 March 2014

I assumed the configuration would be in a DSL, or some other scripting language, so that made me ignore Go.

That said it isn't an unreasonable idea, because I've been interested in exploring Go, and it is threaded and "cool".

I'll take a look at Warp later. For the moment I'm putting the finishing touches to a Key-Value store, which uses an append-only logfile, and is written in .. perl.

icon Paul Tötterman at 21:48 on 22 March 2014

Is tengine any better than nginx wrt configurability? Apparently it's a free fork with many of the features found in the commercial offering.

icon Steve Kemp at 22:23 on 22 March 2014

tengine is something I've not explored. Perhaps I should, but I guess that the configuration will be lacking.

I assume I want to specify caching rules based on:

  • Client-request.
  • Server-response.
  • Conditional on cookies, http-code, etc, etc.

To me that implies some "scripting language", or DSL, rather than a static nginx config. (Which might allow some flexibility, but not enough.)

icon Abe Hassan at 04:03 on 23 March 2014

I was going to comment with either Nginx or Varnish but I suppose a better thing would be to ask you ... which ones have you looked at and why have you ruled them all out? (I ask that, understaind the answer might be "it sounds like a fun problem to try to tackle" which is legitimate too.)

icon Steve at 11:13 on 23 March 2014

nginx is too simple, as mentioned above there are invalidation problems, but equally there is little flexibility.

varnish is something that works really really well, but it is a pain to configure because it is too complex.

I've seen people struggle with it for years, and I don't like it.

So this is a fun problem to have.

icon Anonymous Coward at 06:07 on 24 March 2014

Have you considered using a more exotic language like Erlang or Haskell?

icon Steve Kemp at 09:39 on 24 March 2014

I'd not - just because I want some "scripting" configuration - I'm not sure if Haskell/Erlang would allow that.

Although I wish to avoid using C/C++ it might be that they're the most sane, since I could embed javascript/lua/similar to configure things.