New, improved version of mod_mcpage

Update: And that's what I get for not upgrading to the latest version of libmemcached before making the patch; MEMCACHED_HASH_KETAMA was dropped from libmemached a little while ago, and the previous version of the patch didn't reflect that. A new version of the patch has been uploaded.

Updated Update: Slightly updated patch uploaded again - small bug turned up in production use that didn't show up in testing where pages would occasionally be stored in memcached when they shouldn't have been, despite not actually being served up from memcached down the road. Latest version fixes.

You can get it from here. The patch applies to lighttpd 1.4.19 (I haven't tried it against 1.4.20 yet, but I'd be surprised if it had much trouble). There's a README that explains what it does and how to set it up in more detail, but here's the short version:

With the older version of mod_mcpage, the backend application was responsible for placing the page into memory,  gzipping if necessary, so lighttpd + mod_mcpage could serve it up directly without having to hit the backend at all. Now, no backend changes are required -- mod_mcpage will place pages (and their content types (no more "text/html" only)) into memcached, handle compressing them if the page will be too large, and serve up pages from memcached. If lighttpd + mod_mcpage find a page in memcached, it will decompress (if necessary) the page, set the content type, and return the page to the user. In production use, serving pages up for anonymous users from memcached has been a massive help for Daily Kos. This new version should simplify things a bit by not requiring the backend app and lighttpd to coordinate the page caching. Note: I have only used this with lighttpd running as a proxy in front of a webserver running a backend app (Scoop in this case). Also, mod_mcpage uses libmemcached, not the older libmemcache. Make sure libmemcached is installed before you try to install this.

TODO:

As always, there's more stuff to do. At the moment, expiration headers aren't set. That might be nice, although probably of somewhat limited utility with dynamic pages that are likely to change soon. The plugin only supports reading in write_queue content that's in memory, not from a file. This works fine with content returned by mod_proxy, which is what I wanted it for, but does not currently support putting files that would be served from disk into memcached. In my opinion, that wouldn't be particularly useful, but it might be desirable behavior. Small frequently loaded files can be placed in a tmpfs mount and read out of memory that way without the overhead of memcached. Still more memcached behaviors could be set from the configuration file, but aren't. Only text files are supported currently - binary data gets garbled up. This should be fixed, but the workaround is to exclude those from being served by memcached (and why are you proxying them anyway?).

comments powered by Disqus