Perl Tiered Cache Benchmarks · time to pull the plug

21 Sep 2009, 00:10

These benchmarks are for the Perl tiered caching modules discussed here originally. Ruby tiered cache benchmarks shall follow later.

All tests performed on a Xen virtual machine with 512MB of RAM using one core of an Intel Core2 Duo E8500 @ 3.16GHz. The Perl version is 5.10.0 running on Debian 5.0. The remote memcached server is 1.2.5, local memcached is 1.2.2. The modules tested were Cache::Memcached::Fast, Cache::Mmap, Cache::FastMmap, and the custom Cache::Tiered and Cache::TieredFast modules (modules and benchmarking code are here.

The tiered configurations all use a remote memcached server. Cache::Tiered using a local Cache::Mmap cache, Cache::TieredFast uses a local Cache::FastMmap cache, and Cache::TieredLmc uses a local memcached server.

Benchmarking results below the fold. The first set of benchmarks is read-heavy, setting a value and reading it back 50,000 times (and resetting it if need be), the second set of benchmarks set and read a value 50,000 times. The test value used is the zlib HOWTO html file. NB: Right now, with the tiered cache modules, you pass in the memcached or mmap objects you want to use. Eventually, they will make them themselves.

Update:Forgot to add that the mmap files are stored in /dev/shm, but any memory based filesystem should perform the same. Disk-based mmap were not tested, but they don't seem as likely to give the same speed benefit.

## Read-heavy benchmarks

Using memcached... Memcache set... At 10000 At 20000 At 30000 At 40000 At 50000 Memcached elapsed time: 101.018182 seconds : cache misses: 3 Using local memcached... Local Memcache set... At 10000 At 20000 At 30000 At 40000 At 50000 local memcached elapsed time: 3.473889 seconds : cache misses: 0 Using mmap... Mmap set... At 10000 At 20000 At 30000 At 40000 At 50000 Mmap elapsed time: 5.370702 seconds : cache misses: 0 Using mmap fast... Mmap fast set... At 10000 At 20000 At 30000 At 40000 At 50000 Mmap fast elapsed time: 3.16509 seconds : cache misses: 0 Using Cache::Tiered... Cache::Tiered set... At 10000 At 20000 At 30000 At 40000 At 50000 Tiered elapsed time: 9.488398 seconds : cache misses: 0 Using Cache::TieredFast... Cache::TieredFast set... At 10000 At 20000 At 30000 At 40000 At 50000 TieredFast elapsed time: 5.193063 seconds : cache misses: 0 Using Cache::TieredLmc... Cache::TieredLmc set... At 10000 At 20000 At 30000 At 40000 At 50000 TieredLmc elapsed time: 113.350251 seconds : cache misses: 3 Done with read-heavy benchmarks. ## Write-heavy benchmarks Using memcached... At 10000 At 20000 At 30000 At 40000 At 50000 Memcached elapsed time: 129.66146 seconds Using local memcached... At 10000 At 20000 At 30000 At 40000 At 50000 Local memcached elapsed time: 11.377581 seconds Using mmap... At 10000 At 20000 At 30000 At 40000 At 50000 Mmap elapsed time: 12.923498 seconds Using mmap fast... At 10000 At 20000 At 30000 At 40000 At 50000 Mmap fast elapsed time: 7.368855 seconds Using Cache::Tiered... At 10000 At 20000 At 30000 At 40000 At 50000 Cache::Tiered elapsed time: 44.16001 seconds Using Cache::TieredFast... At 10000 At 20000 At 30000 At 40000 At 50000 Cache::TieredFast elapsed time: 40.125187 seconds Using Cache::TieredLmc... At 10000 At 20000 At 30000 At 40000 At 50000 Cache::TieredLmc elapsed time: 149.111689 seconds Done with write-heavy benchmarks.

Done with benchmarking.

Notes:

All of these results seem to be about what you'd expect, except 1) I didn't think the local memcached would be faster than Cache::Mmap (but it's still slower than Cache::FastMmap) and 2) in light of #1, it's surprising that Cache::TieredLmc with a local memcached cache) would perform so badly. Somehow, the value that's being returned by the local memcached instance is getting blanked before it gets out of the if loop that fetches it, even though there's nothing that looks like it should be doing it. I'm including the code in the tarball for the local memcached tiered cache, in case some enterprising soul is able to fix it. I've looked, but can't take the time to figure it out. The mmap versions perform as expected, fetching the value and keeping it around long enough to return without hitting the remote memcached. Tiering a remote memached with the mmap improves performance, which makes the Cache::TieredLmc results so surprising, since the local memcached by itself performs so well.