Time To Pull The Plug

This is a subtitle. There are many like it, but this one is here.

Intriguing Mod_mcpage Development Using a Local Memcached Instance

No patch yet, but I added a local memcached instance that only stores stuff for a few seconds to mod_mcpage to see how it worked, and got some very interesting benchmarks. My original plan had been to craft up a cache with POSIX shared memory (which is pretty sweet), semaphores, and Glib hashes, but I thought about it some and decided to just let memcached handle the housekeeping stuff, since with a custom shared mem cache I’d have to worry about expiring older stuff and removing unused entries myself.

In summary, though, being served from a paravirtualized one proc Xen VM with 128MB RAM, a local memcached instance of 16MB, and a remote memcached server with 64MB running on a fully virtualized one proc Xen VM with 128MB RAM (processor-wise, the Xen host has a Intel Core2 Duo E8500 @ 3.16GHz):

Running 1000 requests with a concurrency of 1 had 95% of all requests served in 3ms with both local and remote memcacheds, 95% within 17ms with just the remote memcached, and 95% within 780ms with no memcached at all.

Running 1000 requests with a concurrency of 5, with both 95% were within 135ms (80% within 9ms, 50% within 7ms), with just the remote 95% were within 90ms (80% were within 69ms and 50% were within 63ms), and no memcaching at all 95% were within 3766ms (80% were within 3621ms and 50% were within 3554ms).

Mean times for concurrency of one with both, remote only, and no memcached: 4.004ms, 13.878ms, and 778.267ms.

Mean times (across all requests) for concurrency of five with both, remote only, and no memcached: 3.926ms, 13.359ms, and 718.912ms.

Update: As I was cleaning up mod_mcpage to make a patch out of it in the relatively near future, I took out a bunch of random debugging stuff that was going into the log and re-ran ab again, this time with the index page of the site I was testing with using the tiered local + remote caches against the same file being served statically. Over 10,000 requests at a concurrency of one, the tiered cache had a mean time per request of 2.216ms. The static file had a mean time of 2.110ms. Over 10,000 requests with a concurrency of five, the tiered cache had a mean time per request of 1.120ms across all requests. The static file had a mean time of 1.025ms across all requests.

Not too bad. Doing it in shared memory would be blazingly fast, but who knows when I’ll have time to get all the little bits of that done. Something worth shooting for down the road though.

Full benchmarks (including the update) below the fold.

Running ab -n 1000 -c 1 xxxx:8888

With both local memcached and remote memcached:

Server Software:   lighttpd/1.5.0
Server Hostname:    xxxx
Server Port:        8888

Document Path:      /
Document Length:    105458 bytes

Concurrency Level:  1
Time taken for tests:   4.004 seconds
Complete requests:  1000
Failed requests:    0
Write errors:       0
Total transferred:  105719000 bytes
HTML transferred:   105458000 bytes
Requests per second:    249.76 [#/sec] (mean)
Time per request:   4.004 [ms] (mean)
Time per request:   4.004 [ms] (mean, across all concurrent requests)
Transfer rate:      25785.48 [Kbytes/sec] received

Connection Times (ms)
          min  mean[+/-sd] median   max
Connect:    0    0   0.6      0   9
Processing: 2    4  13.0      2 169
Waiting:    1    2  13.0      1 167
Total:      2    4  13.0      3 169

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      3
  90%      3
  95%      3
  98%      4
  99%     75
 100%    169 (longest request)

With just the remote memcached:

Server Software:        lighttpd/1.5.0
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105458 bytes

Concurrency Level:      1
Time taken for tests:   13.878 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      105719000 bytes
HTML transferred:       105458000 bytes
Requests per second:    72.05 [#/sec] (mean)
Time per request:       13.878 [ms] (mean)
Time per request:       13.878 [ms] (mean, across all concurrent requests)
Transfer rate:          7439.02 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    0   0.8      0      10
Processing:    11   14  10.0     13     300
Waiting:        9   12  10.0     11     298
Total:         11   14  10.1     13     300

Percentage of the requests served within a certain time (ms)
 50%     13
 66%     14
 75%     14
 80%     15
 90%     16
 95%     17
 98%     18
 99%     21
100%    300 (longest request)

With no memcaching (ab reports Apache rather than lighttpd because lighttpd is just proxying content back):

Server Software:        Apache/2.2.9
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105427 bytes

Concurrency Level:      1
Time taken for tests:   778.267 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      105739000 bytes
HTML transferred:       105427000 bytes
Requests per second:    1.28 [#/sec] (mean)
Time per request:       778.267 [ms] (mean)
Time per request:       778.267 [ms] (mean, across all concurrent requests)
Transfer rate:          132.68 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    1   2.5      0      20
Processing:   667  778 774.7    709   10898
Waiting:      656  765 774.8    697   10887
Total:        668  778 774.7    710   10899

Percentage of the requests served within a certain time (ms)
 50%    710
 66%    718
 75%    723
 80%    729
 90%    745
 95%    780
 98%    911
 99%    993
100%  10899 (longest request)

Running ab -n 1000 -c 5 xxxx:8888

With both local memcached and remote memcached:

Server Software:        lighttpd/1.5.0
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105458 bytes

Concurrency Level:      5
Time taken for tests:   3.926 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      105719000 bytes
HTML transferred:       105458000 bytes
Requests per second:    254.72 [#/sec] (mean)
Time per request:       19.629 [ms] (mean)
Time per request:       3.926 [ms] (mean, across all concurrent requests)
Transfer rate:          26297.67 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    1   0.6      1       4
Processing:     3   19  38.6      7     217
Waiting:        1    9  30.9      2     211
Total:          4   20  38.6      7     218

Percentage of the requests served within a certain time (ms)
 50%      7
 66%      8
 75%      9
 80%      9
 90%     30
 95%    135
 98%    152
 99%    157
100%    218 (longest request)

With just the remote memcached:

Server Software:        lighttpd/1.5.0
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105458 bytes

Concurrency Level:      5
Time taken for tests:   13.359 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      105779816 bytes
HTML transferred:       105518555 bytes
Requests per second:    74.86 [#/sec] (mean)
Time per request:       66.794 [ms] (mean)
Time per request:       13.359 [ms] (mean, across all concurrent requests)
Transfer rate:          7732.82 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    1   0.7      1      10
Processing:    34   66  27.2     63     385
Waiting:        9   35  20.7     35     341
Total:         35   67  27.2     63     385

Percentage of the requests served within a certain time (ms)
 50%     63
 66%     66
 75%     68
 80%     69
 90%     74
 95%     90
 98%    110
 99%    253
100%    385 (longest request)

With no memcaching (ab reports Apache rather than lighttpd because lighttpd is just proxying content back):

Server Software:        Apache/2.2.9
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105427 bytes

Concurrency Level:      5
Time taken for tests:   718.912 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      105739000 bytes
HTML transferred:       105427000 bytes
Requests per second:    1.39 [#/sec] (mean)
Time per request:       3594.560 [ms] (mean)
Time per request:       718.912 [ms] (mean, across all concurrent requests)
Transfer rate:          143.63 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    0   1.2      0      20
Processing:   678 3577 233.1   3554    7580
Waiting:      668 3565 233.0   3541    7568
Total:        679 3577 233.1   3554    7580

Percentage of the requests served within a certain time (ms)
 50%   3554
 66%   3582
 75%   3608
 80%   3621
 90%   3681
 95%   3766
 98%   3868
 99%   3999
100%   7580 (longest request)

Running ab -n 10000 -c 1 xxxx:8888/

With the local and remote memcaches, after removing the debugging statements going to the error log:

Server Software:        lighttpd/1.5.0
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105458 bytes

Concurrency Level:      1
Time taken for tests:   22.164 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1056910000 bytes
HTML transferred:       1054580000 bytes
Requests per second:    451.17 [#/sec] (mean)
Time per request:       2.216 [ms] (mean)
Time per request:       2.216 [ms] (mean, across all concurrent requests)
Transfer rate:          46567.37 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    0   0.5      0      20
Processing:     2    2   0.6      2      26
Waiting:        0    0   0.3      0      11
Total:          2    2   0.8      2      27

Percentage of the requests served within a certain time (ms)
 50%      2
 66%      2
 75%      2
 80%      2
 90%      2
 95%      2
 98%      3
 99%      4
100%     27 (longest request)

Running ab -n 10000 -c 1 xxxx:8888/testdex.html

The same file as above, but served statically.

Server Software:        lighttpd/1.5.0
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /testdex.html
Document Length:        105428 bytes

Concurrency Level:      1
Time taken for tests:   21.104 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1056640000 bytes
HTML transferred:       1054280000 bytes
Requests per second:    473.84 [#/sec] (mean)
Time per request:       2.110 [ms] (mean)
Time per request:       2.110 [ms] (mean, across all concurrent requests)
Transfer rate:          48894.47 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    0   0.4      0      10
Processing:     2    2   0.2      2      12
Waiting:        0    0   0.1      0      10
Total:          2    2   0.4      2      12

Percentage of the requests served within a certain time (ms)
 50%      2
 66%      2
 75%      2
 80%      2
 90%      2
 95%      2
 98%      3
 99%      3
100%     12 (longest request)

Running ab -n 10000 -c 5 xxxx:8888/

With the local and remote memcaches, after removing the debugging statements going to the error log:

Server Software:        lighttpd/1.5.0
Server Hostname:        xxxx
Server Port:            8888

Document Path:          /
Document Length:        105458 bytes

Concurrency Level:      5
Time taken for tests:   11.200 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1056910000 bytes
HTML transferred:       1054580000 bytes
Requests per second:    892.85 [#/sec] (mean)
Time per request:       5.600 [ms] (mean)
Time per request:       1.120 [ms] (mean, across all concurrent requests)
Transfer rate:          92154.27 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    1   1.5      1      30
Processing:     2    5   4.0      4      72
Waiting:        0    1   1.6      1      30
Total:          3    6   5.2      5      91

Percentage of the requests served within a certain time (ms)
 50%      5
 66%      5
 75%      6
 80%      6
 90%      7
 95%      7
 98%      8
 99%     14
100%     91 (longest request)

Running ab -n 10000 -c 5 xxxx:8888/testdex.html

The same file as above, but served statically.

Server Software:        lighttpd/1.5.0
Server Hostname:        10.250.55.15
Server Port:            8888

Document Path:          /testdex.html
Document Length:        105428 bytes

Concurrency Level:      1
Time taken for tests:   21.104 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1056640000 bytes
HTML transferred:       1054280000 bytes
Requests per second:    473.84 [#/sec] (mean)
Time per request:       2.110 [ms] (mean)
Time per request:       2.110 [ms] (mean, across all concurrent requests)
Transfer rate:          48894.47 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0    0   0.4      0      10
Processing:     2    2   0.2      2      12
Waiting:        0    0   0.1      0      10
Total:          2    2   0.4      2      12

Percentage of the requests served within a certain time (ms)
 50%      2
 66%      2
 75%      2
 80%      2
 90%      2
 95%      2
 98%      3
 99%      3
100%     12 (longest request)

Caveats: ab is not the perfect benchmarking software, these were just meant to get a rough idea how tiering the cache would perform, and it’s liable to have more tweaks done before a patch is made.