Threaded stat() 4

Posted by jan Wed, 31 Jan 2007 11:00:00 GMT

Just as a proof of concept I implemented a threaded stat() call. It is a bit of a hack currently, but it looks promising when I look at the performance data:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.00    0.00   26.60   68.40    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   0.60 66.90  1.60 13019.20   22.40     6.36     0.01   190.39     6.10   88.20  14.49  99.28
sdb          0.00   0.60 66.60  1.60 13061.60   22.40     6.38     0.01   191.85    14.09  208.82  14.67 100.04

In http://blog.lighttpd.net/articles/2007/01/27/accelerating-small-file-transfers we tried the same without a async stat() and with fcgi-stat-accel. With the threaded stat() I moved the code into lighttpd itself which reduces the external communicating and manages everything in lighttpd itself.

name              Throughput  util% iowait%
----------------- ------------ ----- ------------
no stat-accel     12.07MByte/s 81%  
stat-accel (tcp)  13.64MByte/s 99% 45.00%
stat-accel (unix) 13.86MByte/s 99% 53.25%
threaded-stat     14.32MByte/s 99% 68.40%

(larger is better)

Implementation

in stat_cache.c I started a separate thread for handling the stat() call, 4 threads to be exact.

stat_cache_get_entry() checks its cache, if this file is already known. If not, it pushes the filename into the stat_cache_queue and returns HANDLER_WAIT_FOR_EVENT. On the other end of the stat_cache_queue is one of the 4 stat()-threads which runs the stat() and pushs the connection back into the joblist_queue. On the mainloop, just where the poll() call is started is now the handler for this queue which just actives all connections which are in this queue.

This way we made the stat() call itself async and can leave the rest of the code as is. Up to now we only get the inode into the fs-buffers as in the other examples, we are not handling the full stat-cache updates in the thread.

gpointer *stat_cache_thread(gpointer *_srv) {
        server *srv = (server *)_srv;
        stat_job *sj = NULL;

        /* take the stat-job-queue */
        GAsyncQueue * inq = g_async_queue_ref(srv->stat_queue);
        GAsyncQueue * outq = g_async_queue_ref(srv->joblist_queue);

        /* get the jobs from the queue */
        while ((sj = g_async_queue_pop(inq))) {
                /* let's see what we have to stat */
                struct stat st;

                /* don't care about the return code for now */
                stat(sj->name->ptr, &st);

                stat_job_free(sj);

                g_async_queue_push(outq, sj->con);
        }

        return NULL;
}
Trackbacks

Use the following link to trackback from your own site:
http://blog.lighttpd.net/articles/trackback/3254

Comments

Leave a response

  1. FreeDude Wed, 31 Jan 2007 13:37:32 GMT
    Great! This is IMHO a _much_ better solution that hacking it through the fastcgi interface. Now everybody benefits, without needing to complicate their site setup. Now all I have left to wish for in 1.5 is to automatically spawn the php fastgi processes :)
  2. Scott Wed, 31 Jan 2007 17:27:24 GMT
    I kind of like externally spawning fcgi processes. It means that making any changes to the lighttpd config and restarting doesn't also mean the overhead of killing and restarting fcgi handlers each time.
  3. Roze Sun, 04 Feb 2007 07:28:58 GMT
    Scott: how often do you need to restart your lighttpd? :) Once a year?
  4. Andre Bogus Mon, 05 Feb 2007 13:29:28 GMT
    Roze: If you develop with Lighty, you usually have a test environment, which might restart rather often...
Comments