Threaded stat()

jan — Wed, 31 Jan 2007 12:00:00 +0000

Just as a proof of concept I implemented a threaded stat() call. It is a bit of a hack currently, but it looks promising when I look at the performance data:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.00    0.00   26.60   68.40    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   0.60 66.90  1.60 13019.20   22.40     6.36     0.01   190.39     6.10   88.20  14.49  99.28
sdb          0.00   0.60 66.60  1.60 13061.60   22.40     6.38     0.01   191.85    14.09  208.82  14.67 100.04

In http://blog.lighttpd.net/articles/2007/01/27/accelerating-small-file-transfers we tried the same without a async stat() and with fcgi-stat-accel. With the threaded stat() I moved the code into lighttpd itself which reduces the external communicating and manages everything in lighttpd itself.

name              Throughput  util% iowait%
----------------- ------------ ----- ------------
no stat-accel     12.07MByte/s 81%  
stat-accel (tcp)  13.64MByte/s 99% 45.00%
stat-accel (unix) 13.86MByte/s 99% 53.25%
threaded-stat     14.32MByte/s 99% 68.40%

(larger is better)

Implementation

in stat_cache.c I started a separate thread for handling the stat() call, 4 threads to be exact.

stat_cache_get_entry() checks its cache, if this file is already known. If not, it pushes the filename into the stat_cache_queue and returns HANDLER_WAIT_FOR_EVENT. On the other end of the stat_cache_queue is one of the 4 stat()-threads which runs the stat() and pushs the connection back into the joblist_queue. On the mainloop, just where the poll() call is started is now the handler for this queue which just actives all connections which are in this queue.

This way we made the stat() call itself async and can leave the rest of the code as is. Up to now we only get the inode into the fs-buffers as in the other examples, we are not handling the full stat-cache updates in the thread.

gpointer *stat_cache_thread(gpointer *_srv) {
        server *srv = (server *)_srv;
        stat_job *sj = NULL;

        /* take the stat-job-queue */
        GAsyncQueue * inq = g_async_queue_ref(srv->stat_queue);
        GAsyncQueue * outq = g_async_queue_ref(srv->joblist_queue);

        /* get the jobs from the queue */
        while ((sj = g_async_queue_pop(inq))) {
                /* let's see what we have to stat */
                struct stat st;

                /* don't care about the return code for now */
                stat(sj->name->ptr, &st);

                stat_job_free(sj);

                g_async_queue_push(outq, sj->con);
        }

        return NULL;
}

"Threaded stat()" by Andre Bogus

Mon, 05 Feb 2007 14:29:28 +0000

Roze: If you develop with Lighty, you usually have a test environment, which might restart rather often...

"Threaded stat()" by Roze

Sun, 04 Feb 2007 08:28:58 +0000

Scott: how often do you need to restart your lighttpd? :) Once a year?

"Threaded stat()" by Scott

Wed, 31 Jan 2007 18:27:24 +0000

I kind of like externally spawning fcgi processes. It means that making any changes to the lighttpd config and restarting doesn't also mean the overhead of killing and restarting fcgi handlers each time.

"Threaded stat()" by FreeDude

Wed, 31 Jan 2007 14:37:32 +0000