lighty's life

digg.com is currently having lighttpd.net on the front-page It looks like del.icio.us got also “infected” and its users joined right away

The rails-based mephisto-blog which is driving this site is generating static content which is served by lighty. Thanks to darix work on clean-url and mod-magnet impact on the site is minimal.

$ tail -100000 .../access.log |\
   grep -v 'light_button' | \
   awk -F ' ' '{ print $11 }' |\
   grep -v '\(kneschke\|incremental\|lighttpd\.net\|/#"$\)' |\
   sort |\
   uniq -c |\
   sort -r -n | less

   1081 "http://digg.com/" 
    963 "http://www.digg.com/" 
    615 "http://digg.com/linux_unix/lighttpd_Fast_Secure_and_Flexible_Web_Server" 
     84 "http://digg.com/news" 
     78 "http://www.digg.com/linux_unix/lighttpd_Fast_Secure_and_Flexible_Web_Server" 
     58 "http://www.digg.com/view/technology" 
     45 "http://www.digg.com/news" 
     42 "http://del.icio.us/" 
     35 "http://digg.com/view/technology" 
     33 "http://digg.com/view/all" 
     30 "http://www.digg.com/view/all" 
     21 "http://doggdot.us/" 
     15 "http://news.netcraft.com/" 
     14 "http://digg.com/linux_unix" 
     12 "http://www.digg.com" 
      8 "http://www.planetmysql.org/?offset=60" 
      8 "http://www.digg.com/linux_unix" 
      8 "http://diggriver.com/" 

Netcraft reports:

This month the Web Server Survey adds public tracking of lighttpd, an open source server designed for high-performance sites that has been gaining popularity in recent months. Lighttpd is currently detected on 1.38 million sites for a 1.2% share of the web server market, well ahead of Zeus and moving up quickly on Sun. Lighttpd has a relatively small memory footprint and is optimized for a large number of parallel connections, which has made it popular on sites using applications based on AJAX or Ruby on Rails, or hosting environments for virtual private servers.

The news got picked up by:

• http://punto-informatico.it/p.aspx?id=1944508
• http://www.golem.de/0704/51464.html
• http://lwn.net/Articles/228850/
• http://www.linux-magazin.de/news/webstatistik_lighttpd_betritt_die_buehne

With the release of 1.5.0 we aim for higher ranks and try to get ahead of Suns iplanet. You always need a goal: World Domination. :)

After getting covered with benchmark and stumbling over time-outs reported by http_load it was time to look a bit deeper into the problem.

http://www.lighttpd.net/2007/3/5/thread-starvation talks about how you can reduce the probability of time outs reported to the user. With the help of a new timing infrastructure you can now track the time spent a different stages of a request. With the help of gnuplot you can get a feeling of where the time is spent.

To make the long story short: If you use one of the async-io backends in 1.5.0 you want to set the server.max-read-threads to twice the number of disks.

A new pre-release on the road to stabilize 1.5.0.

http://www.lighttpd.net/download

Most of the work went into mod-proxy-core and new platforms:

• m-p-c can handle uploads via APJ13 which required some larger patches
• we should compile fine on solaris and IRIX again
• kqueue should work too again
• we compile natively on win32 (staticly and dynamicly)

The is the first official release that compiles natively on win32. Read the instructions on docs/build-win32.txt if you want to build yourself. It even starts :)

I have to say thanks to all the testers on the IRC channel, to jtiai for the win32 patches and jakabosky for the hard work on mod-proxy-core.

Kiel, Germany. – ... how to start such a press-release ?

According to netcraft’s server-survey Feb’07 lighttpd is now on #6 and has now a market-share of 0.6%. This is more then 700,000 domains which are handled by lighttpd. Last month we were at #12 with 170,000 domains.

image provided by Michael Hayes

Even if I don’t trust this impressive raise I’m none-the-less impressed by it. What comes next ? If we really make it into the top5 for a longer time I will need someone who knows how to write a press-release.

A lot of changes made it into the svn recently. It is time again to spread the code to more testers than just the few early adaptors who are using the svn-trunk/

deflate.mimetypes = ( "text/html" )

Download:

updated lighttpd-1.5.0-r1605.tar.gz

ChangeLog (r1605):

• added check for leading slashes in mod_rewrite
• improved async-io performance for files between 4k and 16k
• fixed compression of static files in mod_deflate
• fixed configure check for the library providing aio_read()
• fixed moddirlisting, modstaticfile and the internal error-pages for mod_deflate
• fixed compilation on FreeBSD and MacOS X

• added O_NOATIME support to the network-backends linux-sendfile and writev
• added a portable, threaded network-backend based on glib’s gthread
• added threaded stat()
• added url.redirect-code to mod_redirect to set other HTTP status-codes than 302
• added filter-API and mod_chunked and mod_deflate for dynamic compression
• added a static balancer for mod-proxy-core
• added $HTTP[“request-method”] and $PHYSICAL[“path”] conditionals
• fixed X-Sendfile support in mod-proxy-core
• fixed crash if mtime is 0
• added cmake as experimental build-system
• fixed urls in AJP13-protocol of mod-proxy-core
• added support for “now” and “weeks” to mod-expire
• added mod-magnet

As you may have noticed, www.lighttpd.net got moved to a new host, sharing the same box as all the other lighttpd.net domains. It is using mephisto as its content-backend to give you the possibility to comment if you want.

I added a first, full benchmark for Async IO and optimal parameters at http://www.lighttpd.net/2007/2/3/raw-io-performance

After a long night we finally have everything in place for a threaded stat() calls. Not only that, we also have a new network backend for all those platforms which have problems with the posix-aio on. You need to have glib2-2.6.0 or higher installed.

server.max-stat-threads = 4
server.max-write-threads = 8
server.network-backend = "gthread-aio" 

Depending on the backend, your OS and the number of disks you might want to raise the two values, but keep in mind that you will get problems if you raise them too much. Performance will decrease again at a given point.

The performance of the different backends is: linux-aio-sendfile, posix-aio, gthread-aio, ...

On the way linux-aio-sendfile and posix-aio should behave better under high concurrent load now. They even got some stats:

server.io.linux-aio.async-read: 1261
server.io.linux-aio.sync-read: 551

Time for benchmarks, check my earlier article about lighty-1-5-0-and-linux-aio and try to generate the same set of testfiles and take http_load to generate random load. It is important that you use more files then you can cache in memory.

Just as a proof of concept I implemented a threaded stat() call. It is a bit of a hack currently, but it looks promising when I look at the performance data:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.00    0.00   26.60   68.40    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   0.60 66.90  1.60 13019.20   22.40     6.36     0.01   190.39     6.10   88.20  14.49  99.28
sdb          0.00   0.60 66.60  1.60 13061.60   22.40     6.38     0.01   191.85    14.09  208.82  14.67 100.04

In http://blog.lighttpd.net/articles/2007/01/27/accelerating-small-file-transfers we tried the same without a async stat() and with fcgi-stat-accel. With the threaded stat() I moved the code into lighttpd itself which reduces the external communicating and manages everything in lighttpd itself.

name              Throughput  util% iowait%
----------------- ------------ ----- ------------
no stat-accel     12.07MByte/s 81%  
stat-accel (tcp)  13.64MByte/s 99% 45.00%
stat-accel (unix) 13.86MByte/s 99% 53.25%
threaded-stat     14.32MByte/s 99% 68.40%

(larger is better)

gpointer *stat_cache_thread(gpointer *_srv) {
        server *srv = (server *)_srv;
        stat_job *sj = NULL;

        /* take the stat-job-queue */
        GAsyncQueue * inq = g_async_queue_ref(srv->stat_queue);
        GAsyncQueue * outq = g_async_queue_ref(srv->joblist_queue);

        /* get the jobs from the queue */
        while ((sj = g_async_queue_pop(inq))) {
                /* let's see what we have to stat */
                struct stat st;

                /* don't care about the return code for now */
                stat(sj->name->ptr, &st);

                stat_job_free(sj);

                g_async_queue_push(outq, sj->con);
        }

        return NULL;
}

Thanks to some help from a irc-channel (#lighttpd at irc.freenode.net) we solved another long-standing problem:

As lighttpd is event-based web-server we have problems when it comes to blocking operations. In 1.5.0 we add async sendfile() operations which helps for large files alot. For small files most of the time is spent on the initial stat() call which has no async interface.

Fobax submitted a nice solution for this problem: move the stat() to a fastcgi app which returns with X-LIGHTTPD-send-file: and hands the request back to lighttpd. The fastcgi can block and spend some time while lighttpd moves on the with other requests. When the fastcgi returns the information for the stat() call is in the fs-buffers and lighttpd doesn’t block on the stat() anymore.

All this is documented by darix in the wiki at HowtoSpeedUpStatWithFastcgi

This works with mod_fastcgi in 1.4.0 or with mod-proxy-core in 1.5.0 + aio.

$ ./build/spawn-fcgi -f ./build/fcgi-stat-accel -p 1029 -C 1

$SERVER["socket"] == ":1025" {
  $HTTP["url"] =~ "^/seek-bound/" {
    proxy-core.protocol = "fastcgi" 
    proxy-core.backends = ( "127.0.0.1:1029" )
    proxy-core.allow-x-sendfile = "enable" 
    proxy-core.max-pool-size = 20
  }
}

$ http_load -parallel 200 -seconds 60 urls.100k

$ iostat -xm 5
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.20    0.00   45.80   45.00    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   0.00 73.00  0.00 13278.40    0.00     6.48     0.00   181.90     7.09   98.30  13.71 100.08
sdb          0.00   0.00 69.20  0.00 12625.60    0.00     6.16     0.00   182.45    13.63  194.71  14.46 100.08

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.19    0.00   38.56   53.25    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00   1.00 67.63  4.30 12533.07   47.95     6.12     0.02   174.91    10.28  144.44  13.89  99.90
sdb          0.00   1.00 66.13  4.30 12442.76   47.95     6.08     0.02   177.35    11.92  168.46  14.18  99.90

name             Throughput    util%
----------------- ------------- ---------
no stat-accel     12.07MByte/s 81%
stat-accel (tcp)  13.64MByte/s 99%
stat-accel (unix) 13.86MByte/s 99%