more threaded io 6

Posted by jan Thu, 01 Feb 2007 11:01:00 GMT

After a long night we finally have everything in place for a threaded stat() calls. Not only that, we also have a new network backend for all those platforms which have problems with the posix-aio on. You need to have glib2-2.6.0 or higher installed.

The new options are: 
server.max-stat-threads = 4
server.max-write-threads = 8
server.network-backend = "gthread-aio"

Depending on the backend, your OS and the number of disks you might want to raise the two values, but keep in mind that you will get problems if you raise them too much. Performance will decrease again at a given point.

The performance of the different backends is: linux-aio-sendfile, posix-aio, gthread-aio, ...

On the way linux-aio-sendfile and posix-aio should behave better under high concurrent load now. They even got some stats:

server.io.linux-aio.async-read: 1261
server.io.linux-aio.sync-read: 551

Time for benchmarks, check my earlier article about lighty-1-5-0-and-linux-aio and try to generate the same set of testfiles and take http_load to generate random load. It is important that you use more files then you can cache in memory.

Trackbacks

Use the following link to trackback from your own site:
http://blog.lighttpd.net/articles/trackback/3259

Comments

Leave a response

  1. Pedro Melo Thu, 01 Feb 2007 16:07:43 GMT
    In a future kernel, you might find this interesting: http://www.ussg.iu.edu/hypermail/linux/kernel/0701.3/2165.html
  2. tbone Thu, 01 Feb 2007 20:22:19 GMT
    Jan, I'll try to get some benchs this weekend for you using a large number of disks in a situation similar to how we'd be running it live (ie. millions of small (10k) files).
  3. Curt Fri, 02 Feb 2007 17:49:57 GMT
    http://lighttpd.net is down (1:48 PM EST, Friday, Feb 2). Does anyone know how to contact an adminstrator? Thank you.
  4. Jan Kneschke Fri, 02 Feb 2007 23:54:22 GMT
    We are moving www.lighttpd.net to this host and are waiting for the DNS to be updated. The old host got already disabled. Bad timing from my side.
  5. Jürgen Herz Sat, 03 Feb 2007 09:07:44 GMT
    Being just an interested amateur I don't have the hardware to do benchmarks myself. But I'm very curious in the improvements regarding aio and threaded stat(). Though not concerning the latter I'd like to know something about the impact on performance of the new 16k lower limit for aio. So I'm particularly interested in benchmarking static delivery throughoutput with files of 16383 bytes in one run and files of 16385 bytes in another run.
  6. Jan Kneschke Sun, 04 Feb 2007 18:05:08 GMT
    I benchmarked it now with 10k files and got 1.8MByte/s with async-read() and 800kbyte/s with sync-read. I lowered the threshold to 4kbyte. That should match the size of the bytes that we got with the read-ahead from the stat() call. ...
Comments