The benchmarks only showed results for small files (100kbyte). Time to add larger files to the pool and talk about the chunk-size.

I just push all the work to the kernel and hope that it does it right.
Currently I allow 64 jobs to be pushed to the kernel. Kernel threads are
more light-weight that “real” threads.

Currently I’m working on a posix AIO version. On linux that is using
threads to handle the read(), let’s see how that works out.

I did a third benchmark round against 1000 10Mbyte files. tibco @ IRC is
running a flv-site in china and said that their files are around 12-17Mb.

Client was a win2003-amd64, dual core box connected via Intel Pro/1000
to the server [raid1 … as before].

linux-aio-sendfile: 52Mbyte/s [reading 1Mbyte chunks]

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.80    0.00   46.20   13.40    0.00   38.60
linux-aio-sendfile: 55Mbyte/s [reading 768kbyte chunks]
avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           2.99    0.00   56.37    4.58    0.00   36.06
linux-aio-sendfile: 58Mbyte/s [reading 512kbyte chunks]
avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.40    0.00   62.67    5.39    0.00   30.54
linux-aio-sendfile: 54Mbyte/s [reading 384kbyte chunks]
avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           5.18    0.00   55.38    1.99    0.00   37.45
linux-aio-sendfile: 21Mbyte/s [reading 256kbyte chunks]
avg-cpu:  %user   %nice %system %iowait  %steal   %idle

          21.00    0.00   28.60    0.80    0.00   49.60

Compared to:

linux-sendfile: 30Mbyte/s

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           1.20    0.00   22.20   71.00    0.00    5.60 

Summary

No matter what, large files or small files, when you disk start to suffer from seeking around AIO will give you, at least in my setup, 80% more throughput.