lighty's life: Optimizing Lighty for high-concurrent, large-file downloads

Optimizing Lighty for high-concurrent, large-file downloads

jan — Fri, 11 Nov 2005 14:28:00 +0000

In lighttpd 1.4.6 we have added some modifications for sites which have handle some 100 files in parallel with size of more than 100Mb each.

The problem in earlier releases was that lighttpd had to wait until the disk had seeked to the right place, read a few 100 kbyte to send it out. And this for each request as this scenario was completly trashing the disk-buffering. The IO-wait went sky-high and we were completly bound to the disk-io.

You could see this by running vmstat:

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  7 306852  51396    152 237004  632  492 34952  7531 6983 11677  3 25  0 72

You see, we have 72% io-wait and 25% are spent in the kernel doing something. Only 3% are spent in userland (lighty).

When we investigated the problem we layed out a plan:

/* Optimizations for the future:
 *
 * adaptive mem-mapping
 *   the problem:
 *     we mmap() the whole file. If someone has alot large files and 32bit
 *     machine the virtual address area will be unrun and we will have a failing
 *     mmap() call.
 *   solution:
 *     only mmap 16M in one chunk and move the window as soon as we have finished
 *     the first 8M
 *
 * read-ahead buffering
 *   the problem:
 *     sending out several large files in parallel trashes the read-ahead of the
 *     kernel leading to long wait-for-seek times.
 *   solutions: (increasing complexity)
 *     1. use madvise
 *     2. use a internal read-ahead buffer in the chunk-structure
 *     3. use non-blocking IO for file-transfers
 *   */

And that’s what we did step-by-step:

instead of mmap()ing the whole file we use a moving window of 512kb
we used madvise() to tell the kernel to load this area into memory NOW
we have a switch to use local-buffering instead of asking the kernel to do so

If you want to use it, set:

server.network-backend = "writev"

If you want to try out the local-buffering define LOCAL_BUFFERING in network_writev.c.

Up to now we don’t put any pressure on async-file-io as the coding impact is quite large and we don’t expect a big win over the current behaviour.

Both ways result in better caching the data we need which reduces the number of seeks and in the end reduces io-wait for us.

Problem solved, next one :)

"Optimizing Lighty for high-concurrent, large-file downloads" by Jamie

Mon, 03 Apr 2006 19:59:52 +0000

Albert, no, SSL requires a unique IP address (or at least a new instance of the server running on a separate port number). SSL cannot work as name-based. Remember that stunnel4 might be any easy way to do SSL if you're not too concerned about performance. Apache requires a unique IP per SSL instance as well. One way around this is to only have a single SSL certificate https://www.example.com/customername and then use mod_rewrite to redirect the request to /var/www/customername/ or something like that.

"Optimizing Lighty for high-concurrent, large-file downloads" by GM

Tue, 29 Nov 2005 01:12:24 +0000

Regarding large-file server troubles... :P When will be a function like this: http://trac.lighttpd.net/trac/ticket/208 developed?

"Optimizing Lighty for high-concurrent, large-file downloads" by Yusuf

Tue, 22 Nov 2005 05:20:17 +0000

Jan, People who are looking for virtual hosting with SSL using one IP are looking for Server Name Indication which is currently only supported by Opera and mod_gnutls. Patches are available for Mozilla. http://paul.querna.org/journal/articles/2005/04/24/tls-server-name-indication?po stid=70 http://paul.querna.org/journal/articles/2005/04/29/sni-support-in-mozilla

"Optimizing Lighty for high-concurrent, large-file downloads" by Albert

Fri, 18 Nov 2005 04:22:02 +0000

I currently use Mathopd, and I can attest to sendfile() being much better in almost all cases. zero-copy is excellent, and Mathopd runs about 3x faster on my current box than Apache 2.x. I will see if I can try Lighttpd and do some benchmarks of it vs Mathopd. I will be running some stuff that might _REQUIRE_ Apache, so all this might end up being moot.. It's actually shameful to me that Apache is the most popular web server in the world, it's slower than most..

"Optimizing Lighty for high-concurrent, large-file downloads" by Jan Kneschke

Thu, 17 Nov 2005 12:47:55 +0000

Peter, linux-sendfile is better 90% of the cases. Only if you suffer a high io-wait value in vmstat, you might want to try writev. *-sendfile offer zero-copy transfers which is just blazing fast. But as soon as you are disk-io-bound they have to wait too and reducing the wait-time becomes more important.

"Optimizing Lighty for high-concurrent, large-file downloads" by Peter

Tue, 15 Nov 2005 23:02:13 +0000

Jan, do you encourage "writev" and discourage OS specific network backend e.g. "linux-sendfile" in 1.4.7? "linux-sendfile" works for me - so should I step to "writev"? Thanks for lighty! :-)

"Optimizing Lighty for high-concurrent, large-file downloads" by Albert

Tue, 15 Nov 2005 10:33:29 +0000

Great question! But I always had one certificate, one name, doesn't that work?? www1.domainname.com = certificate 1 www2.domainname.com = certificate 2 No?? If I have virtual hosting, say: albert.domainname.com bob.domainname.com jack.domainname.com and I want all 3 to be accessable via https:// you are telling me that all 3 have to be on 3 different IP's? I don't think it true, otherwise, most of the hosting companies wouldn't be able to host SSL, because it's just one Apache server with a bunch of virtual hosts.. no?

"Optimizing Lighty for high-concurrent, large-file downloads" by Jan Kneschke

Mon, 14 Nov 2005 19:54:03 +0000

How does apache support name-based virtual hosting ? SSL does the certificate check BEFORE the Host: header is exchanged. It is a problem of the protocol and not a problem of lighty. TLS as the successor of SSLv3 turns it around: First the Host: header and a request to Switch to TLS afterwards. Up to now we don't support TLS.

"Optimizing Lighty for high-concurrent, large-file downloads" by albert@achtung.com

Mon, 14 Nov 2005 08:34:14 +0000

Problem solved, next one :) I have hestitated to try lighty because I need virtual hosting (only have 1 IP) but I need name based hosting AND I need it to support SSL. It says in the SSL docs that it only supports IP based, not name based. Is this still true?? If not, then the docs need to be changed, and I'll try out lighty. If so, there you go, you now have the next problem to solve :-) Basically, I only have 4 requirements for a webserver, and it's amazing to me that short of apache, almost nothing out there meets these requirements: 1) Free! 2) Name based virtual hosting 3) SSL support (also name based virtual hosting) 4) runs PHP I've looked at about 50 web servers, and so far, only apache meets these 4 simple criterias.. So if lighty supports name-based SSL, then I'm switching in a heartbeat..