lighty's life

The Ranking

The Alexa Global Top 500 lists a few sites which are already known from PoweredByLighttpd in our wiki

#6 is youtube who is using lighttpd for sending out the static content (the images, the videos, ...). They use a patched version which handles their load better.

#12 is wikipedia. If you open en.wikipedia.org with firebug and check the “All->Net” tab, you will see upload.wikimedia.org. All the images-work is handed here: thumbnail generation, delivery.

#49 is imageshack.us

SourceForge.net is at #80. The whole site runs on lighttpd. Dynamic content via PHP and the usual static content.

#132 is sendspace.com a huge file-sharing site. They care a lot about concurrent access to a large set of large files. You don’t want to hear their disk-backends seeking. :)

On #221 we have our first brother from the family of torrent sites: mininova.org They are a long running lighttpd user and are leading the group:

• isohunt.com
• torrentreactor.to
• animersion.net
• torrents.md
• torrentz.com

... and those were just the ones I could find right away. :)

The Numbers

The December Survey from Netcraft talks about 178,619 detected servers. I know that quite a few of them are domain-hosters which are using a single IP to park a few thousand domain-names.

The Others

But who is behind those numbers ? I have contact to a some of owners of the sites mentioned above and want to write an article about the different users of lighttpd. How they are using it, what they would like to see, what they love, ... If you are interested and are listed in Alexa before my personal site drop me a mail at jan@kneschke.de or just add a comment with your mail-address below.

And who is on place #15,276 ?

Build Systems

Before you can build a C-program you first have to find out which functions the system you compile on supports. Not only that UNIX has various flavours, no, there is also Windows.

The natural way on UNIX is using autotools (autoconf, automake, ...) which creates a shell script (configure) which generates the Makefiles which are process by make to build the application.

Using automake reduced the pain of autoconf when it was released, but it none the less is still enough pain to look for something else. Not to forget that win32 and shell-scripts aren’t real friends.

scons

SCons going another route. It replaces make and the autotools by a python based build-system. You can do everything with just a few lines of python code. Very neat.

The bad side: its development is more or less sleeping. The unstable 0.96.9x branch which fixes various nasty bugs is unstable since 2004-08-22, the release of 0.96.1.

cmake

cmake is more relaxed and doesn’t want to solve the whole problem. It does the configure checks and leaves the building to the native build-system. On Unix it is either make or kdevelop3, on MacOSX it is XCode and on windows it is nmake.

As extra cmake supports basic packaging and has very nice integration with test-tools like Dart.

Cross-Compiling

As a proof-of-concept I’ve added cmake support to trunk used the openwrt SDK to build lighttpd with openwrt.

$ cmake .
$ rm CMakeCache.txt
$ OPENWRT=.../OpenWrt-SDK-Linux-x86_64-1/staging_dir_mipsel/ \
CC=$OPENWRT/bin/mipsel-linux-gcc \
LD=$OPENWRT/bin/mipsel-linux-uclibc-ld make

Command Line Options

$ cmake -L .
...
CMAKE_INSTALL_PREFIX:PATH=
...
WITH_BZIP:BOOL=OFF
WITH_MYSQL:BOOL=OFF
WITH_OPENSSL:BOOL=OFF
WITH_PCRE:BOOL=ON
WITH_SQLITE3:BOOL=OFF
WITH_WEBPAV_PROPS:BOOL=OFF
WITH_XATTR:BOOL=OFF
WITH_ZLIB:BOOL=ON
$ cmake -DWITH_BZIP:BOOL=ON \
-DCMAKE_INSTALL_PREFIX:PATH=/home/jan/l-1.5.0-cmake/

Static Linking

To make it easier for embedded systems which don’t have support for dlopen() I added the option:

$ cmake -DBUILD_STATIC:BOOL=ON .

It will build all the modules as static libraries which are linked into the server at build-time. server.modules is used to initialized the modules as before.

Large Response content

Robert already did some benchmarking and it looks like you the X-LIGHTTPD-send-temp-file helps if you have to send more than 16kbyte of content. Otherwise it fits into one FastCGI packet and into the TCP recv-buffers. Boring, right ? Let’s take the next step and move the idea a bit further.

Pre-generating content

At the railsconf in June Zed Shaw and I discussed on some ways to accelerate rails apps and funky Web2.0 apps in particular.

Usually you have dependencies between components. Let’s say you are writing a mail app and you are pooling the “you got mail” service and want to show a fade-in showing it and in the same way refresh the folder-list to show the new mail-count.

We want to play nice and only load the components which really have changed, so you have 1 to 3 XMLHTTPRequests:

1. the “you got mail” fade-in
2. if their is new mail, the the folder list gets updated
3. if the new mail is in the the current folder, the folder view gets updated too

In the AJAX world you have smaller requests, but all of them have to load the session first, before they can generate their content. What if you turn it around move the above logic back into the server and the “you got mail” loads the session once and generates the content for all 3 parts as it knows that app will ask for it anyway in the next 1-2 seconds ?

It would make the folder-list and the folder-view requests instant, as they just have to check if the content-file exists and can stream it out with X-LIGHTTPD-send-temp-file right away.

Now you just have to mix in some cleanup on mtime in case the temp-file is not fetched [perhaps memcached ?], have to find a way how to guess the filename without reading the session first [that’s the expensive part we want to get rid off], ...

Read Ahead

In a way this is like read-ahead on Hard Disks or prefetch in CPUs or even browsers

Hot or not ?

Ok, what do you think about this idea ? Do you have problems that could be solved this way ?