lighty's life

lighty developer blog

Delay Request Handling for Stupid Crawlers

I’m sure you know what “Crawl-Delay” is, but you may or may not know that, not all search engine crawlers support this nice stuff.

What to do for those don’t obey the instrustion? They’ll eat all your Mbits/month or slow your webserver down. OK, ban it with url.access-deny. This is the only option u can choose before. But you don’t want to remove your pages from the stupid search engine index, do you?

Here comes another option for you: with this patch, u can delay handling of a specified request for some seconds. Example configuration:

$HTTP[“user-agent”] =~ “stupid-crawler” { connection.delay-seconds = 2 }

OK, here’s the link to the lighttpd-2296-request-handle-delay.patch which applies to branches/lighttpd-1.4.x@2296

Be aware that this patch is to be reviewed before commited to repo.

Growing the Team

The dev team has gotten some new members recently that you might not know of. Here’s a heads up (in alphabetical order):

  • hoffie
  • icy
  • nitrox
  • stbuehler

If you hang out in #lighttpd, you will already know these names.

That being said, there is also some news about the state of the development of lighty.

Lighty is a great webserver and we all love it. But if you use it for quite some time, you’ll eventually find out that it’s not perfect. There are some oddities and shortcomings that cannot be ironed out without rewriting a significant part of the core.

This is where I had the idea to start all over.
With lessons learned from the past, we could write a new version of lighty that fixes the design issues the current versions have. But at the same time keep all the good stuff and remain fairly compatible.
The main idea to be fast and lightweight is still valid and if you are familiar with 1.4.x or 1.5, you will have no problems with the new version.

Jan had the idea to use glib for stuff like strings/buffers and arrays. He wanted to rewrite the corresponding parts and did so in a branch of his.
I proposed to take this even further and use glib throughout the source. It makes coding easier, faster and in the end more secure because you can rely on proven and well tested source.

stbuehler jumped in and created a new branch where he started to hack on an all new lighty using the ideas from above. The following days we discussed a lot – and still do so – about the new design. What we could do better, what shortcomings we wanted to fix.

The result is nothing less but a more flexible and faster design for lighty.

This sounds awesome but where is the catch you might think.
Well, currently we have not too much code ready but we are actively hacking on it. Do not ask when it will be released, there is no estimated date. We are still at the very beginning of the new version but hope to have it some day become the official Lighttpd 2.0

We created a page that lists some of the (technical) plans we currently have. You might find some of them interesting.

This is it for today folks. Hope you like the route we are taking and maybe drop us a line in #lighttpd or in the comments.

- update -

To make it clear: 1.5 or 1..4 have not been dropped. If you think 1.5 is unstable and you get a segfault or something like that, then tell us. File a ticket, attach a stacktrace and document the problem in a detailed way so it can be fixed. Thanks for using lighty.

GSoC: Collecting Ideas

Google is doing a the Summer of Code again and I was approached from different sides to participate this year. There is already a lot going on in SVN again working towards a new 1.4.19 release, but for the future we want to attract more developers for more new, great features.

Before I submit the application request I would like to collect some more ideas what we would like to see to be done. At GSoC2008/Ideas I added my short list of projects that I would like to mentor.

If you would like to be a mentor too, please add your own list.

  • Documentation improvements
  • Integration into web-panels
  • maintaining the OpenWRT port

are just some small ideas where we could take help from non-developers.

Project Glib: It Compiles ... Mostly

Part of my ongoing, hidden effort to port lighttpd to glib I can now at least say: it compiles.

Well, mostly. The core does compile and mod_staticfile is also mostly done. Mostly, as I don’t have decided on how to handle the config-file handling yet.

Weekends Projects: Lua as Config-language

With the first implementation of mod_cml and later mod_magnet smart users asked why we don’t use lua also as config language for lighttpd.

Up to now I was pushing back and saying that a huge change in the configfile would make upgraders very unhappy as they would have to rewrite the config files. Looking at what we have for 1.5.0 right now, we need tweaks anyway.

Weekend Projects: Killing buffer.c and array.c

As part of a weekend project I’m ripping 1.5 apart a bit. I worked with glib for quite some time now and feel the need to replace the base-structures around “buffer” (strings) and “array” (hash) apart and replace it with the glib counter-parts.

To make this a nice experience and that I don’t harm anyone else I imported trunk into a local bzr tree:

bzr clone svn://

to clone the SVN tree.

Let’s see how well that experiment goes.