lighty's life: delay request handling for stupid crawlers

delay request handling for stupid crawlers

moo — Fri, 22 Aug 2008 11:50:00 +0200

I’m sure you know what “Crawl-Delay” is, but you may or may not know that, not all search engine crawlers support this nice stuff.

What to do for those don’t obey the instrustion? They’ll eat all your Mbits/month or slow your webserver down. OK, ban it with url.access-deny. This is the only option u can choose before. But you don’t want to remove your pages from the stupid search engine index, do you?

Here comes another option for you: with this patch, u can delay handling of a specified request for some seconds. Example configuration:

$HTTP["user-agent"] =~ "stupid-crawler" {
    connection.delay-seconds = 2
}

OK, here’s the link to the lighttpd-2296-request-handle-delay.patch which applies to branches/lighttpd-1.4.x@2296

Be aware that this patch is to be reviewed before commited to repo.

"delay request handling for stupid crawlers" by Guy

Sat, 13 Sep 2008 00:16:19 +0200

mOo,
Thanks for creating lighttpd. It's really nice. Typing 'u' instead of 'you' isn't endearing you to anyone and I fear that it would give the impression to some folks that this was put together by an inexperienced, immature kid who couldn't possibly write anything of value.

i.e. - it could needlessly hinder uptake and I'd hate to see that happen to such a fine software package.

"delay request handling for stupid crawlers" by icy

Wed, 10 Sep 2008 23:12:09 +0200

Sorry but I don't get it.
m0o has done a great job creating a nice open source application that you get without paying. And you complain about his usage of "u" instead of "you"?
What about saying "thank you" instead? :)

"delay request handling for stupid crawlers" by master@masters.com

Wed, 10 Sep 2008 19:42:30 +0200

Would it kill you to type out you as opposed to "u"? You come across as a young texting ninja that is out of touch with the english language.

"delay request handling for stupid crawlers" by mOo

Thu, 28 Aug 2008 02:37:46 +0200

it is not using hash table or so, but the timer timeout and timer for connection is not either. see trunk/src/server.c if (handle_sig_alarm) { ... for (ndx = 0; ndx used; ndx++) {

"delay request handling for stupid crawlers" by fcicq

Mon, 25 Aug 2008 16:41:44 +0200

I found mod_evasive (in lighty) is not using a hashtable. I did a benchmark with apache 2.0+mod_evasive & lighty with mod_evasive, in a low concurrent connections both works well, if i have many connections, apache works great but lighty works poor. @mOo: http code 503 is only for Robots :) & if it received a http 503, it will know it exists and perform a crawl later.

"delay request handling for stupid crawlers" by Jan Berger

Sun, 24 Aug 2008 23:04:42 +0200

awesome hint to stop the stupid MSN crawlers as they are still running amok on my site. they produce more hits than my normal users :)

"delay request handling for stupid crawlers" by mOo

Fri, 22 Aug 2008 15:43:41 +0200

"or if the rate of a same IP / Useragent is too high, we can randomly return http code 503? I think it wont hurt. " - u may try mod_evasive.c which does not support "randomly" however. yet i don't think "randomly" is meanful here.

"delay request handling for stupid crawlers" by mOo

Fri, 22 Aug 2008 15:42:21 +0200

reply to "It seems to me that delaying the request by n seconds will not change the overall rate of crawling - it'll just offset the responses by n seconds, and increase the concurrent connections (and thus memory usage) to the server during that time" this may not be true, i don't think any crawler will open 1024 connections to you. well, i mean, they can make less connections if you're fast, or more if you're slow, but they'll open __at most__ like 10 or 20 connections no matter how slow your server responses. or they even open 20 connectios to you no matter how fast it is. how knows. connection resource is cheap in lighttpd itself because it's just light. this is not true in apache/php/mysql. i once have usleep(200) or so in php side for specified bots, it helped a lot reducing system load, but yes it waste some memory by idling as php is heavy. as i said, this patch is to deal with bots, not evil guys who use bot as a tool and adjust the concurrent connection count. u can analyze most of the crawler bots that hit your server and figure out their patten. limit then each individually, e.g.: for bot a which opens 20 connections, delay each request for 20 seconds, so it is 1 req/s in average. for bot opens 3 connections, delay 6 seconds and it will be 6 req/s. perhaps in one day i can implement a score table so it can be sum up for each ip, u can just set n req per m seconds in config.

"delay request handling for stupid crawlers" by mOo

Fri, 22 Aug 2008 15:35:37 +0200

in reply to "Or is there some aspect of the patch that I'm missing": this patch simply just delay n seconds for each individual request, regardless of ip nor connection (keep-alive). i know it's far away to be graceful against the dirty clients from the world. if you need more advanced version, something like request priority may have to be implemented, and score algorithm can be configured or move to plugins blahblah. the hard part is not about implementing the "idea" part in c code, but to design the alghorithm, and write support functions, find code inject points, or even rewrite stuffs, trying to make it the best practical frameworks however i just made a simple version and i hope it's somehow effective

"delay request handling for stupid crawlers" by fcicq

Fri, 22 Aug 2008 13:47:45 +0200

I love the idea too. can we have a first-byte.delay-seconds? or if the rate of a same IP / Useragent is too high, we can randomly return http code 503? I think it wont hurt.

"delay request handling for stupid crawlers" by Paul Annesley

Fri, 22 Aug 2008 13:14:14 +0200

It seems to me that delaying the request by n seconds will not change the overall rate of crawling - it'll just offset the responses by n seconds, and increase the concurrent connections (and thus memory usage) to the server during that time.

Or is there some aspect of the patch that I'm missing?

Also - I seem to remember that some crawlers take site response time into account when ranking results. If that is the case, this could make the site look awfully slow to the crawler.

That said - I like the idea :) But perhaps more of a rate-limiting approach would work better?