lighty's life

It seems to me that delaying the request by n seconds will not change the overall rate of crawling - it'll just offset the responses by n seconds, and increase the concurrent connections (and thus memory usage) to the server during that time.

Or is there some aspect of the patch that I'm missing?

Also - I seem to remember that some crawlers take site response time into account when ranking results. If that is the case, this could make the site look awfully slow to the crawler.

That said - I like the idea :) But perhaps more of a rate-limiting approach would work better?

I love the idea too. can we have a first-byte.delay-seconds? or if the rate of a same IP / Useragent is too high, we can randomly return http code 503? I think it wont hurt.

in reply to "Or is there some aspect of the patch that I'm missing": this patch simply just delay n seconds for each individual request, regardless of ip nor connection (keep-alive). i know it's far away to be graceful against the dirty clients from the world. if you need more advanced version, something like request priority may have to be implemented, and score algorithm can be configured or move to plugins blahblah. the hard part is not about implementing the "idea" part in c code, but to design the alghorithm, and write support functions, find code inject points, or even rewrite stuffs, trying to make it the best practical frameworks however i just made a simple version and i hope it's somehow effective

reply to "It seems to me that delaying the request by n seconds will not change the overall rate of crawling - it'll just offset the responses by n seconds, and increase the concurrent connections (and thus memory usage) to the server during that time" this may not be true, i don't think any crawler will open 1024 connections to you. well, i mean, they can make less connections if you're fast, or more if you're slow, but they'll open __at most__ like 10 or 20 connections no matter how slow your server responses. or they even open 20 connectios to you no matter how fast it is. how knows. connection resource is cheap in lighttpd itself because it's just light. this is not true in apache/php/mysql. i once have usleep(200) or so in php side for specified bots, it helped a lot reducing system load, but yes it waste some memory by idling as php is heavy. as i said, this patch is to deal with bots, not evil guys who use bot as a tool and adjust the concurrent connection count. u can analyze most of the crawler bots that hit your server and figure out their patten. limit then each individually, e.g.: for bot a which opens 20 connections, delay each request for 20 seconds, so it is 1 req/s in average. for bot opens 3 connections, delay 6 seconds and it will be 6 req/s. perhaps in one day i can implement a score table so it can be sum up for each ip, u can just set n req per m seconds in config.

"or if the rate of a same IP / Useragent is too high, we can randomly return http code 503? I think it wont hurt. " - u may try mod_evasive.c which does not support "randomly" however. yet i don't think "randomly" is meanful here.

awesome hint to stop the stupid MSN crawlers as they are still running amok on my site. they produce more hits than my normal users :)

I found mod_evasive (in lighty) is not using a hashtable. I did a benchmark with apache 2.0+mod_evasive & lighty with mod_evasive, in a low concurrent connections both works well, if i have many connections, apache works great but lighty works poor. @mOo: http code 503 is only for Robots :) & if it received a http 503, it will know it exists and perform a crawl later.

it is not using hash table or so, but the timer timeout and timer for connection is not either. see trunk/src/server.c if (handle_sig_alarm) { ... for (ndx = 0; ndx used; ndx++) {

Would it kill you to type out you as opposed to "u"? You come across as a young texting ninja that is out of touch with the english language.

Sorry but I don't get it.
m0o has done a great job creating a nice open source application that you get without paying. And you complain about his usage of "u" instead of "you"?
What about saying "thank you" instead? :)

mOo,
Thanks for creating lighttpd. It's really nice. Typing 'u' instead of 'you' isn't endearing you to anyone and I fear that it would give the impression to some folks that this was put together by an inexperienced, immature kid who couldn't possibly write anything of value.

i.e. - it could needlessly hinder uptake and I'd hate to see that happen to such a fine software package.