lighty's life

I'd like to shine some light on ticket #1270, which seems to be a regression introduced as part of 1.4.16. This is causing all responses from a FCGI server.404-error-handler to have a 404 status code regardless of what's set by the downstream FCGI process. What this basically means for Rails apps (or anything else that requires the 404-error-handler) is that all responses end up having a 404 status. People are reporting this as a problem with redirects because browsers (rightly) choke when you try to redirect with a 404 but what everyone seems to be missing is that *all* their responses coming off of FCGI are 404: even the ones that look normal in the browser. There's been four separate tickets opened in the past four days that can be traced back to this issue: #1272, #1270, #257, and #948. I've been unable to comment or participate otherwise because my IP seems to be blacklisted. At any rate, we brought our staging machines up to 1.4.16, noticed the issue, and have decided to hold off until there's some resolution on this issue. I'm quite surprised this issue hasn't been reported more heavily since the 1.4.16 release. I can't see how any of the lighttpd+fcgi+rails configurations could have possibly upgraded to 1.4.16 and not noticed the issue.

Thanks, Ryan, for your post. You've saved me a lot of pain with our Rails systems! Here's to 1.4.17...

Also, slightly off topic but, what's the process for getting off the "sc.surbl.org" blacklist. I've tried commenting in trac from four different IPs on distant netblocks and always get the rejected for suspected spam due to blacklist message. On the upside, I'm becoming fairly proficient at navigating trac using elinks :)

Ryan, have you tried this diff? http://trac.lighttpd.net/trac/attachment/ticket/1270/connections.c.diff I plan on trying this diff tomorrow. Busy with other things today.

Ryan, have you tried the diff posted by jeremyevans on bug #1270? I plan on trying this diff tomorrow. Busy with other things today.

Patrick: I haven't. I'm setting aside some time this weekend to investigate a couple of different options, including that patch. Unfortunately, it looks like the patch reintroduces another bug so it might not get any traction. From what I've been able to understand from reading the tickets, the bug reintroduced by the patch is much less of a problem than running without the security fixes in 1.4.16 so I'm probably going to run with it if nothing else shakes out.

I'm go glad to see a post on the blog.

It gives me hope that lighttpd is not dead. Any word on 1.5? I'm not asking for a specific "release date", just a time frame.

Will it be weeks away from releasing? Months away?

I am not familiar with rails but why are the apps mentioned relying on the 404-handler so much? I mean the 404-handler is meant to display your own (dynamic) 404 page and should not be used as a core component for your site imho. Don't get me wrong, I'm not denying that this header overwriting is bad but I'm also thinking that it's not that a big problem with the right application design.

icy: I've always thought the name "404-handler" was a poor description of what the thing is used for. Basically, any request that comes in that does not match a static file on disk gets put through the 404-handler. In a typical Rails config (if there is such as thing), the 404-handler is set up such that a collection of FCGI processes receives the request and generates a dynamic response. Basically, *all* dynamic requests go through the 404-handler. That's why I think the name is funny: I'd prefer to call it the "dynamic-request-handler" or some such.

The reason Rails apps tend to use the 404-handler is because the caching system is capable of writing dynamically generated responses to disk after the initial requests, which works well with this config. lighttpd serves the file from disk after the initial dynamic request (i.e., the 404-handler is never triggered after the first response for any given resource).

I should also note that I'm arguing from ignorance a bit since I don't have a great understanding of lighttpd's overall architecture. It could very well be that the 404-handler was intended only to be used as you've described above. It's use in the wild is far more wide ranging, however.

No doubt that using the 404-handler is a nice trick but you can achive the exact same behavior without using it. In my fastcgi engine setup, I rewrite every request to my fastcgi gateway which then decides if to create the page or get it from the cache (I use xcache for in-memory caching). That way, not a single 404 is ever raised. I'm not saying now that this is the only way to go but it is the one that worked for me for quite a long time now.