Compression of dynamic content 11
It looks like a few changes won’t make it into trunk/ before I leave for vacation. But you should know what is in the pipeline and what you want to wait for:
- HTTP Response filtering is implemented
- HTTP/1.1 chunking becomes a module
- compression of dynamic content
This will add compression not only for mod_proxy_core and its backends (FastCGI, SCGI, HTTP, AJP13) but also to internally generated content like the directory listings.
With these changes we will become more and more stream based. Or like JDD called it: The Web is a Pipe
Trackbacks
Use the following link to trackback from your own site:
http://blog.lighttpd.net/articles/trackback/2620
Im sorry, there seems to be a problem leaving comments on firefox =(
Just to say, someone mentioned caching of dynamic pages. My opinion is its prolly not best to do any sort of caching on dynamic text.. Its normally better practise to make faster code than cache sections of it.
However, What i do thing would be a good idea is to determine what is dynamic PHP and static PHP, for example I use PHP files for all my pages, not just dynamic ones. So we could cache anything that does not use a function except echo and print.
Simple =)
e.g.
(a PHP file with no code sections)
<?php $c = 0; $c++; echo $c; ?>
<?php echo "testing..."; ?> can all be cached.
<?php echo rand(); ?> can't.
This gets alittle more complex when you start to think about includes and if statements involving external arguments. but im sure there are solutions for them too.
That is the only case of caching I can think of. I dont know how caching is handled, but i presume that if the cachefile exists it will send it, if the cache file doesnt its tested and cached if pos, so this could easily be implimented.
Anyway, This original thread is about using temp files to send large amounts of data to the core from swarm-fcgi. I believe that the current IPC is via Unix Sockets, which are handled by the kernel and should be the fastest method available. I have a little understanding on the current method used, but what i think would be the fastest would be to
Create a triangle pipe/IPC system, something like
1. Fork() "forwarder"
2. pair TCP/Unix socket with a Pipe[out]
3. Pipe[in] send request to fcgi
4. fcgi send output to "forwarder" on Pipe[out]
5. The "forwarder" thread passes on the information recieved to the TCP/Unix socket.
I had a quick read on mod_fastcgi.c but its a pretty big file. From what i could understand is that this module forks off anyway, and runs the CGI scripts. If so, you could just forward two fd handles to the fork(), 1 for the TCP output, and 1 for the PIPE errors/cmds. This would mean that the "core" would get the page request, forward the fd handle to the cgi module, the cgi module would do its stuff in its own thread, once complete it would send the result down the TCP connection and either close it, itself or hand it back over to the core to close.
Mike
P.s, Give vfork() a go instead of fork(). This should save some memory on each cgi thread as it shares memory and thread's until execve(). It will require removing the code that clears the fd handles etc.
http://www.hmug.org/man/2/vfork.php
- new filter chain API
- mod_chunked (HTTP/1.1 chunked encoding)
- mod_deflate (dynamic compression)
New response filtering modules can be made easily now.mod_ssi (server side includes) should be converted to a filter, that way any dynamic content can include static headers/footers.
For some websites the backend has to do a lot of work. So moving some of the cpu work to the frontend will help balance the work load.
Also mod_deflate can be turned on/off for any mime type, url or user agent.