lighty's life: Growing the team

Growing the team

icy — Wed, 23 Jul 2008 14:43:00 +0000

The dev team has gotten some new members recently that you might not know of. Here’s a heads up (in alphabetical order):

hoffie
icy
nitrox
stbuehler

If you hang out in #lighttpd, you will already know these names.

That being said, there is also some news about the state of the development of lighty.

Lighty is a great webserver and we all love it. But if you use it for quite some time, you’ll eventually find out that it’s not perfect. There are some oddities and shortcomings that cannot be ironed out without rewriting a significant part of the core.

This is where I had the idea to start all over. With lessons learned from the past, we could write a new version of lighty that fixes the design issues the current versions have. But at the same time keep all the good stuff and remain fairly compatible. The main idea to be fast and lightweight is still valid and if you are familiar with 1.4.x or 1.5, you will have no problems with the new version.

Jan had the idea to use glib for stuff like strings/buffers and arrays. He wanted to rewrite the corresponding parts and did so in a branch of his. I proposed to take this even further and use glib throughout the source. It makes coding easier, faster and in the end more secure because you can rely on proven and well tested source.

stbuehler jumped in and created a new branch where he started to hack on an all new lighty using the ideas from above. The following days we discussed a lot – and still do so – about the new design. What we could do better, what shortcomings we wanted to fix.

The result is nothing less but a more flexible and faster design for lighty.

This sounds awesome but where is the catch you might think. Well, currently we have not too much code ready but we are actively hacking on it. Do not ask when it will be released, there is no estimated date. We are still at the very beginning of the new version but hope to have it some day become the official Lighttpd 2.0

We created a page that lists some of the (technical) plans we currently have. You might find some of them interesting.

This is it for today folks. Hope you like the route we are taking and maybe drop us a line in #lighttpd (freenode) or in the comments.

- update -

To make it clear: 1.5 or 1..4 have not been dropped. If you think 1.5 is unstable and you get a segfault or something like that, then tell us. File a ticket, attach a stacktrace and document the problem in a detailed way so it can be fixed. Thanks for using lighty.

"Growing the team" by hoffie

Fri, 15 Aug 2008 10:58:52 +0000

james: I don't know what memory leak you are talking about... 191919, james: Blog comments are not really ideal for bug/request tracking. Please either talk to us on IRC and see if it can be fixed immediately or file a ticket in trac (yes, we know it's slow sometimes).
Or do both -- file a ticket so we can track it and remind us on IRC :p

"Growing the team" by 191919

Thu, 14 Aug 2008 10:17:50 +0000

When 1.5 trunk can be compiled on Leopard 10.5.4?

autogen.sh complains AC_DEFINE not defined.

configure complains PKG_CHECK not defined.

Latest automake/autoconf/libtool installed.

"Growing the team" by james

Tue, 12 Aug 2008 03:07:45 +0000

Think we could have a .20 that patches the sendfile memory leak ?

"Growing the team" by lighty fan

Sun, 10 Aug 2008 03:56:34 +0000

Hope the dev's don't get to disheartened by some of the replies. Please keep up all your good work :) Who cares if there hasn't been a release in 5/6 month or that 1.5 isn't out of the door. As a developer I feel the pain, as I know you can only work on what your interested in. If Jan wants to rip the guts out and replace them, its his choice... If i don;t like it fork lighttpd or move to a different webserver. Me, I'm glad, as i don't want to upgrade my webserver every month to add a new bell or whistle.... although back porting the 1.5 fix for wordpress's image upoader would be nice :) Also can the dev's setup a mailing list so i don't have to keep coming back to check to see if a new release has been made. Thanks + good luck with 1.4/1.5 and 2.0 :)

"Growing the team" by icy

Thu, 07 Aug 2008 11:27:22 +0000

191919: I told you it will not be confusing. You don't know how our new config system works yet :)

john_smith: thanks again for your lengthy feedback!
about GTK: sure you don't have GTK on a server but that wasn't what I meant to say. glib is only part of this bigger project like FireFox is part of Mozilla. It means there are many developers working on it and it will not disappear from one day to the other.

About the dependencies etc: yesterday stbuehler did again some heavy work on 2.0 and implemented static compiling. The resulting binary was 1.9 mb on my system. Mind you: it's a 64bit box and there were NO optimizations to make the filesize smaller. On a 32bit box it's about 1.5 mb. If you compile only the really needed stuff in, it will be smaller.
Sure there is a lot of code to come but that will not change this size too much because the biggest part is the base and the libs. That should be small enough to fit on most embedded devices. We wont make our time harder by coding everything on our own and optimizing that just to make it run on some tiny devices more. It's a tradeoff you have to make. There are always even smaller devices where people want lighty to run on.
Our target systems are anything from your average home router to big servers with 10gbit link and thousands of requests per second.
Long story short: we want to scale.
And btw: the current build uses about 1.5mb RAM (again: 64bit box, less on 32bit)

The buffering: 2.0 will probably make it configurable, the default behaviour is a thing to discuss but shouldn't concern you too much then ;)

P.S. appretiate the mumbling

"Growing the team" by 191919

Thu, 07 Aug 2008 10:11:06 +0000

icy: ordering plugins in config file sometimes is confusing. esp. when several 3rd party plugins are used together.

"Growing the team" by mlx

Wed, 06 Aug 2008 20:16:49 +0000

> We will surely not be the next FireFox that gets started to be lightweight and ends up as bloat.

best line ever :-)

P.S. especially as some of my earlier posts might sound a bit offensive or negative: thanks to everyone contributing to lighttpd. it's much appreciated. i'm still in love with lighty ;-)

"Growing the team" by john_smith

Wed, 06 Aug 2008 18:09:11 +0000

> development for 2.0 is _NOT_ at the cost of development for 1.x
This is known to you.But others can see only the fact that main page of site didn't changed anyhow within half year or so and even blog is quite silent, lacking any news.Oh yeah, I was interested to find truth and found that someone is still alive in comments. I bet that at very best only each 20th visitor will figure this out.

> Others are studying and can't quit just like that too.
Yep, true. But again, being #4 server in the world is not just a word. Many and many people will die without even having chance to achieve something like that where only 3 big names are ahead, so you can be very proud by this fact. So if you're really like this project (or why it has appeared and and so nice and even free?) you may want to think on paying it more attention and with some luck it may happen to became your way of life for some years or so and.As for me, it still looks better to work on thing you're like than boring day-to-day job for some s*king corporations.For example. Google was not a mega-corp at it's start, Apache did not conquered market in one day and many other heavyweights were at some point just usual weaklings. I hope to see you big and lucky in some day or at least properly rewarded for your nice server. And I can see opensource model changes world, in future IMHO there will be fewer coders who work on things on which they're not really interested yourself and do not retain any rights on code (as it was in boring and sucking proprietary world for years and years).

> getting sponsors to pay us to develop lighty as a fulltime job
As for me I will be happy if something like this will happen (if you're happy to work on lighty of course).Sadly all I can help here is small donation. Uhm, well, I wonder why you did not put donation button in visible place and rather hide it so well in menus?Waste majority of users will not view any pages except main page and "download" so these are fair place to put donation button here (and there is nothing wrong in asking for donations for free program, IMHO, especially taking into account that single lighty can power whole huge systems with thousands of users and serious loads and save a decent amount of moneys).

> doesn't sound like an impossible task one might think.
I hope that the day will come and you will be rewarded for your eforts, as talented people like you really should be.

> BTW: Google is not really a competitor
Surely.Google just serving itself, they've crafted all their infrastructure and they're working on improving it. Looks like GWS is part of this work. Actually, they're part of statistic - that's all.

> and b) they use lighty too
Yep. That's hard to ignore and ... hmmm... this is just a very good advertising to your server ;).If google uses something in heavily loaded place (AFAIK, lighty serves such hellish load like YouTube?), it definitely worth of trying for others.

> It's part of the gtk project
As for me, I do not need GTK on servers so this still definitely means something like yet another extra library, "specially for lighty". I hope that it is quite small and does not has bunch of dependencies surrounding it. When talking about lightweight, dependencies are counted as well. If for example .net program is 100Kb in size but requires to install 150+ Mb of runtime sh*t, only complete moron will claim it is "lightweight". Since this still requires 150+ mb of sh*t in your system.

BTW, keep in mind that current lighty is so lightweight and easy that it is heavily used in constrained environments as well. I know at least 2 areas where lighty is unbeatable.

First one is virtualized systems. There is each byte counts. If program eats 1 Mb more RAM and 10Mb more diskspace and you have 20 VMs on host, this at least yelds to 20Mb more RAM used and 200Mb more diskspace wasted.For example lighty often runs on VMs where Apache with its forks will run out of memory even at light loads.On other hand, lighty is a way to go so hosting company can offer cheap VMs with small resources while user can have a decent hosting on such resources with lighty and since lots of such VMs fits to one host, such hosting can be at very competitive prices comparable to shared hosting but with fully isolated VM environments instead.So decent hosting features are getting possible getting possible at very low prices (the only drawback is that users need to be aware about environment limits and how to deal with it but there is still lots of people who is smart enough to save their moneys in such ways).

Second known use includes embedded devices and small consumer devices. I'm serious. Modern router and NAS devices include 2.4 or even 2.6 linux kernel, 16Mb or more RAM (32 Mb is quite common, some with 64 and few even with 128 and 256Mb on board).And even with their small CPU and limited RAM where apache looks like a jerk, lighty still way to go.The only one who can compete here is thttpd.Furthermore, even PHP works great with fastcgi. And thttpd is out of luck here since it's awful when it comes to PHP usage. So... can you imagine a great web-face for autonomous torrent downloader in a small box?And can you imagine that this one powered by just lighty + php?That's where to this world came.Small device can run lighty and it works well even in such constrained environments (because lighty was intended to deal with C10K on quite old hardware, heh). Let's admit that NOR flash ROM still costs something and there is only 8Mb flash IC at very best, often just 4Mb or even 2Mb (with tightly compressed rootfs and other stuff so there is really fits a bit more than that). And yes, there may be NO extra room for additional libs. There may be no such libs ported at all to certain ARM or MIPS based device. Currently lighty fits well anyway due to minimal dependencies and still fast even here and beats thttpd to the hell when it comes to using PHP stuff for web interface. I hope you're not going to broke this. Or lighty will became unsuitable for this area.

> If we would have to implement all the stuff we use
> from that lib on our own, it would take a lot of time
I'm thinking this is not a major issue at least for us, users, as long as there is "current" versions which people can run today are available and maintained (possible with some neat backports\fixes from "future" versions if there is some worth ones).Though I can imagine that if you're like this lib, you want to use it.That's up to you to decide, but when doing decisions please do not forget that lots of people likes lighty because it is really lightweight from ALL sides. Including very few dependencies, simple compilation and very low resources use which allows to extend lighty usage even to small VMs and tiny embedded-class devices. From my ("user" aka "admin") standpoint I like how lighty looks, installs and uses resources today. I manage some lighty servers even including these on small VDS servers with limited RAM and even having a real fun with some lighties embedded into small devices where it still looks like a decent server even with limited horsepower.

> do not think the new lighty will be any less lightweight.
I hope it will be as lightweight as it is already or even better. Time will show...

> Concerning the RAM/CPU/whatever usage:
"Whatever" unfortunately can also mean "disk footprint" and "dependencies" unfortunately. Especially true when it comes to embedded-class devices like SOHO wi-fi routers and NAS boxes where lighty sometimes getting used these days. In these XXI century days even small device should have own web server to control it easily. Sometimes people needs a really decent UI, then PHP comes into the play. Surely with lighttpd. Who else can sit in some 32Mb RAM on 266MHz CPU and still do job in quite decent way?Thttpd?It's works poorly with php.Nginx?It's a bit more fat in RAM terms under light\typical loads and that's matters in such scenarios.

> We will surely not be the next FireFox that gets started to be
> lightweight and ends up as bloat.
Uhm...let's admit that they still managed to boost speed in new release slightly and RAM usage now lower, so as user I can admit there is very nice improvement. I hope they will keep same course and speed since there is surely lots of places for improvements :)

> Yep that buffering is a lot discussed matter
As for me this is the ONLY weak place of lighty I figured out so far.

> My idea would be to make it somehow configurable.
That sounds very cool. My suggestion in this case is following: default setting should prevent high memory usage in ANY scenario. Server is intended to run on it's own and there is not always alive humans who can take care of fastcgi backend if it goes mad for whatever reason and going to flood some gigz into lighty's side causing it to die by horrible death at some point. There is already some nasty nuts who claim that lighty has quite big memleaks. I did dozens of private tests on my own with millions and millions of requests and was not able to find any leaks at least in my configurations. At very most, I found that if you're uploading huge files like a mad, lighty RAM footprint may increase by some static amount (the record is 40Mb RSS memory on x64 system uploading iso-sized crap over a gigabit wire for a while, slightly smaller on 32-bit, normally RSS is something like 4Mb or smaller though).This one looks like it allocates a bit of extra memory for awesome load and then not willing to returnin it to system (at least quickly and easily).But RAM usage never grows more than this static value (still not so great for embedded systems but these are usually do not serve in such hellish conditions and usually 32-bit ones).

P.S. do not take all my mumbling as a total truth - this is just some opinition of yet another guy who uses your server and likes how it works. There is some thousands and thousands of another people who may have different views, goals and whatever.

"Growing the team" by icy

Wed, 06 Aug 2008 15:48:00 +0000

So it really means priority. There's no need for that because the order of the plugin execution is given by either the load order or the place in the config (unlike the current system).
There will be no more confusion about that. It's one of the things we want to make clearer.

"Growing the team" by 191919

Wed, 06 Aug 2008 15:25:52 +0000

the word "merit" was borrowed from a windows directshow filter concept.

it roughly means, if there are two similiar plugins, which will be chosen as the earlier one and which will be the later one.

for example, there are 2 plugins: (a) to replace "abc" with "def", (b) to replace "def" with "abc" and the text is "abcdef".

if (a)'s merit value is greater than (b), the result is "abcabc", otherwise, the result is "defdef".

"Growing the team" by Murat

Wed, 06 Aug 2008 08:43:24 +0000

I'm using 1.5.0-svn for more than a year and never encountered a problem. Its very stable to me (using libaio to serve Images)

"Growing the team" by icy

Wed, 06 Aug 2008 00:44:03 +0000

john_smith: thank you for you feedback!
2) development for 2.0 is _NOT_ at the cost of development for 1.x
We don't want to get you excited about 2.0 because it's new, a rewrite or uses some lib. Those were just technical facts we wanted to provide. Not meant as "cool feature".
About the permanent jobs and makeing it "way of life": Some of us have regular jobs and can't risk quitting those to dedicate to lighty. Others are studying and can't quit just like that too.
Actually I've thought about that foundation thing, getting sponsors to pay us to develop lighty as a fulltime job. Would definetely help but that's not an easy task. And I don't even know what the others think about such a thing so I can only speak for myself.
Lighty is used on hundrets of thousands of servers so getting some companies to sponsor the project doesn't sound like an impossible task one might think.
BTW: Google is not really a competitor because a) their webserver is not available for the public and b) they use lighty too (hey Google, wanna be a sponsor?) :)

3) I can understand you, we are users of other software too. And we will keep compiling as easy as it is now. Right now we use Waf as a build system which makes the whole process a "./waf configure && ./waf" => compiled.
But of course we will also provide autotools and cmake scripts. That should please everyone. And trust me, it's worth depending on glib. If we would have to implement all the stuff we use from that lib on our own, it would take a lot of time. And I have no fear that glib will get abandoned. It's part of the gtk project which is huge and has been around for some time too.
The libev code could even be embedded in the lighty tree but usually package maintainers are not so happy about stuff like that.
Concerning the RAM/CPU/whatever usage: do not think the new lighty will be any less lightweight. In fact I think it will be even more so (the memory management will be better because of the slab-allocator in GSlice e.g.). Time will tell but I have a good feeling.
We will surely not be the next FireFox that gets started to be lightweight and ends up as bloat.

4) Yep that buffering is a lot discussed matter. My idea would be to make it somehow configurable. That would please everybody.

"Growing the team" by ionix

Tue, 05 Aug 2008 20:08:33 +0000

@jan berger I use lighttpd 1.5 r2106 i manually compiled on about 24 of my servers, for one simple reason mod_upload progress this is for 2million pageviews/500,000 visitors site with php 5.2.6 patched with php-fpm patch i recently switched from lighty 1.4.19 tho to nginx 0.6.31 for file serving, static files, fcgi, and everything else most of my issues were with php but recent patching up with php-fpm solved most of these

"Growing the team" by john_smith

Tue, 05 Aug 2008 19:43:37 +0000

Damn, there were dupes. Please kill one duped message? Well, as for me, I do NOT need "totally new cool features" or whatever. To be honest I'm pretty fair with current 1.4 release (1.5 is not release so I do not use it yet). I'm really need a fast lightweight server with MINIMUM possible dependencies (so it is not going to be like overbloated Apache and easy to install), which will serve fast using very few resources. All other stuff I need is FastCGI and\or SCGI to interface to other stuff like PHP and Co. I see only ONE annoyance in lighty at all: as far as I can understand, it caches data from CGI backends in memory, that's awfully bad idea since this may cause lots of RAM consumed. As for me, this is the only annoyance. Let's point out your mistakes from "user" (er, system admin) point of view: 1) Releases... um... where they are?People tend to abandon programs which are no longer actively developed. At least, why don't you provide some news in blogs or so? That's makes you worse since predictability is a very base thing for software. Each admin prefers to rely on software which he uses. If admin will believe that development in stuck or goes nowhere, good admin will evaluate competing products at least. Don't let this to happen. 2) You caused big buzz in the world. Now you're chosen f...ly wrong time for big changes at cost of current versions development. Can't you see people like lighty as it is? I will not bet they (and I am) will like 2.0 just because it is "total rewrite" or "uses glib" or so. As for me, you managed to became #4 server in the world.Not everyone gets so famous.Don't you think that while another 3 are Apache, Google and M$ crap, you can make lighty development something like your permanent jobs and "way of life" rather than work as ordinary workers for stupid companies?Author of nginx is a dedicated server developer in one big www russian company for example(so nginx serves this company of course and best-fits their needs).I'm pretty sure lots of people need custom coding, bunch of improvements installation and configuration, etc today.Why don't make profit from it and not to turn into a fully-powered development company, foundation or whatever you like?I'm pretty sure you will have a very few chances in your lives to have just Google, MS and Apache more popular than your product. I hope you will a proper reward for your efforts, at least. 3) Keep in mind: as admin I'm (and not only I am) absolutely *hate* each extra library needed as well as every additional dependence needed to build program on server side (and sometimes you have no other choice). This means more complicated compilation, extra stuff in system, more bloat and some extra dependence on library authors.Are they going to code stable and fast libs and tools in long term?Are you sure?Then, fine, otherwise such dependencies may turn into nightmare if you happen to depend on no-longer-maintained code. I'd chosen lighty since it's small, fast, not hard to compile and get it running and it serves at blazing speed while needs very few RAM and CPU. Absolutely worst thing you can do is to harm any of these nice features anyhow in new releases. 4) I'm personally don't see *major* issues with stability or security even in current 1.4 version (while nginx is constantly fixes numerous crashes in worker processes...). From my point of view, you're going to fix things which are surely not broken right now.No?At least it looks like this. Let's say as for me to be fully happy with lighty I want to see just one change: no more these stupid huge streams from CGI backends stored in RAM!At most, some fixed buffers.That's where nginx can beat you in RAM consumption even while it starts several processes. That's all what really annoys me. Everything else is optional for me.Nice to have but not mission-critical.

"Growing the team" by john_smith

Tue, 05 Aug 2008 18:23:20 +0000

Hey, guys, Lighttpd is now #4 web server in the world with almost 3 000 000 sites according to netcraft.Congrats :)

"Growing the team" by icy

Sun, 03 Aug 2008 22:27:41 +0000

Flo: Yep it does need better communication. That's the reason for the post because we wanted to let people know what is going on currently.
Looking forward to your contributions.

191919: I would suggest you get one from the svn repository.
About the flexible configuration: the standard config will be something like the current one syntax wise but because of the completely new internal structure, it will be a lot more logical and provide interesting (read useful) new ways to configure lighty.
And if that was not enough, then you can do some really mighty things with the lua frontend stbuehler is building. I think no webserver has this kind of feature :)
About the mini httpd compilation: We will definetely support that as far as possible.
About the private plugin data: yep good idea, worth putting in.
About the plugin-merit support: can you explain what you mean by that?

"Growing the team" by jan berger

Sun, 03 Aug 2008 22:21:25 +0000

first of all i want to give you guys a thanks for contributing to lighty. i'm interested in who is using lighty 1.5 with php in production? any concerns about it? can you recommend it? thanks in advance :)

"Growing the team" by 191919

Fri, 01 Aug 2008 12:31:47 +0000

when we can have a buildable lighty 1.5 snapshot? :)

"Growing the team" by Flo

Thu, 31 Jul 2008 07:41:44 +0000

I second that, as I said this project needs just a better communication. I'm a little busy these days but I plan to contribute, I already have some patches to submit.

"Growing the team" by mlx

Wed, 30 Jul 2008 09:07:52 +0000

"So what does a project need to be considered alive, in your opinion?" Well, especially as you keep on saying that 1.5.x is not dead ... there has been a post back in September 2007 saying "If this release passes your requirements it will be the last 1.5.0 pre-release." There has been zero news, no new pre-releases, no stable release ever since. (or am I just missing something?) I guess even the mailing list was down most the time. I think that's what people are referring to. If I may suggest something ... if you are serious about 1.5.x, give us some news, give us a new pre-release or something. If we knew that there's one more or less stable pre-release from time to time, people might even use that in production and be able to provide further feedback. That's just my 2 cents though.

"Growing the team" by 191919

Wed, 30 Jul 2008 04:45:40 +0000

some suggestions for 2.0:

- more flexiable configuration syntax
- mini-httpd compliation
- plug-in merit support
- plug-in private data support during the whole request processing lifetime (stat/open/read/write/close)

"Growing the team" by sean

Wed, 30 Jul 2008 02:35:42 +0000

i love using lighttpd, it will be quite wonderful if it can be much faster and easier to control, thanks

"Growing the team" by Tim

Tue, 29 Jul 2008 21:04:09 +0000

I'm glad to hear lighttpd is active again, unfornuately - it looks like I'll have to use the light/fast/stable NGINX until Lighttpd 2.0 is ready.

"Growing the team" by nitrox

Tue, 29 Jul 2008 18:38:58 +0000

Lets stop it here, if some people doen´t like glib, its ok. There are quite alot alternatives left. Lighty will always do the wrong steps for some of you and there will always be something to complain about.

But thats what you should fill with good ideas, code contributions, fixes, bugzilla (please spam us, if sth. is not working!) and stuff... and as said, we all do this in our spare time, we don´t get any $$$ for it, so please keep this in mind.

1.5x already makes use of glib, e.g. if you want non-blocking AIO, thats optional atm., but some of you already make use of it.

Brad, please consider this as a place for ideas and join #lighttpd for discussions instead.

"Growing the team" by icy

Tue, 29 Jul 2008 15:05:55 +0000

191919: I'm not angry :)
Brad did call it a "foolish turn"; that's what I was refering to.
I just don't get why some people think the new version will be bloated or - like you've put it - another Apache. What makes you think so? (I really want to know)
We have no plans to make lighty any slower. In fact (imho) the new design will make it even faster while also being more flexible.

Lighty started as a fast and lightweight webserver and at the same time provides features no other webserver has.
And we sure as hell will follow that direction in the future too.