Growing the team 59
- hoffie
- icy
- nitrox
- stbuehler
If you hang out in #lighttpd, you will already know these names.
That being said, there is also some news about the state of the development of lighty.
Lighty is a great webserver and we all love it. But if you use it for quite some time, you’ll eventually find out that it’s not perfect. There are some oddities and shortcomings that cannot be ironed out without rewriting a significant part of the core.
This is where I had the idea to start all over. With lessons learned from the past, we could write a new version of lighty that fixes the design issues the current versions have. But at the same time keep all the good stuff and remain fairly compatible. The main idea to be fast and lightweight is still valid and if you are familiar with 1.4.x or 1.5, you will have no problems with the new version.
Jan had the idea to use glib for stuff like strings/buffers and arrays. He wanted to rewrite the corresponding parts and did so in a branch of his. I proposed to take this even further and use glib throughout the source. It makes coding easier, faster and in the end more secure because you can rely on proven and well tested source.
stbuehler jumped in and created a new branch where he started to hack on an all new lighty using the ideas from above. The following days we discussed a lot – and still do so – about the new design. What we could do better, what shortcomings we wanted to fix.
The result is nothing less but a more flexible and faster design for lighty.
This sounds awesome but where is the catch you might think. Well, currently we have not too much code ready but we are actively hacking on it. Do not ask when it will be released, there is no estimated date. We are still at the very beginning of the new version but hope to have it some day become the official Lighttpd 2.0
We created a page that lists some of the (technical) plans we currently have. You might find some of them interesting.
This is it for today folks. Hope you like the route we are taking and maybe drop us a line in #lighttpd (freenode) or in the comments.
- update -
To make it clear: 1.5 or 1..4 have not been dropped. If you think 1.5 is unstable and you get a segfault or something like that, then tell us. File a ticket, attach a stacktrace and document the problem in a detailed way so it can be fixed. Thanks for using lighty.
Trackbacks
Use the following link to trackback from your own site:
http://blog.lighttpd.net/articles/trackback/5428
-
Lighthttp-Team grows
Even if it's about scrapping everything and rebuilding from scratch and skipping 1.5.
A Grand Plan of Things To Come (with or without timeline) would be most welcome.
I admit I am a bit annoyed by this announce, my hope to see 1.5 release changing my life (nearly) is gone but I think I can wait for an awesome 2.0 release fully featured and blazing speedy. 1.4 is working fine (nginx and apache still are far behind IMO)
Just two wishes for this new development cycle:
- more news about dev
- upload progress in 2.0 :)
that's all, good to see lighty is not dead !1.4 as a stable, fast and reliable release with many features already implemented.
1.5 still in development, lots of changes in queue, cool new features but needs some testing and feedback from you (as user).
2.0 just in the phase of collecting ideas and talking about the future design - not more, not less (and this fucking progressbar, which seems to be the killa-feature :P).
1. new developers have a hard time to figure out how our own api works
2. the glib api is probably better tested (we had a stupid bug with resizing a buffer and a resulting endless loop in the log function)
3. i still don't understand the array implementation - my eyes hurt when i try to read it :)
And btw, perhaps you remember http://blog.lighttpd.net/articles/2008/02/19/project-glib-it-compiles-mostly ;-)
Why libev?
Again, we have of course our own code for similar things, and the first two arguments from above are valid here too (remember out-of-bound index in fdevent?)
So we leave the event handling part to a library which was made for exactly that purpose.
And why do you think lighty isn't "lightweight"?
The tickets... yeah. I just blame trac - it is slow like hell :p
I don't think that does matter because if you compile from source, you surely will have that much free space in your build environment.
The resulting binary is still way smaller than this. In fact I don't think the final binary will be a lot bigger than it is now for 1.x.
Maybe something around 200kb? With glib.so and libev.so together maybe 1mb. Small enough to fit on small devices.
The main thing I think of when saying lightweight, is RAM and CPU consumption. These two are the things you might reach the limits, not hdd space.
To me it looks like there are lots of "inconsistencies" in people's opinions.
On the one hand you want something like 1.4.x (which WILL stay there and get security updates and "simple" fixes), something which is stable, which is stand-alone and well tested. On the other hand you expect certain problems (mod_{proxy,fastcgi,scgi,...} buffering issue, anyone?) to be fixed timely, which is evidently not possible without rethinking and rewriting huge parts of the design (and as such impossible to do in short-term).
So what would your solution be in that case? Concentrate on inventing a time machine, fix the problems in the past and fly back to the present? It's pretty obvious that it won't work like that.
There have to be compromises, and here is ours: we keep 1.4.x as the well-tested and very stable branch, we keep 1.5.x which already has lots of fancy new stuff and just waits on becoming mature enough to finally replace 1.4.x and we have goals which even exceed the time frame of "near future".
So what does a project need to be considered alive, in your opinion? We have something for the "present", something for the near future and some long-time goals. There is not really more we can do, especially considering that lighty's team is rather small actually and this is still a free-time project. In my opinion, the time which is spent on the project is spent well, and you cannot force anyone to work on projects if they don't want to (i.e. increase the time they work on a project).
One thing I agree with though is versioning. I guess in some cases we could have really done better (pr-wise), but I'd simply say that most programmers are probably not too good at pr-related stuff... (no offense intended). This probably also created the myth that lighty is dead, but as a #lighttpd regular and someone talking with those guys who actually do the work I really can only say, that there are lots of things happening (which are not necessarily apparent to the general public).
Brad, I really cannot understand you. As a packager one should be rather happy about being able to use system libraries instead of having software ship their very own implementations with all their problems. At least this is what I feel when packaging certain software.
Static linking is of course not a good idea in general, and I guess this comparison was only used to show you some numbers which you could compare to 1.4.x (which hardly has any external deps).
One last note regarding other web servers: If you find software which fits your needs better than lighty, then go and use it. I don't have any objections or bad feelings about this and I doubt anyone else has. Why make your life harder and force yourself into using software A, even if it lacks feature X, while software B provides it? Doesn't make sense in my opinion.
Sure, it is nice to see that lots of people are using lighty, no doubt about that, but having only unsatisfied users (which are hard to satisfy at all) is not certainly better.
Also please note that I'm solely speaking for myself here, not for the team, which I have apparently become a part of, just by doing some support and giving some input on new ideas. ;)
but for the core components, like the array/buffer/string and the i/o model, i think few people will accept dependencies, no matter the size in kb or in line of them.
being a neat web server is the main reason of the popularity of httpds like lighty and nginx.
You just don't like it as a dependency because it is a dependency. Do I see this correctly?
Package managers can handle dependencies for a reason.
Again: if you don't like glib as a dependency, PROVIDE HARD FACTS why you think so. But calling our decisions foolish without backing up that (insulting) claim, is... well... foolish.
but don't let lighty be another apache.
Brad did call it a "foolish turn"; that's what I was refering to.
I just don't get why some people think the new version will be bloated or - like you've put it - another Apache. What makes you think so? (I really want to know)
We have no plans to make lighty any slower. In fact (imho) the new design will make it even faster while also being more flexible.
Lighty started as a fast and lightweight webserver and at the same time provides features no other webserver has.
And we sure as hell will follow that direction in the future too.
But thats what you should fill with good ideas, code contributions, fixes, bugzilla (please spam us, if sth. is not working!) and stuff... and as said, we all do this in our spare time, we don´t get any $$$ for it, so please keep this in mind.
1.5x already makes use of glib, e.g. if you want non-blocking AIO, thats optional atm., but some of you already make use of it.
Brad, please consider this as a place for ideas and join #lighttpd for discussions instead.
- more flexiable configuration syntax
- mini-httpd compliation
- plug-in merit support
- plug-in private data support during the whole request processing lifetime (stat/open/read/write/close)
Looking forward to your contributions.
191919: I would suggest you get one from the svn repository.
About the flexible configuration: the standard config will be something like the current one syntax wise but because of the completely new internal structure, it will be a lot more logical and provide interesting (read useful) new ways to configure lighty.
And if that was not enough, then you can do some really mighty things with the lua frontend stbuehler is building. I think no webserver has this kind of feature :)
About the mini httpd compilation: We will definetely support that as far as possible.
About the private plugin data: yep good idea, worth putting in.
About the plugin-merit support: can you explain what you mean by that?
2) development for 2.0 is _NOT_ at the cost of development for 1.x
We don't want to get you excited about 2.0 because it's new, a rewrite or uses some lib. Those were just technical facts we wanted to provide. Not meant as "cool feature".
About the permanent jobs and makeing it "way of life": Some of us have regular jobs and can't risk quitting those to dedicate to lighty. Others are studying and can't quit just like that too.
Actually I've thought about that foundation thing, getting sponsors to pay us to develop lighty as a fulltime job. Would definetely help but that's not an easy task. And I don't even know what the others think about such a thing so I can only speak for myself.
Lighty is used on hundrets of thousands of servers so getting some companies to sponsor the project doesn't sound like an impossible task one might think.
BTW: Google is not really a competitor because a) their webserver is not available for the public and b) they use lighty too (hey Google, wanna be a sponsor?) :)
3) I can understand you, we are users of other software too. And we will keep compiling as easy as it is now. Right now we use Waf as a build system which makes the whole process a "./waf configure && ./waf" => compiled.
But of course we will also provide autotools and cmake scripts. That should please everyone. And trust me, it's worth depending on glib. If we would have to implement all the stuff we use from that lib on our own, it would take a lot of time. And I have no fear that glib will get abandoned. It's part of the gtk project which is huge and has been around for some time too.
The libev code could even be embedded in the lighty tree but usually package maintainers are not so happy about stuff like that.
Concerning the RAM/CPU/whatever usage: do not think the new lighty will be any less lightweight. In fact I think it will be even more so (the memory management will be better because of the slab-allocator in GSlice e.g.). Time will tell but I have a good feeling.
We will surely not be the next FireFox that gets started to be lightweight and ends up as bloat.
4) Yep that buffering is a lot discussed matter. My idea would be to make it somehow configurable. That would please everybody.
it roughly means, if there are two similiar plugins, which will be chosen as the earlier one and which will be the later one.
for example, there are 2 plugins: (a) to replace "abc" with "def", (b) to replace "def" with "abc" and the text is "abcdef".
if (a)'s merit value is greater than (b), the result is "abcabc", otherwise, the result is "defdef".
There will be no more confusion about that. It's one of the things we want to make clearer.
> development for 2.0 is _NOT_ at the cost of development for 1.x
This is known to you.But others can see only the fact that main page of site didn't changed anyhow within half year or so and even blog is quite silent, lacking any news.Oh yeah, I was interested to find truth and found that someone is still alive in comments. I bet that at very best only each 20th visitor will figure this out.
> Others are studying and can't quit just like that too.
Yep, true. But again, being #4 server in the world is not just a word. Many and many people will die without even having chance to achieve something like that where only 3 big names are ahead, so you can be very proud by this fact. So if you're really like this project (or why it has appeared and and so nice and even free?) you may want to think on paying it more attention and with some luck it may happen to became your way of life for some years or so and.As for me, it still looks better to work on thing you're like than boring day-to-day job for some s*king corporations.For example. Google was not a mega-corp at it's start, Apache did not conquered market in one day and many other heavyweights were at some point just usual weaklings. I hope to see you big and lucky in some day or at least properly rewarded for your nice server. And I can see opensource model changes world, in future IMHO there will be fewer coders who work on things on which they're not really interested yourself and do not retain any rights on code (as it was in boring and sucking proprietary world for years and years).
> getting sponsors to pay us to develop lighty as a fulltime job
As for me I will be happy if something like this will happen (if you're happy to work on lighty of course).Sadly all I can help here is small donation. Uhm, well, I wonder why you did not put donation button in visible place and rather hide it so well in menus?Waste majority of users will not view any pages except main page and "download" so these are fair place to put donation button here (and there is nothing wrong in asking for donations for free program, IMHO, especially taking into account that single lighty can power whole huge systems with thousands of users and serious loads and save a decent amount of moneys).
> doesn't sound like an impossible task one might think.
I hope that the day will come and you will be rewarded for your eforts, as talented people like you really should be.
> BTW: Google is not really a competitor
Surely.Google just serving itself, they've crafted all their infrastructure and they're working on improving it. Looks like GWS is part of this work. Actually, they're part of statistic - that's all.
> and b) they use lighty too
Yep. That's hard to ignore and ... hmmm... this is just a very good advertising to your server ;).If google uses something in heavily loaded place (AFAIK, lighty serves such hellish load like YouTube?), it definitely worth of trying for others.
> It's part of the gtk project
As for me, I do not need GTK on servers so this still definitely means something like yet another extra library, "specially for lighty". I hope that it is quite small and does not has bunch of dependencies surrounding it. When talking about lightweight, dependencies are counted as well. If for example .net program is 100Kb in size but requires to install 150+ Mb of runtime sh*t, only complete moron will claim it is "lightweight". Since this still requires 150+ mb of sh*t in your system.
BTW, keep in mind that current lighty is so lightweight and easy that it is heavily used in constrained environments as well. I know at least 2 areas where lighty is unbeatable.
First one is virtualized systems. There is each byte counts. If program eats 1 Mb more RAM and 10Mb more diskspace and you have 20 VMs on host, this at least yelds to 20Mb more RAM used and 200Mb more diskspace wasted.For example lighty often runs on VMs where Apache with its forks will run out of memory even at light loads.On other hand, lighty is a way to go so hosting company can offer cheap VMs with small resources while user can have a decent hosting on such resources with lighty and since lots of such VMs fits to one host, such hosting can be at very competitive prices comparable to shared hosting but with fully isolated VM environments instead.So decent hosting features are getting possible getting possible at very low prices (the only drawback is that users need to be aware about environment limits and how to deal with it but there is still lots of people who is smart enough to save their moneys in such ways).
Second known use includes embedded devices and small consumer devices. I'm serious. Modern router and NAS devices include 2.4 or even 2.6 linux kernel, 16Mb or more RAM (32 Mb is quite common, some with 64 and few even with 128 and 256Mb on board).And even with their small CPU and limited RAM where apache looks like a jerk, lighty still way to go.The only one who can compete here is thttpd.Furthermore, even PHP works great with fastcgi. And thttpd is out of luck here since it's awful when it comes to PHP usage. So... can you imagine a great web-face for autonomous torrent downloader in a small box?And can you imagine that this one powered by just lighty + php?That's where to this world came.Small device can run lighty and it works well even in such constrained environments (because lighty was intended to deal with C10K on quite old hardware, heh). Let's admit that NOR flash ROM still costs something and there is only 8Mb flash IC at very best, often just 4Mb or even 2Mb (with tightly compressed rootfs and other stuff so there is really fits a bit more than that). And yes, there may be NO extra room for additional libs. There may be no such libs ported at all to certain ARM or MIPS based device. Currently lighty fits well anyway due to minimal dependencies and still fast even here and beats thttpd to the hell when it comes to using PHP stuff for web interface. I hope you're not going to broke this. Or lighty will became unsuitable for this area.
> If we would have to implement all the stuff we use
> from that lib on our own, it would take a lot of time
I'm thinking this is not a major issue at least for us, users, as long as there is "current" versions which people can run today are available and maintained (possible with some neat backports\fixes from "future" versions if there is some worth ones).Though I can imagine that if you're like this lib, you want to use it.That's up to you to decide, but when doing decisions please do not forget that lots of people likes lighty because it is really lightweight from ALL sides. Including very few dependencies, simple compilation and very low resources use which allows to extend lighty usage even to small VMs and tiny embedded-class devices. From my ("user" aka "admin") standpoint I like how lighty looks, installs and uses resources today. I manage some lighty servers even including these on small VDS servers with limited RAM and even having a real fun with some lighties embedded into small devices where it still looks like a decent server even with limited horsepower.
> do not think the new lighty will be any less lightweight.
I hope it will be as lightweight as it is already or even better. Time will show...
> Concerning the RAM/CPU/whatever usage:
"Whatever" unfortunately can also mean "disk footprint" and "dependencies" unfortunately. Especially true when it comes to embedded-class devices like SOHO wi-fi routers and NAS boxes where lighty sometimes getting used these days. In these XXI century days even small device should have own web server to control it easily. Sometimes people needs a really decent UI, then PHP comes into the play. Surely with lighttpd. Who else can sit in some 32Mb RAM on 266MHz CPU and still do job in quite decent way?Thttpd?It's works poorly with php.Nginx?It's a bit more fat in RAM terms under light\typical loads and that's matters in such scenarios.
> We will surely not be the next FireFox that gets started to be
> lightweight and ends up as bloat.
Uhm...let's admit that they still managed to boost speed in new release slightly and RAM usage now lower, so as user I can admit there is very nice improvement. I hope they will keep same course and speed since there is surely lots of places for improvements :)
> Yep that buffering is a lot discussed matter
As for me this is the ONLY weak place of lighty I figured out so far.
> My idea would be to make it somehow configurable.
That sounds very cool. My suggestion in this case is following: default setting should prevent high memory usage in ANY scenario. Server is intended to run on it's own and there is not always alive humans who can take care of fastcgi backend if it goes mad for whatever reason and going to flood some gigz into lighty's side causing it to die by horrible death at some point. There is already some nasty nuts who claim that lighty has quite big memleaks. I did dozens of private tests on my own with millions and millions of requests and was not able to find any leaks at least in my configurations. At very most, I found that if you're uploading huge files like a mad, lighty RAM footprint may increase by some static amount (the record is 40Mb RSS memory on x64 system uploading iso-sized crap over a gigabit wire for a while, slightly smaller on 32-bit, normally RSS is something like 4Mb or smaller though).This one looks like it allocates a bit of extra memory for awesome load and then not willing to returnin it to system (at least quickly and easily).But RAM usage never grows more than this static value (still not so great for embedded systems but these are usually do not serve in such hellish conditions and usually 32-bit ones).
P.S. do not take all my mumbling as a total truth - this is just some opinition of yet another guy who uses your server and likes how it works. There is some thousands and thousands of another people who may have different views, goals and whatever.
best line ever :-)
P.S. especially as some of my earlier posts might sound a bit offensive or negative: thanks to everyone contributing to lighttpd. it's much appreciated. i'm still in love with lighty ;-)
john_smith: thanks again for your lengthy feedback!
about GTK: sure you don't have GTK on a server but that wasn't what I meant to say. glib is only part of this bigger project like FireFox is part of Mozilla. It means there are many developers working on it and it will not disappear from one day to the other.
About the dependencies etc: yesterday stbuehler did again some heavy work on 2.0 and implemented static compiling. The resulting binary was 1.9 mb on my system. Mind you: it's a 64bit box and there were NO optimizations to make the filesize smaller. On a 32bit box it's about 1.5 mb. If you compile only the really needed stuff in, it will be smaller.
Sure there is a lot of code to come but that will not change this size too much because the biggest part is the base and the libs. That should be small enough to fit on most embedded devices. We wont make our time harder by coding everything on our own and optimizing that just to make it run on some tiny devices more. It's a tradeoff you have to make. There are always even smaller devices where people want lighty to run on.
Our target systems are anything from your average home router to big servers with 10gbit link and thousands of requests per second.
Long story short: we want to scale.
And btw: the current build uses about 1.5mb RAM (again: 64bit box, less on 32bit)
The buffering: 2.0 will probably make it configurable, the default behaviour is a thing to discuss but shouldn't concern you too much then ;)
P.S. appretiate the mumbling
autogen.sh complains AC_DEFINE not defined.
configure complains PKG_CHECK not defined.
Latest automake/autoconf/libtool installed.
Or do both -- file a ticket so we can track it and remind us on IRC :p