lua use in lighttpd cml

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

lua use in lighttpd cml

Klaus Ripke
recently we got some surprising news about lighty
(like this single threaded webserver working
almost the same as the multithreaded zeus),
anyway, I found no track of the use of Lua in it's CML
(on the list nor the wiki nor uses.html).

So for the record
http://lighttpd.net/documentation/cml.html
Jan notes a nice over 30 fold speedup (4900:150)
compared to doing the same in PHP.
Reply | Threaded
Open this post in threaded view
|

Re: lua use in lighttpd cml

Markus Walther
Klaus Ripke wrote:
> recently we got some surprising news about lighty
> (like this single threaded webserver working
> almost the same as the multithreaded zeus),
> anyway, I found no track of the use of Lua in it's CML
> (on the list nor the wiki nor uses.html).
>

Yes, I noticed this too some time ago and even wrote Lua scripts to
abuse it a little :-) See below for a small example.

You need a 1-line patch to enable GET params (script?a=1&b=2&c=3 => Lua
global table get={a=1,b=2,c=3) to work correctly in lighttpd 1.4.9
(patch available from me or the lighttpd developer).

CML stands for Cache Meta Language and is a way to describe
sophisticated caching behaviour of the lighttpd web server, turning
dynamic web pages (driven by PHP, Lua, what have you) into static files
whenever there was no change. So, you get the full power of Lua to
describe when and how to cache, and you communicate with the web server
using a few reserved variables and return values
(output_contenttype,request,get,output_include,CACHE_HIT,CACHE_MISS --
see below). The latest functionality in CML is called PowerMagnet, a way
to do sophisticated URL rewriting AFAIK.

The developer, Jan Kneschke, notes it's _not_ a good idea to (ab)use
this to _also_ generate the dynamic web pages themselves (that's what I
did for fun in my experiments, BTW). This is because lighttpd is
single-threaded for reasons of maximum speed, so CML-based dynamic
content generation happens synchronously, blocking everything until it's
done. He recommends using the FastCGI interface instead, so that's what
I'm doing now.

But yes, for quick Lua web scripting hacks and if you are not too
concerned about their performance, lighttpd is a perfect and very nice
web server that can be built out of the box with Lua.
(Is lighttpd perhaps the only high-performance web server where Lua is
available so directly/not as a module?)

It's a pity a chance was missed to use Lua to also handle the
configuration files. Jan Kneschke says Lua/CML came _after_ that part
was designed/implemented and he doesn't want a dependency on Lua in
embedded situations with tight memory. I may however add this as an
option when I have time.

> So for the record
> http://lighttpd.net/documentation/cml.html
> Jan notes a nice over 30 fold speedup (4900:150)
> compared to doing the same in PHP.
>

That's wrong. The comparison is about simulating caching _in the
scripting language_ (PHP or Lua would make no difference, I believe), or
doing it _in the webserver_! Webservers are most efficient delivering
static content, so it does make sense to move that decision - whether to
send some static cached file or regenerate it by running your PHP,Lua,
etc. (Fast)CGI script - to the webserver. The Lua-based CML merely
expresses the caching rules _and_ allows you to piece together
independent parts of a multi-part web page (output_include below), which
gives you another performance bonus if those parts are modified at
different intervals, i.e. have independent caching characteristics.

--Markus Walther

-- EXAMPLE index.cml for lighttpd web server
output_contenttype = "text/html"
local cwd = request["CWD"]
-- let's generate some dynamic content from within CML
local f=assert(io.open(cwd.."dynamic_content.html","w"))
-- show request and get (URL-encoded) params
f:write("<br><br>\n<b>REQUEST</b><br>\n")
for k,v in pairs(request) do
        f:write(k,"=",v,"\n<br>\n")
end
f:write("<br><b>GET</b><br>\n")
for k,v in pairs(get) do
        f:write(k,"=",v,"\n<br>\n")
end
-- list directory,showing some aux. functions in CML
f:write("<br><b>DIRECTORY</b> ",cwd,"<br>\n")
for file in dir_files(cwd) do
        f:write("<b>",file,
        file_isreg(file)==0 and "/ " or " ",
        "</b> ",
        "MD5 of filename: ",md5(file),"\n<br>\n")
end
f:close()
-- piece together various parts
output_include = {cwd.."static_part1.html",
                  cwd.."dynamic_content.html",
                  cwd.."static_part2.html"}
return CACHE_HIT -- no regeneration of dynamic content
                  -- (we did it ourselves)


Reply | Threaded
Open this post in threaded view
|

Re: lua use in lighttpd cml

Klaus Ripke
On Sun, Feb 05, 2006 at 02:16:43PM +0300, Markus Walther wrote:
> Klaus Ripke wrote:
> >Jan notes a nice over 30 fold speedup (4900:150)
> >compared to doing the same in PHP.
> >
>
> That's wrong. The comparison is about simulating caching _in the
> scripting language_ (PHP or Lua would make no difference, I believe), or
> doing it _in the webserver_!
Err, no, not really.

Clearly the CML has kind of an "unfair advantage" by running
inside the webserver. However, Fast-CGI is not that slow.
If you ever compared Apache's mod_php to PHP via Fast-CGI
you know it is not much of a difference (typically about 10%).

Hence http://lighttpd.net/documentation/cml.html does not read
"the main reason is the Fast-CGI communication overhead"
but instead "... the overhead of the PHP interpreter to start up".
And this does not mean to launch a new process,
it's the reinitialization PHP does for every request
inside the persistent Fast-CGI server process.

In one of our customer's project they use Apache/mod_php,
which takes 50ms CPU time on every request before doing any work,
just to reinit and grok a couple 1000 lines of PHP (on a 2.8GHz XEON).
In the same time I can exec a bunch of Lua CGIs.


Yes, PHP or Lua does make a huge difference.
While on plain execution time PHP may only be
a factor of five or so slower on average
http://shootout.alioth.debian.org/debian/benchmark.php?test=all&lang=lua&lang2=php
the initialization frenzy is an important feature of PHP.


cheers
Klaus
Reply | Threaded
Open this post in threaded view
|

Re: lua use in lighttpd cml

Markus Walther
Klaus Ripke wrote:

> On Sun, Feb 05, 2006 at 02:16:43PM +0300, Markus Walther wrote:
>
>>Klaus Ripke wrote:
>>
>>>Jan notes a nice over 30 fold speedup (4900:150)
>>>compared to doing the same in PHP.
>>>
>>
>>That's wrong. The comparison is about simulating caching _in the
>>scripting language_ (PHP or Lua would make no difference, I believe), or
>>doing it _in the webserver_!
>
> Err, no, not really.

<long explanation about why PHP is worse than Lua for (Fast)CGI
applications deleted>

I'm sorry, but your original post was this:

"[...]
So for the record
http://lighttpd.net/documentation/cml.html
Jan notes a nice over 30 fold speedup (4900:150)
compared to doing the same in PHP."

You didn't provide the context to understand your speedup claim, so I
reacted and also explained things a bit. Fortunately you did provide the
URL which anybody on the list can read for him/herself. Here is what Jan
says in there about CML:

"CML (Cache Meta Language) wants to solves several problems:

         * dynamic content needs caching to perform
         * checking if the content is dirty inside of the application is
usually more expensive than sending out the cached data
         * a dynamic page is usually fragmented and the fragments have
different livetimes
         * the different fragements can be cached independently"

The entire document does not make any comparison Lua versus PHP -
therefore I am correct to say you're wrong, because you cannot draw such
conclusions FROM http://lighttpd.net/documentation/cml.html, and that is
what your sentence, quoted above, suggested to the Lua-L readership.

Of course, it is possible to provide independent data on Lua vs PHP such
as the one you have cited from your own or others' experience and
measurements - and I even found them interesting. But that is an
entirely separate thread and a new topic.

Regards, Markus Walther
Reply | Threaded
Open this post in threaded view
|

Re: lua use in lighttpd cml

Klaus Ripke
dear regular readers of this list, please just skip this message


Not to annoy anybody,
just to clarify the topic for the innocent finding this thread
in the archives when searching for Lua vs. PHP:

On Tue, Feb 07, 2006 at 12:02:12AM +0300, Markus Walther wrote:
...
> The entire document does not make any comparison Lua versus PHP -
> therefore I am correct to say you're wrong, because you cannot draw such
> conclusions FROM http://lighttpd.net/documentation/cml.html,
...
Obviously the document DOES make a comparison Lua versus PHP.
It compares Lua internal vs. PHP external.

It does not compare PHP internal vs. PHP external,
because using PHP internally in a single threaded single
process webserver is not an option.
That's different for both preforked (MP-)Apache and MT-Apache,
but both are much slower than lighty anyway.

As mentioned the comparision Lua external vs. PHP external
would probably show less than a 30 fold speedup,
but still a significant speedup.
But why test Lua external, if Lua, by it's very design, is suitable
to be used internal even in a single threaded single process server?

So the comparision is fair and is really a comparision of the
languages, taking all deployment conditions into account.
Without any external references, for this application,
a single threaded single process webserver, the best you can do
with Lua is 30 times faster than what you can do with PHP.


best regards
Klaus