CGILua future

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

CGILua future

Javier Guerra Giraldez
Hi

I'm evaluating CGILua for a project i'm doing.  Currently, the web part is 
done with PHP, and the back processes that do most of the work are pure Lua.

Unfortunately, extending the web part is getting cumbersome, because most of 
the low level functions are written in Lua.  I find myself reimplementing 
some Lua functions in PHP.  Therefore, i'd want to pick up just one language.  
As much as i like PHP, Lua is the clear winner in this case, so i need CGILua 
or something like that.

Several times before, i've thought of modifying the FastCGI version of PHP to 
allow for persistent processes.  I think this is the only real advantage of 
Java (J2EE, etc) over PHP.  So now i think it's a great opportunity to do it 
with Lua instead of PHP.

Reading the FastCGI version of CGILua, it seems so close to that feature; 
especially after finding that the main event loop is in a Lua script and not 
the C caller.  But there is that venv() call for every event!  totally 
recreating the statelessness of PHP and old-style CGI!

I'm sure i could just take it off, and use a single global environment, but it 
would make a mess when trying to use several different applications on the 
same server.  also, i think it must be there for a good reason that i just 
don't get.

Also, i'm afraid that some queries can take a few seconds to be processed, and 
the single event loop would make every query to wait single-threadedly on a 
queue.

I've just glossed over the fcgi docs, there are a few hints of interleaving 
the HTML requests for a single FastCGI server, but i can't find any useful 
example.

Also, at the kepler project page there's a mention about CGILua alpha4, which 
should do persistent sessions... can we see that code? is it progressing? is 
it stalled?  should i try to do my own session system instead of the current 
"venv() for every request" scheme?

-- 
Javier

Attachment: pgpLHu_l2FmgJ.pgp
Description: PGP signature

Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

Tomas-14
	Hi,

> Reading the FastCGI version of CGILua, it seems so close to that feature;
> especially after finding that the main event loop is in a Lua script and not
> the C caller.  But there is that venv() call for every event!  totally
> recreating the statelessness of PHP and old-style CGI!
	The environment is not totally recreated.  VEnv inherits all
globals from the original environment.  Also, libraries loaded with
require are "cached" avoiding reloading code from the file.

> I'm sure i could just take it off, and use a single global environment, but it
> would make a mess when trying to use several different applications on the
> same server.  also, i think it must be there for a good reason that i just
> don't get.
	The main reason is to protect one script from another.

> Also, i'm afraid that some queries can take a few seconds to be processed, and
> the single event loop would make every query to wait single-threadedly on a
> queue.
	Maybe I've missed the point but currently CGILua does not share
Lua States over threads.

> Also, at the kepler project page there's a mention about CGILua alpha4, which
> should do persistent sessions... can we see that code? is it progressing? is
> it stalled?  should i try to do my own session system instead of the current
> "venv() for every request" scheme?
	We'll release the next version (alpha 4) until the end of
november.  We included a sessions library which still needs thorough tests
and some improvements but is working.  The documentation also lacks some
examples (but this is just another alpha :-) etc.  Well, give it a try and
tell us what you think!

	Regards,
		Tomas

Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

Daniel Silverstone
In reply to this post by Javier Guerra Giraldez
On Tue, 2004-11-23 at 12:58 -0500, Javier Guerra wrote:
> Several times before, i've thought of modifying the FastCGI version of PHP to 
> allow for persistent processes.  I think this is the only real advantage of 
> Java (J2EE, etc) over PHP.  So now i think it's a great opportunity to do it 
> with Lua instead of PHP.

There is Aranha at http://aranha.pepperfish.net/

It's currently being rewritten from the ground-up but it does work well
and is in use on many sites including http://www.pepperfish.net/
http://www.digital-scurf.org/ and http://www.simtec.co.uk/

I wrote most of Aranha and am happy to answer questions.

D.

-- 
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 2BC8 4016 2068 7895



Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

D Burgess-4
In reply to this post by Javier Guerra Giraldez
There is a bit of scope here.
1) CGILua does not allow you access to both FastCGI environments, namely
the startup environment and the request environment, there is only one.
This is imprtant This can be changed in CGILua. When I use the word
"environment" I mean FastCGI environment not Lua.

2) Modify venv.lua and/or the fastcgi launcher.

> 've just glossed over the fcgi docs, there are a few hints of interleaving
> the HTML requests for a single FastCGI server, but i can't find any useful
> example.
FastCGI has a session affinity patch.

On Tue, 23 Nov 2004 12:58:46 -0500, Javier Guerra <[hidden email]> wrote:
> Reading the FastCGI version of CGILua, it seems so close to that feature;
> especially after finding that the main event loop is in a Lua script and not
> the C caller.  But there is that venv() call for every event!  totally
> recreating the statelessness of PHP and old-style CGI!
> 
> I'm sure i could just take it off, and use a single global environment, but it
> would make a mess when trying to use several different applications on the
> same server.  also, i think it must be there for a good reason that i just
> don't get.

-- 
DB

Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

Javier Guerra Giraldez
In reply to this post by Tomas-14
On Tuesday 23 November 2004 12:51 pm, Tomas wrote:
> > totally recreating the statelessness of PHP and old-style CGI!
>
>  The environment is not totally recreated.  VEnv inherits all
> globals from the original environment.  Also, libraries loaded with
> require are "cached" avoiding reloading code from the file.

right, the startup time is far lower than old CGI, the only overhead seems to 
be the creation of the new environment (and maybe garbage-collecting the old 
ones).  but the main advantage of Java that i'd love to see is the 
persistence of in-RAM objects.  J2EE has a lot of complex mechanisms to 
reduce serialization and deserializations, those are possible just because 
the object containers persist from one HTTP request to other.

a simple sessions library, like that of PHP, can make it easier to retrieve 
objects from storage; but maybe the most Lua-like solution would be just a 
global table where i could put any object (or tree of objects) that i want to 
persist for another request.  then, i just need a cookie to store the key.  
maybe some extra features like setting a timeout to purge the object (to have 
expiring sessions)

>  The main reason is to protect one script from another.

that's what i guessed, but thought maybe there was another reason.

> > Also, i'm afraid that some queries can take a few seconds to be
> > processed, and the single event loop would make every query to wait
> > single-threadedly on a queue.
>
>  Maybe I've missed the point but currently CGILua does not share
> Lua States over threads.

what i mean is, given the simple event loop, can i do overlapping requests?  
it's important because some queries can take a few seconds to be processed.  
short enough for a single user; but if another user can't get his requests 
served meanwhile, we're in trouble.

>  We'll release the next version (alpha 4) until the end of
> november.  We included a sessions library which still needs thorough tests
> and some improvements but is working.  The documentation also lacks some
> examples (but this is just another alpha :-) etc.  Well, give it a try and
> tell us what you think!

i'd like to see it!  at least see where the project is headed... my 
requirements aren't too complex, so if your priorities are too different, i'd 
have to roll my own.

is the alpha 4 code available anywhere?

-- 
Javier

Attachment: pgpDNW9c28NWy.pgp
Description: PGP signature

Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

Javier Guerra Giraldez
In reply to this post by Daniel Silverstone
On Tuesday 23 November 2004 1:04 pm, Daniel Silverstone wrote:
> There is Aranha at http://aranha.pepperfish.net/

i'm currently checking it.  still not sure how it works, but so far looks more 
like what i'd want.

> It's currently being rewritten from the ground-up but it does work well

what are the main goals of the rewrite? just code cleaning and update? or the 
dynamic process manager mentioned on the webpage?

and about the several-workers model... can different workers get the same 
persistent objects? or is there any affinity management?


-- 
Javier

Attachment: pgpR3KpdgE5cA.pgp
Description: PGP signature

Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

Jay Carlson
In reply to this post by D Burgess-4
David Burgess wrote:

FastCGI has a session affinity patch.

There have been significant periods when 3skel.com has been losing. I'm glad to see it's back online.

There are two ways of looking at session affinity.

One is that it is a fundamental way of structuring process flow. In this case, the lifetime of a process flow is tied to the lifetime of a particular FastCGI process. Life is relatively simple if you believe this.

The second is that session affinity is just a performance hack. In this view, a computational flow must be able to migrate across engines. You must be able to keep all of your interesting data in some shared repository, but it happens that most of the time you don't need to reinstantiate it.

I honestly don't know which of these I believe in. Given that users are already used to web apps breaking randomly ("Just Another Victim Of The Ambient Morality") it may make sense to punt on migration. The computational model in the second case can get arbitrarily complex, as you may need to do a full Pluto serialization into a db on every context switch---or wuss out, and declare that some tables (the persistent ones) are "special" in the pejorative sense and can only absorb scalars or acyclic tables of scalars. After a delightful bout with ASP where false ~= false, I'm willing to work with ANYTHING that wasn't designed by idiots.

At this point I'm about ready to give up on migration *and* naive session affinity, and force fit the programming model into wiki-on-drugs and call it a day. Oh, and run around quoting more Iain Banks ship names in comments.

-- A Frank Exchange Of Views
-- Grey Area
-- Very Little Gravitas Indeed

Jay

Reply | Threaded
Open this post in threaded view
|

Re: CGILua future

Daniel Silverstone
In reply to this post by Javier Guerra Giraldez
On Tue, 2004-11-23 at 21:03 -0500, Javier Guerra wrote:
> On Tuesday 23 November 2004 1:04 pm, Daniel Silverstone wrote:
> > There is Aranha at http://aranha.pepperfish.net/
> i'm currently checking it.  still not sure how it works, but so far looks more 
> like what i'd want.

If you want more samples; let me know

> > It's currently being rewritten from the ground-up but it does work well
> what are the main goals of the rewrite? just code cleaning and update? or the 
> dynamic process manager mentioned on the webpage?

Lua 5.1, allowing for the dynamic process manager, Better class support
from the ground up, cleaner code. There are many things which will be
better in the new Aranha.

> and about the several-workers model... can different workers get the same 
> persistent objects? or is there any affinity management?

The new aranha will allow object sharing via the cache. Currently if I
want to share data across HTTP requests I either serialise it into form
variables or I use a database session manager.

I have placed a copy of the session manager module at
http://users.pepperfish.net/dsilvers/util.session/ for you to look at.

D.

-- 
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 2BC8 4016 2068 7895



Reply | Threaded
Open this post in threaded view
|

RES: CGILua future

André de Leiradella
In reply to this post by Javier Guerra Giraldez
Javier Guerra [[hidden email]] wrote:
> right, the startup time is far lower than old CGI, the only overhead
> seems to be the creation of the new environment (and maybe
> garbage-collecting the old ones).  but the main advantage of Java
> that i'd love to see is the persistence of in-RAM objects.  J2EE has
> a lot of complex mechanisms to reduce serialization and
> deserializations, those are possible just because the object
> containers persist from one HTTP request to other. 
> 
> a simple sessions library, like that of PHP, can make it easier to
> retrieve objects from storage; but maybe the most Lua-like solution
> would be just a global table where i could put any object (or tree of
> objects) that i want to persist for another request.  then, i just
> need a cookie to store the key. maybe some extra features like
> setting a timeout to purge the object (to have expiring sessions)

Take a look at lper, a persistent Lua implementation that relies on a
malloc library that uses a mmapped file alloc memory. Don't know if it's
easy to make it work with FastCGI or in a multithreaded environment
though...

Andre de Leiradella


Reply | Threaded
Open this post in threaded view
|

Re: RES: CGILua future

Luiz Henrique de Figueiredo
Javier Guerra wrote:
> a simple sessions library, like that of PHP, can make it easier to
> retrieve objects from storage;

André de Leiradella wrote:
> Take a look at lper, a persistent Lua implementation

That works only in Unix.

You can try also a database library, such as lgdbm, which has proxy tables
that make tables persistent.
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: RES: CGILua future

Daniel Silverstone
In reply to this post by André de Leiradella
On Wed, 2004-11-24 at 21:41 -0300, Andrà de Leiradella wrote:
> > a simple sessions library, like that of PHP, can make it easier to
> > retrieve objects from storage; but maybe the most Lua-like solution
> > would be just a global table where i could put any object (or tree of
> > objects) that i want to persist for another request.  then, i just
> > need a cookie to store the key. maybe some extra features like
> > setting a timeout to purge the object (to have expiring sessions)
> Take a look at lper, a persistent Lua implementation that relies on a
> malloc library that uses a mmapped file alloc memory. Don't know if it's
> easy to make it work with FastCGI or in a multithreaded environment
> though...

Aranha has a util.session module (available from
http://users.pepperfish.net/dsilvers/util.session/ ) which uses a
database to store the session contents. Perhaps you could take ideas
from that.

D.

-- 
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 2BC8 4016 2068 7895




Reply | Threaded
Open this post in threaded view
|

Re: RES: CGILua future

Javier Guerra Giraldez
In reply to this post by Luiz Henrique de Figueiredo
On Wednesday 24 November 2004 6:51 pm, Luiz Henrique de Figueiredo wrote:
> Javier Guerra wrote:
> > a simple sessions library, like that of PHP, can make it easier to
> > retrieve objects from storage;
>
> André de Leiradella wrote:
> > Take a look at lper, a persistent Lua implementation
>
> That works only in Unix.
>
> You can try also a database library, such as lgdbm, which has proxy tables
> that make tables persistent.
> --lhf

that's exactly my point: the "venv() for each request" model creates the 
necessity of a persistence engine.  take a look at prevayler, they claim 
enormous speedups, just because they don't have to (de)serialize everything.

what they take advantage of, is that in Java, the object state isn't flushed 
for each request.  the only other system i know that can do that is FastCGI, 
because it's a single process managing requests in an event loop.

what i'm thinking now is to simplify things radically:  i'll take just the 
fastcgi launcher of CGILua, and replace the lua-side with a minimal loop.  to 
keep different applications separate, i'll run a different launcher process 
instance for each application (i think the apache .conf lets me do it easily)

that means that each web app will have it's own lua state; one that doesn't go 
away after each request.

on this environment, a small API would let the app register scripts referenced 
by the URL, and state objects referenced by a session cookie.  no 
(de)serializing needed unless the app wants to make 'permanent' records (to a 
SQL db, usually)

does this sound more Lua-like?

-- 
Javier

Attachment: pgpZUbHJ1CwN5.pgp
Description: PGP signature

Reply | Threaded
Open this post in threaded view
|

Re: RES: CGILua future

D Burgess-4
Seems a very sane approach to me.
> what i'm thinking now is to simplify things radically:  i'll take just the
> fastcgi launcher of CGILua, and replace the lua-side with a minimal loop.  to
> keep different applications separate, i'll run a different launcher process
> instance for each application (i think the apache .conf lets me do it easily)
> 
> that means that each web app will have it's own lua state; one that doesn't go
> away after each request.
> 
> on this environment, a small API would let the app register scripts referenced
> by the URL, and state objects referenced by a session cookie.  no
> (de)serializing needed unless the app wants to make 'permanent' records (to a
> SQL db, usually)
> 
> does this sound more Lua-like?
> 
> --
> Javier
> 
> 
> 


-- 
DB

Reply | Threaded
Open this post in threaded view
|

RE: RES: CGILua future

Andre Carregal
In reply to this post by Javier Guerra Giraldez
While I have to agree with your arguments, may I suggest a different
approach?

Kepler separates the problem in three levels:

- A launcher is the lowest level communication layer, it just talks with the
web server.

- CGILua is a template processor (in fact the templates are called Lua
Pages) library that does not know about the web server and how the request
has been launched.

- A Kepler application is a series of Lua Pages and/or Lua scripts that make
something using HTTP, CGILua and other libraries.


>From what has been said about FastCGI and Session Affinity (SA), I think you
could benefit from CGILua and Lua Page functionality simply implementing a
session server (like the one on http://www.tfarmbruster.com/document.htm)
that presents itself as a CGILua launcher.

As long as you are able to maintain SA using this session server, the use of
VEnv would stop being a hindrance and offer you some scope containment
benefits. Or if you'd prefer, simply remove VEnv from the equation and you
can have Lua Pages using SA on a surely very fast setup.

My insistence on the use of CGILua is simply a matter of template
compatibility, so your solution would add another launching/developing
option to Kepler, instead of becoming another template processing option.

I know that I am obviously biased on my proposal, but I sincerely believe
that Lua Pages are a nice solution to template processing, and that the
faster they are handled the happier their users will be. :o)

Andre



> -----Original Message-----
> From: [hidden email]
> [[hidden email] Behalf Of Javier Guerra
> Sent: Thursday, November 25, 2004 10:02 AM
> To: Luiz Henrique de Figueiredo; Lua list
> Subject: Re: RES: CGILua future
...
> what i'm thinking now is to simplify things radically:  i'll
> take just the
> fastcgi launcher of CGILua, and replace the lua-side with a
> minimal loop.  to
> keep different applications separate, i'll run a different
> launcher process
> instance for each application (i think the apache .conf lets
> me do it easily)
>
> that means that each web app will have it's own lua state;
> one that doesn't go
> away after each request.
>
> on this environment, a small API would let the app register
> scripts referenced
> by the URL, and state objects referenced by a session cookie.  no
> (de)serializing needed unless the app wants to make
> 'permanent' records (to a
> SQL db, usually)
>
> does this sound more Lua-like?
>
> --
> Javier
>


Reply | Threaded
Open this post in threaded view
|

Re: RES: CGILua future

Javier Guerra Giraldez
On Thursday 25 November 2004 11:22 am, Andre Carregal wrote:
> While I have to agree with your arguments, may I suggest a different
> approach?

of course... i have no plans of doing yet another template processor.  i'll 
research a bit more about that CGILua layer to make my system usable as just 
a new optional launcher.

what i have in mind would be something like this:

- mod_CGI with SA
- a fastcgi launcher (i think yours can work as-is)
- a main lua script, that calls a user's script on startup (before any 
request).
- that user script registers some functions on a table, keyed by path strings
- on every request, if the path specified by the URL is on the table, the 
referenced function is called
- if the URL isn't on the table, a Lua Page is called just as in CGILua, but 
the venv() would be optional.

a few additions would be pretty obvious:

- add a user sessions table, with an entry for every web user.  the main loop 
would fetch it before calling the registered functions or Lua Pages, who can 
do anything they want on this object.
- the startup script could create a few venv()s and register them with 
subtrees, so the web app could have a few separate environments.


my long-term goal would be to hide the statelessness of http; and later create 
an API that would talk to a simple JavaScript on the client, to create a 
GUI-like programmin plattform.  In principle, that would make html templates 
far less needed.  most of the programming would be GUI-like, with call that 
create elements, read user input, open windows, get user events, etc.


-- 
Javier

Attachment: pgpAgX6Q4lT4E.pgp
Description: PGP signature