.. garbage collection...

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

.. garbage collection...

Andrew Paton
Hi I'm fairly new to lua and just had a quick question for you guys.  I'm
running a single lua state with a bunch of precompiled small lua scripts.
Now the problem I had was that a single script never went anywhere close to
the default garbage collection threshold.  When running another script
though the threshold was bumped up once again, this eventually cascades
until I'm out of memory (small memory model constraints for what I'm doing).
The question I have was how is the proper way to fix this.  I simply after
running a script forced the threshold to be 1 and life was all good.  Is
this proper?



Reply | Threaded
Open this post in threaded view
|

Re: .. garbage collection...

Roberto Ierusalimschy
> Hi I'm fairly new to lua and just had a quick question for you guys.  I'm
> running a single lua state with a bunch of precompiled small lua scripts.
> Now the problem I had was that a single script never went anywhere close to
> the default garbage collection threshold.  When running another script
> though the threshold was bumped up once again, this eventually cascades
> until I'm out of memory (small memory model constraints for what I'm doing).

Could you send us a minimum version of the scripts that reproduce the
problem? That should not happen :-(

-- Roberto

Reply | Threaded
Open this post in threaded view
|

RE: .. garbage collection...

Andrew Paton
Ok here's a simple script and how were running it...  lets assumed its
pre-compiled..

-- our script
print("hello\n")

//-  now our function is this..

static int Print(lua_State *L)
{
    int n=lua_gettop(L);
    int i;

    printf("Running Script Command : Print\n");
    for (i=1; i<=n; i++)
    {
        if (i>1)
            printf("     ");
        if (lua_isstring(L,i))
            printf("%s",lua_tostring(L,i));
        else

printf("%s:%p",lua_typename(L,lua_type(L,i)),lua_topointer(L,i));
    }
    return 0;
}

// - now this function is in a lib, heres the mappings.


static const struct luaL_reg Our_funcs[] =
{
    {"print",Print},
};


LUALIB_API void lua_Ourlibopen (lua_State *L)
{
  luaL_openl(L, Our_funcs);
}

// - to run this pre compiled script were calling... note gLState has been
pre-initialized.
// and some globals (3 in this case) added to it before running these
scripts.

iReturn = lua_dobuffer (gLState,buff,size,name);

// now after doing this our memory prints keep going up.. and taking a look
at the
// check garbage collector it seems that as nblocks is << threshold for this
script
// and when we call lua_dobuffer it runs the protected parser which does
this.

luaC_checkGC(L);
old_blocks = L->nblocks;
status = luaD_runprotected(L, f_parser, &p);
if (status == 0) {
    /* add new memory to threshold (as it probably will stay) */
    L->GCthreshold += (L->nblocks - old_blocks);

// this adding to the threshold constantly makes it luaC_checkGC(L); not
clean up next time around
// until I'm out of memory... so I had to call this after the do_buffer to
fix it.

    lua_setgcthreshold(gLState,1);








-----Original Message-----
From:	[hidden email] [[hidden email]]
On Behalf Of Roberto Ierusalimschy
Sent:	Friday, January 26, 2001 10:56 AM
To:	Multiple recipients of list
Subject:	Re: .. garbage collection...

> Hi I'm fairly new to lua and just had a quick question for you guys.  I'm
> running a single lua state with a bunch of precompiled small lua scripts.
> Now the problem I had was that a single script never went anywhere close
to
> the default garbage collection threshold.  When running another script
> though the threshold was bumped up once again, this eventually cascades
> until I'm out of memory (small memory model constraints for what I'm
doing).

Could you send us a minimum version of the scripts that reproduce the
problem? That should not happen :-(

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: .. garbage collection...

Roberto Ierusalimschy
> Hi I'm fairly new to lua and just had a quick question for you guys.  I'm
> running a single lua state with a bunch of precompiled small lua scripts.
> Now the problem I had was that a single script never went anywhere close
to
> the default garbage collection threshold.  When running another script
> though the threshold was bumped up once again, this eventually cascades
> until I'm out of memory (small memory model constraints for what I'm doing).

Yeah, this is a tricky bug. When we compile a chunk, we add the memory
used by its code to the threshold, because usually this is not garbage;
we then avoid useless GC cicles. But in your case, you keep running new
chunks over and over, and those chunks do not use any extra memory!
So, the counter nblocks never gets GCthreshold...

One solution to this problem can be to remove this "optimization";
another (maybe better) is to add only a fraction of the code, for
instance to change  ldo:252 to

    L->GCthreshold += (L->nblocks - old_blocks)/2;


-- Roberto

Reply | Threaded
Open this post in threaded view
|

Why no select in luasocket ?

Cary O'Brien
Hi,

We're evaluating Lua for an embedded application, so I'm sure
to have a bunch of questions in the next few weeks.  So,
to start things out...

Why isn't there select() in luasocket?

-- cary
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Why no select in luasocket ?

Diego Nehab-3
Hi,

> [hidden email] wrote:
> We're evaluating Lua for an embedded application, so I'm sure
> to have a bunch of questions in the next few weeks.  So,
> to start things out...
> 
> Why isn't there select() in luasocket?

All TCP/IP I/O in LuaSocket is non-blocking plus all input is buffered
by the toolkit. Timeout values are provided to substitute select where
it would be called to avoid blocking. The library uses select to avoid
busy waits while performing non-blocking I/O.

The only use of select by a script would to find out which of several
sockets is ready for a given operation, thus alowing multiplexing. The
library provides poll for that, and soon multi-threading within Lua will
render this use less important.

I agree using poll will turn the multiplexing into a busy wait, while
select would block the process, but select has several idiosincrasies
and I was not willing to implement my own version of it (because of
buffered I/O, I can't just export it to Lua) and yet keep it's problems.

If there is anything you can't do with the current API, please clarify
and I will try to help.

Regards,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: Why no select in luasocket ?

Edgar Toernig
Hi,

Diego Nehab wrote:
> > Why isn't there select() in luasocket?
> 
> The only use of select by a script would to find out which of several
> sockets is ready for a given operation, thus alowing multiplexing. The
> library provides poll for that, and soon multi-threading within Lua will
> render this use less important.

Oh please, don't shoot pthreads at this.  Is it that difficult to add
for example a wait function that gets a list of sockets?  Polling is
nearly useless for anything but embedded systems or DOS.  And timeouts
add an artificial latency that only some applications may tolerate.

> If there is anything you can't do with the current API, please clarify
> and I will try to help.

I only had a short look on luasocket but I would say: wait for any one
of two sockets to become readable ;-)

Ciao, ET.



Reply | Threaded
Open this post in threaded view
|

Re: Why no select in luasocket ?

Diego Nehab-3
> > > Why isn't there select() in luasocket?
> > The only use of select by a script would to find out which of several
> > sockets is ready for a given operation, thus alowing multiplexing. The
> > library provides poll for that, and soon multi-threading within Lua will
> > render this use less important.
> Oh please, don't shoot pthreads at this.  Is it that difficult to add
> for example a wait function that gets a list of sockets?  Polling is
> nearly useless for anything but embedded systems or DOS.  And timeouts
> add an artificial latency that only some applications may tolerate.

Oh please, better your tone. Some of us that are not native english
speakers might get offended by the (hopefully) aparent sarcasm of your
comments. Try also being less categorical with your statements, lest
they prove unwise.  

I understand the point you are trying to make, and I will think about a
nice and clean way to implement select. It is not as obvious at it may
seem, but then I am not God. See, I don't like select and I don't want
to use the same interface to the functionality it provides. Do you have
any suggestion (besides running toLua in /usr/include)?

> > If there is anything you can't do with the current API, please clarify
> > and I will try to help.
> I only had a short look on luasocket but I would say: wait for any one
> of two sockets to become readable ;-)

That has already been said, by me. If you take better look, you will see
that LuaSocket was created mainly to be used on the client side. The
original idea was to provide support for for HTTP, SMTP and FTP.
Server-side support is being improved with time.  

Someone asked about a HTTP server written in Lua. We might try something
like that soon, multi-threaded most likely. By then, maybe the select
(or whatever) function will be available.

Regards,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: Why no select in luasocket ?

Edgar Toernig
Diego Nehab wrote:
> 
> > > > Why isn't there select() in luasocket?
> > > The only use of select by a script would to find out which of several
> > > sockets is ready for a given operation, thus alowing multiplexing. The
> > > library provides poll for that, and soon multi-threading within Lua will
> > > render this use less important.
> > Oh please, don't shoot pthreads at this.  Is it that difficult to add
> > for example a wait function that gets a list of sockets?  Polling is
> > nearly useless for anything but embedded systems or DOS.  And timeouts
> > add an artificial latency that only some applications may tolerate.
> 
> Oh please, better your tone. Some of us that are not native english
> speakers might get offended by the (hopefully) aparent sarcasm of your
> comments.

I'm sorry.  It was not in my intend to offend anyone.  I'm not a native
english speaker myself and I've "honestly" no clue of what is offensive
in that paragraph.  Is it the "don't shoot pthreads"?  That's because I
saw too much simple problems being solved with pthreads.
Or is it the question "Is it that difficult..."?  It was really just meant
as that, a question if it is that difficult to implement.  I see no
obvious problems at least on Unix systems.

And about the two statements ...
> Try also being less categorical with your statements, lest they prove
> unwise.

If you think they are wrong then correct/flame/shoot/make an idiot of me.

> I understand the point you are trying to make, and I will think about a
> nice and clean way to implement select. It is not as obvious at it may
> seem, but then I am not God. See, I don't like select and I don't want
> to use the same interface to the functionality it provides. Do you have
> any suggestion (besides running toLua in /usr/include)?

Hmm... you don't like the interface...  I made a different interface
for some app.  A register function that gets an fd and a callback function.
And then a generic wait with only a timeout.  The wait will select on
all registered fds and will call the callback of the fd if something
happens. 
Or another one: simulate an event queue.  You register fds.  The next-
event function selects if no events are present and then builds events
on the select result.  (This version may get problems because it
caches results.)

The question is: is it easier to export select and make the register/wait
in the highlevel or to implement it completely in C?

If you want something really different I guess you have to switch to
asynch io and completion signals.  But that's IMHO even uglier, less
portable and much more difficult to implement.

> > > If there is anything you can't do with the current API, please clarify
> > > and I will try to help.
> > I only had a short look on luasocket but I would say: wait for any one
> > of two sockets to become readable ;-)
> 
> That has already been said, by me. If you take better look, you will see
> that LuaSocket was created mainly to be used on the client side. The
> original idea was to provide support for for HTTP, SMTP and FTP.
> Server-side support is being improved with time.

For a single connection everything _is_ there.  Select on one fd is like
your poll.  It's only necessary if you really want to handle multiple
fds.  That's what I wanted to say with "wait for two" ;)  And trying to
solve this with "read with small timeout and poll all others" is IMHO
just an ugly hack.

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Why no select in luasocket ?

Cary O'Brien
> Diego Nehab wrote:
> > 

I wrote a while ago..

> > > > > Why isn't there select() in luasocket?
> > > > The only use of select by a script would to find out which of several
> > > > sockets is ready for a given operation, thus alowing multiplexing. The
> > > > library provides poll for that, and soon multi-threading within Lua will
> > > > render this use less important.
> > > Oh please, don't shoot pthreads at this.  Is it that difficult to add
> > > for example a wait function that gets a list of sockets?  Polling is
> > > nearly useless for anything but embedded systems or DOS.  And timeouts
> > > add an artificial latency that only some applications may tolerate.
> > 

[snip]

> Hmm... you don't like the interface...  I made a different interface
> for some app.  A register function that gets an fd and a callback function.
> And then a generic wait with only a timeout.  The wait will select on
> all registered fds and will call the callback of the fd if something
> happens. 

This sort of event-driven interface is available in TCL (which also
has plain old select) .  It is called "fileevent".  You can set up
callbacks to run when the file is readable or writeable.  This lets
you set up multi-threaded (sort of) programs without threads.  TCL
handles the select loop.  This originally came out of the requirements
of running TCL under X (i.e. TK), but I use it a lot for server
programs that have to manage multiple clients.

> Or another one: simulate an event queue.  You register fds.  The next-
> event function selects if no events are present and then builds events
> on the select result.  (This version may get problems because it
> caches results.)
> 
> The question is: is it easier to export select and make the register/wait
> in the highlevel or to implement it completely in C?
> 

The fileevent interface and having the select loop in the interpreter
allows some incredible things to happen.

I have one very, very complex server implemented with fileevents [1].
I can start TCL from the command line, start the server from this
TCL session, and since the interpreter multiplexes both console and
application IO for me I can examine global data structures and run
functions from the command line WHILE THE SERVER IS RUNNING.

The event-driven framework in TCL is way beyond any other scripting
language, including Python and Perl.

But this is a lot of work to add to LUA probably.

> If you want something really different I guess you have to switch to
> asynch io and completion signals.  But that's IMHO even uglier, less
> portable and much more difficult to implement.
> 

Yuck.

[snip]

[1] Well, not exactly fileevents, but the C library for that particular
protocol had callbacks that work exactly loke file events.

-- cary 

> > > > If there is anything you can't do with the current API, please clarify
> > > > and I will try to help.
> > > I only had a short look on luasocket but I would say: wait for any one
> > > of two sockets to become readable ;-)
> > 
> > That has already been said, by me. If you take better look, you will see
> > that LuaSocket was created mainly to be used on the client side. The
> > original idea was to provide support for for HTTP, SMTP and FTP.
> > Server-side support is being improved with time.
> 
> For a single connection everything _is_ there.  Select on one fd is like
> your poll.  It's only necessary if you really want to handle multiple
> fds.  That's what I wanted to say with "wait for two" ;)  And trying to
> solve this with "read with small timeout and poll all others" is IMHO
> just an ugly hack.
> 
> Ciao, ET.
> 


Reply | Threaded
Open this post in threaded view
|

Re: Why no select in luasocket ?

Adolf Mathias
Cary O'Brien wrote:

> 
> This sort of event-driven interface is available in TCL (which also
> has plain old select) .  It is called "fileevent".  You can set up
> callbacks to run when the file is readable or writeable.  This lets
> you set up multi-threaded (sort of) programs without threads.  TCL
> handles the select loop.  This originally came out of the requirements
> of running TCL under X (i.e. TK), but I use it a lot for server
> programs that have to manage multiple clients.
> 
> > Or another one: simulate an event queue.  You register fds.  The next-
> > event function selects if no events are present and then builds events
> > on the select result.  (This version may get problems because it
> > caches results.)
> >
> > The question is: is it easier to export select and make the register/wait
> > in the highlevel or to implement it completely in C?
> >
> 
> The fileevent interface and having the select loop in the interpreter
> allows some incredible things to happen.
> 
> I have one very, very complex server implemented with fileevents [1].
> I can start TCL from the command line, start the server from this
> TCL session, and since the interpreter multiplexes both console and
> application IO for me I can examine global data structures and run
> functions from the command line WHILE THE SERVER IS RUNNING.
> 
> The event-driven framework in TCL is way beyond any other scripting
> language, including Python and Perl.
> 
> But this is a lot of work to add to LUA probably.

I don't really think that this would be too difficult: Actually, I wrote
such a thing some time ago, although it is not very clean and very C
file descriptor centric, and so I wouldn't give it away right now. But
essentially, it is a generic C representation of an event source (for
now, file descriptor and timer) and a main loop consisting of repetitive
calls to an event dispatcher. And it's easy for the main loop to invoke
e.g. Lua callbacks.

Dolfi

Reply | Threaded
Open this post in threaded view
|

No select in luasocket, yet...

Diego Nehab-3
In reply to this post by Edgar Toernig
Hi all,

I do not plan to implement event-driven I/O in LuaSocket, at least not
in the near future. So, to solve the "wait-on-several-sockets" problem,
I would rather have a cleaner interface to select.

The library caches it's reads from sockets (thus achieving 10-20 fold
increase in the performance of line mode I/0), so that I have to check
if there is data available in socket buffers before calling select.
Because of that, I can't just export select and I am definitely not
exporting FD_* "macros".

I was thinking about a select that receives a table with the socket
objects to be queried and returns a table with those that pass the
query. The function call also receives a string describing the type of
query (receive or send) and an optional timeout value.

    a = s:accept(); b = s:accept(); c = s:accept()
    -- checks who is readable
    r = select({a, b, c}, "r", 2)

After the call, if only b and c were readable, r would equal {b,c} (or
{c,b}). Would everyone be happy with that? Maybe someone has a better
idea...

Regards,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: .. garbage collection...

John Belmonte-2
In reply to this post by Roberto Ierusalimschy
Roberto wrote:
> Yeah, this is a tricky bug. When we compile a chunk, we add the memory
> used by its code to the threshold, because usually this is not garbage;
> we then avoid useless GC cicles. But in your case, you keep running new
> chunks over and over, and those chunks do not use any extra memory!
> So, the counter nblocks never gets GCthreshold...

I just got burned by this also, by having a small lua_dostring in a main loop that just accesses
a variable or calls a function that uses no memory.

In fact if you want to make your computer very unhappy, run this Lua script:

    while not nil do
        dostring("return nil -- have a nice day!")
    end


-John