Redefining functions

classic Classic list List threaded Threaded
12 messages Options
lu
Reply | Threaded
Open this post in threaded view
|

Redefining functions

lu
Hi guys,

I've got a question, when I run this code:

char *t="function foo()\n a=1 \n end\n";

for (int i=0; i < 10000; i++)
  lua_dostring(L, t);


The memory usage of Lua increases dramatically, it seems as though Lua
stores the function "foo" 10.000 times instead of replacing the
previous function. Is this correct ? Is it possible (for me) to
change this behaviour, if so where should i start looking in the
source.

I'm asking this because a program that I'm the author of ( girder ),
is about to switch to Lua as the scripting language, but it turned up
that running the same script (with a function) over and over
again will fill up the memory quickly.

Thanks!

Ron


Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

benjamin sunshine-hill
> char *t="function foo()\n a=1 \n end\n";
>
> for (int i=0; i < 10000; i++)
>   lua_dostring(L, t);

This is bad for two reasons.
First, there are the memory issues you describe. A different copy of the 
function is created each time, and remains until the garbage collector runs.

Even more inneficiently, however, this means that lua has to parse and 
compile your script each time you iterate through the loop! This is obviously 
quite a waste.

For things such as this, I suggest you not use dostring. Instead, before the 
loop use something like "loadstring" (I'm afraid I'm not in front of my API 
reference right now), which will stick the function on the stack. Then, in 
your loop, each time call:

lua_pushvalue(L, 1); // duplicates the function
lua_call(L, 0, 0); // calls the duplicate, leaves the original

after your loop, use lua_pop to remove the original.

Since this merely creates references to the same function, rather than 
copies, it should be much more efficient.

Ben

Reply | Threaded
Open this post in threaded view
|

RE: Redefining functions

Brownsword
In reply to this post by lu
On a related note:

Is it normal operating procedure for people to store their programs as strings, or as pre-compiled bytecode?  I have space and load time concerns and would rather not keep the source files around, if at all possible.  If stored as bytecode, is there any support for handling byte-ordering issues?


-----Original Message-----
From: Ben Sunshine-Hill [[hidden email]]
Sent: Saturday, September 28, 2002 11:48 AM
To: Multiple recipients of list
Subject: Re: Redefining functions


> char *t="function foo()\n a=1 \n end\n";
>
> for (int i=0; i < 10000; i++)
>   lua_dostring(L, t);

This is bad for two reasons.
First, there are the memory issues you describe. A different copy of the 
function is created each time, and remains until the garbage collector runs.

Even more inneficiently, however, this means that lua has to parse and 
compile your script each time you iterate through the loop! This is obviously 
quite a waste.

For things such as this, I suggest you not use dostring. Instead, before the 
loop use something like "loadstring" (I'm afraid I'm not in front of my API 
reference right now), which will stick the function on the stack. Then, in 
your loop, each time call:

lua_pushvalue(L, 1); // duplicates the function
lua_call(L, 0, 0); // calls the duplicate, leaves the original

after your loop, use lua_pop to remove the original.

Since this merely creates references to the same function, rather than 
copies, it should be much more efficient.

Ben

Reply | Threaded
Open this post in threaded view
|

RE: Redefining functions

Basile STARYNKEVITCH
>>>>> "Brownsword," == Brownsword, Andrew <[hidden email]> writes:

    Brownsword,> On a related note: Is it normal operating procedure
    Brownsword,> for people to store their programs as strings, or as
    Brownsword,> pre-compiled bytecode?  I have space and load time
    Brownsword,> concerns and would rather not keep the source files
    Brownsword,> around, if at all possible.  If stored as bytecode,
    Brownsword,> is there any support for handling byte-ordering
    Brownsword,> issues? [...]

Apparently the lua_dostring accept a compiled bytecode string
(compiled with luac). 

Compiled strings starts with the ESC character.

-- 

Basile STARYNKEVITCH         http://starynkevitch.net/Basile/ 
email: basile<at>starynkevitch<dot>net 
alias: basile<at>tunes<dot>org 
8, rue de la Faïencerie, 92340 Bourg La Reine, France

lu
Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

lu
In reply to this post by lu
> This is bad for two reasons.
> First, there are the memory issues you describe. A different copy of the
> function is created each time, and remains until the garbage collector runs.

Ah so if I force the garbage collector to run the memory problems
should go away ? I'll look into that right now.

> Even more inneficiently, however, this means that lua has to parse and
> compile your script each time you iterate through the loop! This is obviously
> quite a waste.

I do realise this, I was only giving an example there, the script that
is being parsed is user supplied (run-time). And it is something different all
the time. But it can be the same and it can be triggered very often
very fast. ( My program reacts to infrared input for example ).
Storing the bytecode is a second step that will be done in
the future.

> For things such as this, I suggest you not use dostring. Instead, before the
> loop use something like "loadstring" (I'm afraid I'm not in front of my API
> reference right now), which will stick the function on the stack. Then, in
> your loop, each time call:

hmm,.. interesting, but probably not doable since I have no knowledge
of what the user wants to do in the script. ( it doesn't have to be a
function ).

Thanks for your suggestions!!

Ron


Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

benjamin sunshine-hill
In reply to this post by Brownsword
On Saturday 28 September 2002 11:52 am, you wrote:
> On a related note:
>
> Is it normal operating procedure for people to store their programs as
> strings, or as pre-compiled bytecode?  I have space and load time concerns
> and would rather not keep the source files around, if at all possible.  If
> stored as bytecode, is there any support for handling byte-ordering issues?

Lua transparently handles loading either source or bytecode. I have my build 
environment set up to include source files in debug builds, and bytecode in 
release builds, with the same filename. Lua has no problem dealing 
identically with each (except that it can't provide as much debugging 
information with stripped bytecode). And bytecode is cross-platform; 
byte-ordering issues are properly handled by the engine.

lu
Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

lu
In reply to this post by lu
>> This is bad for two reasons.
>> First, there are the memory issues you describe. A different copy of the
>> function is created each time, and remains until the garbage collector runs.

> Ah so if I force the garbage collector to run the memory problems
> should go away ? I'll look into that right now.

I tried this:

lua_setgcthreshold (L, lua_getgcthreshold (L));

to force the Garbage collector but still the memory footprint grows
dramatically. Is this the correct way to trigger the Garbage collector
?
Anything else I can do to find out what is going on here ?

Ron


Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

Terence Martin-2
I think what you're shooting for is setting the gc threshold to 0. That
should force a garbage collection cycle.

> > Ah so if I force the garbage collector to run the memory problems
> > should go away ? I'll look into that right now.
>
> I tried this:
>
> lua_setgcthreshold (L, lua_getgcthreshold (L));
>
> to force the Garbage collector but still the memory footprint grows
> dramatically. Is this the correct way to trigger the Garbage collector
> ?
> Anything else I can do to find out what is going on here ?
>
> Ron
>


lu
Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

lu
In reply to this post by lu
> I think what you're shooting for is setting the gc threshold to 0. That
> should force a garbage collection cycle.

I did this and like magic the garbage collector kicks in and cleans
out the stale functions!

     th = lua_getgcthreshold (L);

     lua_setgcthreshold (L, 0);

     lua_setgcthreshold (L, th);


Since the memory footprint was growing by at least 10 megabytes I
figured that the byte count was over the Threshold, but maybe
functions aren't counted in the byte count ? Anyway thanks for the
great help!! ( I'm using version 4.0.1 )

Ron


Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

benjamin sunshine-hill
In reply to this post by lu
> hmm,.. interesting, but probably not doable since I have no knowledge
> of what the user wants to do in the script. ( it doesn't have to be a
> function ).

When you load a lua script into the interpreter, it is _always_ loaded as a 
function. Basically, any block of Lua code is a function, including the top 
level one. So, for instance, this script:

function foo()
    -- do something
end

a = 3
foo()


...consists of two functions: the function "foo", and the top level function. 
So yes, this will always work.

Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

Eero Pajarre-2
In reply to this post by lu
[hidden email] wrote:
Since the memory footprint was growing by at least 10 megabytes I
figured that the byte count was over the Threshold, but maybe
functions aren't counted in the byte count ? Anyway thanks for the
great help!! ( I'm using version 4.0.1 )


Lua parser (at least on my lua 4.x version) seems to
increase gc threshold on purpose after parsing some
new code.

from ldo.c:
  if (status == 0) {
    /* add new memory to threshold (as it probably will stay) */
    L->GCthreshold += (L->nblocks - old_blocks);
  }


In the specific case you described this leads to
a leakage like behavior. I think forcing GC as
you now do is the correct fix.


		Eero



Reply | Threaded
Open this post in threaded view
|

Re: Redefining functions

Björn De Meyer
Eero Pajarre wrote:
> 
> Lua parser (at least on my lua 4.x version) seems to
> increase gc threshold on purpose after parsing some
> new code.
/snip
> In the specific case you described this leads to
> a leakage like behavior. I think forcing GC as
> you now do is the correct fix.
> 
>                 Eero

Of course, in the generic case, what lua does is correct. 
When a function is defined /usually/ it will remain defined.
It is therefore /usually/ a good idea to increase the gc 
threshold to prevent garbage collection whilst the function 
is defined. I think the case where one and the same
function is redefined more than, about 10 times is 
a "pathological" case. Such a situation should be avoided
as much as possible.



-- 
"No one knows true heroes, for they speak not of their greatness." -- 
Daniel Remar.
Björn De Meyer 
[hidden email]