User setting of GC Threshold

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

User setting of GC Threshold

Peter Hill-2
There was an email to the PLua list recently (PLua = Lua on the PalmOS)
relating to GC Thresholds that I think deserves mentioning.

Basically the person was writting a Lua app which uses more than half of the
availble memory (palmtops being rather limited in memory) and the "used x 2"
auto threshold was too great.

It seems rather limiting to me to just allow this one predefined
auto-threshold. It would be more flexible to permit the user to define an
automatic post-GC function which is called upon completion of GC with the
current memory info, and which returns the new desired threshold.

Cheers,
Peter Hill.



Reply | Threaded
Open this post in threaded view
|

Re: User setting of GC Threshold

Roberto Ierusalimschy
> It seems rather limiting to me to just allow this one predefined
> auto-threshold. It would be more flexible to permit the user to define an
> automatic post-GC function which is called upon completion of GC with the
> current memory info, and which returns the new desired threshold.

1) Programming is rather limiting.

2) "The cheapest, fastest, and more reliable components of a system are
those that aren't there." Or, as Asko pointed out, "Every additional
feature also adds to the learning curve, documentation pages, etc.
etc.".

3) When we stop asking for new features, we have more time (and
motivation) to realize how to implement them.

(I leave this "automatic post-GC function" as an exercise. Hints:
* "automatic post-GC function" -> gc metamethod
* "current memory info" -> lua_getgcthreshold/count"
* "returns the new desired threshold" -> lua_setgcthreshold
)

-- Roberto


Reply | Threaded
Open this post in threaded view
|

RE: User setting of GC Threshold

Bilyk, Alex
In reply to this post by Peter Hill-2
Good points. On the relating note I would like to assert again that line 443 in ldo.c saying 

	G(L)->GCthreshold += (G(L)->nblocks - old_blocks);

will lead to eventual consumption of ALL available computer memory under the right conditions regardless of the platform. 

It had happened in one of our biggest commercial projects about a week prior to release date. It was easily fixed (by removing this line and afterwards by avoiding too much parsing) but it scared the hell out of just about everyone. I would not want to imagine what would have happened if this problem was discovered after our release. The problem is that under certain conditions the line above effectively disables GC.

AB

.

-----Original Message-----
From: Roberto Ierusalimschy [[hidden email]]
Sent: Tuesday, February 25, 2003 3:33 AM
To: Multiple recipients of list
Subject: Re: User setting of GC Threshold 

> It seems rather limiting to me to just allow this one predefined
> auto-threshold. It would be more flexible to permit the user to define an
> automatic post-GC function which is called upon completion of GC with the
> current memory info, and which returns the new desired threshold.

1) Programming is rather limiting.

2) "The cheapest, fastest, and more reliable components of a system are
those that aren't there." Or, as Asko pointed out, "Every additional
feature also adds to the learning curve, documentation pages, etc.
etc.".

3) When we stop asking for new features, we have more time (and
motivation) to realize how to implement them.

(I leave this "automatic post-GC function" as an exercise. Hints:
* "automatic post-GC function" -> gc metamethod
* "current memory info" -> lua_getgcthreshold/count"
* "returns the new desired threshold" -> lua_setgcthreshold
)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: User setting of GC Threshold

Peter Hill-2
In reply to this post by Roberto Ierusalimschy
Peter Hill:
> It seems rather limiting to me to just allow this one predefined
> auto-threshold. It would be more flexible to permit the user to define an
> automatic post-GC function which is called upon completion of GC with the
> current memory info, and which returns the new desired threshold.

Roberto Ierusalimschy:
> (I leave this "automatic post-GC function" as an exercise. Hints:
> * "automatic post-GC function" -> gc metamethod
> * "current memory info" -> lua_getgcthreshold/count"
> * "returns the new desired threshold" -> lua_setgcthreshold

Ok, I'll pass that on to the PLua list. It isn't a solution I normally would
have considered because (a) it is not obvious from the manual (and hence not
to be relied upon) that the new post-GC threshold level would already have
been determined & available before the __gc metamethods were run and (b)
since setting a new threshold can cause an immediate GC, it seems rather
dangerous to me to do so inside a __gc metamethod (which itself is called as
an ongoing part of the *current* GC).


> 1) Programming is rather limiting.

??? 8-O


> 2) "The cheapest, fastest, and more reliable components of a system are
> those that aren't there." Or, as Asko pointed out, "Every additional
> feature also adds to the learning curve, documentation pages, etc.
> etc.".
>
> 3) When we stop asking for new features, we have more time (and
> motivation) to realize how to implement them.

True. I agree with both of those in principle, and I heartily support the
goal of minimising Lua.

But... "Every additional feature also adds to the learning curve,
documentation pages, etc. etc." is not always the case.

In some cases removed features can be so commonly used (and hence
re-implemented by the user) that they become default features despite not
being official. So one still has to wade through documentation... except now
there may be several slightly different versions :-(.

And in some cases adding a feature may reduce documentation. To add the GC
threshold solution you have suggested above requires that the Lua manual
contain additional documentation explaining at what phase during GCing the
threshold is set, and what limitations are placed upon setting the threshold
(and hence potentially initiating a new GC cycle) within a __gc metamethod.
Or you could add the line "Upon completion of GC the method _THRESHOLD is
called to establish the new threshold..." and skip all that. :-)

*cheers*
Peter Hill.



Reply | Threaded
Open this post in threaded view
|

Re: User setting of GC Threshold

Roberto Ierusalimschy
In reply to this post by Bilyk, Alex
> Good points. On the relating note I would like to assert again that   
> line 443 in ldo.c saying                                              
>
>         G(L)->GCthreshold += (G(L)->nblocks - old_blocks);
>
> will lead to eventual consumption of ALL available computer memory
> under the right conditions regardless of the platform.

This is one of the nine known bugs of Lua 4.0. I think it is corrected
in 4.0.1. (The line is still there, but not the bug.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

WinHeap

Erik Hougaard
Have anyone tried Lua with WinHeap (www.winheap.com) ?

Some of their sounds like it could give a boost to a lua based
application...

/Erik


Reply | Threaded
Open this post in threaded view
|

Registering a C function into a Lua table

Nick Davies
In reply to this post by Roberto Ierusalimschy
Hi!
I'm a little new at Lua, so I probably missed something obvious, but I can't
seem to register a C function into a Lua table. What I want to do is
register the static function State::DoubleNumber into the Lua table State so
that I can call it from Lua code as State.DoubleNumber. I'm using Lua
5.0-beta. This is the C++ code that I'm using:

...
    Command("State = {}");
    Register(State::DoubleNumber, "State", "DoubleNumber");
...
void State::Command(const char* str)
{
    luaL_loadbuffer(L, str, strlen(str), "Command");
    lua_call(L, 0, 0);
}
...
void State::Register(Method method, const char* module, const char* name)
{
    lua_pushstring(L, module);
    lua_pushstring(L, name);
    lua_pushcfunction(L, method);
    lua_settable(L, 1);        // << crashes here
    lua_pop(L, 1);
}
...
Method is a typedef:
...
    typedef int (*Method)(lua_State*);
...

The code crashes on the commented line in State::Register. Unfortunately, I
have to go to school now, so I can't be any more verbose, but does anyone
know what I'm doing wrong? I have also checked that the stack item at
position 1 is indeed a string with the value "State" just before the line
that crashes.

Thanks,
Nick



Reply | Threaded
Open this post in threaded view
|

RE: Registering a C function into a Lua table

Vincent Penquerc'h-4
Title: RE: Registering a C function into a Lua table

>     lua_pushstring(L, module);
>     lua_pushstring(L, name);
>     lua_pushcfunction(L, method);
>     lua_settable(L, 1);        // << crashes here

(at least in old versions) lua_settable expects a table as first
argument, rather than a string.

--
Vincent Penquerc'h

Reply | Threaded
Open this post in threaded view
|

Re: WinHeap

Björn De Meyer
In reply to this post by Erik Hougaard
Erik Hougaard wrote:
> 
> Have anyone tried Lua with WinHeap (www.winheap.com) ?
> 
> Some of their sounds like it could give a boost to a lua based
> application...
> 
> /Erik

Sorry, but that site was a waste of time to me. There are tons of other 
"heap managers" -read malloc() implementations- out there, many of them 
free software, and most of them better than those of Microsoft. 
It's one of the reasons non-MS OSes usually run faster. I'll take
amply-tested open software solutions, like Lua or a GNU malloc, over 
some obscure proprietary product with rather absurd claims.

-- 
"No one knows true heroes, for they speak not of their greatness." -- 
Daniel Remar.
Björn De Meyer 
[hidden email]

Reply | Threaded
Open this post in threaded view
|

SV: WinHeap

Erik Hougaard
The main reason for my interest, is the fact that it directly aimed at
multithreaded/Multi CPU Win32 applications. Must of the free software heap
managers are not finetuned for Win32 multithreaded applications.

But this should turn into a proprietary vs. open source discusion. I'm
developing commercial software and if I can buy something that makes me earn
a bigger paycheck - I will go for that (Need to eat and make
house-payments)! And my customers wants Win32 applications so thats what I'm
developing - So please drop the political attitude - I'm looking for
technical information!

/Erik

-----Oprindelig meddelelse-----
Fra: [hidden email]
[[hidden email]; vegne af Björn De Meyer
Sendt: 26. februar 2003 14:10
Til: Multiple recipients of list
Emne: Re: WinHeap


Erik Hougaard wrote:
>
> Have anyone tried Lua with WinHeap (www.winheap.com) ?
>
> Some of their sounds like it could give a boost to a lua based
> application...
>
> /Erik

Sorry, but that site was a waste of time to me. There are tons of other
"heap managers" -read malloc() implementations- out there, many of them
free software, and most of them better than those of Microsoft.
It's one of the reasons non-MS OSes usually run faster. I'll take
amply-tested open software solutions, like Lua or a GNU malloc, over
some obscure proprietary product with rather absurd claims.

--
"No one knows true heroes, for they speak not of their greatness." --
Daniel Remar.
Björn De Meyer
[hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: WinHeap

Nick Trout-5
In reply to this post by Erik Hougaard

> On Behalf Of Erik Hougaard
> Sent: February 26, 2003 4:47 AM
> Subject: SV: WinHeap
> 
> The main reason for my interest, is the fact that it directly aimed at
> multithreaded/Multi CPU Win32 applications. Must of the free 
> software heap
> managers are not finetuned for Win32 multithreaded applications.
> 
> But this should turn into a proprietary vs. open source discusion. I'm
> developing commercial software and if I can buy something 
> that makes me earn
> a bigger paycheck - I will go for that (Need to eat and make
> house-payments)! And my customers wants Win32 applications so 
> thats what I'm
> developing - So please drop the political attitude - I'm looking for
> technical information!


http://lua-users.org/wiki/MemoryAllocation

Hoard [3] "Hoard is a fast, scalable and memory-efficient allocator for
multiprocessors. Hoard solves the heap contention problem caused when
multiple threads call dynamic memory allocation functions like malloc()
and free() (or new and delete). Hoard can dramatically improve the
performance of multithreaded programs running on multiprocessors." 
http://www.cs.utexas.edu/users/emery/hoard/

Please feed back any information to
http://lua-users.org/wiki/MemoryAllocation

Regards,
Nick




Reply | Threaded
Open this post in threaded view
|

Re: Registering a C function into a Lua table

Nick Davies
In reply to this post by Vincent Penquerc'h-4
RE: Registering a C function into a Lua table> (at least in old versions)
lua_settable expects a table as first
> argument, rather than a string.

It's just that I didn't know how to get the table State onto the stack. But,
I do now, because Asko Kauppi told me how.

Thanks, Vincent and Asko, I managed to get things working. :) I did this by
adding a single line of code, making my "register Lua function" code look
like this:

...
void State::Register(Method method, const char* module, const char* name)
{
    lua_pushstring(L, module);
    lua_gettable(L, LUA_GLOBALSINDEX);    // < new line
    lua_pushstring(L, name);
    lua_pushcfunction(L, method);
    lua_settable(L, 1);
    lua_pop(L, 1);
}
...

Thanks again!

Nick



Reply | Threaded
Open this post in threaded view
|

SV: WinHeap

Erik Hougaard
In reply to this post by Nick Trout-5
-----Oprindelig meddelelse-----
>Hoard [3] "Hoard is a fast, scalable and memory-efficient allocator for
>multiprocessors. Hoard solves the heap contention problem caused when
>multiple threads call dynamic memory allocation functions like malloc()
>and free() (or new and delete). Hoard can dramatically improve the
>performance of multithreaded programs running on multiprocessors."
>http://www.cs.utexas.edu/users/emery/hoard/

Hoard is also on our evaluation list and is looking good. Our legal
department is checking their commerical license model. They need to make
sure that the modified LGPL does not conflict with the commercial license.

/Erik