Memory woes

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory woes

Brett Bibby
We have a PS2 game that is in Beta and close to shipping now and I'm still having trouble tracking/controlling Lua's memory usage. Any tips would be greatly appreciated.

Notes:
1. We use Lua 5.0.3 to script the entire game. Each script is an object that is loaded, run and collected individually as needed. We are loading a few hundred lua scripts for the non-game, once the player starts the game we unload all of these, and load a few hundred game scripts. Once game ends, reverse and repeat the process.

2. We do a memory dump from main menu, then run the game, then exit the game and do another dump from the main menu. Each time we do this cycle, memory associated with Lua has increased 145KB to 155KB. The things (quantity and exact items) being loaded both in game and out are fixed.

3. We do manual garbage collections after all unloads, prior to loads, and once again after the load has completed.

4. We have verified that the exact same objects are in memory at each snapshot.

5. We dumped the globals and registry tables and verified that the exact same keys are in lua at each snapshot.

6. The memory reported by lua as being in use doesn't change much (a few KB between runs).

7. Our internal memory manager traps all lua allocations and tags them so we know they are coming from lua. The increase in memory is memory used only. Fragmentation wouldn't effect the number.

8. Shutting down lua and restarting it isn't an option this late in the project.

Debugging Lua isn't the easiest thing I have ever done. If anybody can suggest any additional checks or debugging strategies it would be greatly appreciated.

Are there any code commentary docs anywhere that explain the internals of Lua?

Thanks,
Brett

Reply | Threaded
Open this post in threaded view
|

RE: Memory woes

Richard Ranft
This might be useless, but why not simply start a new Lua VM when leaving
the menu to start the game, then dump the menu VM.  On game exit, start a
new Lua VM again for the menu and just drop the old "game" VM.  I can't see
how this would take any longer than dumping all scripts and then loading a
new set at each transition.  Unless memory allocation is so dreadfully slow
on the PS2 that it makes retaining the memory pool across states mandatory.

Just a thought, and probably not something you haven't already thought of.

Rich


> Subject: Memory woes
>
>
> We have a PS2 game that is in Beta and close to shipping now and
> I'm still
> having trouble tracking/controlling Lua's memory usage. Any tips would be
> greatly appreciated.
>
> Notes:
> 1. We use Lua 5.0.3 to script the entire game. Each script is an
> object that
> is loaded, run and collected individually as needed. We are loading a few
> hundred lua scripts for the non-game, once the player starts the game we
> unload all of these, and load a few hundred game scripts. Once game ends,
> reverse and repeat the process.
>
> 2. We do a memory dump from main menu, then run the game, then
> exit the game
> and do another dump from the main menu. Each time we do this
> cycle, memory
> associated with Lua has increased 145KB to 155KB. The things
> (quantity and
> exact items) being loaded both in game and out are fixed.
>
> 3. We do manual garbage collections after all unloads, prior to
> loads, and
> once again after the load has completed.
>
> 4. We have verified that the exact same objects are in memory at each
> snapshot.
>
> 5. We dumped the globals and registry tables and verified that the exact
> same keys are in lua at each snapshot.
>
> 6. The memory reported by lua as being in use doesn't change much
> (a few KB
> between runs).
>
> 7. Our internal memory manager traps all lua allocations and tags
> them so we
> know they are coming from lua. The increase in memory is memory
> used only.
> Fragmentation wouldn't effect the number.
>
> 8. Shutting down lua and restarting it isn't an option this late in the
> project.
>
> Debugging Lua isn't the easiest thing I have ever done. If anybody can
> suggest any additional checks or debugging strategies it would be greatly
> appreciated.
>
> Are there any code commentary docs anywhere that explain the internals of
> Lua?
>
> Thanks,
> Brett
>


Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Brett Bibby
If I could use multiple VMs, I'd just shut down the one I have and restart it. The problem is that some state data needs to be shared, and while I know there's lots of possibilities, all are ugly at this point.

Does anybody know if there any tech docs available on how Lua is implemented? I find the source isn't very easier to "read"...

Thanks,
Brett

----- Original Message ----- From: "Richard Ranft" <[hidden email]>
To: "Lua list" <[hidden email]>
Sent: Saturday, July 01, 2006 11:41 PM
Subject: RE: Memory woes


This might be useless, but why not simply start a new Lua VM when leaving
the menu to start the game, then dump the menu VM.  On game exit, start a
new Lua VM again for the menu and just drop the old "game" VM. I can't see
how this would take any longer than dumping all scripts and then loading a
new set at each transition. Unless memory allocation is so dreadfully slow on the PS2 that it makes retaining the memory pool across states mandatory.

Just a thought, and probably not something you haven't already thought of.

Rich


Subject: Memory woes


We have a PS2 game that is in Beta and close to shipping now and
I'm still
having trouble tracking/controlling Lua's memory usage. Any tips would be
greatly appreciated.

Notes:
1. We use Lua 5.0.3 to script the entire game. Each script is an
object that
is loaded, run and collected individually as needed. We are loading a few
hundred lua scripts for the non-game, once the player starts the game we
unload all of these, and load a few hundred game scripts. Once game ends,
reverse and repeat the process.

2. We do a memory dump from main menu, then run the game, then
exit the game
and do another dump from the main menu. Each time we do this
cycle, memory
associated with Lua has increased 145KB to 155KB. The things
(quantity and
exact items) being loaded both in game and out are fixed.

3. We do manual garbage collections after all unloads, prior to
loads, and
once again after the load has completed.

4. We have verified that the exact same objects are in memory at each
snapshot.

5. We dumped the globals and registry tables and verified that the exact
same keys are in lua at each snapshot.

6. The memory reported by lua as being in use doesn't change much
(a few KB
between runs).

7. Our internal memory manager traps all lua allocations and tags
them so we
know they are coming from lua. The increase in memory is memory
used only.
Fragmentation wouldn't effect the number.

8. Shutting down lua and restarting it isn't an option this late in the
project.

Debugging Lua isn't the easiest thing I have ever done. If anybody can
suggest any additional checks or debugging strategies it would be greatly
appreciated.

Are there any code commentary docs anywhere that explain the internals of
Lua?

Thanks,
Brett





Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Aaron Brown
Brett wrote:

Does anybody know if there any tech docs available on how
Lua is implemented? I find the source isn't very easier to
"read"...

True.  I haven't seen anything that really gets into the
nitty-gritty, but the following resources may be helpful, at
least to give you a bit more context to do your own digging:

If I remember correctly, "The Implementation of Lua 5.0" has
detailed coverage of things like how upvalues are closed:

 http://www.tecgraf.puc-rio.br/~lhf/ftp/doc/jucs05.pdf

Hopefully (for your purposes) it talks in detail about Lua
5.0 GC too.

(I say "if I remember correctly" because I can't find my
local copy of it, and www.tecgraf.puc-rio.br seems to be
down right now.)

Kein-Hong Man's "No-Frills Guide" explains what all the VM
opcodes do:

 http://luaforge.net/docman/?group_id=83

--
Aaron

Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Asko Kauppi
In reply to this post by Brett Bibby

If all the state you need can be expressed in "INI-like" key, value format, then it wouldn't be a huge endeavour to write C functions capable of accessing such "shared" space. This would be handy otherwise, too, especially if the system would support informing Lua machines of changes to the values (similar to the way GConf is used, between Gtk+ processes).

Perhaps I'll do this kind of a system, but .. won't be there for your launch date (hmm... unless you hire me for this :)

-asko


Research kirjoitti 2.7.2006 kello 10.26:

If I could use multiple VMs, I'd just shut down the one I have and restart it. The problem is that some state data needs to be shared, and while I know there's lots of possibilities, all are ugly at this point.



Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Edgar Toernig
In reply to this post by Brett Bibby
Research wrote:
>
> [...] Each time we do this cycle, memory associated with Lua has
> increased 145KB to 155KB.

> 6. The memory reported by lua as being in use doesn't change much (a few KB 
> between runs).
> 
> 7. Our internal memory manager traps all lua allocations and tags them so we 
> know they are coming from lua. The increase in memory is memory used only. 
> Fragmentation wouldn't effect the number.

As there are, as far as I know, no memory leaks in Lua, I would look
at your memory manager for leaks.

One possible problem, depending on your implementation of memory management:
Lua frees memory by realloc'ing to size 0.  Some implementations of realloc
do not free memory completely in this case but keep a small memory block
allocated.

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Wim Couwenberg-2
Brett,

>> [...] Each time we do this cycle, memory associated with Lua has
>> increased 145KB to 155KB.

>> 6. The memory reported by lua as being in use doesn't change much (a few KB
>> between runs).

> As there are, as far as I know, no memory leaks in Lua, I would look
> at your memory manager for leaks.

I agree with Edgar on the "no memory leaks in Lua" part.  Here are
just some random remarks, based on your observations:

1.  It can take two subsequent gc cycles to collect everything.  This
one is easy to check by doing a double gc when you start/quit your
game.

2.  If you have a stable life time cycle of objects, then Lua should
*not* report "a few KB" increase between runs (I gather your obtain
this via gcinfo), it should report the exact same amount.  The GC in
Lua 5.0 runs atomically and is completely deterministic.  Realloc-ing
to 0 (zero) as Edgar suggested cannot cause this increase.

3.  A few KB increase in Lua objects can easily lead to an increase of
145 to 155 KB in allocated memory if some "heavy" userdata is not
collected that refers to C/C++ objects allocated outside the userdata.
Since Lua doesn't know about the size of such objects it cannot report
it.

4.  There *is* a way to create uncollectable cycles in Lua: if (k, v)
is a key value pair in a weak valued table and v refers (directly or
indirectly) back to k then that pair will never be removed from the
table, even if v is the *only* reference to k.

--
Wim



Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Wim Couwenberg-2
> 4.  There *is* a way to create uncollectable cycles in Lua: if (k, v)
> is a key value pair in a weak valued table and v refers (directly or
> indirectly) back to k then that pair will never be removed from the
> table, even if v is the *only* reference to k.

Correction: this holds for *weak keyed tables* and not (as stated) for
*weak valued tables*.  Sorry.

--
Wim



Reply | Threaded
Open this post in threaded view
|

RE: Memory woes

Richard Ranft
In reply to this post by Asko Kauppi
I haven't looked, but isn't there some way to pull environment data out of a
Lua state?  This could be held while the VM is restarted, then pushed into
the new VM....

I'm just throwing things out - and probably making no sense.  But, if I say
something that spurs you to think of a solution, all's well that ends well -
lol.  I find that I solve problems faster if I try to explain what I'm doing
to a buddy of mine (he has minimal scripting/programming background) - by
the time I explain my problem to him I usually see the solution.

Rich

> -----Original Message-----
> From: [hidden email]
> [[hidden email] Behalf Of Asko Kauppi
> Sent: Sunday, July 02, 2006 9:22 AM
> To: Lua list
> Subject: Re: Memory woes
>
>
>
> If all the state you need can be expressed in "INI-like" key, value
> format, then it wouldn't be a huge endeavour to write C functions
> capable of accessing such "shared" space. This would be handy
> otherwise, too, especially if the system would support informing Lua
> machines of changes to the values (similar to the way GConf is used,
> between Gtk+ processes).
>
> Perhaps I'll do this kind of a system, but .. won't be there for your
> launch date (hmm... unless you hire me for this :)
>
> -asko
>
>
> Research kirjoitti 2.7.2006 kello 10.26:
>
> > If I could use multiple VMs, I'd just shut down the one I have and
> > restart it. The problem is that some state data needs to be shared,
> > and while I know there's lots of possibilities, all are ugly at
> > this point.
> >
>


Reply | Threaded
Open this post in threaded view
|

Re: Memory woes

Brett Bibby
In reply to this post by Wim Couwenberg-2
Thanks everybody for your replies...

As there are, as far as I know, no memory leaks in Lua, I would look
at your memory manager for leaks.

I agree with Edgar on the "no memory leaks in Lua" part.  Here are
just some random remarks, based on your observations:

This is good to know. I was also refering to "soft" leaks like when symbols don't get collected for whatever reason.


1.  It can take two subsequent gc cycles to collect everything.  This
one is easy to check by doing a double gc when you start/quit your
game.

I wondered about this. We have put in 2 gcs and will see if the behaivor changes.

2.  If you have a stable life time cycle of objects, then Lua should
*not* report "a few KB" increase between runs (I gather your obtain
this via gcinfo), it should report the exact same amount.  The GC in
Lua 5.0 runs atomically and is completely deterministic.  Realloc-ing
to 0 (zero) as Edgar suggested cannot cause this increase.

I thought so too. Good to know it shouldn't.

4.  There *is* a way to create uncollectable cycles in Lua: if (k, v)
is a key value pair in a weak valued table and v refers (directly or
indirectly) back to k then that pair will never be removed from the
table, even if v is the *only* reference to k.
Correction: this holds for *weak keyed tables* and not (as stated) for
*weak valued tables*.

This is interesting. Is this the _only_ way to create uncollectable items? Are there others?

Thanks!