Ever-Increasing Memory Usage in 5.0

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Ever-Increasing Memory Usage in 5.0

Gavin Kistner
We're using Lua 5.0 in our runtime with tolua++, and I'm not sure that I understand some of the GC stuff I'm seeing.
 
Frequently (500-1000 times per second), our application runs a Lua function that has code like this:
 
function foo( )
 m = Matrix.new( )
 m = nil
 
 if g_collectNOW then
  collectgarbage( )
 end
end
 
Matrix.new() returns a userdata object. Lua's gc threshold starts out around 82k. We are pretty sure that there are no memory leaks on the C++ side of things.
 
If g_collectNOW is always true, memory stays at a consistent level.
If g_collectNOW is always false, memory grows linearly over time.
 
Question #1: Why does memory grow linearly over time? Shouldn't I see a sawtooth pattern of memory rising to a particular threshold, gc occuring, and memory dropping back down? I could even understand it (but not like it) if the sawtooth perhaps peaked at every-increasing values as the threshold got set to ever-increasing values. Instead, I see a constant linear growth, at the rate of a few megabytes per minute.
 
If I alternate g_collectNOW between false for a while (memory grows), then to true (gc every frame), the level that memory drops to for each gc-each-frame section is higher than the previous. Memory seems to be being irretrievably leaked.
 
Question #2: Uhm...what's going on? Where did that memory go, and why can't I get it back?

Thanks in advance for any help!

--
Gavin Kistner
.

Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Thomas Harning Jr.
On 10 Apr 2006 19:43:40 -0400
[hidden email] wrote:

> Matrix.new() returns a userdata object. Lua's gc threshold starts out
> around 82k. We are pretty sure that there are no memory leaks on the
> C++ side of things. If g_collectNOW is always true, memory stays at a
> consistent level. If g_collectNOW is always false, memory grows
Lua may not be seeing that the memory usage by the Matrix is high, so
it doesnt see a need to GC.  If I understand how the GC works, it will
keep track of the space taken by Lua objects and cycle when there is a
certain size limit.  Since a userdata object is quite small in Lua, it
oughtta take quite a while before that limit is reached.
If you can manage to allocate the Matrix data inside the userdata, then
Lua may have a better chance of seeing the rapid increase in space.

--
Thomas Harning Jr.
Fortune:
Wish and hope succeed in discerning signs of paranormality where reason
and careful scientific procedure fail.
- James E. Alcock, The Skeptical Inquirer, Vol. 12

signature.asc (207 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Gavin Kistner
On Apr 10, 2006, at 6:56 PM, Thomas Harning Jr. wrote:

> On 10 Apr 2006 19:43:40 -0400
> [hidden email] wrote:
>
>> Matrix.new() returns a userdata object. Lua's gc threshold starts out
>> around 82k. We are pretty sure that there are no memory leaks on the
>> C++ side of things. If g_collectNOW is always true, memory stays at a
>> consistent level. If g_collectNOW is always false, memory grows
> Lua may not be seeing that the memory usage by the Matrix is high, so
> it doesnt see a need to GC.  If I understand how the GC works, it will
> keep track of the space taken by Lua objects and cycle when there is a
> certain size limit.  Since a userdata object is quite small in Lua, it
> oughtta take quite a while before that limit is reached.
> If you can manage to allocate the Matrix data inside the userdata,  
> then
> Lua may have a better chance of seeing the rapid increase in space.

I'm unsure where the Matrix data is allocated, so thanks for these  
thoughts. It very well may be that the Lua side of the memory growth  
is quite tiny. (I'm just the humble scripter, not the engineer who  
works on the integration.) I will run a test tomorrow and let it run  
a very long time. However, my hunch is that it will not suddenly hit  
a GC point and collect, but will interminably grow.

In either case, this doesn't explain the results I'm seeing with the  
increasing 'plateaus' which GC brings mem usage down to. Any thoughts  
on that?
Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Daniel Collins
In reply to this post by Gavin Kistner

> I'm unsure where the Matrix data is allocated, so thanks for these  
> thoughts. It very well may be that the Lua side of the memory growth  
> is quite tiny. (I'm just the humble scripter, not the engineer who  
> works on the integration.) I will run a test tomorrow and let it run  
> a very long time. However, my hunch is that it will not suddenly hit  
> a GC point and collect, but will interminably grow.

Where are you getting the amount of allocated memory from? If it is from

collectgarbage("count")
then my understanding is that only includes the memory allocated from
Lua. So that function should tell you how much memory is allocated on
the "Lua side". This might help you decide where the allocated memory
is.

- DC

Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Daniel Silverstone
In reply to this post by Gavin Kistner
On Mon, 2006-04-10 at 23:07 -0600, Gavin Kistner wrote:
> I'm unsure where the Matrix data is allocated, so thanks for these  
> thoughts. It very well may be that the Lua side of the memory growth  
> is quite tiny. (I'm just the humble scripter, not the engineer who  
> works on the integration.) I will run a test tomorrow and let it run  
> a very long time. However, my hunch is that it will not suddenly hit  
> a GC point and collect, but will interminably grow.

My guess is that you're boxing the matrix pointer in a userdata, and as
such, your userdata is four bytes big. This is all Lua takes into
account so I'm guessing your sawtooth will be present, just incredibly
large.

> In either case, this doesn't explain the results I'm seeing with the  
> increasing 'plateaus' which GC brings mem usage down to. Any thoughts  
> on that?

Simple memory fragmentation at a guess. At points less regularly than
once per frame you are allocating an object which is longer lived than
the per-frame objects.

When I last worked on a game engine, we would force a garbage-collect
once per second which was less irritating than every frame but often
enough to keep memory use down. But we weren't using Lua so I don't know
how Lua would behave in this instance.

Remember that as Thomas said, the size of the userdata from the point of
view of the garbage collector is the number of bytes you allocated (4 or
8 for a boxed pointer) and the size of the Udata header which is
probably somewhere in the region of 16 or 20 bytes on most common 32bit
architectures.

Regards,

Daniel

--
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 2BC8 4016 2068 7895


Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Roberto Ierusalimschy
In reply to this post by Gavin Kistner
> If I alternate g_collectNOW between false for a while (memory grows),
> then to true (gc every frame), the level that memory drops to for each
> gc-each-frame section is higher than the previous. Memory seems to be
> being irretrievably leaked.

I assume your userdata has __gc metamethods, is that right? Then, they
are collected in two phases: in a first GC cycle, their metamethods are
called; only in the next cycle they are actually deallocated (if the
metamethods did not "ressurect" them). That may (or may not :) explain
part of that behavior.

-- Roberto
Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Chris Marrin
In reply to this post by Daniel Silverstone
Daniel Silverstone wrote:
> ...
> When I last worked on a game engine, we would force a garbage-collect
> once per second which was less irritating than every frame but often
> enough to keep memory use down. But we weren't using Lua so I don't know
> how Lua would behave in this instance.

One of the most incredible things about Lua 5.1 is its incremental GC.
We have a rendering engine which uses Lua, so we have a rendering loop
that runs at ~60fps. After each loop I added this:

     double ct = getCurrentTime();
     lua_gc(myState, LUA_GCSTEP, 10);
     ct = getCurrentTime() - ct;

The resultant ct (collect time) is close to 0% when the system was at
idle and rarely got above 1% of total frame time at high load. I also
monitored memory usage:

     int memused = lua_gc(myState, LUA_GCCOUNT, 0);

and found that memory would peak after some high load frames and then
take around 2 seconds or so (120 frames) to get back to its quiescent
state. This is great for any real-time system like this. Memory usage is
kept under control, but GC never affects frame rate and therefore
animation stays smooth.

So, switch to 5.1 :-)

--
chris marrin              ,""$, "As a general rule,don't solve puzzles
[hidden email]        b`    $  that open portals to Hell" ,,.
         ,.`           ,b`    ,`                            , 1$'
      ,|`             mP    ,`                              :$$'     ,mm
    ,b"              b"   ,`            ,mm      m$$    ,m         ,`P$$
   m$`             ,b`  .` ,mm        ,'|$P   ,|"1$`  ,b$P       ,`  :$1
  b$`             ,$: :,`` |$$      ,`   $$` ,|` ,$$,,`"$$     .`    :$|
b$|            _m$`,:`    :$1   ,`     ,$Pm|`    `    :$$,..;"'     |$:
P$b,      _;b$$b$1"       |$$ ,`      ,$$"             ``'          $$
  ```"```'"    `"`         `""`        ""`                          ,P`
Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Gavin Kistner
In reply to this post by Daniel Collins
On Apr 11, 2006, at 1:02 AM, Daniel Collins wrote:
> Where are you getting the amount of allocated memory from? If it is  
> from
> collectgarbage("count")
> then my understanding is that only includes the memory allocated from
> Lua. So that function should tell you how much memory is allocated on
> the "Lua side". This might help you decide where the allocated memory
> is.

I'm using 5.0, so gcinfo() would be the appropriate analog.

However, I was seeing these results in 'real-world' numbers (the  
amount of memory reported by Process Explorer's "private bytes"  
value) and our own internal memory tracking counters.

I'll run some more tests today with gcinfo() watching the Lua pool  
for a more accurate view of just the Lua world.
Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Gavin Kistner
In reply to this post by Daniel Silverstone
On Apr 11, 2006, at 3:47 AM, Daniel Silverstone wrote:
> My guess is that you're boxing the matrix pointer in a userdata,  
> and as
> such, your userdata is four bytes big. This is all Lua takes into
> account so I'm guessing your sawtooth will be present, just incredibly
> large.

I'll verify your guess today, but let's assume that it's true, as  
it's a "worst case" scenario for the discrepancy between Lua's size  
and the C++ memory side. Let's assume that a Matrix is 80 bytes in C+
+, and 4 bytes in Lua. (I don't really know, but it's at least 16x4  
bytes of data, and we'll throw in some overhead to be sure.) As I  
mentioned before, the GC threshold starts out at 82k.

Absent any calls to GC, that means we can have roughly 21,000  
Matrices hanging around in Lua before the threshold will be hit. At  
80 bytes per, that's 1.6MB of memory waiting on the C++ side of  
things. At this particular threshold, I should see a sawtooth pattern  
on the order of a couple megabytes. However, I've watched the linear  
memory increase go for at least 5 megabytes.

Further, when I call collectgarbage() the first time, that's an  
implicit parameter of 0, forcing garbage collection. According to the  
5.0 documentation, the garbage collection should run (leaving a  
single Matrix around, and perhaps a few other Lua objects in this  
test), update the count, and set the new threshold to twice the  
count. At this point, my threshold should be very small, and (if the  
auto-GC was working) I should see a much smaller sawtooth. But, a  
single call to collectgarbage() does not induce any sawtooth later  
on, again as memory grows by multiple megabytes.


> Simple memory fragmentation at a guess. At points less regularly than
> once per frame you are allocating an object which is longer lived than
> the per-frame objects.

Unless we've missed something, this particular guess shouldn't be the  
case. We have internal memory trackers, and (separate from this  
issue) the simulation is running at a steady state, with 0 memory  
allocations after startup.

(Hrm...though Heisenberg may be smacking us upside the head. The act  
of flipping the gc-now flag was being done through a scriptlet  
mechanism that does use memory allocations. I'll have to try a pure-
code flipper and see how that goes. Thanks for the idea.)



> When I last worked on a game engine, we would force a garbage-collect
> once per second which was less irritating than every frame but often
> enough to keep memory use down. But we weren't using Lua so I don't  
> know
> how Lua would behave in this instance.
>
> Remember that as Thomas said, the size of the userdata from the  
> point of
> view of the garbage collector is the number of bytes you allocated  
> (4 or
> 8 for a boxed pointer) and the size of the Udata header which is
> probably somewhere in the region of 16 or 20 bytes on most common  
> 32bit
> architectures.

Thanks so much for all your help. This is, in fact, for a game engine  
(sort of). As I've noted, I'll get stronger numbers today as to where  
memory is stored, what the gcinfo() threshold and usage is reporting  
during the increase, and so on.
Reply | Threaded
Open this post in threaded view
|

Re: Ever-Increasing Memory Usage in 5.0

Mark Hamburg-4
If you've got significant data accessed from Lua via stub userdata objects
-- i.e., objects just containing pointers to the real data -- it is
definitely a good idea to give the GC a little more help. In our case, it's
huge bitmaps that create the issue.

Under Lua 5.1 our solution is to run the GC forward a bit as we allocate
these. How much to run it forward was determined essentially empirically
(haphazardly?) to balance between memory consumption and GC time.

Under Lua 5.0, we built up a count of the number of bytes allocated and
forced a full GC when that number crossed a threshold.

Mark