Research on Lua programs with excessive memory consumption

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Research on Lua programs with excessive memory consumption

Pablo Musa-2
Hi everybody,
I am doing some research about memory problems in Lua, more specifically on memory waste due to objects that will never be used again. My idea is to create a library to help "memory leak"* detection in Lua.

In this research I would like to first analyze some real problems and then propose a solution.
If you have a program which is using more memory than it should, can you please send it to me with as many details as possible?!!

* I do not like the term "memory leak" for this problem, but that is not the focus of this email.

Thanks in advance,
Pablo Musa
Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Ico Doornekamp
* On 2014-05-02 21:22:27 +0200, Pablo Musa wrote:
 
> Hi everybody, I am doing some research about memory problems in Lua,
> more specifically on memory waste due to objects that will never be
> used again. My idea is to create a library to help "memory leak"*
> detection in Lua.
>
> In this research I would like to first analyze some real problems and
> then propose a solution.  

Can you elaborate on what kind of problems you are looking for?

I have written a few somewhat large (>30k lines) Lua applications over
the last few years, ment to run 'indefinitely' in embedded environments,
and I have never found any signs of leaking memory in Lua.

Every time I found these applications were leaking memory, I had myself
to blame because of lingering references to objects which I should have
cleaned up.

(Maybe the only exception is the case that Lua does not automatically
shrink tables when elements are removed, but I've learned to work around
this by forcing a rehash of the table when memory is tight)

Ico

--
:wq
^X^Cy^K^X^C^C^C^C

Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Matthew Wild
In reply to this post by Pablo Musa-2
Hi,

On 2 May 2014 20:22, Pablo Musa <[hidden email]> wrote:
> Hi everybody,
> I am doing some research about memory problems in Lua, more specifically on
> memory waste due to objects that will never be used again. My idea is to
> create a library to help "memory leak"* detection in Lua.

See:

  - http://code.matthewwild.co.uk/luatraverse/
  - http://code.matthewwild.co.uk/lua-getsize/

> In this research I would like to first analyze some real problems and then
> propose a solution.
> If you have a program which is using more memory than it should, can you
> please send it to me with as many details as possible?!!

Occasionally we have people with memory usage issues in Prosody. For
debugging these we have a small script based on the above projects
which will dump the Lua state as a graph to a file, with each entry
containing data such as the object type, size, etc. We then analyze
these using scripts and even a GUI tool to determine which tables are
accounting for the most memory usage, etc.

I don't have a sample application to give you, but I have a real-world
story of a "memory leak" we had once in Prosody. LuaExpat creates
"parser" objects, which call Lua handler functions when various events
occur. It stores these using luaL_ref(). Because of this there is
always a strong reference to the handler functions, as far as the GC
is concerned. The parser objects do remove this reference in their
__gc, however if the handler functions have the parser as an upvalue
(or, as in our case, hold a reference to it indirectly through another
table or function) then __gc won't get called. Then we have a cyclical
reference that can never be cleaned up by the GC. The result was a
slow leak of parser objects and handler functions.

> * I do not like the term "memory leak" for this problem, but that is not the
> focus of this email.

I believe Roberto likes to call it "hoarding":
http://lua-users.org/lists/lua-l/2012-03/msg00482.html :)

Regards,
Matthew

Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Ignacio Burgueño-2
I'd love a tool that combines luatraverse, lua-getsize and lua-microscope.

Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Philipp Janda
Am 02.05.2014 23:21 schröbte Ignacio Burgueño:
> I'd love a tool that combines luatraverse, lua-getsize and lua-microscope.

Your wish is my command: I've added an option to microscope which uses
lua-getsize if available in CPATH on userdatas, tables, functions, and
threads (Lua 5.1 only obviously). An example can be seen here[1].
However, I have no idea how an integration with luatraverse would look
like (I do my own traversal), and I have some unresolved issues with
lua-getsize:

*   Currently it returns only the size of a userdata's payload without
the size of the userdata value itself (that's 40 bytes here).
*   I'm not sure if/how we can encounter a Proto object on the Lua stack.
*   It reports 104 bytes for an empty table (which would indicate one
allocated hash slot), but my own measurements show a size of 64 bytes
for empty tables (without any hash slots allocated). I will investigate
further which one is right ...
*   For value types it reports the size of the corresponding C type
(lua_Number, int, void*, etc.), not the size of the Lua type which is
always TValue (16 bytes on my machine).
*   I'd love support for Lua 5.2+, and maybe a rockspec for Christmas
... ;-)


Regarding memory issues in Lua: The only annoyances I had so far have to
do with tables keeping allocated hash/array slots, even if the keys are
set to nil. This mostly happens with weak tables used as global caches
in my modules (I do that less often now ...).


Philipp

   [1]:  http://siffiejoe.github.io/lua-microscope/example1_sizes.gif




Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Philipp Janda
Am 03.05.2014 12:19 schröbte Philipp Janda:
>
> *   It reports 104 bytes for an empty table (which would indicate one
> allocated hash slot), but my own measurements show a size of 64 bytes
> for empty tables (without any hash slots allocated). I will investigate
> further which one is right ...

As a follow-up:
Empty tables use a static dummy node instead of an allocated one, so 64
bytes would be correct (on my machine). The downside is that there seems
to be no easy way to detect that from outside of `ltable.c` ...

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Ignacio Burgueño-2
In reply to this post by Philipp Janda
On Sat, May 3, 2014 at 7:19 AM, Philipp Janda <[hidden email]> wrote:

Your wish is my command: I've added an option to microscope which uses lua-getsize if available in CPATH on userdatas, tables, functions, and threads (Lua 5.1 only obviously). An example can be seen here[1]. However, I have no idea how an integration with luatraverse would look like (I do my own traversal), and I have some unresolved issues with lua-getsize:

Having written many long running services with luanode, and spent hours debugging memory consumption issues, having this tool available is great. Many thanks.
 
Reply | Threaded
Open this post in threaded view
|

Re: Research on Lua programs with excessive memory consumption

Pablo Musa-2
Thanks everyone for the answers.

Yes, we call it memory hoarding. It is not a definitive name yet, but we are working on that.

I will take a look at the options for tracking the problem.

But what I am actually looking for are running programs with the symptoms (huge increasingly memory use), so I can analyze them and improve my research. If anyone has a program with excessive memory consumption synmptons and is whilling to share, that would be perfect.

Thanks again,
Pablo Musa
Em Domingo, 4 de Maio de 2014 12:29, Ignacio Burgueño <[hidden email]> escreveu:
On Sat, May 3, 2014 at 7:19 AM, Philipp Janda <[hidden email]> wrote:

Your wish is my command: I've added an option to microscope which uses lua-getsize if available in CPATH on userdatas, tables, functions, and threads (Lua 5.1 only obviously). An example can be seen here[1]. However, I have no idea how an integration with luatraverse would look like (I do my own traversal), and I have some unresolved issues with lua-getsize:

Having written many long running services with luanode, and spent hours debugging memory consumption issues, having this tool available is great. Many thanks.