problem with the garbage collecting.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

problem with the garbage collecting.

Thierry-2
Hi,

I'm using Lua-5.3.0.

I integrated Lua in a C software. I created objects for mapping my own
C structs. Some object uses a lot of resources like memory and file
descriptor.

I built this design hoping that the Lua GC clean release my object when
it is no longer used by the Lua code. Of course, I use the special
function '__gc' for cleaning my own resources.

I run some Lua function in the same time, so I use coroutine.

This system run perfectly while I do not encounter errors.

When a function launched in a coroutine is terminated, I start the gc
and I see my resources freed. It is the expected behavior. I use as
parameter of the lua_gc function the "lua_State" returned by the
function lua_newthread() function. previously I remove the last
reference of my coroutine/thread in the main Lua stack.

When the function is not terminated because it yield, or when the
function is terminated with an error, the lua_gc() doesn't release my
resources. I must run the lua_gc with the main stack as parameter.

It is the normal behaviour ? Do you have some tips for avoiding to call
the lua_gc function with the main thread when a error occurs ?

NOTE: it is mandatory to force the GC after the execution of some
Lua code because the allocated resources can be heavy and must be freed
ASAP.

Thank you,
Thierry

Reply | Threaded
Open this post in threaded view
|

Re: problem with the garbage collecting.

Javier Guerra Giraldez
On Sat, Sep 26, 2015 at 9:57 AM, Thierry <[hidden email]> wrote:
> NOTE: it is mandatory to force the GC after the execution of some
> Lua code because the allocated resources can be heavy and must be freed
> ASAP.


short answer: if you need to release resources deterministically (of
even just eagerly), don't rely on GC. it's better to do a :close()
method that you call as soon as you don't need your object.


long answer:

it is in the nature of GC that there's no guarantees of _when_ it will
collect objects.  on some languages, it's a stated goal to be as eager
as possible, sometimes even resorting to reference counting despite
all its known drawbacks; this is not the case in Lua, where GC tries
to do as little work as possible, and objects can (and do) remain in
memory for a long time after not being referenced anywhere.

the Lua GC is driven by "memory pressure", but it doesn't have any
access to C pointers, or other resources.  the userdata objects are
treated as black boxes, and they are accounted only for their declared
size.  In many cases, a userdata object is only a pointer to the real
data, so it exerts extremely low "pressure".

it's possible to have many of these very small objects, the rest of
the system is choking full of malloc()ated memory, lots of files open,
and the Lua VM is totally "too bad, not my problem, I only have these
thousands of tiny objects".


oh, and about handling errors: yes, Lua doesn't have a 'finally'
clause, like it doesn't have 'try...catch', but it's not hard to
implement either with pcall().  it's not just for "dangerous code",
it's also quite good to get the chance to release resources.


cheers,

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: problem with the garbage collecting.

Thierry-2
On Sat, 26 Sep 2015 10:14:20 -0500
Javier Guerra Giraldez <[hidden email]> wrote:

> On Sat, Sep 26, 2015 at 9:57 AM, Thierry <[hidden email]> wrote:
> > NOTE: it is mandatory to force the GC after the execution of some
> > Lua code because the allocated resources can be heavy and must be freed
> > ASAP.
>
>
> short answer: if you need to release resources deterministically (of
> even just eagerly), don't rely on GC. it's better to do a :close()
> method that you call as soon as you don't need your object.
>
>
> long answer:
>
> it is in the nature of GC that there's no guarantees of _when_ it will
> collect objects.  on some languages, it's a stated goal to be as eager
> as possible, sometimes even resorting to reference counting despite
> all its known drawbacks; this is not the case in Lua, where GC tries
> to do as little work as possible, and objects can (and do) remain in
> memory for a long time after not being referenced anywhere.
>
> the Lua GC is driven by "memory pressure", but it doesn't have any
> access to C pointers, or other resources.  the userdata objects are
> treated as black boxes, and they are accounted only for their declared
> size.  In many cases, a userdata object is only a pointer to the real
> data, so it exerts extremely low "pressure".
>
> it's possible to have many of these very small objects, the rest of
> the system is choking full of malloc()ated memory, lots of files open,
> and the Lua VM is totally "too bad, not my problem, I only have these
> thousands of tiny objects".
>
>
> oh, and about handling errors: yes, Lua doesn't have a 'finally'
> clause, like it doesn't have 'try...catch', but it's not hard to
> implement either with pcall().  it's not just for "dangerous code",
> it's also quite good to get the chance to release resources.


Thank you for the response. I'm a little bit annoyed because only the
Lua knows the usage of my variable.

In brief, my program associated file descriptor to an Lua abject. If
the user forgot to close the fd, and cannot close because an error
occurs, I expect that the GC function __gc() close it. A challenge is to
close the fd as soon as possible for prevent memory leaks.

I called functions executed in a coroutine. Maybe the developper copy
the object containig the FD in the global env, or maybe not. It cant
know this information, so I can't close himself the fd. Only the gc and
the refcount system (I guess) know the information.

I resolve my issue forcing a full garbage collection cycle after each
functions using these type of object (I detect the object creation).

Maybe a quick __gc() of some object call will be welcome. it may be
declared with  anything like "__mode" = "q" (similar of weak
references) ?

Maybe it have another way to known the usage of a variable ?

Thierry

Reply | Threaded
Open this post in threaded view
|

Re: problem with the garbage collecting.

Javier Guerra Giraldez
On Tue, Oct 13, 2015 at 3:20 AM, Thierry FOURNIER <[hidden email]> wrote:
> I called functions executed in a coroutine. Maybe the developper copy
> the object containig the FD in the global env, or maybe not. It cant
> know this information, so I can't close himself the fd. Only the gc and
> the refcount system (I guess) know the information.


there's no refcount system in Lua.

but you can implement one: on your object initialization you can
register all 'open' objets in a global table, and another table where
you keep those that should stay open.  periodically (or at the pcall()
exit)  close any object that is still in the 'all objects' table and
not in the 'currently needed' table.

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: problem with the garbage collecting.

Brian Maher-3

You can also change your api. Instead of returning an object that you need to close, you run a callback and pass in the object. When the callback returns, close the object. Some notes:

* when closing the object, set a flag so you know that the object is closed and throw an error from methods to avoid dereferencing the closed object.

* try to avoid calling the callback from c code otherwise it won't play nice with coroutines.

On Oct 13, 2015 1:32 AM, "Javier Guerra Giraldez" <[hidden email]> wrote:
On Tue, Oct 13, 2015 at 3:20 AM, Thierry FOURNIER <[hidden email]> wrote:
> I called functions executed in a coroutine. Maybe the developper copy
> the object containig the FD in the global env, or maybe not. It cant
> know this information, so I can't close himself the fd. Only the gc and
> the refcount system (I guess) know the information.


there's no refcount system in Lua.

but you can implement one: on your object initialization you can
register all 'open' objets in a global table, and another table where
you keep those that should stay open.  periodically (or at the pcall()
exit)  close any object that is still in the 'all objects' table and
not in the 'currently needed' table.

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: problem with the garbage collecting.

zejian ju
In reply to this post by Thierry-2
I made a change to lua5.1's GC and add strict reference counting machanism:


In some condition, it is much quicker than normal GC.

Regards,
Zenk Ju



2015-10-13 16:20 GMT+08:00 Thierry FOURNIER <[hidden email]>:
On Sat, 26 Sep 2015 10:14:20 -0500
Javier Guerra Giraldez <[hidden email]> wrote:

> On Sat, Sep 26, 2015 at 9:57 AM, Thierry <[hidden email]> wrote:
> > NOTE: it is mandatory to force the GC after the execution of some
> > Lua code because the allocated resources can be heavy and must be freed
> > ASAP.
>
>
> short answer: if you need to release resources deterministically (of
> even just eagerly), don't rely on GC. it's better to do a :close()
> method that you call as soon as you don't need your object.
>
>
> long answer:
>
> it is in the nature of GC that there's no guarantees of _when_ it will
> collect objects.  on some languages, it's a stated goal to be as eager
> as possible, sometimes even resorting to reference counting despite
> all its known drawbacks; this is not the case in Lua, where GC tries
> to do as little work as possible, and objects can (and do) remain in
> memory for a long time after not being referenced anywhere.
>
> the Lua GC is driven by "memory pressure", but it doesn't have any
> access to C pointers, or other resources.  the userdata objects are
> treated as black boxes, and they are accounted only for their declared
> size.  In many cases, a userdata object is only a pointer to the real
> data, so it exerts extremely low "pressure".
>
> it's possible to have many of these very small objects, the rest of
> the system is choking full of malloc()ated memory, lots of files open,
> and the Lua VM is totally "too bad, not my problem, I only have these
> thousands of tiny objects".
>
>
> oh, and about handling errors: yes, Lua doesn't have a 'finally'
> clause, like it doesn't have 'try...catch', but it's not hard to
> implement either with pcall().  it's not just for "dangerous code",
> it's also quite good to get the chance to release resources.


Thank you for the response. I'm a little bit annoyed because only the
Lua knows the usage of my variable.

In brief, my program associated file descriptor to an Lua abject. If
the user forgot to close the fd, and cannot close because an error
occurs, I expect that the GC function __gc() close it. A challenge is to
close the fd as soon as possible for prevent memory leaks.

I called functions executed in a coroutine. Maybe the developper copy
the object containig the FD in the global env, or maybe not. It cant
know this information, so I can't close himself the fd. Only the gc and
the refcount system (I guess) know the information.

I resolve my issue forcing a full garbage collection cycle after each
functions using these type of object (I detect the object creation).

Maybe a quick __gc() of some object call will be welcome. it may be
declared with  anything like "__mode" = "q" (similar of weak
references) ?

Maybe it have another way to known the usage of a variable ?

Thierry