coroutines local userdata and GC

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

coroutines local userdata and GC

arioch82
Hi,

first of all sorry for the long email but i could really use some help here...

i am running lua 5.1 in a C++ environment and i am having a problem with some user data not being garbage collected and i have been trying to figure out why for a while without much luck.

the following code is all abstracted but the flow is representative of production code, some of this may look weird without context but please i am not looking to any comment on code design, just trying to understand how the userdata/GC works when you have local references inside coroutines

-----------------------------------------------------------------------
1) i have a "class.lua" module that defines a "New" function for OOP like:

function New(self, obj)
    local obj = obj or {}
    setmetatable(obj, self)
    self.__index = self
    return obj
end

2) i have another module, the one leaking (let's call it "base_module.lua"), that has the following code

local class = require "class"

MyTable = class:New {
    update = function(self)
        if self.co == nil then
            self.co = coroutine.create(self.coroutine_update, self)
        end

        coroutine.resume(self.co, self)
    end,

    coroutine_update = function(self)
        local myref = cbinding.CreateMyUserData()

        while true do
            do_something(myref)
            coroutine.yield()
        end
    end,
}

3) i have one lua file that contains something like:

local base_module = require "base_module"

local tableref = nil

function enter()
    tableref = base_module.MyTable:New {}
end

function update()
    tableref:update()
end

function exit()
    tableref = nil
end
-----------------------------------------------------------------------

now the last lua file is the one that i am actually running from the C side, every time i decide to run it i call the enter function then i start calling update into a loop and when i don't need it anymore i call the exit function.

When i call the exit function what I would expect is for tableref to be collected -> "tableref.co" coroutine to be collected since there are no more references to it -> userdata "myref" inside the coroutine_update to be collected... which is not happening.

I have then tried to associate the userdata to the object removing the creation from the coroutine

-----------------------------------------------------------------------
MyTable = class:New {
    New = function (self, obj)
        -- create the userdata at creation time
        local obj = class.New(self, obj)
        obj.myref = cbinding.CreateMyUserData()
        return obj
    end,

    update = function(self)
        if self.co == nil then
            self.co = coroutine.create(self.coroutine_update, self)
        end

        coroutine.resume(self.co, self)
    end,

    coroutine_update = function(self)
        -- reference the userdata associated to the object
        local myref = self.myref

        while true do
            do_something(myref)
            coroutine.yield()
        end
    end,
}
-----------------------------------------------------------------------

This didn't fix the problem.

Then i thought that since functions are first class values in lua maybe the object "MyTable" in the "base_module.lua" itself could be keeping a reference through the "coroutine_update" variable so i have moved it outside the table in the same file hoping to fix it

-----------------------------------------------------------------------
local function coroutine_update(self)
    -- reference the userdata associated to the object
    local myref = self.myref

    while true do
        do_something(myref)
        coroutine.yield()
    end
end

MyTable = class:New {
    New = function (self, obj)
        -- create the userdata at creation time
        local obj = class.New(self, obj)
        obj.myref = cbinding.CreateMyUserData()
        return obj
    end,

    update = function(self)
        if self.co == nil then
            -- removed the self. from coroutine_update
            self.co = coroutine.create(coroutine_update, self)
        end

        coroutine.resume(self.co, self)
    end,

}
-----------------------------------------------------------------------

which of course didn't fix it either.
From what i understand if the coroutine is freed then all local variables inside that function should be garbage collected, at that point the userdata shouldn't have any references anymore and should be collected too but for some reason this is not happening.

I will keep investigating but any help/suggestions/thoughts will be highly appreciated.

Thank you
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Sean Conner
It was thus said that the Great arioch82 once stated:

> Hi,
>
> first of all sorry for the long email but i could really use some help
> here...
>
> i am running lua 5.1 in a C++ environment and i am having a problem with
> some user data not being garbage collected and i have been trying to figure
> out why for a while without much luck.
>
> the following code is all abstracted but the flow is representative of
> production code, some of this may look weird without context but please i am
> not looking to any comment on code design, just trying to understand how the
> userdata/GC works when you have local references inside coroutines

  [ code snipped ]

> function exit()
>     tableref = nil
> end

  Setting a userdata to nil doesn't cause the GC to run immediately
(although it will eventually be called).  As a test of your __gc() method on
"myref", try the following:

        function exit()
          tableref = nil
          collectgarbage('collect')
        end

  Calling collectgarbag() will force a GC cycle.  If it is reclaimed, then
you have two choices:  either just wait for Lua to run a GC cycle with the
understanding that it's not always immedate, or force a GC cycle.

  If it's not collected, then you are leaking references somewhere.

  -spc (Chasing down stray references isn't fun ... )


Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
Hi Sean,

thanks for your reply.

I have tried forcing several collections manually on the C side after that call and the userdata are not collected, there are no more references to that userdata as the one shown in the snippets that's why i feel i am missing something.

I was thinking that maybe since the coroutine is in a yield status in that and since it has the "self" parameter it keeps a reference to the "tableref" that will never be cleared?
What would be the right approach in the case? I know that coroutines cannot be killed manually.

Thank you again
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Anton Titov
In reply to this post by arioch82
On 18.11.2014 06:47, arioch82 wrote:
> >From what i understand if the coroutine is freed then all local variables
> inside that function should be garbage collected, at that point the userdata
> shouldn't have any references anymore and should be collected too but for
> some reason this is not happening.

Every running function (and all the functions that called it) are
referenced from the Lua stack and would not be collected even if there
are no other references to them. This is a good thing that prevents
crashes ;)

Corotines are actually functions called in separate Lua (sub-)states
with a corotine library doing some scheduling. Therefore even yielded
corotine is "running function" for the purposes in the first paragraph.

Your solution is to replace "while true do" with some more meaningful
condition and call the corotine one more time when destroying to let it
exit.

Best,
Anton

Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Philipp Janda
In reply to this post by arioch82
Am 18.11.2014 um 08:19 schröbte arioch82:
> Hi Sean,

Hi!

>
> thanks for your reply.
>
> I have tried forcing several collections manually on the C side after that
> call and the userdata are not collected, there are no more references to
> that userdata as the one shown in the snippets that's why i feel i am
> missing something.

Given the right options microscope[1] can display registry table,
environments, upvalues, locals (in the main thread and in other
coroutines), i.e. most places where a leaked reference could be.
(Beware: Images of complete Lua states tend to be huge, and Graphviz[3]
has issues with rendering huge graphs in bitmap formats). luatraverse[2]
can do similar things in plain text format (so without the Graphviz
dependency).

>
> I was thinking that maybe since the coroutine is in a yield status in that
> and since it has the "self" parameter it keeps a reference to the "tableref"
> that will never be cleared?
> What would be the right approach in the case? I know that coroutines cannot
> be killed manually.

If the suspended coroutine is still reachable and has a reference to the
userdata on its stack, then the userdata won't be collected. You'd have
to make sure that there are no more reference to the coroutine to free
that userdata reference on the coroutine stack.

>
> Thank you again
>

Philipp

   [1]: https://github.com/siffiejoe/lua-microscope/
   [2]: http://code.matthewwild.co.uk/luatraverse/
   [3]: http://www.graphviz.org/



Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Roberto Ierusalimschy
In reply to this post by arioch82
> I have tried forcing several collections manually on the C side after that
> call and the userdata are not collected, there are no more references to
> that userdata as the one shown in the snippets that's why i feel i am
> missing something.
>
> I was thinking that maybe since the coroutine is in a yield status in that
> and since it has the "self" parameter it keeps a reference to the "tableref"
> that will never be cleared?
> What would be the right approach in the case? I know that coroutines cannot
> be killed manually.

Did you see the problem in the code you sent, or only in the original,
non-simplified code?  I ran the code that you sent, and 'tableref' is
being collected the first time the GC runs.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

where.lua (was: coroutines local userdata and GC)

mniip
In reply to this post by Sean Conner
On 11/18/2014 10:14 AM, Sean Conner wrote:
> If it's not collected, then you are leaking references somewhere.
> -spc (Chasing down stray references isn't fun ... )

I've spent a little time and made a script [1] that traverses a lua
state searching for an object. Unless you mess with the ldblib/lbaselib
functions I can say with confidence that if my script doesn't find it -
it will be garbage-collected.

Example usage:

    local where=require"where"
    local hash
    do
        local t={}
        hash = tostring(t)
        function string.foo()
            t=t
        end
    end
    -- at this point we have no access to t, only to its address
    print(where.where(hash))

Prints: select(2,debug.getupvalue(_G.string.foo,1))
More info on the usage is in the source.

    [1] https://gist.github.com/mniip/70497b2a39b486a120e6

/* mniip */

Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
In reply to this post by Roberto Ierusalimschy
Hi Roberto,

I wrote that code on the fly while i was writing the email, i have only tested the production code but i can give it a try.
the problem tho is not about the collection of the "tableref" but the one of the the userdata created and referenced inside the coroutine that is yielding.
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
In reply to this post by Philipp Janda
those libraries will be really useful, thanks for the links!

the only reference to the suspended coroutine is "tableref.co" so if i set "tableref = nil" i won't have any reference to it once tableref gets collected (or at least i would hope so...)
This may fight tho with the "running coroutine references" that Anton Titov was mentioning, so tableref may be collected but the actually coroutine will never be because of the "lua stack system-reference"?
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
In reply to this post by Anton Titov
Are you sure that this applies to coroutines?
It sounds really odd to me, a coroutine may be a complex function with several nested loops in it; i would need to add a "CanRun()" condition to every single nested loop (or a "if not CanRun() then return" kind of thing inside every single loop) and keep calling the coroutine until the resume fails before being able of collecting the table that contains the reference to that coroutine...

it doesn't really seem like a nice approach, am i missing something?
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Anton Titov
In reply to this post by Anton Titov


On 18.11.2014 06:47, arioch82 wrote:
> >From what i understand if the coroutine is freed then all local variables
> inside that function should be garbage collected, at that point the userdata
> shouldn't have any references anymore and should be collected too but for
> some reason this is not happening.

Every running function (and all the functions that called it) are
referenced from the Lua stack and would not be collected even if there
are no other references to them. This is a good thing that prevents
crashes ;)

Corotines are actually functions called in separate Lua (sub-)states
with a corotine library doing some scheduling. Therefore even yielded
corotine is "running function" for the purposes in the first paragraph.

Your solution is to replace "while true do" with some more meaningful
condition and call the corotine one more time when destroying to let it
exit.

Best,
Anton




Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Dong Feng
In reply to this post by Anton Titov


2014-11-18 1:14 GMT-08:00 Anton Titov <[hidden email]>:
On 18.11.2014 06:47, arioch82 wrote:
>From what i understand if the coroutine is freed then all local variables
inside that function should be garbage collected, at that point the userdata
shouldn't have any references anymore and should be collected too but for
some reason this is not happening.

Every running function (and all the functions that called it) are referenced from the Lua stack and would not be collected even if there are no other references to them. This is a good thing that prevents crashes ;)

Corotines are actually functions called in separate Lua (sub-)states with a corotine library doing some scheduling. Therefore even yielded corotine is "running function" for the purposes in the first paragraph.

Your solution is to replace "while true do" with some more meaningful condition and call the corotine one more time when destroying to let it exit.

Best,
Anton


I doubt that a yielding coroutine is considered as a "running function". A yielding coroutine should logically be a "returned function" which could be picked up from the point it has yielded as a semi-continuation. It can be resumed, but from the point of view of GC, it is a returned function until it is resumed again.

Therefore, if a yielding coroutine lose all references pointing to it, it can never be resumed again and therefore should be GC-claimable.

Is my understanding above correct or not?

Thank you,
Dong



Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
that's my understanding as well.

I was thinking that the problem could be that since functions are first class values, having the function defined in a module like the "coroutine_update" in the snippet actually means keeping a reference to that function.
Now when I run coroutine.create(coroutine_update) i am actually adding a reference to the same function through the coroutine so even if the coroutine is collected the function will still be referenced through the "coroutine_update" variable in the module, therefore the local userdata will never be garbage collected.

But i am not sure if the way i think function values are referenced is correct, maybe they are "instanced" when referenced by coroutines?
If someone could shred some light on this it would be great.
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Dong Feng


2014-11-18 11:41 GMT-08:00 arioch82 <[hidden email]>:
that's my understanding as well.

I was thinking that the problem could be that since functions are first
class values, having the function defined in a module like the
"coroutine_update" in the snippet actually means keeping a reference to that
function.
Now when I run coroutine.create(coroutine_update) i am actually adding a
reference to the same function through the coroutine so even if the
coroutine is collected the function will still be referenced through the
"coroutine_update" variable in the module, therefore the local userdata will
never be garbage collected.

But i am not sure if the way i think function values are referenced is
correct, maybe they are "instanced" when referenced by coroutines?
If someone could shred some light on this it would be great.



I believe they are "referenced" rather than "instanced".

Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
if that's the case and i create and run multiple coroutines on the same non-anonymous function how would all variables local to the function behave?
they would all resume each other and share the same data?
that would also mean that all locally created userdata/tables will never be free since there will always be a variable in the module pointing to that function
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Dong Feng


2014-11-18 16:04 GMT-08:00 arioch82 <[hidden email]>:
if that's the case and i create and run multiple coroutines on the same
non-anonymous function how would all variables local to the function behave?
they would all resume each other and share the same data?
that would also mean that all locally created userdata/tables will never be
free since there will always be a variable in the module pointing to that
function


--
View this message in context: http://lua.2524044.n2.nabble.com/coroutines-local-userdata-and-GC-tp7664107p7664135.html
Sent from the Lua-l mailing list archive at Nabble.com.



Local variables do not belong to a function (or closure, strictly speaking). Instead, they reside on stack therefore they belong to one "invocation" to a function. When two coroutine share the same function, each still maintain a separated stack of that function. So the two share the same set of upvals but have separated local vars.



Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Andrew Starks
In reply to this post by arioch82


On Tue, Nov 18, 2014 at 6:04 PM, arioch82 <[hidden email]> wrote:
if that's the case and i create and run multiple coroutines on the same
non-anonymous function how would all variables local to the function behave?
they would all resume each other and share the same data?
that would also mean that all locally created userdata/tables will never be
free since there will always be a variable in the module pointing to that
function



--
View this message in context: http://lua.2524044.n2.nabble.com/coroutines-local-userdata-and-GC-tp7664107p7664135.html
Sent from the Lua-l mailing list archive at Nabble.com.

The function is a single reference, but when it runs, its arguments and locals are executed in its own space.

The upvalues of this function are shared. And so:

```
local a = 1

my_func = function (b)
  b = tonumber(b) or 1
  local c = 1
  for i = 1, 10 do
    a = a + 1
    b = b + 1
    c = c + 1
    print('a', a, 'b', b, 'c', c)
    coroutine.yield()
  end
end


---make a bunch of coroutines with my_func as the main

---resume them. you'll see that each coroutine will cause the single `a` variable to increment
---but `b` and `c` are all independent.

```

-Andrew
  



Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
In reply to this post by Dong Feng
Thanks for the explanation Andrew/Dong.

so those userdata should be cleared as i initially supposed when the coroutine is collected but for some reason they aren't, i have event tried setting "tableref.co = nil" just before "tableref = nil" in the "exit" function but that doesn't fix it either.

I am running out of ideas...
Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

Jerome Vuarand
2014-11-19 1:28 GMT+00:00 arioch82 <[hidden email]>:
> Thanks for the explanation Andrew/Dong.
>
> so those userdata should be cleared as i initially supposed when the
> coroutine is collected but for some reason they aren't, i have event tried
> setting "tableref.co = nil" just before "tableref = nil" in the "exit"
> function but that doesn't fix it either.
>
> I am running out of ideas...

As Roberto did before, I've put your code in actual files, added a
"cbinding" module and a main script, and it shows the userdata in
myref is collected at the first garbage collection cycle. I've put it
in a repo online: https://bitbucket.org/doub/tmp-coroutine-collection.
Code works in both Lua 5.1 and Lua 5.2.

So if you want more help on that subject rather than random
suggestions, you need to craft demo code that actually shows the
problem, so we can look at it and tell you where it's wrong. Chances
are by isolating the problem you'll find the issue yourself.

Reply | Threaded
Open this post in threaded view
|

Re: coroutines local userdata and GC

arioch82
It was the binding library i am using (a modified version of luawrapper), it was keeping an internal cache that was preventing references from being collected.

Thank you all for your support!
12