Replacement for unlocked refs?

classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Replacement for unlocked refs?

J. Perkins-2
I just discovered that unlocked refs are no longer supported
in the work3 release. I was using them to allow C++ objects to
keep track of the table which represented them in Lua. What's
the workaround to allow this in work3?

Jason


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Diego Nehab-3
Hi,

> I just discovered that unlocked refs are no longer supported
> in the work3 release. I was using them to allow C++ objects to
> keep track of the table which represented them in Lua. What's
> the workaround to allow this in work3?

You can  have, for instance, a  locked reference  to a  table associating
your objects userdata with the corresponding object tables. Then set the
mode of this table to  weak keys so that  when your userdata is  collected,
the whole pair is removed  automatically from the table and  associated
object table will also be collected.

Regards,
Diego.


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Ugh. If this is the only way to do it, I'll probably add a
table of weak keys and rewrite luaL_ref/luaL_unref to use
it. Sure feels like a big hack.

If you'll accept my $0.02, the current implementation of
ref/unref seems a big hack. The ability to associate C and
script objects is an awfully important feature for an
embedded language, but the current method (indices into
the registry) seems an afterthought. At the very least, a
dedicated table should be set aside for refs, freeing the
registry for application use, and support re-added for
unlocked refs.

All IMO, of course. I realize that this probably isn't a
big deal for people who only call C code from Lua, and
rarely Lua code from C.

Jason



it would seem that the ability
to associate C and Lua objects would be an important
function that should be built into the language,

Okay, I can do this but it seems a bit like a hack.


Actually
I will probably add a table of weak keys to Lua and hide it behind
the luaL_ref/luaL_unref methods. This really does seem like something that should be built into the language -- Lua being an
embedded language and all -- and not something hacked onto the
registry table, which I thought was supposed to be reserved for
my application's use.



Diego Nehab wrote:

I just discovered that unlocked refs are no longer supported
in the work3 release. I was using them to allow C++ objects to
keep track of the table which represented them in Lua. What's
the workaround to allow this in work3?

You can  have, for instance, a  locked reference  to a  table associating
your objects userdata with the corresponding object tables. Then set the
mode of this table to  weak keys so that  when your userdata is  collected,
the whole pair is removed  automatically from the table and  associated
object table will also be collected.



Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Oops, sorry about that extra rambling at the end there, I
accidentally sent before I was done.

Jason



Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Roberto Ierusalimschy-5
In reply to this post by J. Perkins-2
On Thu, 31 Jan 2002, J. Perkins wrote:

> If you'll accept my $0.02, the current implementation of
> ref/unref seems a big hack. The ability to associate C and
> script objects is an awfully important feature for an
> embedded language, but the current method (indices into
> the registry) seems an afterthought. At the very least, a
> dedicated table should be set aside for refs, freeing the
> registry for application use, and support re-added for
> unlocked refs.

In the work version, the old ref/unref are just macros that use
luaL_ref, luaL_unref, and lua_rawgeti over the registry. These functions
can operate over any table. So, it is trivial to put your references in
your own dedicated table (which can be weak).

-- Roberto


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
But really two tables are required, right? One for locked refs
and another for unlocked? And then the reference value must also
indicate which table it is stored in, by using negative numbers
for unlocked refs or something like that. Doable, but really ugly.
However it is actually implemented, this should be, IMHO, a part
of Lua itself and not something the end-user should have to write.

Is there some reason why unlocked refs were dropped?

Jason




Roberto Ierusalimschy wrote:

On Thu, 31 Jan 2002, J. Perkins wrote:
If you'll accept my $0.02, the current implementation of
ref/unref seems a big hack. The ability to associate C and
script objects is an awfully important feature for an
embedded language, but the current method (indices into
the registry) seems an afterthought. At the very least, a
dedicated table should be set aside for refs, freeing the
registry for application use, and support re-added for
unlocked refs.

In the work version, the old ref/unref are just macros that use
luaL_ref, luaL_unref, and lua_rawgeti over the registry. These functions
can operate over any table. So, it is trivial to put your references in
your own dedicated table (which can be weak).



Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Roberto Ierusalimschy-5
On Thu, 31 Jan 2002, J. Perkins wrote:

> But really two tables are required, right? One for locked refs
> and another for unlocked? And then the reference value must also
> indicate which table it is stored in, [...]

Usually you know whether a reference points to a locked or unlocked object,
so it does not need to include that information.

(If you really want to emulate the old behavior, you can use two tables,
one weak and one strong. You put all elements in the weak table,
but only locked ones go to the other table. That way, your references
are always to the weak table; you do not need negative indices. To unref,
you set the ref to nil in both tables. But, as I said, I think most
people don't need that.)



> Is there some reason why unlocked refs were dropped?

It was replaced by weak references (and, unlike some other languages, we
do not need more than one way to do something ;-). But also the whole
system of references was somewhat outdated by multiple states. And there
is a tendency in Lua to unificate all data structures into tables.
Globals went to tables, now tag methods and references are going, too.

-- Roberto


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Keeping all references in the weak table is a good idea, I will
do that. Still, I stand by my assertion that this is a critical
function of an embedded language which should be a part of the
library, not "emulated" in user code. But since I seem to be the
only one concerned by it, I will go suffer in silence  ;)

Thanks for the help,

Jason



Roberto Ierusalimschy wrote:

On Thu, 31 Jan 2002, J. Perkins wrote:


But really two tables are required, right? One for locked refs
and another for unlocked? And then the reference value must also
indicate which table it is stored in, [...]


Usually you know whether a reference points to a locked or unlocked object,
so it does not need to include that information.

(If you really want to emulate the old behavior, you can use two tables,
one weak and one strong. You put all elements in the weak table,
but only locked ones go to the other table. That way, your references
are always to the weak table; you do not need negative indices. To unref,
you set the ref to nil in both tables. But, as I said, I think most
people don't need that.)




Is there some reason why unlocked refs were dropped?


It was replaced by weak references (and, unlike some other languages, we
do not need more than one way to do something ;-). But also the whole
system of references was somewhat outdated by multiple states. And there
is a tendency in Lua to unificate all data structures into tables.
Globals went to tables, now tag methods and references are going, too.

-- Roberto







Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Diego Nehab-3
Hi,

> Keeping all references in the weak table is a good idea, I will
> do that. Still, I stand by my assertion that this is a critical
> function of an embedded language which should be a part of the
> library, not "emulated" in user code. But since I seem to be the
> only one concerned by it, I will go suffer in silence  ;)

You can, on the other hand, keep a strong reference, and unref it during
the garbage colection of your userdatum. That is, as long as you do not
reference it in the associated table, thus forming a cycle :-).

Regards,
Diego.


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Diego Nehab wrote:

You can, on the other hand, keep a strong reference, and unref it during
the garbage colection of your userdatum. That is, as long as you do not
reference it in the associated table, thus forming a cycle :-).


The table *is* the script interface to the object. The table does contain a reference to the userdata, in fact it has the *only* reference to the userdata, so I must rely on the Lua table to determine the lifespan of the object.

I could use the userdata as the object interface in Lua, but then every access would have to go look up the associated table, which seems worse than reimplementing lua_ref(0) myself.

Thanks for the help,

Jason





Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Diego Nehab-3
Hi,

> The table *is* the script interface to the object. The table does 
> contain a reference to the userdata, in fact it has the *only* 
> reference to the userdata, so I must rely on the Lua table to 
> determine the lifespan of the object.
> 
> I could use the userdata as the object interface in Lua, but then 
> every access would have to go look up the associated table, which 
> seems worse than reimplementing lua_ref(0) myself.

The methods will have to be looked up in the table anyways. In Lua
4.1-work3, you can create a table with all the methods and set it as the
index field of the eventtable. The call

    userdatum:method(...)

will be translated to 

    userdatum.method(userdatum, ...) 

The dot will see its a userdatum and check the eventtable for the userdatum
and return the field 'method' of the index table.

This could be your C function that does whatever it wants with the pointer
it receives. 

Even if you don't have eventtables, the method table could be passed as a
closure parameter to the userdatum index tag method.

I don't see how this is too bad. Maybe you need the references outside of C
functions called by lua methods.

Regards,
Diego.



Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
You're thinking method tables, but I'm talking about "instance tables". The instance table contains the component's local data, which includes a pointer to the C++ object instance associated with it. The event table of the index table is the method table. Get it? ;) You're thinking that the Lua table is just a list of C functions. I'm thinking that the Lua table could contain it's own data and functions, which work on top of the C functions.

I could still make the userdata the "main" object, with a custom "index" method that attempts to look up the object first in the method table, and then in the "local instance" table. Since most access will be methods anyway, it's probably not a big loss.

I think I'm cranky about this mostly because I had it all working (rolled my own event tables, basically), and what I thought was a minor version upgrade is turning out to be a complete rewrite of my scripting interface. Perhaps it would be better named Lua 5.0-work3.

Jason




Diego Nehab wrote:

The methods will have to be looked up in the table anyways. In Lua
4.1-work3, you can create a table with all the methods and set it as the
index field of the eventtable. The call

    userdatum:method(...)

will be translated to userdatum.method(userdatum, ...)
The dot will see its a userdatum and check the eventtable for the userdatum
and return the field 'method' of the index table.

This could be your C function that does whatever it wants with the pointer
it receives.
Even if you don't have eventtables, the method table could be passed as a
closure parameter to the userdatum index tag method.

I don't see how this is too bad. Maybe you need the references outside of C
functions called by lua methods.

Regards,
Diego.








Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Diego Nehab-3
Hi,

Last try... I think we missunderstood each other. :-)

So you have a C++ object and an  associated Lua table. From Lua you want
to be able to reference associated table as a full fledged Lua table and
also the C++ object (and for that you need the pointer to the C++ object
somewhere as a userdatum). From C++ you also want to have full access to
everything.

You problem now that there are no weak refs is that if you place the C++
object  userdatum as  a field  of the  associated table  (say, the  self
field) AND store a strong ref to  the associated field in the C++ object
structure they will never be  collected (cycle). Furthermore, in the Lua
side, the object is represented by its associated table.

Is it not  so? If not, ignore the  rest of the message and  I will never
talk about this issue again. :-)

My suggestion  is that you use  the userdatum directly to  reference the
object from Lua.  You can create an associated table  and place a strong
ref  to it  as a  field in  the  C++ object.  You set  the gettable  and
settable methods for  the userdatum to access the  associated table. You
set the garbage collection method  to unref the associated table. (there
is no self field in the associated table, there is no cycle)

Your object will then work as a table for Lua scripts and as a userdatum
for C++ code.  Lua can see the  userdatum value directly and  pass it to
C++ code. C++ code has the ref and  thus the associated table too. It is
fast and clean, I believe.

But you do have to change your code.

Regards,
Diego.


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Diego Fernandes Nehab wrote:

Last try... I think we missunderstood each other. :-)


[...]


My suggestion  is that you use  the userdatum directly to  reference the
object from Lua.  You can create an associated table  and place a strong
ref  to it  as a  field in  the  C++ object.  You set  the gettable  and
settable methods for  the userdatum to access the  associated table. You
set the garbage collection method  to unref the associated table. (there
is no self field in the associated table, there is no cycle)

There are still some synchronization problems with this approach, though they too could be worked around. Say I have a component, which has both Lua and C++ parts. The C++ part is being reference counted.

Lua could hold a reference to the object while the C++ refcount goes to zero. At this point I could let it sit around until it's GCed.

Lua could try to GC an object whose refcount > 0. I can't release the Lua table because the object could be passed back into Lua later, and I don't want to lose any information that's been stored there.

So say I have a C++/Lua object. The userdata gets GCed, but I ignore it because the C++ refcount > 0. Now I make a call into Lua and pass my object in as a parameter. The call returns, and the C++ refcount gets decremented to zero. Can I delete the objects? C++ no longer needs it, and Lua tried to GC it earlier. But Lua could have kept a reference to the table when I made the Lua call. Or maybe it didn't.

I do understand you're approach and it does work, but it assumes that Lua "owns" the object, which is not the case in this instance. I really need the ability to store both weak and strong references, which I can get with a pair of tables in my scripting interface (although...well, I think I made my case earlier).

Thanks for knocking this around with me, I hadn't considered your approach before and it did get me thinking.

Jason


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Diego Nehab-3
Hi,

Didn't know you were refcounting stuff. I tried that once, while writing
a raytracer  that described  its scenes in  Lua. C++  objects refcounted
their relations, and Lua had its own bc, of course. It was not a serious
project though, and I didn't have to worry about memory leaks as long as
it didn't crash. But I vaguely remember it worked.

> I do understand you're approach and it does work, but it assumes that 
> Lua "owns" the object, which is not the case in this instance. I really 
> need the ability to store both weak and strong references, which I can 
> get with a pair of tables in my scripting interface (although...well, I 
> think I made my case earlier).

What if you increment your object's refcount  by one when you first pass
it to Lua to take into account that  Lua has a reference to it? Lua's gc
would then  only need to decrement  the refcount by one.  Then, when the
count  reaches zero,  the  C++ object  releases the  strong  ref to  the
associated  table, so  that  the table  can  also  be collected,  before
scheduling itself for destruction ('delete  this' is probably not a good
idea). The table  can't be collected before Lua  collects the associated
object's userdatum since  the object has a strong ref  to it. The object
refcount cannot reach zero before Lua collects it, since Lua incremented
it once.

Nobody owns anybody that  way, and it might work. If  it doesn't, my old
raytracer has a leak :-)

> Thanks for knocking this around with me, I hadn't considered your 
> approach before and it did get me thinking.

Yes, I guess I am on a nerdish  mood today :-). Anyways, I seldom answer
something verbosely, and I thought I needed to make myself clearer.

Best regards,
Diego.


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Peter Loveday-2
> What if you increment your object's refcount  by one when you first pass
> it to Lua to take into account that  Lua has a reference to it? Lua's gc
> would then  only need to decrement  the refcount by one.

I do something identical to this in a project; the userdata pointer in Lua
is exactly the same as any C++ reference to the object in the way it is
reference counted.  As you say, the gc just decrements the count.  This
also works well with multiple Lua references to the same object via
different userdata objects (in 4.1), as each different one just holds a
ref on the object.

I don't have the requirement of C++ having a Lua table, as in this example,
but I can't see why that wouldn't work more or less the same way; Lua maintains
the refcount, C++ can effectively hold a ref as long as it likes, and release
it when it wants.

Love, Light and Peace,
- Peter Loveday
eyeon Software



Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Peter Loveday wrote:

I do something identical to this in a project; the userdata pointer in Lua
is exactly the same as any C++ reference to the object in the way it is
reference counted.  As you say, the gc just decrements the count.  This
also works well with multiple Lua references to the same object via
different userdata objects (in 4.1), as each different one just holds a
ref on the object.


How do you pass your userdata object into Lua calls? That is, say I have C++ code that wants to call a function like:

int doSomething(MyObject* obj);

...where obj exists in both C++ and Lua. If doSomething were a wrapper to a Lua function, you would have to push the name of the function, and then the argument. If you can't keep a weak reference to the userdata object in Lua, how would you push obj onto the Lua stack?

Hmm, now that I think about it...I guess I could just call newuserdatabox(), as there's no reason why I need to use the exact same Lua object for the call. Then I can increment my refcount, and GC will be called once for every Lua instance of the userdata. That *could* create a whole mess of duplicate userdata's that need collecting but it would work. I'll have to get a rough estimate of how many extra objects I'd be creating with this approach.

Thanks for the insight, much appreciated.

Jason




I don't have the requirement of C++ having a Lua table, as in this example,
but I can't see why that wouldn't work more or less the same way; Lua maintains
the refcount, C++ can effectively hold a ref as long as it likes, and release
it when it wants.

Love, Light and Peace,
- Peter Loveday
eyeon Software








Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Diego Nehab-3
> How do you pass your userdata object into Lua calls?  That is, say I 
> have C++ code that wants to call a function like:
> 
> int doSomething(MyObject* obj);

Now you caught  me... What you would like  to have is strong  ref to the
C++ object stored in it. But  that would kill garbage collection because
of the cycle. I agree that here  a weak ref would be greatly appreciated
:-(.

Using lua_pushuserdatabox too much might become a problem. I will try to
find an elegant solution and post it if I find it.

Regards,
Diego.


Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

Peter Loveday-2
In reply to this post by J. Perkins-2
> How do you pass your userdata object into Lua calls?  That is, say I 
> have C++ code that wants to call a function like:
> 
> int doSomething(MyObject* obj);

Yep, in my case the number of calls like this is relatively minimal.  I
do indeed just push a newuserdatabox() every time.

I haven't found this to be a problem for what I'm doing, but I can see
that it could become prohibitive in other applications.

Love, Light and Peace,
- Peter Loveday
eyeon Software



Reply | Threaded
Open this post in threaded view
|

Re: Replacement for unlocked refs?

J. Perkins-2
Peter Loveday wrote:

How do you pass your userdata object into Lua calls? That is, say I have C++ code that wants to call a function like:

int doSomething(MyObject* obj);

Yep, in my case the number of calls like this is relatively minimal.  I
do indeed just push a newuserdatabox() every time.

I haven't found this to be a problem for what I'm doing, but I can see
that it could become prohibitive in other applications.

It turns out I only do this a few places in my code, but
it gets run every frame (it's a game project) which adds
up quickly. The number of such calls is likely to grow
significantly once it gets out to the endusers, so it looks
like I'll have to roll-my-own weak reference system.

It's not that it's all that hard to do, just that it
seems like something that should be built in.

Jason


123