when do programs set metatable?

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

when do programs set metatable?

Roberto Ierusalimschy
More often than not, programs set the metatable of a userdata right
after creating the userdata. Does anyone know of relevant counterexamples
to this pattern?

More specifically: if Lua puts a cost on setting the metatable of an
object long after its creation (the longer the more expensive), do you
know of programs that would be hurt?

Yet more specifically, I am interested only in cases where the metatable
has a __gc metamethod.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

RE: when do programs set metatable?

Jerome Vuarand-2
Roberto Ierusalimschy wrote:
> More often than not, programs set the metatable of a userdata right
> after creating the userdata. Does anyone know of relevant
> counterexamples to this pattern?  
> 
> More specifically: if Lua puts a cost on setting the metatable of an
> object long after its creation (the longer the more expensive), do
> you know of programs that would be hurt?  
> 
> Yet more specifically, I am interested only in cases where the
> metatable has a __gc metamethod. 

I did that at some point for a userdata which is an hybrid between a
pointer and a dynamic array. That is the same userdata contain just a
pointer, and can be owner or not of the pointed data. So whether or not
we are the owner we have or not the __gc metamethod, and that can change
over time.

I don't use that system anymore, because I needed at some point to have
per-class metatables rather than per object (for luaL_checkudata). With
my new system, I allocate all buffers through lua_newuserdata instead of
malloc, and I store the buffer ud as the environment of the pointer ud
to prevent its collection as long as the pointer exists and points to
it.


Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Wesley Smith
In reply to this post by Roberto Ierusalimschy
On Feb 8, 2008 6:45 AM, Roberto Ierusalimschy <[hidden email]> wrote:
> More often than not, programs set the metatable of a userdata right
> after creating the userdata. Does anyone know of relevant counterexamples
> to this pattern?


What thoughts are behind this question?  I'm curious.
best,
wes

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Mark Meijer-2
On 08/02/2008, Wesley Smith <[hidden email]> wrote:
> On Feb 8, 2008 6:45 AM, Roberto Ierusalimschy <[hidden email]> wrote:
> > More often than not, programs set the metatable of a userdata right
> > after creating the userdata. Does anyone know of relevant counterexamples
> > to this pattern?
>
>
> What thoughts are behind this question?  I'm curious.
> best,
> wes

I'm guessing it involves an idea for optimization.

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Doug Rogers-4
In reply to this post by Wesley Smith
Wesley Smith wrote:
> On Feb 8, 2008 6:45 AM, Roberto Ierusalimschy <[hidden email]> wrote:
>> More often than not, programs set the metatable of a userdata right
>> after creating the userdata. Does anyone know of relevant counterexamples
>> to this pattern?

> What thoughts are behind this question?  I'm curious.
> best,

Sounds like an optimization for allowing the GC to make assumptions
about metatableless userdata. I wonder what it is.

Would it be possible to deprecate the __gc metamethod and simply provide
a C function to be called in lua_newuserdata()? It actually sounds like
a pretty good idea. To be backwards compatible, though, it would require
a little more work - perhaps the penalty?

The __gc metamethod does seem to be an exceptional case. It is only
available for userdata, which are only available to the C API. Seems
ripe for simplification.

Doug

-- 
Innovative Concepts, Inc. www.innocon.com 703-893-2007 x220

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Roberto Ierusalimschy
In reply to this post by Mark Meijer-2
> > > More often than not, programs set the metatable of a userdata right
> > > after creating the userdata. Does anyone know of relevant counterexamples
> > > to this pattern?
> >
> > What thoughts are behind this question?  I'm curious.
> 
> I'm guessing it involves an idea for optimization.

Yes. The point would be to avoid traversing all userdata, looking
for the ones that need finalization, at the atomic phase of each GC
cycle. We could keep them (the userdata that need finalization) in
a different list.

To add another "next" pointer to each userdata would increase memory
consumption, so one way to add userdata to a diferent list is to remove
them from the rootgc list. Because this list is singled linked, Lua
should have to search the userdata from the beginning of the list to
remove it.  If it was created recently, it would be near the beginning
(probably the first one yet), so the search would be cheap.

Among other things, this change could open the doors for __gc
metamethods for tables.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Javier Guerra Giraldez
On 2/8/08, Roberto Ierusalimschy <[hidden email]> wrote:
> Among other things, this change could open the doors for __gc
> metamethods for tables.

hurray!!

-- 
Javier

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Doug Rogers-4
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:

> Among other things, this change could open the doors for __gc
> metamethods for tables.

Ah, the other direction!

When I ran my first experiments with __gc, I had encapsulated my
userdatum in a table so that I could attach other fields from Lua. I was
surprised when the C objects were not collected (the metatable was
attached to the table, not the userdatum). Then I read the fine print,
so I removed the table and used the userdatum directly.

I'm sure many would love to have this feature.

Doug

-- 
Innovative Concepts, Inc. www.innocon.com 703-893-2007 x220

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Mike Pall-82
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:
> To add another "next" pointer to each userdata would increase memory
> consumption [...]

Probably not feasible here, but FYI:
  http://en.wikipedia.org/wiki/XOR_linked_list

> Among other things, this change could open the doors for __gc
> metamethods for tables.

Are you sure you want to open this door? You may never be able to
close it again.

--Mike

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Mark Hamburg-4
In reply to this post by Roberto Ierusalimschy
If we explicitly invalidate a proxy object in Lightroom, we will set the
metatable on the proxy to a new metatable representing the invalid proxy.
But this isn't an incredibly common operation.

Mark

on 2/8/08 6:45 AM, Roberto Ierusalimschy at [hidden email] wrote:

> More often than not, programs set the metatable of a userdata right
> after creating the userdata. Does anyone know of relevant counterexamples
> to this pattern?
> 
> More specifically: if Lua puts a cost on setting the metatable of an
> object long after its creation (the longer the more expensive), do you
> know of programs that would be hurt?
> 
> Yet more specifically, I am interested only in cases where the metatable
> has a __gc metamethod.
> 
> -- Roberto



Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Mark Hamburg-4
In reply to this post by Roberto Ierusalimschy
So, it would presumably remain cheap to go from having a __gc metamethod to
having a different __gc metamethod. It could even be cheap to going from
having a __gc metamethod to not having a __gc metamethod provided the
finalization code dealt with the missing method case. What would be
expensive would be gaining a __gc metamethod well after creation? That seems
fairly reasonable.

Mark



Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Enrico Tassi-3
In reply to this post by Mike Pall-82
On Fri, Feb 08, 2008 at 08:13:25PM +0100, Mike Pall wrote:
> Roberto Ierusalimschy wrote:
> > To add another "next" pointer to each userdata would increase memory
> > consumption [...]
> 
> Probably not feasible here, but FYI:
>   http://en.wikipedia.org/wiki/XOR_linked_list

Thanks for the link, this made my day!

Cheers
-- 
Enrico Tassi

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Roberto Ierusalimschy
In reply to this post by Mark Hamburg-4
> So, it would presumably remain cheap to go from having a __gc metamethod to
> having a different __gc metamethod. It could even be cheap to going from
> having a __gc metamethod to not having a __gc metamethod provided the
> finalization code dealt with the missing method case. What would be
> expensive would be gaining a __gc metamethod well after creation? That seems
> fairly reasonable.

Yes. More specificaly, what would be expensive would be going from a
metatable without a __gc metamethod to a new metatable with a __gc
metamethod well after creation. Note that, if you change the metatable
itself, creating a __gc metamethod, Lua could not detect the change (and
so would not run the finalizer). (This would be an incompatibility...)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Doug Rogers-4
In reply to this post by Enrico Tassi-3
Enrico Tassi wrote:
> On Fri, Feb 08, 2008 at 08:13:25PM +0100, Mike Pall wrote:
>> Probably not feasible here, but FYI:
>>   http://en.wikipedia.org/wiki/XOR_linked_list
> Thanks for the link, this made my day!

I was just about to write a similar thing, "Learn something new every
day!", but then I got caught up in XOR swap, symmetric difference, etc.
Next thing you know my 'cleartool mklabel -r' command actually finished
and I had to get back to work! Clearcase gives a lot, but it's not the
lightest of tools. Too bad there was no svn when this project started...

I had never heard of doubly linked lists in a single xor'd pointer. It's
the kind of clever thing I might find being used by one of the DSP
programmers around here, before memory was even determined to be an
issue. Diabolically clever.

Doug

-- 
Innovative Concepts, Inc. www.innocon.com 703-893-2007 x220

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Mike Pall-82
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:
> Yes. More specificaly, what would be expensive would be going from a
> metatable without a __gc metamethod to a new metatable with a __gc
> metamethod well after creation.

To avoid deteriorating to O(#userdatas * #gcobjects) behaviour
you may want to use a lazy approach: if the userdata object is
not one of the first (say) 5 objects in the rootgc chain (i.e.
recently created) then stop the search. Instead set a global flag
which causes fallback to the old behaviour of atomically scanning
the whole rootgc chain.

> Note that, if you change the metatable
> itself, creating a __gc metamethod, Lua could not detect the change (and
> so would not run the finalizer). (This would be an incompatibility...)

A similar restriction already exists for __mode. I've not heard
of any problems.

--Mike

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Ariel Manzur
In reply to this post by Roberto Ierusalimschy
Most of the time (at least for me, and that's how tolua does it) all
instances of the same class share the same metatable, which always has
a __gc. If it's decided that a particular instance becomes "owned" by
lua after creation, it's registered on some table where the __gc
metamethod can search to see if it has to delete it.
Still, all instances have a __gc, so I assume if we wanted to keep
them out of atomic() we'd have to somehow add and remove __gc from the
userdatas, and take the penalty. The most common case is still that we
already know that the instance will be owned or not at the moment of
creation.

On Feb 8, 2008 12:45 PM, Roberto Ierusalimschy <[hidden email]> wrote:
> More often than not, programs set the metatable of a userdata right
> after creating the userdata. Does anyone know of relevant counterexamples
> to this pattern?
>
> More specifically: if Lua puts a cost on setting the metatable of an
> object long after its creation (the longer the more expensive), do you
> know of programs that would be hurt?
>
> Yet more specifically, I am interested only in cases where the metatable
> has a __gc metamethod.
>
> -- Roberto
>

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Edgar Toernig
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:
>
> More specifically: if Lua puts a cost on setting the metatable of an
> object long after its creation (the longer the more expensive), do you
> know of programs that would be hurt?

On manual finalization (something like fh:close()) I reset the metatable
of that userdata to nil ("I'm done with it - never wanna see it again.").

Modules using this scheme of manual finalization have as many "early
setting" setmetable calls as "late resetting" setmetable calls.
*Both* should be reasonably fast.

Ciao, ET.

PS: setmetatable shouldn't raise out-of-memory errors ...

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Roberto Ierusalimschy
> On manual finalization (something like fh:close()) I reset the metatable
> of that userdata to nil ("I'm done with it - never wanna see it again.").

That would not be a problem. What would be a problem is the reverse,
to put a metatable (with a __gc) into a userdata that did not have a
metatable (or that had a metatable without a __gc).

(Moreover, Mike Pall's suggestion (of turning to the current behavior if
the search gets too long) seems a nice compromise to avoid worst-case
behavior even in that situation.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Diego Nehab-3
Hi,

On manual finalization (something like fh:close()) I
reset the metatable of that userdata to nil ("I'm done
with it - never wanna see it again.").

That would not be a problem. What would be a problem is
the reverse, to put a metatable (with a __gc) into a
userdata that did not have a metatable (or that had a
metatable without a __gc).

I tend to do that too. Furthermore, I sometimes change the
metatable of an object throughout its life. For example, a
newly created socket object may have a certain set of
methods. Once you bind it to an address, or when you connect
it to some other socket, the set of available methods
changes to reflect that.  In my specific case, the __gc
metamethod is always present, and never changed.

I don't mind changing this behavior in exchange for some
functionality I may grow fond of in the future. :)

Kind regards,
Diego

Reply | Threaded
Open this post in threaded view
|

Re: when do programs set metatable?

Peter Loveday
In reply to this post by Roberto Ierusalimschy
More specifically: if Lua puts a cost on setting the metatable of an
object long after its creation (the longer the more expensive), do you
know of programs that would be hurt?

Not for us, we always set metatable right away.

Love, Light and Peace,
- Peter Loveday
Chief Software Architect, eyeon Software