Lua 5 and tags

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lua 5 and tags

Vincent Penne
Hi all,

I have been using lua 4 for some time and love it. I wanted to give a try to lua 5 however there is something that bugs me.

I have actually already used lua 5, but only as a standalone language. I think the language itself is even better than lua 4 used to be, particularly because each function now can have its own global variable environment.

Now, what interests me is to embed it and particularly to create my own new types (userdata or lightuserdata). And here is my problem, in lua 4, I think it was very very simple to do that with tags and tag methods. In particular, it was very easy to test the type of a userdata in C by simply looking at its tag.

But with lua 5, there aren't any more tags, so how do I check easily what is the type of my userdata ? I know I could look into the methatable for some custom entry that I would call "type" for example, but it looks overly complicated and inefficient. Furthermore, there aren't any unified way to generate new type values (like the lua_newtag function) so two libraries may accidentally generate two different kind of userdata with the same "type" value in their metatable, so it would not be safe either. Probably I have missed something ...

Another question : I see that there are light userdata, which are much like the userdata in lua 4, except that they don't have tags. The documentation doesn't seem clear about these light userdata, can I associate a metatable to them or not ? If not, how can we use them since there are absolutely no way we could test the type of these light userdata without metatable nor tag.

Vincent Penne.



Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Alex Sandro Queiroz e Silva-2
Hallo,

Vincent Penne wrote:
 > But with lua 5, there aren't any more tags, so how do I check easily
what is the type of my userdata ? I know I could look into the methatable for some custom entry that I would call "type" for example, but it looks overly complicated and inefficient. Furthermore, there aren't any unified way to generate new type values (like the lua_newtag function) so two libraries may accidentally generate two different kind of userdata with the same "type" value in their metatable, so it would not be safe either. Probably I have missed something ...


You can say two objects (userdata or tables) are of the same type if they share the same metatable. If want to refer to your types by name, you can create entries in the global namespace using the type name as a key and the metatable as a value. I guess luaL_newmetatable and luaL_getmetatable do just that.

Another question : I see that there are light userdata, which are much like the userdata in lua 4, except that they don't have tags. The documentation doesn't seem clear about these light userdata, can I associate a metatable to them or not ? If not, how can we use them since there are absolutely no way we could test the type of these light userdata without metatable nor tag.


You can't associate metatables with light userdata, but they are useful in other contexts. The I/O standard library stores FILE* pointers in light user data.

Now that Programming in Lua is online (http://www.lua.org/pil/), it's definitely a excellent read for anyone studying Lua.

-alex

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Vincent Penne
Alex Sandro Queiroz e Silva wrote:

Hallo,

You can say two objects (userdata or tables) are of the same type if they share the same metatable. If want to refer to your types by name, you can create entries in the global namespace using the type name as a key and the metatable as a value. I guess luaL_newmetatable and luaL_getmetatable do just that.


Ok, I see the idea. I was not aware of these luaL functions, this is what I'm going to need. There is even a luaL_checkudata which does exactly what I want, that is checking the type of my userdata.

You can't associate metatables with light userdata, but they are useful in other contexts. The I/O standard library stores FILE* pointers in light user data.

Ok ... I can certainly see some use for them, but I suppose you have to be careful.

Now that Programming in Lua is online (http://www.lua.org/pil/), it's definitely a excellent read for anyone studying Lua.


I will ! Thanks for the answer.

Vincent Penne.



Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Jamie Webb-3
On Sun, Jul 25, 2004 at 01:02:16AM +0200, Vincent Penne wrote:
> Alex Sandro Queiroz e Silva wrote:
> 
> >Hallo,
> >
> >    You can say two objects (userdata or tables) are of the same type 
> >if they share the same metatable. If want to refer to your types by 
> >name, you can create entries in the global namespace using the type 
> >name as a key and the metatable as a value. I guess luaL_newmetatable 
> >and luaL_getmetatable do just that.
>
> Ok, I see the idea. I was not aware of these luaL functions, this is 
> what I'm going to need. There is even a luaL_checkudata which does 
> exactly what I want, that is checking the type of my userdata.

Unfortunately, neither the reference nor PIL covers them. I think
there are some brief descriptions on the lua-users.org Wiki though.

> >    You can't associate metatables with light userdata, but they are 
> >useful in other contexts. The I/O standard library stores FILE* 
> >pointers in light user data.

Actually, the FILE pointers are stored in pointer-sized full userdata.
Hence you can do file:read(), etc.

> Ok ... I can certainly see some use for them, but I suppose you have to 
> be careful.

Light userdatas are intended for internal use by a module, where a
full userdata would be overkill. They shouldn't be exposed to user
code, for exactly the reason that you can't typecheck them.

-- Jamie Webb

Reply | Threaded
Open this post in threaded view
|

light userdata (was: Lua 5 and tags)

Wim Couwenberg-4
In reply to this post by Vincent Penne
> If not, how can we use them since there are 
> absolutely no way we could test the type of
> these light userdata without metatable nor tag.

I tend to use light userdata in the following
situations:

1) As key for the registry.  This is a very convenient
way to make sure your registry keys are private.  An
example:

    instead of:

    /* retrieve private data */
    lua_pushstring(L, "mykey");
    lua_gettable(L, LUA_REGISTRYINDEX);

    I do it like this:

    /* in file scope: */
    static char mykey[] = "mykey";

    /* retrieve private data */
    lua_pushlightuserdata(L, mykey);
    lua_gettable(L, LUA_REGISTRYINDEX);


2) If garbage collection is not an issue you can
implement types purely in scripts and add the actual
C/C++ object as light userdata, e.g. by assigning it
to a "__data" key or such.  This can save some hassle
to create, administrate and access user data.

--
Wim



		
__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Alex Sandro Queiroz e Silva-2
In reply to this post by Jamie Webb-3
Hallo,

Jamie Webb wrote:

You can't associate metatables with light userdata, but they are useful in other contexts. The I/O standard library stores FILE* pointers in light user data.


Actually, the FILE pointers are stored in pointer-sized full userdata.
Hence you can do file:read(), etc.


	Oops, my bad! Thanks for correcting me.

-alex

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Mark Hamburg-4
In reply to this post by Vincent Penne
The Lua 5 approach is somewhat less efficient than one might like. If
efficiency is a concern, there are a variety of things one can do:

* Rather than using strings to identify types, use light userdata. This
assumes that you have some permanent address to use for this purpose. This
will avoid some string manipulation when doing type-testing.

* If one counts on the garbage collector not moving things or at least on
lua_topointer being stable, you could use do a comparison on the result of
lua_topointer with the metatable. Note that this only works if your program
limits itself to creating only one Lua universe.

* I actually modified the Lua sources to allow a one-byte type tag on full
userdata. I use this to recognize some specific common cases where I can't
just do a metatable comparison. For example, I'm proxying to Objective-C
objects and I wanted a quick way to confirm that a full userdata was a proxy
for such an object before extracting the object pointer. I feel ever so
slightly dirty for having done this but it's a big win when crossing the
Lua/Objective-C boundary frequently (particularly if lua_lock is doing a
mutex lock).

One thing that would really help in the Lua specification would be a way to
bind methods to userdata such that they couldn't be pulled off and used
elsewhere -- i.e., they would be accessible as ud:method() but not as
ud.method. This would allow one to bypass type-tests in native method calls.
To go with this, one would also want something like:

    bindmethod( obj, method )

This would return a function to call method on obj.

Now, the Lua authors can rightfully ask whether profiling has really shown
these issues to be problems and I have to admit that most of my judgment
(after getting over the mutex overhead issues) stemmed from the observation
that for at least some native methods over the half the code was being spent
doing type-testing that could have been avoided rather than from actual
profiling.

Mark


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Alex Sandro Queiroz e Silva-2
Hallo,

Mark Hamburg wrote:
The Lua 5 approach is somewhat less efficient than one might like. If
efficiency is a concern, there are a variety of things one can do:

* Rather than using strings to identify types, use light userdata. This
assumes that you have some permanent address to use for this purpose. This
will avoid some string manipulation when doing type-testing.


	But in Lua a string comparison is just a pointer comparison, isn't it?

-alex

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Jamie Webb-3
On Sun, Jul 25, 2004 at 02:39:01PM -0300, Alex Sandro Queiroz e Silva wrote:
> Hallo,
> 
> Mark Hamburg wrote:
> >The Lua 5 approach is somewhat less efficient than one might like. If
> >efficiency is a concern, there are a variety of things one can do:
> >
> >* Rather than using strings to identify types, use light userdata. This
> >assumes that you have some permanent address to use for this purpose. This
> >will avoid some string manipulation when doing type-testing.
> >
> 
> 	But in Lua a string comparison is just a pointer comparison, isn't 
> 	it?

Yes, but when testing a string on the C-side, you either have to use
strcmp or obtain a Lua string to test against from the registry
or somesuch. For short strings, strcmp would be faster.

Actually, you can just store the address of the Lua string in C, and
as long as you ensure that the string isn't garbage-collected, the
address won't change. That behaviour isn't specced though, and is
functionally the same as using light userdata.

-- Jamie Webb

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Mark Hamburg-4
In reply to this post by Alex Sandro Queiroz e Silva-2
Only between two Lua strings. If one of the strings is coming from C, it
needs to at least go through strcmp if not being actually converted to a Lua
string.

Mark

on 7/25/04 10:39 AM, Alex Sandro Queiroz e Silva at
[hidden email] wrote:

> Hallo,
> 
> Mark Hamburg wrote:
>> The Lua 5 approach is somewhat less efficient than one might like. If
>> efficiency is a concern, there are a variety of things one can do:
>> 
>> * Rather than using strings to identify types, use light userdata. This
>> assumes that you have some permanent address to use for this purpose. This
>> will avoid some string manipulation when doing type-testing.
>> 
> 
> But in Lua a string comparison is just a pointer comparison, isn't it?
> 
> -alex


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Mark Hamburg-4
In reply to this post by Jamie Webb-3
Warning untested (and uncompiled) code:

LUALIB_API int luaL_newmetatable_ud (lua_State *L, void *tident) {
  lua_pushlightuserdata(L, tident);
  lua_rawget(L, LUA_REGISTRYINDEX);  /* get registry[ tident ] */
  if (!lua_isnil(L, -1))  /* tident already in use? */
    return 0;  /* leave previous value on top, but return 0 */
  lua_pop(L, 1);
  lua_newtable(L);  /* create metatable */
  lua_pushlightuserdata(L, tident);
  lua_pushvalue(L, -2);
  lua_rawset(L, LUA_REGISTRYINDEX);  /* registry[ tident ] = metatable */
  lua_pushvalue(L, -1);
  lua_pushlightuserdata(L, tident);
  lua_rawset(L, LUA_REGISTRYINDEX);  /* registry[metatable] = tident */
  return 1;
}


LUALIB_API void  luaL_getmetatable_ud (lua_State *L, void *tident) {
  lua_pushlightuserdata(L, tident);
  lua_rawget(L, LUA_REGISTRYINDEX);
}


LUALIB_API void *luaL_checkudata_ud (lua_State *L, int ud, void *tident) {
  if (!lua_getmetatable(L, ud)) return NULL;  /* no metatable? */
  lua_rawget(L, LUA_REGISTRYINDEX);  /* get registry[metatable] */
  if (lua_touserdata(L, -1) == tident) {
    lua_pop(L, 1);
    return lua_touserdata(L, ud);
  }
  else {
    lua_pop(L, 1);
    return NULL;
  }
}

(Interesting. luaL_checkudata doesn't throw on errors. It should probably be
named luaL_toudata. If the check routine did throw, one might then want a
name either at creation or check time but just for constructing the error
message.)

Mark


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Mike Pall-44
In reply to this post by Mark Hamburg-4
Hi,

Mark Hamburg wrote:
> Only between two Lua strings. If one of the strings is coming from C, it
> needs to at least go through strcmp if not being actually converted to a Lua
> string.

In fact when I was profiling Diego's Luasocket, I noticed that one of my
test cases was spending 12% of its time in luaS_newlstr() (due to the
specific way Luasocket checks its userdata arguments). Converting C strings
to Lua strings all the time is pretty expensive.

I think the fastest solution is to register all methods with an upvalue
that holds a reference to the metatable. Storing and retrieving the
metatable to/from the registry is not necessary. And you don't have to
provide a unique name or a unique memory address for all of your userdata
types.

The performance sensitive code path is then:

void *luaL_checkudata_upval(lua_State *L, int ud, int uv, char *msg)
{
  void *u;
  if (lua_getmetatable(L, ud) &&
      lua_rawequal(L, -1, lua_upvalueindex(uv)) &&
      (u = lua_touserdata(L, ud))) {
    lua_pop(L, 1);
    return u;
  }
  luaL_argerror(L, 0, msg); /* Never returns. */
  return NULL;
}

lua_rawequal() is certainly a lot faster than lua_rawget() + lua_tostring() +
strcmp() (the lauxlib string key solution) or just lua_rawget() (the
lightuserdata plus fixed pointer solution). Making the check an inline
function or macro will save a few more cycles.

The standard usage is:

#define foo_checkudata(L) \
  ((foo_t *)luaL_checkudata_upval((L), 1, 1, "foo expected"))

int foo_method(lua_State *L)
{
  foo_t *foo = foo_checkudata(L);
  ...
}


Adding the following to lapi.c would allow for an even faster solution:

void *lua_touserdata_mc(lua_State *L, int indexu, int indexm)
{
  StkId uo = luaA_index(L, indexu);
  StkId mo = luaA_index(L, indexm);
  if (!ttisuserdata(uo) || !ttistable(mo) ||
      uvalue(uo)->metatable != hvalue(mo)) return NULL;
  return rawuvalue(uo) + 1;
}

[untested]

Bye,
     Mike

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Vincent Penne
In reply to this post by Mark Hamburg-4
Mark Hamburg wrote:

The Lua 5 approach is somewhat less efficient than one might like. If
efficiency is a concern, there are a variety of things one can do:

* Rather than using strings to identify types, use light userdata. This
assumes that you have some permanent address to use for this purpose. This
will avoid some string manipulation when doing type-testing.

Indeed. I was thinking about using a number (similar to lua 4 tag), with the difference that it would now be a float, but a light userdata would be even better here. However, it still involves the lookup into the registry when converting the metatable to this ID.

* If one counts on the garbage collector not moving things or at least on
lua_topointer being stable, you could use do a comparison on the result of
lua_topointer with the metatable. Note that this only works if your program
limits itself to creating only one Lua universe.

Does lua_topointer return a pointer on the internal lua structure of the lua object ? (the metatable in this case) If so, that would work perfectly. It does not even limit you to one lua state, if your C code store these metatable adresses in different structures for each lua state.

However, you talk about the garbage collector. Is it possible that a lua object that is never garbage collected get moved in memory ? Is it a new feature of lua 5 ?



Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Vincent Penne
In reply to this post by Mike Pall-44
Mike Pall wrote:

Hi,

Mark Hamburg wrote:
Only between two Lua strings. If one of the strings is coming from C, it
needs to at least go through strcmp if not being actually converted to a Lua
string.

In fact when I was profiling Diego's Luasocket, I noticed that one of my
test cases was spending 12% of its time in luaS_newlstr() (due to the
specific way Luasocket checks its userdata arguments). Converting C strings
to Lua strings all the time is pretty expensive.

I think the fastest solution is to register all methods with an upvalue
that holds a reference to the metatable. Storing and retrieving the
metatable to/from the registry is not necessary. And you don't have to
provide a unique name or a unique memory address for all of your userdata
types.

The problem with that solution is that it complicates greatly the registration of functions, because you have to think in advance of all types that are going to be checked by your function (for example let's say that your function takes 10 arguments, each of different types). You would need to have 10 upvalues for your function, furthermore you would need to remember which upvalue index corresponds to which type.

However, a similar idea would be to create lua references on the metatables with lua_ref , then you use lua_rawequals like you suggested. However I have no idea of the efficiency of lua_ref / lua_getref functions (haven't looked at the code yet).




Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Mike Pall-44
Hi,

Vincent Penne wrote:
> The problem with that solution is that it complicates greatly  the 
> registration of functions, because you have to think in advance of all 
> types that are going to be checked by your function (for example let's 
> say that your function takes 10 arguments, each of different types). You 
> would need to have 10 upvalues for your function, furthermore you would 
> need to remember which upvalue index corresponds to which type.

Of all the Lua libraries I have come across recently, most methods need
exactly one userdata argument (i.e. 'self'). Only a few need two.

Maybe that's because 'doing it the Lua way' suggest a different API style
(e.g. using strings as options or keys, using multiple arguments instead
of wrapping them up into userdata structures).

The case you mention is somewhat common when you have to create a binding
to huge OO-style libraries that have not been designed exclusively for Lua
(GUI libraries come to mind). I'm pretty sure it pays off to autogenerate
the wrapper code in this case. Then juggling around with different metatables
and upvalues really doesn't matter. But performance does.

> However, a similar idea would be to create lua references on the 
> metatables with lua_ref , then you use lua_rawequals like you suggested. 
> However I have no idea of the efficiency of lua_ref / lua_getref 
> functions (haven't looked at the code yet).

2* lua_rawgeti() + 2* lua_rawseti() plus a few other API calls for the
common code path in luaL_ref(). I guess this is even slower than any
of the registry approaches.

Bye,
     Mike

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Vincent Penne
Mike Pall wrote:

Hi,

Vincent Penne wrote:
The problem with that solution is that it complicates greatly the registration of functions, because you have to think in advance of all types that are going to be checked by your function (for example let's say that your function takes 10 arguments, each of different types). You would need to have 10 upvalues for your function, furthermore you would need to remember which upvalue index corresponds to which type.

Of all the Lua libraries I have come across recently, most methods need
exactly one userdata argument (i.e. 'self'). Only a few need two.

Maybe that's because 'doing it the Lua way' suggest a different API style
(e.g. using strings as options or keys, using multiple arguments instead
of wrapping them up into userdata structures).

The case you mention is somewhat common when you have to create a binding
to huge OO-style libraries that have not been designed exclusively for Lua
(GUI libraries come to mind). I'm pretty sure it pays off to autogenerate
the wrapper code in this case. Then juggling around with different metatables
and upvalues really doesn't matter. But performance does.

Actually, that's exactly what I had in mind :) So, yes, it really depends on the context. If your only userdata is self then it's definitely worth putting it as upvalue.

2* lua_rawgeti() + 2* lua_rawseti() plus a few other API calls for the
common code path in luaL_ref(). I guess this is even slower than any
of the registry approaches.


I mean lua_getref, not luaL_ref. Actually, lua_getref is just a lua_rawgeti into the registry (it's a #define). It is faster than the registry approach, I just benchmarked it. It is not surprising because lua_rawgeti is faster than lua_rawget (I think) and there is no string comparison. It's probably slower than the upvalue method however, but I cannot test it simply at the moment since I have a lot of arguments with different types in my functions :)

Here is the code that do what I suggest :

void *luaL_checkudata_ref(lua_State *L, int ud, int ref, char *msg)
{
 void *u;
 if (lua_getmetatable(L, ud)) && (lua_getref(L, ref), 1) &&
     lua_rawequal(L, -1, -2) &&
     (u = lua_touserdata(L, ud))) {
   lua_pop(L, 2);
   return u;
 }
 luaL_argerror(L, 0, msg); /* Never returns. */
 return NULL;
}




Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Edgar Toernig
In reply to this post by Mike Pall-44
Mike Pall wrote:
>
> I think the fastest solution is to register all methods with an upvalue
> that holds a reference to the metatable. Storing and retrieving the
> metatable to/from the registry is not necessary. And you don't have to
> provide a unique name or a unique memory address for all of your userdata
> types.

That's what I'm using.  It works well.  But, as Vincent mentioned, it
becomes really ugly once you put a lot of stuff into upvalues (incl.
string constants etc).  Managing which function needs which upvalues
becomes tricky and the memory overhead grows a lot because these values
are copied for each closure.

My suggestion: a (module) private registry as I outlined some weeks
ago and a new special index 'lua_registryindex(i)' which addresses
numeric indexes in that private registry.

That way one could put all that module-global stuff (like meta tables,
string constants, mappings, etc) once into the registry and access
them everywhere within the module via numerical constants.
Speed would be nearly as fast as upvalues.

> Adding the following to lapi.c would allow for an even faster solution:
> 
> void *lua_touserdata_mc(lua_State *L, int indexu, int indexm)

Yeah ;-)

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Jamie Webb-3
On Mon, Jul 26, 2004 at 02:03:35AM +0200, Edgar Toernig wrote:
> My suggestion: a (module) private registry as I outlined some weeks
> ago and a new special index 'lua_registryindex(i)' which addresses
> numeric indexes in that private registry.
> 
> That way one could put all that module-global stuff (like meta tables,
> string constants, mappings, etc) once into the registry and access
> them everywhere within the module via numerical constants.
> Speed would be nearly as fast as upvalues.

I'd like to add that adding an array part to userdatas would also
provide fast typing: put a unique light userdata at some known index.
And indeed, if you could access that array using only the userdata
pointer, a userdata could serve as a module registry: just get at it
using a C global variable.

-- Jamie Webb

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Edgar Toernig
Jamie Webb wrote:
>
> I'd like to add that adding an array part to userdatas would also
> provide fast typing:

Hmm... that gets me back to the old idea of merging userdata and
tables: give every table an additional void* field and remove the
userdata type completely *g*

>                       put a unique light userdata at some known index.
> And indeed, if you could access that array using only the userdata
> pointer, a userdata could serve as a module registry: just get at it
> using a C global variable.

Hmm, sounds dangerous and not really handy...

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5 and tags

Mark Hamburg-4
In reply to this post by Vincent Penne
on 7/25/04 12:56 PM, Vincent Penne at [hidden email] wrote:

>> * If one counts on the garbage collector not moving things or at least on
>> lua_topointer being stable, you could use do a comparison on the result of
>> lua_topointer with the metatable. Note that this only works if your program
>> limits itself to creating only one Lua universe.
>>  
>> 
> 
> Does lua_topointer return a pointer on the internal lua structure of the
> lua object ? (the metatable in this case) If so, that would work
> perfectly. It does not even limit you to one lua state, if your C code
> store these metatable adresses in different structures for each lua state.

I think lua_topointer returns a pointer to the current location of some
piece of data associated with the object such as the internal data
structures for a table. This pointer will be unique across Lua universes,
but suffers from the problem that if you cache it in your native data
structures -- e.g., in a global variable -- you need to cache a different
value for each Lua universe.

> However, you talk about the garbage collector. Is it possible that a lua
> object that is never garbage collected get moved in memory ? Is it a new
> feature of lua 5 ?

The current Lua garbage collector does not move objects but that isn't
promised anywhere and there are plenty of garbage collectors that do move
objects. The Microsoft CLR, for example, I believe uses a copying collector.
A promise in the API that even if a copying collector were added that
lua_topointer would continue to return the same value for any particular
object would potentially make lua_topointer more useful.

Mark


12