Feature request: userdata slice

classic Classic list List threaded Threaded
90 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Feature request: userdata slice

云风 Cloud Wu
Lua is a embedded language, it uses powerful userdata to wrap the c object as the lua object. But sometimes it's not enough.

For example, if we have a C struct like this :

struct foo {
  struct foo1 foo1;
  struct foo2 *foo2;
};

We can use userdata to store struct foo in lua variables : 

struct foo * f = lua_newuserdata(L, sizeof(*f));

How to reference sub struct foo1/foo2 in lua ? maybe we can give the userdata a metatable, implementing an __index metamethod.

But foo.foo1 or foo.foo2 should be a new userdata. It needs more complex and expensive way :

Creating a proxy userdata to access a part of struct foo, put the reference of proxy userdata in uservalue of userdata foo to prevent collecting foo before foo1/foo2 proxy, etc.

So I suggest introduce a new mechanism of userdata to simplify them. I call it userdata slice (maybe it's not a good name).

If we can associate an integer to the userdata value, the userdata whould be more flexible. It is different from uservalue, because uservalue is in userdata object, not in value.

An userdata object has only one uservalue , but each userdata value can has an unique associated integer.

Two C API needed:

int lua_getuserslice(lua_State *L, int index);
void lua_setuserslice(lua_State *L, int index, int slice);

Any value with type LUA_TUSERDATA in lua is an two 32bit integer tuple ( or two 16bit integer for small lua), one is the userdata (32bit int) handle , another is the associated integer.

Using handle instead of really address of userdata object because we can put two 32bit integer in 64bit lua value.

I think it's easy to implement and keep the c api compatibility with the previous version. 

Storing C/C++ objects in GUI or 3d engines could be more simple : OBJECT.X and OBJECT.X.Y can be the same userdata with OBJECT without creating new userdata, only the slice is different.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Daurnimator
On 19 August 2015 at 13:18, 云风 Cloud Wu <[hidden email]> wrote:

> Lua is a embedded language, it uses powerful userdata to wrap the c object
> as the lua object. But sometimes it's not enough.
>
> For example, if we have a C struct like this :
>
> struct foo {
>   struct foo1 foo1;
>   struct foo2 *foo2;
> };
>
> We can use userdata to store struct foo in lua variables :
>
> struct foo * f = lua_newuserdata(L, sizeof(*f));
>
> How to reference sub struct foo1/foo2 in lua ? maybe we can give the
> userdata a metatable, implementing an __index metamethod.
>
> But foo.foo1 or foo.foo2 should be a new userdata. It needs more complex and
> expensive way :
>
> Creating a proxy userdata to access a part of struct foo, put the reference
> of proxy userdata in uservalue of userdata foo to prevent collecting foo
> before foo1/foo2 proxy, etc.
>
> So I suggest introduce a new mechanism of userdata to simplify them. I call
> it userdata slice (maybe it's not a good name).
>
> If we can associate an integer to the userdata value, the userdata whould be
> more flexible. It is different from uservalue, because uservalue is in
> userdata object, not in value.
>
> An userdata object has only one uservalue , but each userdata value can has
> an unique associated integer.
>
> Two C API needed:
>
> int lua_getuserslice(lua_State *L, int index);
> void lua_setuserslice(lua_State *L, int index, int slice);

You're saying the slice offset would be stored in the userdata itself?
This would imply that setting a slice would mutate the userdata object?

If not, your slice still needs to create an extra object,
in which case, how is this better than the uservalue option?
i.e.

void makeslice(lua_State *L, int idx, ssize_t offset) {
    idx = lua_absindex(L, idx);
    ssize_t* slice = lua_newuserdata(L, sizeof(ssize_t));
    *slice = offset;
    lua_pushvalue(L, idx);
    lua_setuservalue(L, -2);
}
void* getslice(lua_State *L, int idx) {
    ssize_t offset = *(ssize_t*)lua_touserdata(L, idx);
    void* udata = lua_touserdata(L, lua_getuservalue(L, idx));
    return udata+offset;
}

You would also need to add argument checking to the above.
But you *still* need to add metamethods to these objects.
e.g. You have a slice, but you need to define a getter/setters for the
struct components.
At which point, you might as well write a 'slice' customised for your
needs, as there won't be a general one-size-fits-all solution.

> Any value with type LUA_TUSERDATA in lua is an two 32bit integer tuple ( or
> two 16bit integer for small lua), one is the userdata (32bit int) handle ,
> another is the associated integer.
>
> Using handle instead of really address of userdata object because we can put
> two 32bit integer in 64bit lua value.
>
> I think it's easy to implement and keep the c api compatibility with the
> previous version.

I'm not sure how this would work?
You need 64bits to store a userdata address on a 64 bit architecture.

> Storing C/C++ objects in GUI or 3d engines could be more simple : OBJECT.X
> and OBJECT.X.Y can be the same userdata with OBJECT without creating new
> userdata, only the slice is different.

But you would still need a different object.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu


Daurnimator <[hidden email]>于2015年8月19日周三 上午11:50写道:

You're saying the slice offset would be stored in the userdata itself?
This would imply that setting a slice would mutate the userdata object?

If not, your slice still needs to create an extra object,
in which case, how is this better than the uservalue option?
i.e.

void makeslice(lua_State *L, int idx, ssize_t offset) {
    idx = lua_absindex(L, idx);
    ssize_t* slice = lua_newuserdata(L, sizeof(ssize_t));
    *slice = offset;
    lua_pushvalue(L, idx);
    lua_setuservalue(L, -2);
}
void* getslice(lua_State *L, int idx) {
    ssize_t offset = *(ssize_t*)lua_touserdata(L, idx);
    void* udata = lua_touserdata(L, lua_getuservalue(L, idx));
    return udata+offset;
}


I don't want to create an extra object, because it's more expensive. and we should cache all the extra objects because the extra object with the same slice offset should not be create more than once, if we use them as a table key .
 
I'm not sure how this would work?
You need 64bits to store a userdata address on a 64 bit architecture.

We can create the array in global state to store all the userdata pointer, and use the index (handle) in lua value, 32bit index is enough. and lua_touserdata can be also return the pointer by lookup the array with index.
 
> Storing C/C++ objects in GUI or 3d engines could be more simple : OBJECT.X
> and OBJECT.X.Y can be the same userdata with OBJECT without creating new
> userdata, only the slice is different.

But you would still need a different object.


We don't need a different object, an associated integer  is enough. In the metamethod, we can get the associated integer (It can be offset or a index number) first, and then convert it to the sub object's real address with a map (in C) in root object. 

This can be reduce the number of lua userdata, we don't need create large number of userdata for C objects , many sub objects may only access once (especially in 3d engine bindings), but we should also create an userdata for them now, and should do more for collecting them correctly.
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Tim Hill
In reply to this post by 云风 Cloud Wu

> On Aug 18, 2015, at 8:18 PM, 云风 Cloud Wu <[hidden email]> wrote:
>
>
> So I suggest introduce a new mechanism of userdata to simplify them. I call it userdata slice (maybe it's not a good name).
>
> If we can associate an integer to the userdata value, the userdata whould be more flexible. It is different from uservalue, because uservalue is in userdata object, not in value.
>
> An userdata object has only one uservalue , but each userdata value can has an unique associated integer.
>
> Two C API needed:
>
> int lua_getuserslice(lua_State *L, int index);
> void lua_setuserslice(lua_State *L, int index, int slice);
>

-1 on this. One of the advantages (for me) of userdata in Lua is that there is *no* language defined way that userdata can be modified. This means a C library can use Lua as the container for opaque data and be assured that whatever it stores there is not meddled with. If you want this sort of functionality on some specific userdata, then it’s pretty easy to expose an appropriate custom API for that purpose.

—Tim



Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Daurnimator
In reply to this post by 云风 Cloud Wu
On 19 August 2015 at 14:07, 云风 Cloud Wu <[hidden email]> wrote:

>
>
> Daurnimator <[hidden email]>于2015年8月19日周三 上午11:50写道:
>>
>>
>> You're saying the slice offset would be stored in the userdata itself?
>> This would imply that setting a slice would mutate the userdata object?
>>
>> If not, your slice still needs to create an extra object,
>> in which case, how is this better than the uservalue option?
>> i.e.
>>
>> void makeslice(lua_State *L, int idx, ssize_t offset) {
>>     idx = lua_absindex(L, idx);
>>     ssize_t* slice = lua_newuserdata(L, sizeof(ssize_t));
>>     *slice = offset;
>>     lua_pushvalue(L, idx);
>>     lua_setuservalue(L, -2);
>> }
>> void* getslice(lua_State *L, int idx) {
>>     ssize_t offset = *(ssize_t*)lua_touserdata(L, idx);
>>     void* udata = lua_touserdata(L, lua_getuservalue(L, idx));
>>     return udata+offset;
>> }
>>
>
> I don't want to create an extra object, because it's more expensive. and we
> should cache all the extra objects because the extra object with the same
> slice offset should not be create more than once, if we use them as a table
> key .
>
>>
>> I'm not sure how this would work?
>> You need 64bits to store a userdata address on a 64 bit architecture.
>
>
> We can create the array in global state to store all the userdata pointer,
> and use the index (handle) in lua value, 32bit index is enough. and
> lua_touserdata can be also return the pointer by lookup the array with
> index.

Having an extra layer of indirection via some global index of
userdatas seems a terrible idea.
It would have to be paid by *every* userdata access.

>> > Storing C/C++ objects in GUI or 3d engines could be more simple :
>> > OBJECT.X
>> > and OBJECT.X.Y can be the same userdata with OBJECT without creating new
>> > userdata, only the slice is different.
>>
>> But you would still need a different object.
>>
>
> We don't need a different object, an associated integer  is enough. In the
> metamethod, we can get the associated integer (It can be offset or a index
> number) first, and then convert it to the sub object's real address with a
> map (in C) in root object.
>
> This can be reduce the number of lua userdata, we don't need create large
> number of userdata for C objects , many sub objects may only access once
> (especially in 3d engine bindings), but we should also create an userdata
> for them now, and should do more for collecting them correctly.

You do need a separate object, or the objects are the same....
I doubt you want rawequal(object.foo1, object.foo2) == true, or
rawequal(object, object.foo1) == true.


-1 on this whole idea.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
In reply to this post by Tim Hill
-1 on this. One of the advantages (for me) of userdata in Lua is that there is *no* language defined way that userdata can be modified. This means a C library can use Lua as the container for opaque data and be assured that whatever it stores there is not meddled with. If you want this sort of functionality on some specific userdata, then it’s pretty easy to expose an appropriate custom API for that purpose.


But we use full userdata rather than light userdata because full userdata is a gc object.

In  many cases, we need reference the different parts of an object, but they are the same one object (to garbage collector). If we create dozens of userdata for them,  we need more expensive works : cache these reference userdata in a weak table, put the root object in their uservalue, etc.


Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
In reply to this post by Daurnimator
Having an extra layer of indirection via some global index of
userdatas seems a terrible idea.
It would have to be paid by *every* userdata access.

Lua don't need access userdata, only C function needs. If you call lua_touserdata to access userdata, an simple extra layer is not more expensive.

And all the userdata with __gc metamethod  are already in a seperated list. 
 
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
In reply to this post by Daurnimator
You do need a separate object, or the objects are the same....
I doubt you want rawequal(object.foo1, object.foo2) == true, or
rawequal(object, object.foo1) == true.


object.foo1 is not equal to object.foo2, because the slice of values are different.  Comparing two values with type LUA_TUSERDATA need not change: just compare two tuples (A tuple is combine with two 32bit integers)

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Sean Conner
In reply to this post by 云风 Cloud Wu
It was thus said that the Great 云风 Cloud Wu once stated:

> Lua is a embedded language, it uses powerful userdata to wrap the c object
> as the lua object. But sometimes it's not enough.
>
> For example, if we have a C struct like this :
>
> struct foo {
>   struct foo1 foo1;
>   struct foo2 *foo2;
> };
>
> We can use userdata to store struct foo in lua variables :
>
> struct foo * f = lua_newuserdata(L, sizeof(*f));
>
> How to reference sub struct foo1/foo2 in lua ? maybe we can give the
> userdata a metatable, implementing an __index metamethod.
>
> But foo.foo1 or foo.foo2 should be a new userdata. It needs more complex
> and expensive way :
>
> Creating a proxy userdata to access a part of struct foo, put the reference
> of proxy userdata in uservalue of userdata foo to prevent collecting foo
> before foo1/foo2 proxy, etc.

  No.  Just create two userdatas, one of struct foo1 and one for struct
foo2.  struct foo can then be converted to a plain Lua table with two
fields, foo1 and foo2, both using the appropriate custom user data.

  Or, convert both struct foo1 and struct foo2 to full Lua tables with more
primitive types.  I do this *all the time* for stuff at work.  In one
instance, it's a binary file containing around 100,000,000 records.  The
file itself is mmap()ed into memory and wrapped up into a userdata type.
The fields themselves aren't complicated and the index method just creates a
Lua table with a string and bunch of booleans (stored internally as a 32-bit
value).  Another instance is a binary file of around 44k in size that is
broken down into primitive Lua types (numbers, strings, tables, etc) when
read.  Yeah, it's quite a bit of code (going to and from Lua) but once done,
it's done, and performance hasn't been an issue.

> So I suggest introduce a new mechanism of userdata to simplify them. I call
> it userdata slice (maybe it's not a good name).
>
> If we can associate an integer to the userdata value, the userdata whould
> be more flexible. It is different from uservalue, because uservalue is in
> userdata object, not in value.
>
> An userdata object has only one uservalue , but each userdata value can has
> an unique associated integer.
>
> Two C API needed:
>
> int lua_getuserslice(lua_State *L, int index);
> void lua_setuserslice(lua_State *L, int index, int slice);
>
> Any value with type LUA_TUSERDATA in lua is an two 32bit integer tuple ( or
> two 16bit integer for small lua), one is the userdata (32bit int) handle ,
> another is the associated integer.
>
> Using handle instead of really address of userdata object because we can
> put two 32bit integer in 64bit lua value.

  -1.  

  AN ADDRESS IS NOT AN INTEGER!

  I remember this crap going from 16 bit to 32 bit systems [1] and frankly,
I'd rather not have to live through that again going to 64 bit [3][4].

  Also, I don't think I'm following your API there.  I have this structure:

        typedef union sockaddr_all
        {
          struct sockaddr     sa;  
          struct sockaddr_in  sin;
          struct sockaddr_in6 sin6;
          struct sockaddr_un  ssun;
        } sockaddr_all__t;

which I'm using for a userdata.  I have a custom __index method associated
with this that gives me the following "fields":

        addr - text representation of IPv4/IPv6 address
        daddr - other text representation of IPv4/IPv6 address [5]
        addrbits - The raw binary bits that make up an IPv4/IPv6 address
        port - port number
        family - string, either 'ip', 'ip6' or 'unix'.
        display - textual representation of address and port.

  How would your system with with that?

  -spc

[1] Want to know how false this is?  I can name a mainstream system that
        used C, that could have both 16 *AND* 32 bit pointers *IN THE SAME
        PROGRAM* and where ints were 16 *BITS* long. [2]

[2] Ever wonder what FAR means in zlib.h? [6] Yeah, it comes from that
        time frame.  Never want to go back to it.

[3] One my my pet peeves with LuaJIT is the 2G limit on memory
        allocation, because of tricks played with pointers.

[4] I have an old program written in the early 90s that I'm having a
        hell of a time porting to 64 bit systems because of the assumption
        that an integer and pointers are all 32 bits and are completely
        interchangable.

[5] The difference between the two only show up for IPv6 addresses.  The
        addr field will return "fc00::1" while the daddr field will return
        "[fc00::1]".  

[6] Are you SURE you want to ask?  Because I can and will answer that.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu

  AN ADDRESS IS NOT AN INTEGER!

  I remember this crap going from 16 bit to 32 bit systems [1] and frankly,
I'd rather not have to live through that again going to 64 bit [3][4].

An address in not an integer, but a handle can be.
For example, in posix , file handle is an integer , and file is an object ,too.

We don't need create more than 2^32 userdata in one lua vm, so we can create an array in lua global state ( G(L) ),  and use an integer index to indicate the userdata object's address.

It would be like this:

union Value {
  GCObject *gc;    /* collectable objects */
  void *p;         /* light userdata */
  int b;           /* booleans */
  lua_CFunction f; /* light C functions */
  lua_Integer i;   /* integer numbers */
  lua_Number n;    /* float numbers */
  struct {
      int handle;  
      int slice ;
   } u;   /* full userdata with an integer */
};

and add an array in global_State 

typedef struct global_State {
    int userdata_cap;
    struct Udata *userdata_array;
    ....
};

The lua_touserdata could be:

LUA_API void *lua_touserdata (lua_State *L, int idx) {
  StkId o = index2addr(L, idx);
  switch (ttnov(o)) {
    case LUA_TUSERDATA: return G(L)->userdata_array[o->u.handle];
    case LUA_TLIGHTUSERDATA: return pvalue(o);
    default: return NULL;
  }
}

  Also, I don't think I'm following your API there.  I have this structure:

        typedef union sockaddr_all
        {
          struct sockaddr     sa;
          struct sockaddr_in  sin;
          struct sockaddr_in6 sin6;
          struct sockaddr_un  ssun;
        } sockaddr_all__t;

which I'm using for a userdata.  I have a custom __index method associated
with this that gives me the following "fields":

        addr            - text representation of IPv4/IPv6 address
        daddr           - other text representation of IPv4/IPv6 address [5]
        addrbits        - The raw binary bits that make up an IPv4/IPv6 address
        port            - port number
        family          - string, either 'ip', 'ip6' or 'unix'.
        display         - textual representation of address and port.

  How would your system with with that?

[1]     Want to know how false this is?  I can name a mainstream system that
        used C, that could have both 16 *AND* 32 bit pointers *IN THE SAME
        PROGRAM* and where ints were 16 *BITS* long. [2]
 

You can use 1 for sa, 2 for sin, 3 for sin6 , and 4 for ssun. 0 means the sockaddr_all.

In lua, you  can use

local sall = create_sockaddr(...) -- sa is an userdata that store sockaddr_all__t
local sin = sall.sin  
print(sin.port) 

How to implement these by new api lua_getuserslice and lua_setuserslice ?

#define SOCKADDR 0
#define SOCKADDR_SIN 1
#define SOCKADDR_SIN6 2
#define SOCKADDR_UN 3

int sa_root_index(lua_State *L) {
  const char * f = luaL_checkstring(L, 2);
  int slice = 0;
  if (strcmp(f, "sin")== 0)
    slice = 1;
  else  if (strcmp(f, "sin6") == 0)
    slice = 2;
  else if (strcmp(f, "un") == 0)
    slice = 3;
  else
    return luaL_error(L, "never here");
  lua_settop(L,1);
  lua_setuserslice(L,1,slice);
  return 1;
}

int sa_sin_index(lua_State *L) {
  const char * f = luaL_checkstring(L, 2);
  sockaddr_all_t *obj = lua_touserdata(L,1);
  if (strcmp(f, "port") == 0)
    lua_pushinteger(L, obj->sin.sin_port);
  else {
    // todo: supoort sin.sin_family ... etc
  }
  return 1;
}

int sockaddr_index(lua_State *L) {
  // todo: do more type check
  sockaddr_all_t *obj = lua_touserdata(L,1);
  int slice = lua_getuserslice(L,1);
  switch (slice) {
  case SOCKADDR :
    return sa_root_index(L);
  case SOCKADDR_SIN:
    return sa_sin_index(L);
  case SOCKADDR_SIN6:
    return sa_sin6_index(L);
  case SOCKADDR_UN:
    return sa_un_index(L);
  }
  return luaL_error(L, "never here");
}


[3]     One my my pet peeves with LuaJIT is the 2G limit on memory
        allocation, because of tricks played with pointers.

I don't like 2G limit of LuaJIT, too. and I spent months to remove the limit , but failed.
 
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Tim Hill
In reply to this post by 云风 Cloud Wu

On Aug 18, 2015, at 9:19 PM, 云风 Cloud Wu <[hidden email]> wrote:

-1 on this. One of the advantages (for me) of userdata in Lua is that there is *no* language defined way that userdata can be modified. This means a C library can use Lua as the container for opaque data and be assured that whatever it stores there is not meddled with. If you want this sort of functionality on some specific userdata, then it’s pretty easy to expose an appropriate custom API for that purpose.


But we use full userdata rather than light userdata because full userdata is a gc object.

In  many cases, we need reference the different parts of an object, but they are the same one object (to garbage collector). If we create dozens of userdata for them,  we need more expensive works : cache these reference userdata in a weak table, put the root object in their uservalue, etc.



OK, but my comments apply equally to full and light userdata.

I’m not clear what your actual problem is, as you seem to be suggesting a solution without really saying what the underlying problem you are trying to solve is. What I *think* you are saying is that you wish to expose two distinct userdata values that have the same lifetime from a GC standpoint, but I’m not clear why you need to do this.

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
I’m not clear what your actual problem is, as you seem to be suggesting a solution without really saying what the underlying problem you are trying to solve is. What I *think* you are saying is that you wish to expose two distinct userdata values that have the same lifetime from a GC standpoint, but I’m not clear why you need to do this.

I think this is a common case. 

For example, we have a tree based document in C, (maybe an xml document,  a scene object in 3d engine, or a window in GUI system). and then we want to access a part of it.

We want to write like this in lua :

local t = obj.x.y.z  -- obj is an userdata

We can't create separated userdata stores the separated part of obj , because only one object in C. So create a proxy userdata for obj.x , obj.x.y  and obj.x.y.z is a solution, each proxy referenced a part of obj. 

It's not enough, we also need create a weak table in obj to cache "x" "x.y" and "x.y.z" proxy userdata, because we can't returns different proxy userdata for obj.x when we call obj.x more than once.  (obj.x should be unique, when we need use it as a table key later)

If this tree based document is large and deep, we need create too many small full userdata to access each part of it. It's very expensive now , and not necessary to clone the whole tree from C to lua with lua table.

We had try to use lightuserdata to simulate the idea (userdata slice) above, but gc doesn't mark light userdata,  we should be very careful to manage the lifetime. and lightuserdata has no user type, it's very dangerous to work with other library. 
 
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Philipp Janda
In reply to this post by Tim Hill
Am 19.08.2015 um 09:53 schröbte Tim Hill:
>
> I’m not clear what your actual problem is, as you seem to be
> suggesting a solution without really saying what the underlying
> problem you are trying to solve is. What I *think* you are saying is
> that you wish to expose two distinct userdata values that have the
> same lifetime from a GC standpoint, but I’m not clear why you need to
> do this.

Imagine you have a struct type in C that you want to manipulate from Lua
as a userdata:

     typedef struct {
       /* contents not really important here */
     } A;

The C library can give you an object of that type in many different ways:

1.  The "usual" way: You allocate the memory yourself (via
`lua_newuserdata` in this case) and the C library provides you with an
"A_init" and an "A_clear" function that you call at appropriate times.

2.  As a pointer: The constructor C function (let's call it "A_new")
allocates the necessary memory itself and handles you an `A*`. "A_free"
takes that pointer and destroys the object. You could have multiple
"A_new"/"A_free" pairs (e.g. the `FILE*` type in the Lua io library).

3.  As a preexisting C object: It could be a global variable in C or
just an object that outlives the Lua state. This is similar to 2. in
that you put a pointer in your userdata, but you don't have
constructor/destructor functions (and if you store the destructor in the
userdata, this case can be handled like 2. using a NULL pointer as the
destructor).

4.  As part of another userdata:
     typedef struct {
       A a; /* could also be `A* a` */
     } B;
This is similar to 3., but the lifetime of the A pointer depends on the
lifetime of the B userdata: The B userdata must not be collected as long
as the A userdata is still in use. You don't need
constructors/destructors, because the B userdata already handles this.
This can get complicated if you bind a function to Lua that may
indirectly invalidate the A userdata (e.g. if B is a tagged union and
changes type, or via a "close"-style function for B): Then all further
accesses to the A userdata should throw an error.

You could have all four cases *for one type* *in the same library*, and
you want to treat all those A userdata the same (only one metatable).
The current proposal tries to address the fourth case (one object
embedded in another), but ...

*   It doesn't handle cases 2 and 3.
*   It doesn't handle embedding via pointer (the offset approach fails
one level down).
*   It doesn't handle invalidation of the container userdata.
*   Just an offset is not enough (may be the same in unions anyway), you
also need different metatables, which means that you'll have extra
memory allocation for the embedded userdata anyway, because you can't
fit the handle, the offset, and the metatable pointer in one TValue.

>
> —Tim
>

Philipp




Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
*   Just an offset is not enough (may be the same in unions anyway), you
also need different metatables, which means that you'll have extra
memory allocation for the embedded userdata anyway, because you can't
fit the handle, the offset, and the metatable pointer in one TValue.

Offset is only an option,  an enum (32bit int) would be better. We can use the same metatable, and use the enum to dispatch the different cases.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Dirk Laurie-2
2015-08-19 11:52 GMT+02:00 云风 Cloud Wu <[hidden email]>:
(and also numerous previous emails)
[diatribe deleted]

The whole idea sucks, in spades. Userdata means user data.
Lua is not supposed to know how its implementation details.
The metatable is there to provide an interface. Light userdata
is there when no interface beyond the ability to distinguish between
instances is required.

There has been NO support and plenty of -1's. The OP is rapidly
acquiring an image of someone that has fallen in love with a pet
idea and can't accept that nobody else likes it.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
The whole idea sucks, in spades. Userdata means user data.
Lua is not supposed to know how its implementation details.
The metatable is there to provide an interface. Light userdata
is there when no interface beyond the ability to distinguish between
instances is required.

I don't think you really understand the idea. 

Why a lua function need upvalues ? And why there is a type LUA_TPROTO internal ?
We can associate different upvalues with the same proto to build different closures (lua function).

A full userdata is like the value with typed LUA_TPROTO, with an associated integer (like upvalue) would be more flexible.

 
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Dirk Laurie-2
2015-08-19 13:49 GMT+02:00 云风 Cloud Wu <[hidden email]>:
>> The whole idea sucks, in spades. Userdata means user data.
>> Lua is not supposed to know how its implementation details.
>> The metatable is there to provide an interface. Light userdata
>> is there when no interface beyond the ability to distinguish between
>> instances is required.
>
>
> I don't think you really understand the idea.

Quite possibly. In that case, the idea is too complicated for someone
like me to understand.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

William Ahern
In reply to this post by 云风 Cloud Wu
On Wed, Aug 19, 2015 at 07:21:42AM +0000, 云风 Cloud Wu wrote:

> >   AN ADDRESS IS NOT AN INTEGER!
> >
> >   I remember this crap going from 16 bit to 32 bit systems [1] and frankly,
> > I'd rather not have to live through that again going to 64 bit [3][4].
> >
>
> An address in not an integer, but a handle can be.
> For example, in posix , file handle is an integer , and file is an object
> ,too.
>
> We don't need create more than 2^32 userdata in one lua vm, so we can
> create an array in lua global state ( G(L) ),  and use an integer index
> to indicate the userdata object's address.
>
> It would be like this:
>
> union Value {
>   GCObject *gc;    /* collectable objects */
>   void *p;         /* light userdata */
>   int b;           /* booleans */
>   lua_CFunction f; /* light C functions */
>   lua_Integer i;   /* integer numbers */
>   lua_Number n;    /* float numbers */
>   struct {
>       int handle;
>       int slice ;
>    } u;   /* full userdata with an integer */
> };

This seems needlessly complicated.

Firstly, I don't agree that 2^32 handles is enough. Lua tables might be
limited to 2^32-1 entries, but you can have many tables. Rather, if anything
2^32 slices is too much, at least in the scenarios you've presented. (OTOH,
slices of C userdata arrays might prove quite popular, and being limited to
2^32 would _definitely_ be problematic.)

Secondly, the pointer indirection and handle management seems unnecessary. A
slice is an annotation on the underlying value, similar in some respects to
it's type. Have you tried storing the slice data with the value type tag
(TValue), instead? There appear to be over 20 unused bits.

For true array slices you'll need to double or maybe even triple the size of
a TValue, anyhow, so you can store a vector. Short of that, repurposing the
type tag bits would seem more than sufficient. And it would work just as
well on 32-bit platforms as 64-bit platforms--IIUC your proposed solution
limits you to 16-bit tuples on 32-bit machines.

I actually started this e-mail to criticize the whole idea, but then I
started to see the problems and possibilities when playing around with
metatables on lightuserdata out of curiosity. Then it all clicked. :)

Still, as a practical matter I think (as do most others, probably including
yourself) that bindings which expose the low-level details (such as
hierarchy) of C data structures are a poor idea in general. For my money the
better remedy for all the garbage generated by object.x.y.z indexing is to
improve the semantics of the application's Lua and C libraries so there's
fewer discrete loads and stores of C member values in the Lua code,
especially in critical areas.

Even absent the garbage objects, all those indexing operations aren't
helping performance, and this solution doesn't remedy that at all. Using
(abusing?) Lua as a convenient interface to poke and prod C data structures
is a poor use of both Lua and C. And I think that's why, fundamentally, this
notion of slices is problematic--it's most useful for all the wrong reasons.
(Except, perhaps, for array slices, but your proposal doesn't satisfactorily
solve that.)

If there's simply no way to refactor the code there's always LuaJIT FFI.

Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

云风 Cloud Wu
Even absent the garbage objects, all those indexing operations aren't
helping performance, and this solution doesn't remedy that at all. Using
(abusing?) Lua as a convenient interface to poke and prod C data structures
is a poor use of both Lua and C. And I think that's why, fundamentally, this
notion of slices is problematic--it's most useful for all the wrong reasons.
(Except, perhaps, for array slices, but your proposal doesn't satisfactorily
solve that.)

So slice is not a good name, let's change a name. 

I think one integer can use to present the array slices. We don't need use it for begin and end position of slice, just record the index of slices. The slices of array is also store in userdata. I don't think you will create 2^32 slice of one userdata.








 
 
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: userdata slice

Dirk Laurie-2
In reply to this post by 云风 Cloud Wu
2015-08-19 10:33 GMT+02:00 云风 Cloud Wu <[hidden email]>:

> local t = obj.x.y.z  -- obj is an userdata
>
> We can't create separated userdata stores the separated part of obj ,
> because only one object in C. So create a proxy userdata for obj.x , obj.x.y
> and obj.x.y.z is a solution, each proxy referenced a part of obj.
>
> It's not enough, we also need create a weak table in obj to cache "x" "x.y"
> and "x.y.z" proxy userdata, because we can't returns different proxy
> userdata for obj.x when we call obj.x more than once.  (obj.x should be
> unique, when we need use it as a table key later)
>
> If this tree based document is large and deep, we need create too many small
> full userdata to access each part of it. It's very expensive now , and not
> necessary to clone the whole tree from C to lua with lua table.

This particular use case can be handled in a straightforward way
by a method.

local t = obj:retrieve(x,y,z)

12345