Fastest way to transfer data from C to Lua ?

classic Classic list List threaded Threaded
36 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Tim Hill
Agreed, the real problem is without __gc on light userdata you can never be sure when Lua has released the last pointer, so it is of limited use unless the pointer is to static memory or heap memory that will not be released until after the Lua state is closed. I'm with you, I tend to wrap C pointers in full userdata so that I can track GC correctly.

--Tim

On Jun 9, 2013, at 2:01 AM, David Heiko Kolf <[hidden email]> wrote:

In my opinion the term "light userdata" is a bit misleading, as I see
only a very limited value in using it for storing actual data.  You
cannot assign different metatables to each light userdata.  That means
it is later very hard to find out what was actually stored in it.  And I
don't see any way of making sure that Lua has no stray pointers left --
except by closing the entire Lua state.  So even when I need to do
memory handling in C I would store a pointer in a full userdata.

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Dirk Laurie-2
In reply to this post by David Heiko Kolf-2
2013/6/9 David Heiko Kolf <[hidden email]>:

> Instead of "light userdata" maybe "typeless user pointer" would be a
> clearer term.

Or "pointless user type".

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Kevin Martin
In reply to this post by Tim Hill

On 9 Jun 2013, at 12:05, Tim Hill wrote:

> so it is of limited use unless the pointer is to static memory or heap memory that will not be released until after the Lua state is closed. I'm with you.

This isn't true if the light userdata are stored in a table which has a gc metamethod to call the appropriate free function.

I hardly ever use full userdata as an object, but instead use a table and put light userdata in it. I find it much more flexible than full userdata.

Kevin
Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Philipp Janda
Am 09.06.2013 17:26 schröbte Kevin Martin:
>
> On 9 Jun 2013, at 12:05, Tim Hill wrote:
>
>> so it is of limited use unless the pointer is to static memory or heap memory that will not be released until after the Lua state is closed. I'm with you.
>
> This isn't true if the light userdata are stored in a table which has a gc metamethod to call the appropriate free function.
>
> I hardly ever use full userdata as an object, but instead use a table and put light userdata in it. I find it much more flexible than full userdata.

But then you need to maintain a set of valid userdata handles for each
class to make it safe, or else:

     do
       local o1, o2 = class1.new(), class2.new()
       o1.udata, o2.udata = o2.udata, o1.udata
     end
     collectgarbage()
     collectgarbage()
     -- boom

... and __gc for tables only works in Lua 5.2 and up.

>
> Kevin
>

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Kevin Martin

On 9 Jun 2013, at 18:42, Philipp Janda wrote:

> But then you need to maintain a set of valid userdata handles for each class to make it safe, or else:
>
>    do
>      local o1, o2 = class1.new(), class2.new()
>      o1.udata, o2.udata = o2.udata, o1.udata
>    end
>    collectgarbage()
>    collectgarbage()
>    -- boom

I don't understand the point of your example. udata is a private member of the object, not meant to be altered in Lua code (I tend to prefix with _). If you go and mess with it, then that's your fault, and so is the resultant crash.

With full userdata, I believe you have the same problem if someone does

do
        local o1, o2 = class1.new(), class2.new()
        local o1_mt, o2_mt = getmetatable(o1), getmetatable(o2)
        o1_mt.__gc, o2_mt = o2_mt.__gc, o1_mt.__gc
end

Now, I think you can use the __metatable metamethod to stop this, but then you lose the ability to modify the metatable from Lua which loses a lot of flexibility.

> ... and __gc for tables only works in Lua 5.2 and up.

Fair enough, I didn't know that, but I only use Lua 5.2

Kevin
Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Wesley Smith
In reply to this post by Dirk Laurie-2
>> Instead of "light userdata" maybe "typeless user pointer" would be a
>> clearer term.
>
> Or "pointless user type".


It's quite useful in LuaJIT where you can cast pointers/lightuserdata
to cdata types.

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Philipp Janda
In reply to this post by Kevin Martin
Am 09.06.2013 20:35 schröbte Kevin Martin:

>
> On 9 Jun 2013, at 18:42, Philipp Janda wrote:
>
>> But then you need to maintain a set of valid userdata handles for each class to make it safe, or else:
>>
>>     do
>>       local o1, o2 = class1.new(), class2.new()
>>       o1.udata, o2.udata = o2.udata, o1.udata
>>     end
>>     collectgarbage()
>>     collectgarbage()
>>     -- boom
>
> I don't understand the point of your example. udata is a private member of the object, not meant to be altered in Lua code (I tend to prefix with _). If you go and mess with it, then that's your fault, and so is the resultant crash.
>

I can't find a quote right now, but once there was this guarantee that
you cannot crash a program from within Lua code. I find this guarantee
very useful, so I try to make sure that my C modules follow this
practice. (There are some unsafe modules, where this isn't possible:
e.g. alien).
Btw., a crash is only one possibility, stack/data/heap corruption would
be even worse ...

> With full userdata, I believe you have the same problem if someone does
>
> do
> local o1, o2 = class1.new(), class2.new()
> local o1_mt, o2_mt = getmetatable(o1), getmetatable(o2)
> o1_mt.__gc, o2_mt = o2_mt.__gc, o1_mt.__gc
> end

Only if you don't use luaL_checkudata in your __gc metamethods. In that
case you only get a Lua error and some resource leaks, which leaves us
with ...

>
> Now, I think you can use the __metatable metamethod to stop this, but then you lose the ability to modify the metatable from Lua which loses a lot of flexibility.

Why would you want this flexibility in Lua code? You don't have direct
access to the object from Lua anyway, so you need accessor methods for
everything which breaks encapsulation. If you just want to add methods,
you can put your index table into the __metatable field.

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Tim Hill
In reply to this post by Dirk Laurie-2
Is that a type to be used by pointless users? :)

--Tim

On Jun 9, 2013, at 6:55 AM, Dirk Laurie <[hidden email]> wrote:

Or "pointless user type".

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Tim Hill
In reply to this post by Kevin Martin
Not true, there is nothing to prevent you taking a copy of the light user data from the table. The point is that neither Lua nor C can know when the item is freed at the language level .. your application might have business rules that controls the lifecycle, which is fine (just like you have to manage pointers by hand in C).

--Tim


On Jun 9, 2013, at 8:26 AM, Kevin Martin <[hidden email]> wrote:

>
> On 9 Jun 2013, at 12:05, Tim Hill wrote:
>
>> so it is of limited use unless the pointer is to static memory or heap memory that will not be released until after the Lua state is closed. I'm with you.
>
> This isn't true if the light userdata are stored in a table which has a gc metamethod to call the appropriate free function.
>
> I hardly ever use full userdata as an object, but instead use a table and put light userdata in it. I find it much more flexible than full userdata.
>
> Kevin


Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Coda Highland
In reply to this post by Wesley Smith
On Sun, Jun 9, 2013 at 11:58 AM, Wesley Smith <[hidden email]> wrote:
>>> Instead of "light userdata" maybe "typeless user pointer" would be a
>>> clearer term.
>>
>> Or "pointless user type".
>
>
> It's quite useful in LuaJIT where you can cast pointers/lightuserdata
> to cdata types.
>

Even without LuaJIT I use lightuserdata fairly frequently as an opaque
handle type when Lua is accessing -- but not OWNING -- a C-side data
structure. The C-side code is responsible for informing the Lua-side
code when the object has been destroyed.


/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Dirk Laurie-2
2013/6/10 Coda Highland <[hidden email]>:

> Even without LuaJIT I use lightuserdata fairly frequently as an opaque
> handle type when Lua is accessing -- but not OWNING -- a C-side data
> structure. The C-side code is responsible for informing the Lua-side
> code when the object has been destroyed.

Actually I have used it. Once. But normally I can't do without
a metatable. Life without __tostring is just unbearable.

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Luiz Henrique de Figueiredo
In reply to this post by Philipp Janda
> I can't find a quote right now, but once there was this guarantee that
> you cannot crash a program from within Lua code.

This is mandatory for an embedded language.

If you can crash the host from a Lua script, it's either a bug in Lua
or more proably in a C library exported to Lua. So, C libraries should
ensure the safety of the host, as far as they are concerned.

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

David Demelier
In reply to this post by Tim Hill
2013/6/9 Tim Hill <[hidden email]>:

> Agreed, the real problem is without __gc on light userdata you can never be
> sure when Lua has released the last pointer, so it is of limited use unless
> the pointer is to static memory or heap memory that will not be released
> until after the Lua state is closed. I'm with you, I tend to wrap C pointers
> in full userdata so that I can track GC correctly.
>
> --Tim
>
> On Jun 9, 2013, at 2:01 AM, David Heiko Kolf <[hidden email]> wrote:
>
> In my opinion the term "light userdata" is a bit misleading, as I see
> only a very limited value in using it for storing actual data.  You
> cannot assign different metatables to each light userdata.  That means
> it is later very hard to find out what was actually stored in it.  And I
> don't see any way of making sure that Lua has no stray pointers left --
> except by closing the entire Lua state.  So even when I need to do
> memory handling in C I would store a pointer in a full userdata.
>

For me, lightuserdata is best to use with closure, I usually use them
to store a temporary buffer which will be only used by the closure as
an upvalue, so the script will never know the usage of that userdata
and will never rely on it at anytime. I think nobody should give
lightuserdata to a script just like a variable.


--
Demelier David

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Luiz Henrique de Figueiredo
> I think nobody should give lightuserdata to a script just like a variable.

I'm not sure what you mean exactly here, but giving userdata, light or
not, to a Lua script is totally harmless because userdata has no instrinsic
operations, except for equality test. The real issue is sending userdata
to the host: C code must ensure the validity of the corresponding pointer.

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

Roberto Ierusalimschy
> > I think nobody should give lightuserdata to a script just like a variable.
>
> I'm not sure what you mean exactly here, but giving userdata, light or
> not, to a Lua script is totally harmless because userdata has no instrinsic
> operations, except for equality test. The real issue is sending userdata
> to the host: C code must ensure the validity of the corresponding pointer.

I guess what he means is that it is not trivial to ensure the validity
of a light userdata. So, more often than not, when a lightuserdata
goes to a script, it ends up being used by its corresponding C library
without a proper validation.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: light userdata (Was: Fastest way to transfer data from C to Lua ?)

David Demelier
2013/6/20 Roberto Ierusalimschy <[hidden email]>:

>> > I think nobody should give lightuserdata to a script just like a variable.
>>
>> I'm not sure what you mean exactly here, but giving userdata, light or
>> not, to a Lua script is totally harmless because userdata has no instrinsic
>> operations, except for equality test. The real issue is sending userdata
>> to the host: C code must ensure the validity of the corresponding pointer.
>
> I guess what he means is that it is not trivial to ensure the validity
> of a light userdata. So, more often than not, when a lightuserdata
> goes to a script, it ends up being used by its corresponding C library
> without a proper validation.
>
> -- Roberto
>

Yes, just like Roberto said, I'm more talking about an "ownership" of
pointer. If a C function pass a lightuserdata to Lua and this script
store it, the pointer may be invalidated by the C host at any time.
That's why I advise to use real userdata when we don't know when the
data will be deleted.

Sorry for not being concise, I'm not native english speaker..

Regards,

--
Demelier David

12