Wrapping a Lua table around a C array

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Wrapping a Lua table around a C array

Dimitris Papavasiliou
Hi all,

I need to specify arrays through Lua which later will be converted to normal C arrays.  The trouble is that these arrays may potentially be pretty large and some of them may only need to hold small (byte or short sized) integers.  Therefore, given that numbers in Lua are doubles which I don't want to change, using normal Lua tables to do the job results in a 8x overhead in terms of memory use.  This is not that bad since the values are converted back to unsigned chars where needed when reading the data from C but these arrays are contained inside Lua scripts and these scripts therefore end up being eight times larger than they need to be.

The way I've handled this up to now is to create Lua functions which take an array as binary data (a Lua string) and convert it into a table on the fly.  So instead of containing something like this:

foo = {1, 2, 3, 4, 5, ...}

the script now contains:

foo = frombytes "\001\002\003\004..."

This approach has the merit that it is completely transparent.  You can still specify a normal Lua table (you actually do that, but frombytes() creates the table for you) but the script file isn't larger than needed.  The problem is that you now have the overhead of converting binary data into a Lua table and back to binary data later inside the application.  This is necessary because it allows you to process the data inside Lua before passing them on to the application but when that is not necessary you're still left with a (potentially big) slowdown when loading the data to perform the needless conversions.

So I thought I would only keep the C arrays and make frombytes() and other similar functions create empty tables or userdata with metamethods to allow you to get and set individual array elements.  When read from the application the array can then be used without conversion where possible.  These tables or userdata need to behave like normal tables in order to keep the transparency of the previous approach.  Although getting and setting is easy enough, I'm having trouble with the table length.  The __len metamethod doesn't work at all for tables so that #foo would always return 0 and although it works for the # operator with userdata it doesn't work when calling lua_objlen from inside C.

So the question is: apart from the obvious solutions of implementing my own lua_objlen, hacking the internals of Lua etc. can you think of a slick way to create a table or userdata object that will contain a reference to a string with binary data and provide set, get and length access to it so that it can pose as a normal Lua table?

Thanks,
Dimitris

PS: I know that most of these issues are addressed in Lua 5.2 but since we have no idea when that will be released I'd rather find a more immediate solution.
Reply | Threaded
Open this post in threaded view
|

Re: Wrapping a Lua table around a C array

Jim Pryor-2-3
On Sat, Dec 12, 2009 at 07:08:27PM +0200, Dimitris Papavasiliou wrote:
>    So the question is: apart from the obvious solutions of implementing my
>    own lua_objlen, hacking the internals of Lua etc. can you think of a slick
>    way to create a table or userdata object that will contain a reference to
>    a string with binary data and provide set, get and length access to it so
>    that it can pose as a normal Lua table?

Here's a sketch. You may want to do some of this in C.

local function byte_array_get(self, k)
    ...
end

local function byte_array_set(self, k, val)
    ...
end

local function byte_array_len(self)
    ...
end

local byte_array_prototype=newproxy(true)
local mt = getmetatable(byte_array_prototype)
mt.__index = byte_array_get
mt.__newindex = byte_array_set
mt.__len = byte_array_len

function byte_array()
    return newproxy(byte_array_prototype)
end

local b = byte_array()
b[1]=2
print(b[1]) -- 2
print(#b) -- 1

--
Jim Pryor
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Wrapping a Lua table around a C array

Dimitris Papavasiliou
Hello Jim,

thanks for the reply but that's the approach I've tried so far although I implemented it in C.  The problem with this approach is that this proxy does not completely behave like a table.  I need to find the length of the array from within C when reading the data so I use lua_objlen for this.  (Remember that I need to be able to accept normal tables as well as wrapped C arrays).  Unfortunately objlen doesn't honor the __len metamethod so the result won't be correct.

Dimitris.

On Sun, Dec 13, 2009 at 12:35 AM, Jim Pryor <[hidden email]> wrote:
On Sat, Dec 12, 2009 at 07:08:27PM +0200, Dimitris Papavasiliou wrote:
>    So the question is: apart from the obvious solutions of implementing my
>    own lua_objlen, hacking the internals of Lua etc. can you think of a slick
>    way to create a table or userdata object that will contain a reference to
>    a string with binary data and provide set, get and length access to it so
>    that it can pose as a normal Lua table?

Here's a sketch. You may want to do some of this in C.

local function byte_array_get(self, k)
   ...
end

local function byte_array_set(self, k, val)
   ...
end

local function byte_array_len(self)
   ...
end

local byte_array_prototype=newproxy(true)
local mt = getmetatable(byte_array_prototype)
mt.__index = byte_array_get
mt.__newindex = byte_array_set
mt.__len = byte_array_len

function byte_array()
   return newproxy(byte_array_prototype)
end

local b = byte_array()
b[1]=2
print(b[1]) -- 2
print(#b) -- 1

--
Jim Pryor
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Wrapping a Lua table around a C array

Linker
In my view, crossing lua/c bound is a expensive thing.
May you have to implement whole module in c, and wrap some simple interfaces to lua .
Take care of GC, do not make a lot of garbage.

On Mon, Dec 14, 2009 at 15:43, Dimitris Papavasiliou <[hidden email]> wrote:
Hello Jim,

thanks for the reply but that's the approach I've tried so far although I implemented it in C.  The problem with this approach is that this proxy does not completely behave like a table.  I need to find the length of the array from within C when reading the data so I use lua_objlen for this.  (Remember that I need to be able to accept normal tables as well as wrapped C arrays).  Unfortunately objlen doesn't honor the __len metamethod so the result won't be correct.

Dimitris.


On Sun, Dec 13, 2009 at 12:35 AM, Jim Pryor <[hidden email]> wrote:
On Sat, Dec 12, 2009 at 07:08:27PM +0200, Dimitris Papavasiliou wrote:
>    So the question is: apart from the obvious solutions of implementing my
>    own lua_objlen, hacking the internals of Lua etc. can you think of a slick
>    way to create a table or userdata object that will contain a reference to
>    a string with binary data and provide set, get and length access to it so
>    that it can pose as a normal Lua table?

Here's a sketch. You may want to do some of this in C.

local function byte_array_get(self, k)
   ...
end

local function byte_array_set(self, k, val)
   ...
end

local function byte_array_len(self)
   ...
end

local byte_array_prototype=newproxy(true)
local mt = getmetatable(byte_array_prototype)
mt.__index = byte_array_get
mt.__newindex = byte_array_set
mt.__len = byte_array_len

function byte_array()
   return newproxy(byte_array_prototype)
end

local b = byte_array()
b[1]=2
print(b[1]) -- 2
print(#b) -- 1

--
Jim Pryor
[hidden email]




--
Regards,
Linker Lin
[hidden email]