luajit: any way to make array initialization zero-based?

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

luajit: any way to make array initialization zero-based?

Chris-41
In LuaJIT there is no penalty for using zero-based indexing, it's even
recommended but the problem is that the Lua tables (non-FFI) are not
by default zero-based.  For example:
foo = { 0, 1, 2, 3 }

You will get a one-based array.  Understandable to be compatible with
regular Lua code but annoying if you have no need for that. It's very
confusing for end users when you're using LuaJIT as a scripting
language and using FFI.  In fact it's confusing for me too, I can't
count how many times I use the wrong index.  Forcing the FFI types to
use one-based indexing doesn't work either when that data needs to go
to a native API (eg. OpenGL vertex data or something;  it can be faked
but you take a serious performance hit).  Off the top of my head I
can't think of any other of the API functions that are one-based by
default but those would need to be patched as well if there are any.

Anybody have a patch to "fix" this?  I suppose something to make the
FFI arrays one-based would work just as well.  I would just like it to
be consistent across the board, whatever scheme. Compile option for
LuaJIT would be awesome, maybe it's already there.

Thanks
CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Coda Highland
On Wed, Jun 13, 2012 at 5:07 PM, Chris <[hidden email]> wrote:

> In LuaJIT there is no penalty for using zero-based indexing, it's even
> recommended but the problem is that the Lua tables (non-FFI) are not
> by default zero-based.  For example:
> foo = { 0, 1, 2, 3 }
>
> You will get a one-based array.  Understandable to be compatible with
> regular Lua code but annoying if you have no need for that. It's very
> confusing for end users when you're using LuaJIT as a scripting
> language and using FFI.  In fact it's confusing for me too, I can't
> count how many times I use the wrong index.  Forcing the FFI types to
> use one-based indexing doesn't work either when that data needs to go
> to a native API (eg. OpenGL vertex data or something;  it can be faked
> but you take a serious performance hit).  Off the top of my head I
> can't think of any other of the API functions that are one-based by
> default but those would need to be patched as well if there are any.
>
> Anybody have a patch to "fix" this?  I suppose something to make the
> FFI arrays one-based would work just as well.  I would just like it to
> be consistent across the board, whatever scheme. Compile option for
> LuaJIT would be awesome, maybe it's already there.
>
> Thanks
> CR
>

foo = { [0]=0, 1, 2, 3 }

Problem solved.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Coda Highland
On Wed, Jun 13, 2012 at 5:15 PM, Coda Highland <[hidden email]> wrote:

> On Wed, Jun 13, 2012 at 5:07 PM, Chris <[hidden email]> wrote:
>> In LuaJIT there is no penalty for using zero-based indexing, it's even
>> recommended but the problem is that the Lua tables (non-FFI) are not
>> by default zero-based.  For example:
>> foo = { 0, 1, 2, 3 }
>>
>> You will get a one-based array.  Understandable to be compatible with
>> regular Lua code but annoying if you have no need for that. It's very
>> confusing for end users when you're using LuaJIT as a scripting
>> language and using FFI.  In fact it's confusing for me too, I can't
>> count how many times I use the wrong index.  Forcing the FFI types to
>> use one-based indexing doesn't work either when that data needs to go
>> to a native API (eg. OpenGL vertex data or something;  it can be faked
>> but you take a serious performance hit).  Off the top of my head I
>> can't think of any other of the API functions that are one-based by
>> default but those would need to be patched as well if there are any.
>>
>> Anybody have a patch to "fix" this?  I suppose something to make the
>> FFI arrays one-based would work just as well.  I would just like it to
>> be consistent across the board, whatever scheme. Compile option for
>> LuaJIT would be awesome, maybe it's already there.
>>
>> Thanks
>> CR
>>
>
> foo = { [0]=0, 1, 2, 3 }
>
> Problem solved.
>
> /s/ Adam

Aside: If you use this scheme, use numeric for instead of ipairs().
Not only will ipairs() miss the 0 index, but numeric for is faster
anyway (which is clearly important to you).

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Chris-41
In reply to this post by Coda Highland
On Wed, Jun 13, 2012 at 6:15 PM, Coda Highland <[hidden email]> wrote:
> foo = { [0]=0, 1, 2, 3 }
>
> Problem solved.

Not really.  Again, for end users (ie. people without a lot of
programming experience) it's confusing and doesn't make any sense.
It's easy to forget even if you do know what you're doing.

Plus unpack and probably other stuff doesn't work (ipairs like you mentioned).
foo = unpack{[0]=0,1,2}
foo == 1 -- oops

It really needs to be integrated into the system, I think I have tried
all the hacks working within the system.

CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Chris-41
In reply to this post by Chris-41
On Wed, Jun 13, 2012 at 6:07 PM, Chris <[hidden email]> wrote:
> I suppose something to make the
> FFI arrays one-based would work just as well.

After looking at the Lua libraries some more I think making Lua
zero-based might be too problematic.  If that's the case then making
LuaJIT's FFI interface one-based would be better.  So the ctype[1]
element would actually be the array[0] at the C level.  I believe a
while ago I looked into trying to make a change like that to LuaJIT
but it wasn't obvious how to do it.

CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Dimiter 'malkia' Stanev
LuaJIT FFI handles "C" types, and in "C" arrays start from 0.

The solution is to let your users be aware of this from the start.
No better solution than this.

On 6/13/2012 3:46 PM, Chris wrote:

> On Wed, Jun 13, 2012 at 6:07 PM, Chris<[hidden email]>  wrote:
>> I suppose something to make the
>> FFI arrays one-based would work just as well.
>
> After looking at the Lua libraries some more I think making Lua
> zero-based might be too problematic.  If that's the case then making
> LuaJIT's FFI interface one-based would be better.  So the ctype[1]
> element would actually be the array[0] at the C level.  I believe a
> while ago I looked into trying to make a change like that to LuaJIT
> but it wasn't obvious how to do it.
>
> CR
>
>

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Geoff Leyland
In reply to this post by Coda Highland
On 14/06/2012, at 10:22 AM, Coda Highland wrote:

> numeric for is faster anyway (which is clearly important to you).

Where does this idea come from?  If speed is important to you (and you're doing so little work in your loop that overheads matter), then you should probably benchmark.  I did a silly little benchmark [1] and with LuaJIT, there's not much of a difference between ipairs and a numeric for.

[1] https://gist.github.com/2927017


Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Vadi
In reply to this post by Chris-41
Why are end-users caring about 0 indexing? The API presented to them
should be counting from 1, as Lua has it, which is natural and not
confusing.

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Chris-41
In reply to this post by Dimiter 'malkia' Stanev
On Wed, Jun 13, 2012 at 6:52 PM, Dimiter 'malkia' Stanev
<[hidden email]> wrote:
> LuaJIT FFI handles "C" types, and in "C" arrays start from 0.
>
> The solution is to let your users be aware of this from the start.
> No better solution than this.

It's much more complicated than that.  Often it's hard to tell what is
FFI and what is regular Lua.  There is not always a clear separation
because usually a combination of FFI and Lua types are used all at
once.  I run in to this a lot when managing multiple FFI types in a
regular Lua table and I often introduce bugs because of it (and I
supposedly know what I am doing; no hope for a regular user).

Unfortunately there doesn't seem to be a clean solution because even
changing LuaJIT to use one-based indexing will have problems with
pointer arithmetic and such.

Honestly I think a top to bottom reworking of Lua to use zero-based
indexes is the only clean solution.  That's not so easy, especially if
doing it in the LuaJIT codebase because every library needs to switch
to the zero-based mechanism (even the stack, etc; everything would
have to change).

CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Chris-41
In reply to this post by Vadi
On Wed, Jun 13, 2012 at 7:06 PM, Vadim Peretokin <[hidden email]> wrote:
> Why are end-users caring about 0 indexing? The API presented to them
> should be counting from 1, as Lua has it, which is natural and not
> confusing.

Because wrapping every single FFI data access call has a significant
performance hit when working with very large datasets.

CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Geoff Leyland
In reply to this post by Chris-41
On 14/06/2012, at 10:07 AM, Chris wrote:

> Forcing the FFI types to
> use one-based indexing doesn't work either when that data needs to go
> to a native API (eg. OpenGL vertex data or something;  it can be faked
> but you take a serious performance hit).

Have you considered allocating your FFI arrays to be one longer than they need to be, and then remembering to offset the pointer to the array by one when you pass it to external C functions.  Not exactly pretty, I know, but perhaps less worse than some alternatives.

(Also, LuaJIT has its own mailing list now: http://luajit.org/list.html)
Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Chris-41
On Wed, Jun 13, 2012 at 7:16 PM, Geoff Leyland
<[hidden email]> wrote:
> Have you considered allocating your FFI arrays to be one longer than they need to be, and then remembering to offset the pointer to the array by one when you pass it to external C functions.  Not exactly pretty, I know, but perhaps less worse than some alternatives.
>
> (Also, LuaJIT has its own mailing list now: http://luajit.org/list.html)

Ah, thanks I didn't know there was a LuaJIT list now.  I will have to
post there in the future.

I have tried the offset by one method but again run into issues with
the confusing nature of it.  If an end user has to make a native call
(eg. OpenGL) they have to remember to offset the data which is
unexpected.  A solution is to wrap all external API calls, but again:
performance hit when making lots of calls.  That's really the main
issue with all of this, it's just (inconsistent and confusing) or
(slow[er]).

CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Geoff Leyland
On 14/06/2012, at 11:24 AM, Chris wrote:

> I have tried the offset by one method but again run into issues with
> the confusing nature of it.  If an end user has to make a native call
> (eg. OpenGL) they have to remember to offset the data which is
> unexpected.  A solution is to wrap all external API calls, but again:
> performance hit when making lots of calls.

Have you checked that?  I would hope that LJ would effectively inline the wrapper.  Ideally there's only the (smaller) performance hit of the add.  (I'm not saying it's the best solution in the world, just curious)


Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Chris-41
On Wed, Jun 13, 2012 at 7:29 PM, Geoff Leyland
<[hidden email]> wrote:
> On 14/06/2012, at 11:24 AM, Chris wrote:
> Have you checked that?  I would hope that LJ would effectively inline the wrapper.  Ideally there's only the (smaller) performance hit of the add.  (I'm not saying it's the best solution in the world, just curious)


Yes, it's not horrible.  It's just "something" which I don't like
because I'm already bumping up against performance issues because of
the shear amount of data that is being pushed around.  At this point
almost all my overhead is in the management of all this data.

I think what I'm going to do is just wrap every FFI call for now and
hopefully some day there will be a solution.  I just tested using the
ffi.metatype stuff as a wrapper and it seems pretty fast.  A lot of
unnecessary types, allocations, and wrapping of data but possibly
acceptable for now depending on how many of these wrappers need to be
instantiated.

CR

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Dimiter 'malkia' Stanev
In reply to this post by Chris-41
How about this workaround (ugly, but still could work):

local ffi = require( "ffi" )

ffi.cdef[[
    typedef struct vec3 {
      float x, y, z;
    } vec3;
]]

local a = ffi.new( "vec3[?]", 1024 + 1 )
local b = ffi.cast( "vec3*", ffi.cast("size_t", a) - ffi.sizeof("vec3"))

print(a)
print(b)

for k = 1, 30 do
    b[k].x = k
end

for k = 0, 29 do
    print( a[k].x )
end

The trick here is that 'a' has to be kept, so that 'b' would work (not
sure, Mike would know better)

       
On 6/13/2012 4:07 PM, Chris wrote:

> On Wed, Jun 13, 2012 at 6:52 PM, Dimiter 'malkia' Stanev
> <[hidden email]>  wrote:
>> LuaJIT FFI handles "C" types, and in "C" arrays start from 0.
>>
>> The solution is to let your users be aware of this from the start.
>> No better solution than this.
>
> It's much more complicated than that.  Often it's hard to tell what is
> FFI and what is regular Lua.  There is not always a clear separation
> because usually a combination of FFI and Lua types are used all at
> once.  I run in to this a lot when managing multiple FFI types in a
> regular Lua table and I often introduce bugs because of it (and I
> supposedly know what I am doing; no hope for a regular user).
>
> Unfortunately there doesn't seem to be a clean solution because even
> changing LuaJIT to use one-based indexing will have problems with
> pointer arithmetic and such.
>
> Honestly I think a top to bottom reworking of Lua to use zero-based
> indexes is the only clean solution.  That's not so easy, especially if
> doing it in the LuaJIT codebase because every library needs to switch
> to the zero-based mechanism (even the stack, etc; everything would
> have to change).
>
> CR
>
>

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Daurnimator
On 14 June 2012 12:47, Dimiter 'malkia' Stanev <[hidden email]> wrote:

> How about this workaround (ugly, but still could work):
>
> local ffi = require( "ffi" )
>
> ffi.cdef[[
>   typedef struct vec3 {
>     float x, y, z;
>   } vec3;
> ]]
>
> local a = ffi.new( "vec3[?]", 1024 + 1 )
> local b = ffi.cast( "vec3*", ffi.cast("size_t", a) - ffi.sizeof("vec3"))
>
> print(a)
> print(b)
>
> for k = 1, 30 do
>   b[k].x = k
> end
>
> for k = 0, 29 do
>   print( a[k].x )
> end
>
> The trick here is that 'a' has to be kept, so that 'b' would work (not sure,
> Mike would know better)
>

Why not just use pointer arithmetic?
b = a-1

Same rule for keeping a around applies.

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Dimiter 'malkia' Stanev
Uh. Yes :)

I should've tried this first, but thought it might not work.

Well, thanks for that one!

On 6/13/2012 8:36 PM, Daurnimator wrote:

> On 14 June 2012 12:47, Dimiter 'malkia' Stanev<[hidden email]>  wrote:
>> How about this workaround (ugly, but still could work):
>>
>> local ffi = require( "ffi" )
>>
>> ffi.cdef[[
>>    typedef struct vec3 {
>>      float x, y, z;
>>    } vec3;
>> ]]
>>
>> local a = ffi.new( "vec3[?]", 1024 + 1 )
>> local b = ffi.cast( "vec3*", ffi.cast("size_t", a) - ffi.sizeof("vec3"))
>>
>> print(a)
>> print(b)
>>
>> for k = 1, 30 do
>>    b[k].x = k
>> end
>>
>> for k = 0, 29 do
>>    print( a[k].x )
>> end
>>
>> The trick here is that 'a' has to be kept, so that 'b' would work (not sure,
>> Mike would know better)
>>
>
> Why not just use pointer arithmetic?
> b = a-1
>
> Same rule for keeping a around applies.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: luajit: any way to make array initialization zero-based?

Tony Finch
In reply to this post by Chris-41
Chris <[hidden email]> wrote:

>
> It's much more complicated than that.  Often it's hard to tell what is
> FFI and what is regular Lua.  There is not always a clear separation
> because usually a combination of FFI and Lua types are used all at
> once.  I run in to this a lot when managing multiple FFI types in a
> regular Lua table and I often introduce bugs because of it (and I
> supposedly know what I am doing; no hope for a regular user).
>
> Unfortunately there doesn't seem to be a clean solution because even
> changing LuaJIT to use one-based indexing will have problems with
> pointer arithmetic and such.

There are four points where Lua has one-based table indexes:

(1) In the implementation, which is invisible apart from the performance
effects. LuaJIT uses a zero base for the array part of its tables.

(2) In the syntax for array-like initialization of tables. You can use the
following pattern to construct zero-based arrays:

  t = { [0] = a, b, c }

(3) The length operator.

(4) In the library. Which is actually lots of points :-)

So if you are willing to wrap or reimplement the bits of the library that
you need, it should be fairly easy to adapt your coding style for
zero-based indexing.

Tony.
--
f.anthony.n.finch  <[hidden email]>  http://dotat.at/
Rockall, Malin: East or northeast 5 or 6, increasing 7 or gale 8, perhaps
severe gale 9 later. Moderate or rough, occasionally very rough later.
Occasional rain. Moderate or good.