String parameters when calling C from Lua

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

String parameters when calling C from Lua

Daniel Collins
When calling a C function from lua with a string argument, is that
string copied onto the stack that the C function gets? Or does the stack
contain a pointer to the original string? I am wondering about the most
efficient way to implement localisable string tables. I can see two ways
of doing this but I want to use the most efficient implementation. The
two methods can be boiled down to:

Method One:
-- StringTable.lua
stringtable =
{
    mainmenu = "main",
    quit = "quit",
    yes = "yes"
}

-- SomeMenu.lua
require "StringTable"
callSomeCFunction(stringtable.yes)


Method Two:
-- StringTable.lua
LoadStringData("stringdata.bin")    -- loads the actual strings into a C
array

stringtable =
{
    mainmenu = 1,
    quit = 2,
    yes = 3
}

-- SomeMenu.lua
require "StringTable"
callSomeCFunction(stringtable.yes)    -- integer argument indexes the C
array loaded earlier

- DC

Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Luiz Henrique de Figueiredo
> When calling a C function from lua with a string argument, is that
> string copied onto the stack that the C function gets? Or does the stack
> contain a pointer to the original string?

Neither. The stack contains a Lua value, which can be converted to
a C string (const char*) with lua_tostring. The pointer returned by
lua_tostring points to the internal copy of the string inside Lua.
--lhf
Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Daniel Collins
In reply to this post by Daniel Collins
> > When calling a C function from lua with a string argument, is that
> > string copied onto the stack that the C function gets? Or does the
> > stack contain a pointer to the original string?
>
> Neither. The stack contains a Lua value, which can be
> converted to a C string (const char*) with lua_tostring. The
> pointer returned by lua_tostring points to the internal copy
> of the string inside Lua. --lhf

Let me ask a different question:

Out of the two implementations I sketched out, would there be any
significant performance difference between them? Basically I am
interested in the relative efficiency of passing strings vs numbers from
lua to C.

>From what you say I suspect that there is no real difference. Which
means I should go with the solution that will be easiest to maintain.

- DC

Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Luiz Henrique de Figueiredo
> Out of the two implementations I sketched out, would there be any
> significant performance difference between them? Basically I am
> interested in the relative efficiency of passing strings vs numbers from
> lua to C.

>From Lua to C, they have the same perfomance. From C to Lua, strings
have to hashed, even if they already exist in Lua. If you already have
a string as a Lua value and you want to pass it back to Lua, then you
can use lua_pushvalue and avoid the hash.

Back to your question, I think the main point is what you want to do
with strings in C. If you want to make decisions based on them and
have to strcmp them, then it'd be easier for C to do a switch on
integers. The Lua standard libraries convert strings to numbers for that;
see luaL_checkoption.
Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Mike Pall-5-2
In reply to this post by Daniel Collins
Hi,

Daniel Collins wrote:
> When calling a C function from lua with a string argument, is that
> string copied onto the stack that the C function gets? Or does the stack
> contain a pointer to the original string?

No. Yes.

The Lua stack is just an array of TValue's. In your case this
boils down to an int (for the type) and a pointer (to the string
plus a header) for each stack slot.

Lua strings are immutable, shared and pre-hashed. Constant
strings (i.e. t.key or t["key"]) are hashed once when the script
is loaded.

A string lookup is then just a couple instructions in the fast
path. See luaH_getstr() and realize that the hashstr() macro is
just a bitmask operation and an array lookup. Hashing is done
once when a Lua string is created (see luaS_newlstr()), not when
it is used as a key.

A corollary of this is that you should avoid converting C strings
to Lua strings over and over again. lua_push[l]string() has to
rehash it and lookup it up in the shared string table. Try to
keep repeatedly used Lua strings in upvalues or arrays. Or,
better yet, in Lua code.

> I am wondering about the most
> efficient way to implement localisable string tables. I can see two ways
> of doing this but I want to use the most efficient implementation. The
> two methods can be boiled down to:
>
> Method One:
> -- StringTable.lua
> stringtable =
> {
>     mainmenu = "main",
>     quit = "quit",
>     yes = "yes"
> }
>
> -- SomeMenu.lua
> require "StringTable"
> callSomeCFunction(stringtable.yes)

One hash table lookup.

> Method Two:
> -- StringTable.lua
> LoadStringData("stringdata.bin")    -- loads the actual strings into a C
> array
>
> stringtable =
> {
>     mainmenu = 1,
>     quit = 2,
>     yes = 3
> }
>
> -- SomeMenu.lua
> require "StringTable"
> callSomeCFunction(stringtable.yes)    -- integer argument indexes the C
> array loaded earlier

One hash table lookup and on the C side a double->int conversion
and one array lookup. Slower.

General rule: Lua's hash tables are very fast. Particularly when
used with constant strings. Use them if you can.

Trying to avoid a hash lookup does not pay off. And a workaround
may be slower due to other issues (as seen above).

Bye,
     Mike
Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Daniel Collins
In reply to this post by Daniel Collins

Actually thinking about this more (with much help from Mike and Luiz) I
realise that my particular needs rule out keeping the string in lua.

The strings are immutable but need to persist in C longer than the
duration of the API function called from Lua. So the string parameter
passed in my method one would need to be copied into another buffer.

Whereas with method two, keeping the arrays in a C array outside of lua
then passing integers will avoid the string allocation and copy in my C
code. And the double to int conversion wont even occur since lua_number
is already set to int in my project.

Thanks for the help!

- DC

Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Mike Pall-5-2
In reply to this post by Luiz Henrique de Figueiredo
Hi,

Luiz Henrique de Figueiredo wrote:
> Back to your question, I think the main point is what you want to do
> with strings in C. If you want to make decisions based on them and
> have to strcmp them, then it'd be easier for C to do a switch on
> integers. The Lua standard libraries convert strings to numbers for that;
> see luaL_checkoption.

Ahemm. luaL_checkoption does a linear loop with lots of strcmps.
This is ok only for very few elements or in non-performance
critical paths.

For more elements use a private table in an upvalue of a C
function. Add the string keys plus target values. Either numbers
or lightuserdata may be suitable.

Oh, and another hint: don't attempt to outsmart the method lookup
logic (a table of functions indexed by strings) with the above
plus a switch statement. This is unlikely to yield better
performance.

Bye,
     Mike
Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Mike Pall-5-2
In reply to this post by Daniel Collins
Hi,

Daniel Collins wrote:
> The strings are immutable but need to persist in C longer than the
> duration of the API function called from Lua. So the string parameter
> passed in my method one would need to be copied into another buffer.

Ok, a returned string pointer is guaranteed to be valid as long
as the corresponding Lua string exists in the Lua stack (see the
description of lua_tolstring() in the Lua 5.1 manual).

But the current GC algorithm (as of Lua 5.0/5.1) does not move
objects. So right now you can also rely on the fact that the
string pointer is valid as long as there is _any_ reference to it
(i.e. from the table you use to store them).

Yes, the latter means relying on implementation internals. But
nobody forces you to upgrade to a newer Lua version. The C API
usually changes over major versions, anyway.

Bye,
     Mike
Reply | Threaded
Open this post in threaded view
|

RE: String parameters when calling C from Lua

Daniel Collins
In reply to this post by Daniel Collins
> But the current GC algorithm (as of Lua 5.0/5.1) does not
> move objects. So right now you can also rely on the fact that
> the string pointer is valid as long as there is _any_
> reference to it (i.e. from the table you use to store them).
>
> Yes, the latter means relying on implementation internals.
> But nobody forces you to upgrade to a newer Lua version. The
> C API usually changes over major versions, anyway.

Interesting. This project will be lua5.1 for life, but relying on an
implementation detail like this just seems naughty to me.

- DC

Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Asko Kauppi
In reply to this post by Mike Pall-5-2

Lua authors!!!:
  :)

Would it be possible to have a "lua_pushstring_fixed()"
API function, to cover this issue.  That is, any pushings
of a string that _relies_ on the pointer not to change
after exit back to Lua world, would use such function to
make the point. Normal string pushing would be seen as
temporary.

With current (5.1) version, the call could be simply a
macro doing the normal push.

-asko


On Fri, 19 May 2006 13:21:29 +0200
  Mike Pall <[hidden email]> wrote:

> Hi,
>
> Daniel Collins wrote:
>> The strings are immutable but need to persist in C
>>longer than the
>> duration of the API function called from Lua. So the
>>string parameter
>> passed in my method one would need to be copied into
>>another buffer.
>
> Ok, a returned string pointer is guaranteed to be valid
>as long
> as the corresponding Lua string exists in the Lua stack
>(see the
> description of lua_tolstring() in the Lua 5.1 manual).
>
> But the current GC algorithm (as of Lua 5.0/5.1) does
>not move
> objects. So right now you can also rely on the fact that
>the
> string pointer is valid as long as there is _any_
>reference to it
> (i.e. from the table you use to store them).
>
> Yes, the latter means relying on implementation
>internals. But
> nobody forces you to upgrade to a newer Lua version. The
>C API
> usually changes over major versions, anyway.
>
> Bye,
>     Mike

Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Javier Guerra Giraldez
On Friday 19 May 2006 7:41 am, [hidden email] wrote:
> Would it be possible to have a "lua_pushstring_fixed()"
> API function, to cover this issue.  That is, any pushings

why?  getting again the string isn't so slow, it's the hashing you want to
avoid, that's done at pushstring() time.  getstring() gives you a pointer, no
copying, hashing, or any linear-time operation involved.

--
Javier

attachment0 (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: String parameters when calling C from Lua

Luiz Henrique de Figueiredo
In reply to this post by Asko Kauppi
> Would it be possible to have a "lua_pushstring_fixed()"
> API function, to cover this issue.

If the string is already in Lua, then the best way is to use a reference.
--lhf