Lua C Functions

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua C Functions

ThePhD
Hello!

     I have a question regarding the implementation details of the returns for C functions into Lua. As of right now, I use the C API to return one or more values back to Lua, but only after popping all of the arguments given to Lua just before I do this. This seems to be what is necessary, given the examples I have seen in PIL and other places.

     My usage gets a bit different when I have optional arguments. I pop a fixed number of arguments from the stack for certain functions after performing the call, but some parameters have provided-defaults if I can't get the item off the stack.

     Lua (5.1, 5.2, and 5.3), when I pop more than what's on the stack from the stack, seem to protect against me ruining the stack by "overpopping". LuaJIT, however, crashes because the stack is not protected from these over-pops.

     Some solutions I imagine to my problem would be to just always pop "int arg_count = lua_gettop(L)" values from the stack. But, I'm trying to get near-zero-overhead, and that's another (even if dirt cheap) API call I have to make.

    My other "solution" is maybe being cautious and only popping arguments I know are "required", but I do not know if the implementation is required to clean up the stack after a C function call and all arguments are returned.

     Are implementations required to clean up the stack if there's more than "return-count" things left on the stack before a C function returns?

Sincerely,
ThePhD
Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

Alysson Cunha

You dont need to pop the arguments from stack to rerurn values....

If the function rerurns 3, the top 3 elements of the stack will be the returning values of the function.  You dont need to pop the arguments

Em 09/07/2016 12:37, "ThePhD" <[hidden email]> escreveu:
Hello!

     I have a question regarding the implementation details of the returns for C functions into Lua. As of right now, I use the C API to return one or more values back to Lua, but only after popping all of the arguments given to Lua just before I do this. This seems to be what is necessary, given the examples I have seen in PIL and other places.

     My usage gets a bit different when I have optional arguments. I pop a fixed number of arguments from the stack for certain functions after performing the call, but some parameters have provided-defaults if I can't get the item off the stack.

     Lua (5.1, 5.2, and 5.3), when I pop more than what's on the stack from the stack, seem to protect against me ruining the stack by "overpopping". LuaJIT, however, crashes because the stack is not protected from these over-pops.

     Some solutions I imagine to my problem would be to just always pop "int arg_count = lua_gettop(L)" values from the stack. But, I'm trying to get near-zero-overhead, and that's another (even if dirt cheap) API call I have to make.

    My other "solution" is maybe being cautious and only popping arguments I know are "required", but I do not know if the implementation is required to clean up the stack after a C function call and all arguments are returned.

     Are implementations required to clean up the stack if there's more than "return-count" things left on the stack before a C function returns?

Sincerely,
ThePhD
Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

ThePhD
Dear Alysson,

     Thank you for the clarification! I was unsure, since almost every example I had seen did this and it was not really specified anywhere exactly. My fear was that if the function called it might "leave" the arguments on the stack below where the results were, and eventually trash the VM with overflows. I'm glad this is not the case, and this will allow me to improve my code.

Sincerely,
ThePhD

On Sat, Jul 9, 2016 at 12:02 PM, Alysson Cunha <[hidden email]> wrote:

You dont need to pop the arguments from stack to rerurn values....

If the function rerurns 3, the top 3 elements of the stack will be the returning values of the function.  You dont need to pop the arguments

Em 09/07/2016 12:37, "ThePhD" <[hidden email]> escreveu:
Hello!

     I have a question regarding the implementation details of the returns for C functions into Lua. As of right now, I use the C API to return one or more values back to Lua, but only after popping all of the arguments given to Lua just before I do this. This seems to be what is necessary, given the examples I have seen in PIL and other places.

     My usage gets a bit different when I have optional arguments. I pop a fixed number of arguments from the stack for certain functions after performing the call, but some parameters have provided-defaults if I can't get the item off the stack.

     Lua (5.1, 5.2, and 5.3), when I pop more than what's on the stack from the stack, seem to protect against me ruining the stack by "overpopping". LuaJIT, however, crashes because the stack is not protected from these over-pops.

     Some solutions I imagine to my problem would be to just always pop "int arg_count = lua_gettop(L)" values from the stack. But, I'm trying to get near-zero-overhead, and that's another (even if dirt cheap) API call I have to make.

    My other "solution" is maybe being cautious and only popping arguments I know are "required", but I do not know if the implementation is required to clean up the stack after a C function call and all arguments are returned.

     Are implementations required to clean up the stack if there's more than "return-count" things left on the stack before a C function returns?

Sincerely,
ThePhD

Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

Niccolo Medici
On 7/9/16, ThePhD <[hidden email]> wrote:
>      Thank you for the clarification! I was unsure, since almost every
> example I had seen did this
> [...]
> examples I have seen in PIL and other places.

It's likely that you misunderstood the examples. Can you quote two examples?

> On Sat, Jul 9, 2016 at 12:02 PM, Alysson Cunha <[hidden email]>
> wrote:
>
>> You dont need to pop the arguments from stack to rerurn values....
>>
>> Em 09/07/2016 12:37, "ThePhD" <[hidden email]> escreveu:
>>
>> Hello!
>>
>>      I have a question regarding the implementation details of the
>> returns
>> for C functions into Lua. As of right now, I use the C API to return one
>> or
>> more values back to Lua, but only after popping all of the arguments
>> given
>> to Lua just before I do this. This seems to be what is necessary, given
>> the
>> examples I have seen in PIL and other places.
[...]
>>      Lua (5.1, 5.2, and 5.3), when I pop more than what's on the stack
>> from the stack, seem to protect against me ruining the stack by
>> "overpopping". LuaJIT, however, crashes because the stack is not
>> protected
>> from these over-pops.

Even if this were true, that you need to pop all values before your
function returns, you could simply do "lua_settop(L, 0)". No need to
calculate anything.

Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

ThePhD
Dear Niccolo,

     You're right. When I looked back I misunderstood the examples provided in both PIL and the Lua Manual. Embarrassing, on my part!

     Thanks for the "lua_settop(L, 0)" advice; I had not considered using that function for when I needed to clear the stack.

Sincerely,
ThePhD

On Sat, Jul 9, 2016 at 5:57 PM, Niccolo Medici <[hidden email]> wrote:
On 7/9/16, ThePhD <[hidden email]> wrote:
>      Thank you for the clarification! I was unsure, since almost every
> example I had seen did this
> [...]
> examples I have seen in PIL and other places.

It's likely that you misunderstood the examples. Can you quote two examples?

> On Sat, Jul 9, 2016 at 12:02 PM, Alysson Cunha <[hidden email]>
> wrote:
>
>> You dont need to pop the arguments from stack to rerurn values....
>>
>> Em 09/07/2016 12:37, "ThePhD" <[hidden email]> escreveu:
>>
>> Hello!
>>
>>      I have a question regarding the implementation details of the
>> returns
>> for C functions into Lua. As of right now, I use the C API to return one
>> or
>> more values back to Lua, but only after popping all of the arguments
>> given
>> to Lua just before I do this. This seems to be what is necessary, given
>> the
>> examples I have seen in PIL and other places.
[...]
>>      Lua (5.1, 5.2, and 5.3), when I pop more than what's on the stack
>> from the stack, seem to protect against me ruining the stack by
>> "overpopping". LuaJIT, however, crashes because the stack is not
>> protected
>> from these over-pops.

Even if this were true, that you need to pop all values before your
function returns, you could simply do "lua_settop(L, 0)". No need to
calculate anything.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

Liam Devine

ThePhD,

As you now already know, leaving arguments on the stack with return values is generally not a problem. However, if this function manipulates the stack then their is the potential for a problem to arise. This is because the stack space given to each function is a fixed size (20 IIRC, with an additional five for play). For functions that are going to use so much stack, you have to call lua_checkstack[1]

[1] http://www.lua.org/manual/5.3/manual.html#lua_checkstack

Regards,

Liam


On 09/07/2016 23:08, ThePhD wrote:
Dear Niccolo,

     You're right. When I looked back I misunderstood the examples provided in both PIL and the Lua Manual. Embarrassing, on my part!

     Thanks for the "lua_settop(L, 0)" advice; I had not considered using that function for when I needed to clear the stack.

Sincerely,
ThePhD

On Sat, Jul 9, 2016 at 5:57 PM, Niccolo Medici <[hidden email]> wrote:
On 7/9/16, ThePhD <[hidden email]> wrote:
>      Thank you for the clarification! I was unsure, since almost every
> example I had seen did this
> [...]
> examples I have seen in PIL and other places.

It's likely that you misunderstood the examples. Can you quote two examples?

> On Sat, Jul 9, 2016 at 12:02 PM, Alysson Cunha <[hidden email]>
> wrote:
>
>> You dont need to pop the arguments from stack to rerurn values....
>>
>> Em 09/07/2016 12:37, "ThePhD" <[hidden email]> escreveu:
>>
>> Hello!
>>
>>      I have a question regarding the implementation details of the
>> returns
>> for C functions into Lua. As of right now, I use the C API to return one
>> or
>> more values back to Lua, but only after popping all of the arguments
>> given
>> to Lua just before I do this. This seems to be what is necessary, given
>> the
>> examples I have seen in PIL and other places.
[...]
>>      Lua (5.1, 5.2, and 5.3), when I pop more than what's on the stack
>> from the stack, seem to protect against me ruining the stack by
>> "overpopping". LuaJIT, however, crashes because the stack is not
>> protected
>> from these over-pops.

Even if this were true, that you need to pop all values before your
function returns, you could simply do "lua_settop(L, 0)". No need to
calculate anything.



Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

William Ahern
In reply to this post by ThePhD
On Sat, Jul 09, 2016 at 12:29:25PM -0400, ThePhD wrote:
> Dear Alysson,
>
>      Thank you for the clarification! I was unsure, since almost every
> example I had seen did this and it was not really specified anywhere
> exactly. My fear was that if the function called it might "leave" the
> arguments on the stack below where the results were, and eventually trash
> the VM with overflows. I'm glad this is not the case, and this will allow
> me to improve my code.

When you return from a function Lua will clean up the stack, capturing the
top N values on the stack according to the return value and discarding the
remainder, if any.

However, where you can run into trouble is repeatedly calling from the same
call frame context lua_call, lua_resume (whether or not from the same C
function invocation), and similar routines. You _can_ easily overflow the
stack if, for example, you don't the pop return values from prior calls.

This happened to me when I first began using Lua. I was issuing callbacks on
a coroutine and not managing the stack correctly. After too many callbacks
I'd overflow the stack. But it didn't always happen and it was a headache to
track down as I didn't have a strong grasp on Lua's stack semantics. It was
made more complex case because some callbacks would often yield to be
resumed by an event loop, so the stack management code was split across
multiple different C functions.

This is the exception to the rule. When you `return lua_yield(L, N)` from
your C routine, either directly or indirectly from another function
invocation, then Lua will preserve the stack and only pop the number of
values (N) you want to yield. This is by design because the stack is
normally where you keep your state for when the coroutine is resumed. That's
in contrast to, e.g., a simple closure used as an iterator, where you keep
your state in upvalues. (Neither strategy is mutually exclusive--a closure
invocation can be yielded and resumed like any other function, and also make
use of its upvalues before and after resumption.)

My quick fix back then was to do `lua_settop(L, 0)` before each callback.
Since then I prefer to add an assert that verifies my stack discipline code
is working correctly.

Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

ThePhD
Dear William,

     That's actually a good point. I was thinking I might do `lua_settop(L, 0)` to avoid stack overflows since I don't quite have to deal with coroutines just yet... BUT! I do want to support a generic way to do that in the future from the library.

     The reason this came up for me was because I was using some magic to basically count the number of arguments in a C function and then auto-wrap it. Problem is, certain types can be "left out" (e.g., a pointer is automatically left out and becomes nil/nullptr), and while the getters for lua types using "lua_to{x}" worked just fine with some type checking, the magic stuff would still try to pop exactly N arguments, even if only a few were left on the stack. I guess I could just do a `min(arg_count, lua_gettop(L))` so I don't over-pop the stack, and so I can handle the case of coroutines nicely too?

    Just ideas, really.

Sincerely,
ThePhD

On Mon, Jul 11, 2016 at 3:35 PM, William Ahern <[hidden email]> wrote:
On Sat, Jul 09, 2016 at 12:29:25PM -0400, ThePhD wrote:
> Dear Alysson,
>
>      Thank you for the clarification! I was unsure, since almost every
> example I had seen did this and it was not really specified anywhere
> exactly. My fear was that if the function called it might "leave" the
> arguments on the stack below where the results were, and eventually trash
> the VM with overflows. I'm glad this is not the case, and this will allow
> me to improve my code.

When you return from a function Lua will clean up the stack, capturing the
top N values on the stack according to the return value and discarding the
remainder, if any.

However, where you can run into trouble is repeatedly calling from the same
call frame context lua_call, lua_resume (whether or not from the same C
function invocation), and similar routines. You _can_ easily overflow the
stack if, for example, you don't the pop return values from prior calls.

This happened to me when I first began using Lua. I was issuing callbacks on
a coroutine and not managing the stack correctly. After too many callbacks
I'd overflow the stack. But it didn't always happen and it was a headache to
track down as I didn't have a strong grasp on Lua's stack semantics. It was
made more complex case because some callbacks would often yield to be
resumed by an event loop, so the stack management code was split across
multiple different C functions.

This is the exception to the rule. When you `return lua_yield(L, N)` from
your C routine, either directly or indirectly from another function
invocation, then Lua will preserve the stack and only pop the number of
values (N) you want to yield. This is by design because the stack is
normally where you keep your state for when the coroutine is resumed. That's
in contrast to, e.g., a simple closure used as an iterator, where you keep
your state in upvalues. (Neither strategy is mutually exclusive--a closure
invocation can be yielded and resumed like any other function, and also make
use of its upvalues before and after resumption.)

My quick fix back then was to do `lua_settop(L, 0)` before each callback.
Since then I prefer to add an assert that verifies my stack discipline code
is working correctly.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C Functions

William Ahern
On Mon, Jul 11, 2016 at 03:45:15PM -0400, ThePhD wrote:
> Dear William,
<snip>
>      The reason this came up for me was because I was using some magic to
> basically count the number of arguments in a C function and then auto-wrap
> it. Problem is, certain types can be "left out" (e.g., a pointer is
> automatically left out and becomes nil/nullptr), and while the getters for
> lua types using "lua_to{x}" worked just fine with some type checking, the
> magic stuff would still try to pop exactly N arguments, even if only a few
> were left on the stack. I guess I could just do a `min(arg_count,
> lua_gettop(L))` so I don't over-pop the stack, and so I can handle the case
> of coroutines nicely too?

It's hard to say without knowing exactly what you're trying to solve. But if
you don't have have some kind of loop in the C function(s), then you
wouldn't need to worry about coroutines.

FWIW, I think the learned experience of many regulars here is to avoid magic
functions, wrappers, and binding generators. I've worked extensively with
Perl XS, which is almost entirely magic wrappers and generators around an
underspecified C interface. Other languages are similar, though not quite as
bad. I've run into more trouble trying to simplify Lua's C API or trying to
automate bindings than just accepting a little redundancy. The Lua VM and
language semantics were designed to make the C API and C modules first-class
citizens. That's unique among scripting languages--languages like Python and
JavaScript cannot add Lua-style asymmetric coroutines, for example, because
both the external and internal APIs are implicitly bound to the C stack. And
that's because they were too concerned too early in the design cycle with
convenience--you can prematurely optimize for "convenient" APIs just as you
can prematurely optimize for performance. If you try to be too clever too
early, you'll find that you've coded yourself into a corner, and your future
feature and performance improvements will be dictated by prior bad
decisions.

Along those lines, another piece of advice commonly heard on this list: when
binding C or C++ code, it's often better to keep the bindings relatively
simple with 1:1 mappings. Then wrap those bindings with Lua code to make the
module API more Lua-like. So for big modules you'll often see the C bindings
in a module named foo.core (foo/core.so), which is loaded and wrapped by a
Lua-script module, foo (foo.lua).

For those cases where you do need to add some magic to the C bindings to
make the module work well (like yielding at the C level, creating C
closures, specialized ownership patterns, etc), binding generators will
often get in the way. They make the easy stuff easier, but the difficult
stuff more difficult or impossible. That's not a good trade-off. Direct use
of the Lua API is usually the best balance, IME.