Lua C API can't be used from other languages without C wrappers

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lua C API can't be used from other languages without C wrappers

Soni "They/Them" L.
It was recently brought to my attention that the Lua C API is strictly a
C API, and not as universal as some ppl claim it to be.[1]

More specifically, it's unsafe to directly call this API from any sort
of FFI!
This includes LuaJIT FFI, Python CFFI, any of the many FFI wrappers for
Java, and so on!

Would it be possible to have a built-in "Universal API with C calling
conventions"? Thanks. :)

[1]: https://github.com/chucklefish/rlua/issues/71

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Dirk Laurie-2
2018-02-18 17:23 GMT+02:00 Soni "They/Them" L. <[hidden email]>:

> It was recently brought to my attention that the Lua C API is strictly a C
> API, and not as universal as some ppl claim it to be.[1]

It works perfectly with Free Pascal too.

http://lua-users.org/wiki/LuaInFreePascal

> More specifically, it's unsafe to directly call this API from any sort of FFI!
> This includes LuaJIT FFI, Python CFFI, any of the many FFI wrappers for Java, and so on!

Would not that be more a defect of the FFI than of the Lua API?

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Albert Chan
In reply to this post by Soni "They/Them" L.
I have trouble using Lua C stack, even in C

Say, I have a c function A that uses lua stack, that call another c function B
that also uses c lua stack.  however, B did a lua_settop of 1, then return 1.

Sine both c functions share the same stack, A loses everything except the result of B

It would not be hard to populate the stack again if the values are string, numbers ...
but some stack entries are user data, I don't know how to put it back to stack.

Rework b to restore the stack back before it was called work, but that is a lot of work.
Besides, what happen if I have C, D, E, ... that also mess up the stack ?





Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Dirk Laurie-2
2018-02-18 18:37 GMT+02:00 Albert Chan <[hidden email]>:
> I have trouble using Lua C stack, even in C
>
> Say, I have a c function A that uses lua stack, that call another c function B
> that also uses c lua stack.  however, B did a lua_settop of 1, then return 1.
>
> Sine both c functions share the same stack, A loses everything except the result of B

They do not. Every call generates a new stack frame, and no function
can see further back than its own arguments, except those values that
have been exported into its closure as upvalues. What you describe
cannot happen.

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Soni "They/Them" L.
In reply to this post by Dirk Laurie-2


On 2018-02-18 12:39 PM, Dirk Laurie wrote:

> 2018-02-18 17:23 GMT+02:00 Soni "They/Them" L. <[hidden email]>:
>
>> It was recently brought to my attention that the Lua C API is strictly a C
>> API, and not as universal as some ppl claim it to be.[1]
> It works perfectly with Free Pascal too.
>
> http://lua-users.org/wiki/LuaInFreePascal
>
>> More specifically, it's unsafe to directly call this API from any sort of FFI!
>> This includes LuaJIT FFI, Python CFFI, any of the many FFI wrappers for Java, and so on!
> Would not that be more a defect of the FFI than of the Lua API?
>

What happens if you "pcall(C.lua_error(some_L))" from within LuaJIT?
does the pcall catch the lua_error? are the results the same between
windows and linux? etc.

I seriously doubt it's a defect of the FFI. There's only so much an FFI
can (or should) do, and handling all forms of unwinding, including
setjmp/longjmp *and* C++ exceptions *and* the OS's exception propagation
mechanism, is not one of them.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Albert Chan
In reply to this post by Dirk Laurie-2
On Feb 18, 2018, at 11:54 AM, Dirk Laurie <[hidden email]> wrote:

> 2018-02-18 18:37 GMT+02:00 Albert Chan <[hidden email]>:
>> I have trouble using Lua C stack, even in C
>>
>> Say, I have a c function A that uses lua stack, that call another c function B
>> that also uses c lua stack.  however, B did a lua_settop of 1, then return 1.
>>
>> Sine both c functions share the same stack, A loses everything except the result of B
>
> They do not. Every call generates a new stack frame, and no function
> can see further back than its own arguments, except those values that
> have been exported into its closure as upvalues. What you describe
> cannot happen.
>

you are right, if the call is from lua calling c-function.

If c-function A (from inside A) call B, however, we are sharing the same stack.
--> B removed all arguments except its result, did a lua_settop(L,1); return 1.
--> all stack arguments in A is GONE (except, result from B)

there is a blog about this problem: https://julien.danjou.info/blog/2011/why-not-lua

And this actualy happens to me recently. I had to fix B to leave the stack alone.
My question is, what if B is off-limit, how can it restore the stack from A side ?

more specifically, how to push userdata back to the stack ?



Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Soni "They/Them" L.


On 2018-02-18 02:37 PM, albertmcchan wrote:

> On Feb 18, 2018, at 11:54 AM, Dirk Laurie <[hidden email]> wrote:
>
>> 2018-02-18 18:37 GMT+02:00 Albert Chan <[hidden email]>:
>>> I have trouble using Lua C stack, even in C
>>>
>>> Say, I have a c function A that uses lua stack, that call another c function B
>>> that also uses c lua stack.  however, B did a lua_settop of 1, then return 1.
>>>
>>> Sine both c functions share the same stack, A loses everything except the result of B
>> They do not. Every call generates a new stack frame, and no function
>> can see further back than its own arguments, except those values that
>> have been exported into its closure as upvalues. What you describe
>> cannot happen.
>>
> you are right, if the call is from lua calling c-function.
>
> If c-function A (from inside A) call B, however, we are sharing the same stack.
> --> B removed all arguments except its result, did a lua_settop(L,1); return 1.
> --> all stack arguments in A is GONE (except, result from B)
>
> there is a blog about this problem: https://julien.danjou.info/blog/2011/why-not-lua
>
> And this actualy happens to me recently. I had to fix B to leave the stack alone.
> My question is, what if B is off-limit, how can it restore the stack from A side ?
>
> more specifically, how to push userdata back to the stack ?
>
>
>

Ah I miss Lua 5.1's lua_cpcall...

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

szbnwer@gmail.com
hi there!

1st of all, its a good idea, i already had this in my mind, but never
gone farer than an idea of what about using them from ffi...

sorry if im saying 100% bullshit, i've never used the c api, used only
a bit of lj ffi, and ive got no a clear vision about the internals,
but i have a kinda weak vision about these, and what could be the
actual problem..... so what i have is only some ideas :D

what i think is that there is no impossible, but whatever someone
would make with these actions, that could mess up things in the stack
- if im right.
i think some kinda wrapper/workaround can solve it.

however i can imagine that it can be used if there is a way to restore
the previous state of the lua stack, and then everything is going fine
after the action. so i think these steps can solve it: make a place
where the actual data will be stored with some simple valid lua, this
can be something like an another lua state that have a complete
accessor (i think there is such a toy out there, if not, then it would
be a nice stuff, it can be made even from pure ffi i think :) ) or a
user data that will be populated. then make a snapshot or some notes
for the only necessary data about the current state of lua. then you
can play whatever you want, or at least some limited stuffs that wont
harm the chance of a nice restore. populate your previously made place
with the new data. restore the previous point, and use your obtained
data as u like it. this is limited to obtain info, however after the
game u can use it for any purpose, so it maybe can give an unlimited
tool for do whatever, but kinda complicated... and also this way one
only can count on what will happen on lua side, so i think this is
still harmful if there is any magic that affects anything outside the
stack, like when you have an isolated data with a setter/getter, but
the original place is not directly accessible on the stack...

an another method is to follow up the changes and make everything fine
on a step by step manner for lua, as you can see all the internal
data, and u know how lua acts, i think you can count everything that
is needed, but i think that it takes a smaller bigger wrapper for
every single api function, so i think this will suffer under harder
tasks, but could be nice for experimenting at least. and maybe it can
even have an even smaller wrapper if any job can be accumulated for
the end of a bunch of the api calls.

and what i didnt calculate on the way is that it requires to play from
the stack from both lua and a kinda related tool for making api calls,
or the latter could be completely separated like a whole handover for
the time it acts, but i believe it is possible, as a lua c function (i
mean something that is accessible from lua side but the func is
defined on c side) can do kinda much anything without the chance of
getting into trouble as anything else than that would like to touch
the stack.

so i think its not impossible, but impossible on the raw way, but
possible if well baked, but this takes a whole expert toolset.

start it in small and achieve 2-3 usable c functions with the right
workaround to make everything smooth and slowly possibilities and the
right path will be crystallized, and wont forget to share the results,
this would be a so much cool toy :)

i hope i could say anything useful that have any relation with the
actual reality :D
bests 4 all! :)

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Andrew Gierth
In reply to this post by Albert Chan
>>>>> "Albert" == Albert Chan <[hidden email]> writes:

 Albert> I have trouble using Lua C stack, even in C

 Albert> Say, I have a c function A that uses lua stack, that call
 Albert> another c function B that also uses c lua stack. however, B did
 Albert> a lua_settop of 1, then return 1.

If A calls B _as a C function_, then B should not be using settop or
making assumptions about the start of the lua stack.

If you want B to be able to use the stack freely without affecting the
caller, then you just call it with lua_pushcfunction and lua_call rather
than directly.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Andrew Gierth
In reply to this post by Soni "They/Them" L.
>>>>> "Soni" == Soni \"They/Them\" L <[hidden email]> writes:

 Soni> Ah I miss Lua 5.1's lua_cpcall...

Why?

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Soni "They/Them" L.


On 2018-02-18 07:12 PM, Andrew Gierth wrote:
>>>>>> "Soni" == Soni \"They/Them\" L <[hidden email]> writes:
>   Soni> Ah I miss Lua 5.1's lua_cpcall...
>
> Why?
>

Except for its limitations, it was a nice way to say "this is how you're
supposed to call C functions that deal with the Lua stack"...

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Russell Haley
In reply to this post by Dirk Laurie-2
On Sun, Feb 18, 2018 at 7:39 AM, Dirk Laurie <[hidden email]> wrote:
> 2018-02-18 17:23 GMT+02:00 Soni "They/Them" L. <[hidden email]>:
>
>> It was recently brought to my attention that the Lua C API is strictly a C
>> API, and not as universal as some ppl claim it to be.[1]
>
> It works perfectly with Free Pascal too.
>
> http://lua-users.org/wiki/LuaInFreePascal

+1. Free Pascal is on my todo list...

>> More specifically, it's unsafe to directly call this API from any sort of FFI!
>> This includes LuaJIT FFI, Python CFFI, any of the many FFI wrappers for Java, and so on!
>
> Would not that be more a defect of the FFI than of the Lua API?
>

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Andrew Gierth
In reply to this post by Soni "They/Them" L.
>>>>> "Soni" == Soni \"They/Them\" L <[hidden email]> writes:

 Soni> It was recently brought to my attention that the Lua C API is
 Soni> strictly a C API, and not as universal as some ppl claim it to
 Soni> be.[1]
 Soni> [1]: https://github.com/chucklefish/rlua/issues/71

This issue isn't really about C vs non-C, it's about longjmp
compatibility with the surrounding environment, which can be an issue
even with pure C code. (In embedding Lua in Postgres, which is a pure C
project, I also have this problem since Postgres has its _own_
setjmp/longjmp exception system, and hilarity ensues if you throw lua
errors across pg catch blocks or pg errors across Lua catch blocks.)

Like it or not, longjmp is part of standard C.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Soni "They/Them" L.


On 2018-02-18 11:03 PM, Andrew Gierth wrote:

>>>>>> "Soni" == Soni \"They/Them\" L <[hidden email]> writes:
>   Soni> It was recently brought to my attention that the Lua C API is
>   Soni> strictly a C API, and not as universal as some ppl claim it to
>   Soni> be.[1]
>   Soni> [1]: https://github.com/chucklefish/rlua/issues/71
>
> This issue isn't really about C vs non-C, it's about longjmp
> compatibility with the surrounding environment, which can be an issue
> even with pure C code. (In embedding Lua in Postgres, which is a pure C
> project, I also have this problem since Postgres has its _own_
> setjmp/longjmp exception system, and hilarity ensues if you throw lua
> errors across pg catch blocks or pg errors across Lua catch blocks.)
>
> Like it or not, longjmp is part of standard C.
>

It's called "use setjmp/longjmp internally, but don't leak it across API
boundaries".

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

ThePhD
Perhaps tangential to the discussion, but longjmp / setjmp has bitten me a lot too. From not running C++ destructors to making cleanup code get skipped and generally surprising behavior, if there's any mechanism I'd avoid in any API -- even with plain C where destructors and stack unwinding isn't a thing -- it's these.

Unfortunately, that means your entire API has to be rewritten from the ground up to propagate errors properly to every external function, meaning the internals have to be written to handle it. This is its own challenge, and is extremely hard to do properly throughout an entire API.

On Sun, Feb 18, 2018 at 9:06 PM, Soni "They/Them" L. <[hidden email]> wrote:


On 2018-02-18 11:03 PM, Andrew Gierth wrote:
"Soni" == Soni \"They/Them\" L <[hidden email]> writes:
  Soni> It was recently brought to my attention that the Lua C API is
  Soni> strictly a C API, and not as universal as some ppl claim it to
  Soni> be.[1]
  Soni> [1]: https://github.com/chucklefish/rlua/issues/71

This issue isn't really about C vs non-C, it's about longjmp
compatibility with the surrounding environment, which can be an issue
even with pure C code. (In embedding Lua in Postgres, which is a pure C
project, I also have this problem since Postgres has its _own_
setjmp/longjmp exception system, and hilarity ensues if you throw lua
errors across pg catch blocks or pg errors across Lua catch blocks.)

Like it or not, longjmp is part of standard C.


It's called "use setjmp/longjmp internally, but don't leak it across API boundaries".


--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.



Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

nobody
On 2018-02-19 04:03, ThePhD wrote:
> Perhaps tangential to the discussion, but longjmp / setjmp has bitten
> me a lot too. From not running C++ destructors to making cleanup code
> get skipped and generally surprising behavior, if there's any
> mechanism I'd avoid in any API -- even with plain C where destructors
> and stack unwinding isn't a thing -- it's these.

IIRC, you can compile Lua as C++, in which case it won't use setjmp /
longjmp but do The C++ Thing, and then all of that should Just Work.

(If you look at the top of `ldo.c`, all the ugly stuff is tucked away in
a bunch of small macros and should be relatively easy to change, if an
alternate exception mechanism is available.)

(@Soni: Have you tried messing with that or do you already know that
this doesn't help with the Rust situation?)

-- nobody

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Soni "They/Them" L.


On 2018-02-19 02:09 AM, nobody wrote:

> On 2018-02-19 04:03, ThePhD wrote:
>> Perhaps tangential to the discussion, but longjmp / setjmp has bitten
>> me a lot too. From not running C++ destructors to making cleanup code
>> get skipped and generally surprising behavior, if there's any
>> mechanism I'd avoid in any API -- even with plain C where destructors
>> and stack unwinding isn't a thing -- it's these.
>
> IIRC, you can compile Lua as C++, in which case it won't use setjmp /
> longjmp but do The C++ Thing, and then all of that should Just Work.
>
> (If you look at the top of `ldo.c`, all the ugly stuff is tucked away
> in a bunch of small macros and should be relatively easy to change, if
> an alternate exception mechanism is available.)
>
> (@Soni: Have you tried messing with that or do you already know that
> this doesn't help with the Rust situation?)
>

Rust doesn't have a well-defined panic handling mechanism. And you'd
still need to mark functions as #[unwind], which is an unstable (only
available on nightly) attribute.

> -- nobody
>

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Tim Hill
In reply to this post by ThePhD


> On Feb 18, 2018, at 7:03 PM, ThePhD <[hidden email]> wrote:
>
> Perhaps tangential to the discussion, but longjmp / setjmp has bitten me a lot too. From not running C++ destructors to making cleanup code get skipped and generally surprising behavior, if there's any mechanism I'd avoid in any API -- even with plain C where destructors and stack unwinding isn't a thing -- it's these.
>
> Unfortunately, that means your entire API has to be rewritten from the ground up to propagate errors properly to every external function, meaning the internals have to be written to handle it. This is its own challenge, and is extremely hard to do properly throughout an entire API.
>

+1 .. the C function calling model in Lua is very easy to use for trivial C functions, and very hard to use for non-trivial ones.

By non-trivial, I mean any C function that must manage dynamic resources during its execution, such as memory, file handles etc. These must be cleaned up before the C function returns to Lua, and this is where the problems start. I’m not talking about C functions that maintain state BETWEEN calls (userdata can handle that case), I mean ones that allocate temporary resources just for the duration of the C function call.

The problem, os course, is that if you call almost any Lua function while executing the C function, Lua can throw an error and silently demolish the C stack. This leaks any resources that the function is using, a very nasty bug indeed.

It seems to me there are only two solutions to this:

(A) Allocate a full userdata that stores (or references) the allocated resources. Then setup a meltable for the userdata with a __gc() C function that can cleanup (idempotently) all the resources.

(B) Create a proxy C function that Lua calls instead of the real C function, which then allocates the resources before using lua_pcall() to call the real C function. This allows the C proxy to catch any Lua errors and perform cleanup before returning to Lua.

Both are cumbersome to use from C code, and are tricky to get right. Both add significant overhead to the C function at run-time. Both are inelegant. Solution (A) also suffers from not performing cleanup until the next GC.

I find it interesting that PiL is totally silent on this subject, and somewhat worrying since without either of the above strategies, code will silently leak resources, a type of bug that typically only shows up under stress (e.g. low memory) and is very hard to isolate.

I won’t suggest a solution here (several occur to me), but I do think this is one of the weaker points of Lua.

—Tim







Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Pierre Chapuis
On Mon, Feb 19, 2018, at 20:29, Tim Hill wrote:

> I find it interesting that PiL is totally silent on this subject, and
> somewhat worrying since without either of the above strategies, code
> will silently leak resources, a type of bug that typically only shows up
> under stress (e.g. low memory) and is very hard to isolate.

PiL is not silent on the subject, it advocates your second solution.
It is chapter 27.3 of PiL 4, for instance.

There are two caveats, however:

- The solution only works since Lua 5.2, because afaik before Lua 5.2
  lua_pushcfunction allocated memory. lua_cpcall had to be used
  instead.

- Several code examples in PiL will leak resources if allocation fails,
  e.g. the readdir example in listing 29.1 will leak the directory handler.

That being said, it depends on your context, but on a lot of modern
OSs there is nothing much to do if allocation fails, so defining a custom
Lua allocator or just using lua_atpanic may be enough if you control
your Lua code. If you write a sandbox however, there is no way to avoid
dealing with this

To be honest I have dealt with languages that use a different approach
to bindings such as Ruby and Python, and I still prefer the Lua solution.

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: Lua C API can't be used from other languages without C wrappers

Russell Haley
In reply to this post by Tim Hill
On Mon, Feb 19, 2018 at 11:29 AM, Tim Hill <[hidden email]> wrote:

>
>
>> On Feb 18, 2018, at 7:03 PM, ThePhD <[hidden email]> wrote:
>>
>> Perhaps tangential to the discussion, but longjmp / setjmp has bitten me a lot too. From not running C++ destructors to making cleanup code get skipped and generally surprising behavior, if there's any mechanism I'd avoid in any API -- even with plain C where destructors and stack unwinding isn't a thing -- it's these.
>>
>> Unfortunately, that means your entire API has to be rewritten from the ground up to propagate errors properly to every external function, meaning the internals have to be written to handle it. This is its own challenge, and is extremely hard to do properly throughout an entire API.
>>
>
> +1 .. the C function calling model in Lua is very easy to use for trivial C functions, and very hard to use for non-trivial ones.
>
> By non-trivial, I mean any C function that must manage dynamic resources during its execution, such as memory, file handles etc. These must be cleaned up before the C function returns to Lua, and this is where the problems start. I’m not talking about C functions that maintain state BETWEEN calls (userdata can handle that case), I mean ones that allocate temporary resources just for the duration of the C function call.
>
> The problem, os course, is that if you call almost any Lua function while executing the C function, Lua can throw an error and silently demolish the C stack. This leaks any resources that the function is using, a very nasty bug indeed.
>
> It seems to me there are only two solutions to this:
>
> (A) Allocate a full userdata that stores (or references) the allocated resources. Then setup a meltable for the userdata with a __gc() C function that can cleanup (idempotently) all the resources.
>
> (B) Create a proxy C function that Lua calls instead of the real C function, which then allocates the resources before using lua_pcall() to call the real C function. This allows the C proxy to catch any Lua errors and perform cleanup before returning to Lua.
>
> Both are cumbersome to use from C code, and are tricky to get right. Both add significant overhead to the C function at run-time. Both are inelegant. Solution (A) also suffers from not performing cleanup until the next GC.
>
> I find it interesting that PiL is totally silent on this subject, and somewhat worrying since without either of the above strategies, code will silently leak resources, a type of bug that typically only shows up under stress (e.g. low memory) and is very hard to isolate.
>
> I won’t suggest a solution here (several occur to me), but I do think this is one of the weaker points of Lua.
>
> —Tim

I'm reading PIL right now to learn about the C API, with an a focus on
using it in .Net and C++ (and Free Pascal hopefully one day too!). It
occurred to me last night that the C API has the same design decisions
as filesystem support and user input via readline: "Not our problem".
This was hammered home for me when I watched ThePhD's Lua Workshop
presentation on sol2 and how clean and beautiful it is; but it took a
DOCTOR to put that together. The counter point can be seen in .Net
implementations that leak memory even in their own tests. Another
counterpoint can be seen in Javascript Interpreter Shell (jsish.org)
that uses the TCL model. Jsi provied C structures and methods for
interacting with the embedded interpreter.

I wanted to use Lua to *simply* things, not manage a stack. I have to
say Lua integration has been a bit of a heart breaker. After the PhDs
presentation, I'm now wondering about ditching the C API all together
and re-writing .Net integration using Sol2 instead. Ah, dreams...

Russ

>
>
>
>
>
>

12