C/C++ interoperability

[Viacheslav Usov]

When embedding Lua into an application written in C++, it is pretty much a requirement to compile the Lua library in the C++ mode, to ensure that Lua's error mechanism interoperates properly with C++'s destructors. Lua supports being compiled as C++, and once the compiler is instructed to compile the code as C++, it will automatically use throw/try/catch.

This, however, has a side effect: all of Lua's functions are then defined, in C++ terms, as extern "C++". Which is, unfortunately, a problem when Lua is linked statically [1] and then another C-language library using Lua needs to be linked statically with the same application. Linking such a library (let's call it X) with the rest of the application (containing Lua compiled in the C++ mode) will fail, because the Lua API functions will be decorated according to extern "C++", while X expects extern "C". It may be argued that X could be recompiled as C++, but C and C++ are not fully compatible, sometimes with run-time differences only [2], so this is generally not advisable.

In a project where I had this problem, I resolved it by altering the following definition in luaconf.h:

original:

#define LUA_API extern

modified:

#ifdef __cplusplus 

#define LUA_API extern "C"

#else

#define LUA_API extern

#endif

As far as I can tell, nothing else is needed to resolve the issue. I suggest that this be incorporated into Lua's official source.

I have thought about another way of fixing this, but I am unsure about its true merit. The try/catch/throw keywords, when compiled in the C++ mode, are only used once each, and in a very simple way, without much dependency on the rest of the library. Therefore, this C++ code can be extracted and placed in a separate file, one with a C++ extension, and only be compiled when the C++ error behaviour is needed. I do not think Lua needs to be compiled as C++ for any other reason. Then the rest of library need only support being compiled strictly as C, which may require less of an effort to ensure C/C++ standard conformance. As stated above, I am not really sure whether the benefit is real.

Cheers,

V.

[1] I emphasise static linking because this is the environment where I encountered the issue. I have not examined the shared/dynamic library case. I expect platform-dependent variance here. I do not think the proposed solution will make the situation worse in this respect.

[2] https://en.wikipedia.org/wiki/Compatibility_of_C_and_C%2B%2B

[Alexey Melnichuk-2]

Здравствуйте, Viacheslav.

Вы писали 20 октября 2015 г., 18:28:28:

> When embedding Lua into an application written in C++, it is pretty
> much a requirement to compile the Lua library in the C++ mode, to
> ensure that Lua's error mechanism interoperates properly with C++'s
> destructors. Lua supports being compiled as C++, and once the
> compiler is instructed to compile the code as C++, it will
> automatically use throw/try/catch.

You can build Lua as C library and use lua_pcall.
So  you  create C function which did not have any C++ object in stack.
This function interact with Lua API.
And you can call this function from C++ function using lua_pcall.
You can pass any C++ objects as pointer via upvalues.
Of  course you have chance get error on lua_push***. But I just ignore
this.

--
С уважением,
 Alexey                          mailto:[hidden email]


---
Это сообщение проверено на вирусы антивирусом Avast.
https://www.avast.com/antivirus

[Philipp Janda]

Am 20.10.2015 um 17:28 schröbte Viacheslav Usov:
> When embedding Lua into an application written in C++, it is pretty much a
> requirement to compile the Lua library in the C++ mode, to ensure that
> Lua's error mechanism interoperates properly with C++'s destructors.

I think another safe way is to not use objects with non-trivial
destructors in your binding functions (i.e. wherever `lua_error()` might
be called), and transform all exceptions thrown by your application code
into Lua errors.

> Lua supports being compiled as C++, and once the compiler is
> instructed to compile the code as C++, it will automatically use
> throw/try/catch.
>
> This, however, has a side effect: all of Lua's functions are then defined,
> in C++ terms, as extern "C++". Which is, unfortunately, a problem when Lua
> is linked statically [1] and then another C-language library using Lua
> needs to be linked statically with the same application. Linking such a
> library (let's call it X) with the rest of the application (containing Lua
> compiled in the C++ mode) will fail, because the Lua API functions will be
> decorated according to extern "C++", while X expects extern "C". It may be
> argued that X could be recompiled as C++, but C and C++ are not fully
> compatible, sometimes with run-time differences only [2], so this is
> generally not advisable.
>
> In a project where I had this problem, I resolved it by altering the
> following definition in luaconf.h:
>
> original:
>
> #define LUA_API extern
>
> modified:
>
> #ifdef __cplusplus
> #define LUA_API extern "C"
> #else
> #define LUA_API extern
> #endif
>
> As far as I can tell, nothing else is needed to resolve the issue. I
> suggest that this be incorporated into Lua's official source.

You shouldn't throw exceptions from `extern "C"` functions, especially
if real C code is involved. So the whole point of compiling Lua as C++
becomes moot. Also `extern "C" int (*)( lua_State* )` and `extern "C++"
int (*)( lua_State* )` are distinct types not convertible to each other,
and currently you can only pass one *or* the other to the Lua API
functions, depending on how the Lua library was compiled. So if you have
C code *and* C++ code, and both use `lua_CFunction`s, your only
standards compliant option is to compile Lua as C and use `extern "C"`
in your C++ code for your binding functions.

>
> I have thought about another way of fixing this, but I am unsure about its
> true merit. The try/catch/throw keywords, when compiled in the C++ mode,
> are only used once each, and in a very simple way, without much dependency
> on the rest of the library. Therefore, this C++ code can be extracted and
> placed in a separate file, one with a C++ extension, and only be compiled
> when the C++ error behaviour is needed. I do not think Lua needs to be
> compiled as C++ for any other reason. Then the rest of library need only
> support being compiled strictly as C, which may require less of an effort
> to ensure C/C++ standard conformance. As stated above, I am not really sure
> whether the benefit is real.

Real C code is incompatible with exceptions, so most Lua API functions
will need to be `extern "C++"` anyway just because they could be on the
call stack when an error/exception is thrown.

>
> Cheers,
> V.
>

Philipp

[Viacheslav Usov]

On Tue, Oct 20, 2015 at 7:33 PM, Philipp Janda <[hidden email]> wrote:

[...]

You shouldn't throw exceptions from `extern "C"` functions, especially if real C code is involved. So the whole point of compiling Lua as C++ becomes moot.

This is false. Oh, and "real C" is not defined, too. The current C++ standard states, in clause 7.5.1: "Some of the properties associated with an entity with language linkage are specific to each implementation and are not described here". Subsequent clauses on language linkage do not introduce any exception-related restrictions. So exception propagation over C/C++ boundaries is both standard compliant and entirely implementation specific. There are in fact C/C++ implementations that have consistent and documented behaviour with respect to exception propagation over C/C++ boundaries. With such implementations, the try/catch/throw mechanics as used in Lua is perfectly interoperable with C-language libraries that are designed correctly for use in Lua. Designed correctly for use in Lua implies that any call such a library makes into Lua or into a user-provided function is expected to result in a non-local transfer via lua_error().

Bottom line, compiling Lua as C++ with try/catch/throw mechanics and using C-language Lua libraries is both standard compliant and well defined in particular C/C++ implementations, and is a valuable feature.

Also `extern "C" int (*)( lua_State* )` and `extern "C++" int (*)( lua_State* )` are distinct types not convertible to each other, and currently you can only pass one *or* the other to the Lua API functions, depending on how the Lua library was compiled.

That is true, even though at least some C++ implementations, in non-pedantic modes, do not enforce this distinction. I have missed this aspect; so I need to spend some more time to develop a standard compliant solution for this.

 

So if you have C code *and* C++ code, and both use `lua_CFunction`s, your only standards compliant option is to compile Lua as C and use `extern "C"` in your C++ code for your binding functions.

This is not the only option. One can still compile Lua as C++ with lua_CFunction and other function types defined as extern "C".

Cheers,

V.

[Philipp Janda]

Am 21.10.2015 um 15:45 schröbte Viacheslav Usov:

> On Tue, Oct 20, 2015 at 7:33 PM, Philipp Janda <[hidden email]> wrote:
>
> [...]
>
> You shouldn't throw exceptions from `extern "C"` functions, especially if
>> real C code is involved. So the whole point of compiling Lua as C++ becomes
>> moot.
>
>
> This is false.  Oh, and "real C" is not defined, too.

"Real C" for the scope of this thread is defined (by me) as code
compiled by a C compiler in contrast to an `extern "C"` function that is
compiled by a C++ compiler. E.g. gcc by default does "The Right Thing"
(`-fexceptions`) for C++ code, but not for C code. That's why I made the
distinction. In theory it isn't needed, because the behavior is
undefined by the C++ and C standards in both cases.

> The current C++ standard states, in clause 7.5.1: "Some of the
> properties associated with an entity with language linkage are
> specific to each implementation and are not described here".
> Subsequent clauses on language linkage do not introduce any
> exception-related restrictions.

Meaning the C++ standard doesn't define those properties.

> So exception propagation over C/C++ boundaries is both standard
> compliant and entirely implementation specific.

Any C++ compiler/implementation is allowed to define behavior that is
undefined by the C++ standard.

>
> [...]
>
>
>> So if you have C code *and* C++ code, and both use `lua_CFunction`s, your
>> only standards compliant option is to compile Lua as C and use `extern "C"`
>> in your C++ code for your binding functions.
>>
>
> This is not the only option. One can still compile Lua as C++ with
> lua_CFunction and other function types defined as extern "C".

`lua_pcall()` (C++) -> Lua code -> `lua_CFunction` (`extern "C"`) ->
`lua_error()` (C++) -> `throw`.
So we have undefined behavior again because the exception passes
language boundaries. Whether it works in practice depends on
compiler/implementation, command line flags, whether it's "real C" code
the exception is passing through, etc.

>
> Cheers,
> V.
>

Philipp

[Paco Zamora-Martínez]

We are using an approach similar to what Philipp comments in APRIL-ANN:

I think another safe way is to not use objects with non-trivial destructors in your binding functions (i.e. wherever `lua_error()` might be called), and transform all exceptions thrown by your application code into Lua errors.

We have a binding library which wraps every C++ call with try{}catch{} blocks, and every time we capture an exception we change the value of a flag, and when the value of the flag is true we transform the C++ exception into a Lua error. Something similar to this:

bool exception_thrown=false;

try {

  // here C++ call, potentially exceptions can be thrown

}

catch (whatever) {

  exception_thrown=true;

}

if (exception_thrown) {

  lua_pushstring(L, "some error");

  return lua_error(L);

}

The reason to use the flag variable is to execute the longjmp of lua_error() outside the C++ exception handler, otherwise you can end up with memory leaks.

The problem arises when you call from C++ a Lua function, you need to wrap the call always with a pcall in order to translate the Lua exception into a C++ exception.

Doing this way the library can be loaded in a Lua binary compiled in C, without C++ support.

Lua supports being compiled as C++, and once the compiler is
instructed to compile the code as C++, it will automatically use
throw/try/catch.

This, however, has a side effect: all of Lua's functions are then defined,
in C++ terms, as extern "C++". Which is, unfortunately, a problem when Lua
is linked statically [1] and then another C-language library using Lua
needs to be linked statically with the same application. Linking such a
library (let's call it X) with the rest of the application (containing Lua
compiled in the C++ mode) will fail, because the Lua API functions will be
decorated according to extern "C++", while X expects extern "C". It may be
argued that X could be recompiled as C++, but C and C++ are not fully
compatible, sometimes with run-time differences only [2], so this is
generally not advisable.

In a project where I had this problem, I resolved it by altering the
following definition in luaconf.h:

original:

#define LUA_API extern

modified:

#ifdef __cplusplus
#define LUA_API extern "C"
#else
#define LUA_API extern
#endif

As far as I can tell, nothing else is needed to resolve the issue. I
suggest that this be incorporated into Lua's official source.

You shouldn't throw exceptions from `extern "C"` functions, especially if real C code is involved. So the whole point of compiling Lua as C++ becomes moot. Also `extern "C" int (*)( lua_State* )` and `extern "C++" int (*)( lua_State* )` are distinct types not convertible to each other, and currently you can only pass one *or* the other to the Lua API functions, depending on how the Lua library was compiled. So if you have C code *and* C++ code, and both use `lua_CFunction`s, your only standards compliant option is to compile Lua as C and use `extern "C"` in your C++ code for your binding functions.

I have thought about another way of fixing this, but I am unsure about its
true merit. The try/catch/throw keywords, when compiled in the C++ mode,
are only used once each, and in a very simple way, without much dependency
on the rest of the library. Therefore, this C++ code can be extracted and
placed in a separate file, one with a C++ extension, and only be compiled
when the C++ error behaviour is needed. I do not think Lua needs to be
compiled as C++ for any other reason. Then the rest of library need only
support being compiled strictly as C, which may require less of an effort
to ensure C/C++ standard conformance. As stated above, I am not really sure
whether the benefit is real.

Real C code is incompatible with exceptions, so most Lua API functions will need to be `extern "C++"` anyway just because they could be on the call stack when an error/exception is thrown.

Cheers,
V.

Philipp

[Viacheslav Usov]

On Wed, Oct 21, 2015 at 6:46 PM, Philipp Janda <[hidden email]> wrote:

 

"Real C" for the scope of this thread is defined (by me) as code compiled by a C compiler in contrast to an `extern "C"` function that is compiled by a C++ compiler. E.g. gcc by default does "The Right Thing" (`-fexceptions`) for C++ code, but not for C code. That's why I made the distinction. In theory it isn't needed, because the behavior is undefined by the C++ and C standards in both cases.

That is again not true. "Undefined behaviour" has a very specific meaning in the current C++ standard (clause 1.3.24) and the passage quoted earlier does not use that term. Your definition of "Real C" misses the point, because this area, as stated earlier, is implementation specific. It is sufficient for me that my implementation specifies a behaviour that is suitable for my use. Whatever behaviour observed with your implementation is immaterial to me.

`lua_pcall()` (C++) -> Lua code -> `lua_CFunction` (`extern "C"`) -> `lua_error()` (C++) -> `throw`.
So we have undefined behavior

No, as explained above.

 

Whether it works in practice depends on compiler/implementation, command line flags, whether it's "real C" code the exception is passing through, etc.

Correct.

Cheers,

V.

[Philipp Janda]

Am 21.10.2015 um 19:13 schröbte Viacheslav Usov:

> On Wed, Oct 21, 2015 at 6:46 PM, Philipp Janda <[hidden email]> wrote:
>
>
>> "Real C" for the scope of this thread is defined (by me) as code compiled
>> by a C compiler in contrast to an `extern "C"` function that is compiled by
>> a C++ compiler. E.g. gcc by default does "The Right Thing" (`-fexceptions`)
>> for C++ code, but not for C code. That's why I made the distinction. In
>> theory it isn't needed, because the behavior is undefined by the C++ and C
>> standards in both cases.
>>
>
> That is again not true. "Undefined behaviour" has a very specific meaning
> in the current C++ standard (clause 1.3.24) and the passage quoted earlier
> does not use that term.

Clause 1.3.24 says:
> undefined behavior
> behavior for which this International Standard imposes no requirements
> [ Note: Undefined behavior may be expected when this International
> Standard omits any explicit definition of behavior or when a program
> uses an erroneous construct or erroneous data. [...] ]
>

but I only have the draft standard, maybe they changed it for the final
version.

> Your definition of "Real C" misses the point,
> because this area, as stated earlier, is implementation specific.

And I gave one popular specific implementation where that definition of
"Real C" is relevant.

> It is
> sufficient for me that my implementation specifies a behaviour that is
> suitable for my use. Whatever behaviour observed with your implementation
> is immaterial to me.

You suggested "that this be incorporated into Lua's official source." So
this concerns more than just your implementation. That's the main reason
why I replied in the first place.

>
> Cheers,
> V.
>

Philipp

[Viacheslav Usov]

On Thu, Oct 22, 2015 at 1:09 AM, Philipp Janda <[hidden email]> wrote:

[...]

 

You suggested "that this be incorporated into Lua's official source." So this concerns more than just your implementation.

Fair enough. So let's look at our options. Today, Lua's source code, out of the box, supports three major modes:

Pure C: The Lua library, the host, and any other library interacting with Lua are written in standard compliant C and are built as such.

Pure C++: The Lua library, the host, and any other library interacting with Lua are written in standard compliant C++ and are built as such.

Implementation specific: none of the above.

Now I have a particular need: my host is written in and built as standard compliant C++, I want to use Lua, and I want them to interoperate with Lua libraries written in standard compliant C (and built as such, for reasons indicated in the original post). Requirements re-formulated:

Hybrid mode: Lua libraries, written in and built as standard compliant C or C++, shall all interoperate with Lua (no requirement as to whether it is C or C++).

There have been suggestions in the thread to relax this requirement by replacing "standard compliant C++" with, essentially, "crippled C++". That is not being considered, but it may be further required that C++ libraries shall not throw exceptions at Lua except by calling lua_error().

 

The pure C and pure C++ options do not work by definition. So I am left only with the "implementation specific" option given Lua's current source code, whether you like it or not.

But even if I am in the "implementation specific" area, I still need to meet my requirements. Unless I have missed something big, the only reasonable course of action here is to build Lua as standard compliant C++, which will interoperate properly with the C++ libraries, but that will not interoperate with C libraries at all because of the language linkage, as explained originally. Lua's current source, out of the box, makes this impossible regardless of my implementation. This is the problem that I suggested in the original post should be fixed.

The change as suggested originally, however, has a drawback: it will be applied unconditionally in the pure C++ mode, making it implementation specific. This could be controlled by another define, say CPP_HYBRID, with the code modified as follows:

#if defined(__cplusplus) && defined(CPP_HYBRID)

#define LUA_API  extern "C"

#define LUA_LINKAGE      extern "C"

#else

#define LUA_API  extern

#define LUA_LINKAGE

#endif

LUA_LINKAGE is to be injected into the definitions of the function types such as lua_CFunction as follows:

LUA_LINKAGE typedef int (*lua_CFunction) (lua_State *L);

So the hybrid mode, when desired and supported by the implementation, can be invoked via a single define, without affecting other modes. Comments?

Cheers,

V.

[Roberto Ierusalimschy]

> So the hybrid mode, when desired and supported by the implementation, can
> be invoked via a single define, without affecting other modes. Comments?

As its name implies, 'luaconf.h' has the specific purpose of
accomodating this kind of adaptations that are too particular for a
user. IF all you need to do is to change a few lines there, do change
those lines.

-- Roberto