Use of pointer comparisons to static data in Lua C code

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Use of pointer comparisons to static data in Lua C code

Tom Sutcliffe
Hi list,

An interesting problem just hit me when trying to embed Lua in a large and complicated Win32 application, which boils down to Lua making some unfortunate assumptions in its C implementation. They aren't unreasonable assumptions to make, but since they turn out to be pretty critical and I stumbled into a situation which violates them I'd be interested in what people think about it.

I have a Windows project which embeds Lua 5.3. Because of the structure of the project, Lua is built as a static library (.lib in Windows parlance) which is eventually linked into the final EXE. This EXE hosts the Lua runtime and loads scripts etc. So far so good.

Unfortunately I also want these scripts to (in some circumstances) be able to load additional native code libraries, which are build as DLLs which also (because Lua is built as a static library) end up including a copy of the Lua runtime. This is hardly ideal from a disk usage point of view, and I hope to eventually clean this up, but I figured that since Lua doesn't use any global data it won't actually matter providing I can guarantee that the 2 versions of the Lua code are identical and I don't mix unrelated lua_States.

However there are a couple of places where Lua uses pointer comparisons to static data - in luaO_nilobject and dummynode are the two I found after a quick search. Because there are now multiple copies of the Lua code and static data, one coming from the lib compiled into the host EXE and one from the lib in the extension DLL, the static data will have different pointer addresses depending on which code is checking it, which means that code which checks for "if (x == luaO_nilobject)" and similar will do the wrong thing leading to almost immediate crashes.

In an ideal world I would build Lua as a DLL, have both the EXE and the extension DLL link against the Lua DLL, then there'd only ever be 1 version of the static data as nature intended. However the size and structure of the project and where the Lua runtime fits into it makes this rather tricky without a massive refactor. Ideally I'd like for Lua to be able to tolerate my slightly weird setup as it does so successfully in all the other crazy situations I've thrown at it.

I had a go at modifying isdummy(n) so that it checked for the key type being nil rather than just comparing its pointer value against &dummynode_, which seemed like it might work, but the places luaO_nilobject is used looked a bit too complicated for me to try hacking with. Maybe something could be added to the global_State which would remain unique across multiple instances of the code, or something. I still haven't really decided whether what I'm doing is reasonable or not - on the one hand I'm running multiple copies of the Lua code at different addresses in a single process and passing lua_States between them, which is a weird thing to be doing. On the other hand, my mental model of how generally stateless a lua_State is makes me feel like this should work. Your thoughts appreciated!

Cheers,

Tom


Reply | Threaded
Open this post in threaded view
|

Re: Use of pointer comparisons to static data in Lua C code

Nagaev Boris
On Tue, Jul 12, 2016 at 1:41 PM, Tom Sutcliffe <[hidden email]> wrote:

> Hi list,
>
> An interesting problem just hit me when trying to embed Lua in a large and complicated Win32 application, which boils down to Lua making some unfortunate assumptions in its C implementation. They aren't unreasonable assumptions to make, but since they turn out to be pretty critical and I stumbled into a situation which violates them I'd be interested in what people think about it.
>
> I have a Windows project which embeds Lua 5.3. Because of the structure of the project, Lua is built as a static library (.lib in Windows parlance) which is eventually linked into the final EXE. This EXE hosts the Lua runtime and loads scripts etc. So far so good.
>
> Unfortunately I also want these scripts to (in some circumstances) be able to load additional native code libraries, which are build as DLLs which also (because Lua is built as a static library) end up including a copy of the Lua runtime. This is hardly ideal from a disk usage point of view, and I hope to eventually clean this up, but I figured that since Lua doesn't use any global data it won't actually matter providing I can guarantee that the 2 versions of the Lua code are identical and I don't mix unrelated lua_States.
>
> However there are a couple of places where Lua uses pointer comparisons to static data - in luaO_nilobject and dummynode are the two I found after a quick search. Because there are now multiple copies of the Lua code and static data, one coming from the lib compiled into the host EXE and one from the lib in the extension DLL, the static data will have different pointer addresses depending on which code is checking it, which means that code which checks for "if (x == luaO_nilobject)" and similar will do the wrong thing leading to almost immediate crashes.
>
> In an ideal world I would build Lua as a DLL, have both the EXE and the extension DLL link against the Lua DLL, then there'd only ever be 1 version of the static data as nature intended. However the size and structure of the project and where the Lua runtime fits into it makes this rather tricky without a massive refactor. Ideally I'd like for Lua to be able to tolerate my slightly weird setup as it does so successfully in all the other crazy situations I've thrown at it.
>
> I had a go at modifying isdummy(n) so that it checked for the key type being nil rather than just comparing its pointer value against &dummynode_, which seemed like it might work, but the places luaO_nilobject is used looked a bit too complicated for me to try hacking with. Maybe something could be added to the global_State which would remain unique across multiple instances of the code, or something. I still haven't really decided whether what I'm doing is reasonable or not - on the one hand I'm running multiple copies of the Lua code at different addresses in a single process and passing lua_States between them, which is a weird thing to be doing. On the other hand, my mental model of how generally stateless a lua_State is makes me feel like this should work. Your thoughts appreciated!
>
> Cheers,
>
> Tom
>
>

Hi!

> In an ideal world I would build Lua as a DLL, have both the EXE and the extension DLL link against the Lua DLL, then there'd only ever be 1 version of the static data as nature intended.

You can compile Lua twice:
  - as DLL and link Lua C modules against that DLL,
  - then compile Lua statically as a part of your main executable.


--


Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

Re: Use of pointer comparisons to static data in Lua C code

Philipp Janda
Am 12.07.2016 um 15:00 schröbte Nagaev Boris:

> On Tue, Jul 12, 2016 at 1:41 PM, Tom Sutcliffe <[hidden email]> wrote:
>> Hi list,
>>
>
> Hi!
>
>> In an ideal world I would build Lua as a DLL, have both the EXE and the extension DLL link against the Lua DLL, then there'd only ever be 1 version of the static data as nature intended.
>
> You can compile Lua twice:
>    - as DLL and link Lua C modules against that DLL,
>    - then compile Lua statically as a part of your main executable.

Well, and the above mentioned DLL should be a proxy DLL. See here[1]
(also look at the links at the bottom).

AFAIK, it's also possible to export the Lua API from the main executable
and link the extension modules directly against said executable instead
of a DLL, but I've never done that myself.


Philipp

   [1]: http://lua-users.org/wiki/LuaProxyDll



Reply | Threaded
Open this post in threaded view
|

Re: Use of pointer comparisons to static data in Lua C code

steve donovan
On Tue, Jul 12, 2016 at 4:35 PM, Philipp Janda <[hidden email]> wrote:
> AFAIK, it's also possible to export the Lua API from the main executable and
> link the extension modules directly against said executable instead of a
> DLL, but I've never done that myself.

This is typically how SciTE is compiled on Windows, and it works fine.
The issue with Windows of course is that the extension DLLs need to
link against a specific exe/dll; Unix .so files are much more
promiscuous ;)

Reply | Threaded
Open this post in threaded view
|

Re: Use of pointer comparisons to static data in Lua C code

Tom Sutcliffe
In reply to this post by Philipp Janda
On 12 Jul 2016, at 3:35 pm, Philipp Janda <[hidden email]> wrote:

>
> Am 12.07.2016 um 15:00 schröbte Nagaev Boris:
>> On Tue, Jul 12, 2016 at 1:41 PM, Tom Sutcliffe <[hidden email]> wrote:
>>> In an ideal world I would build Lua as a DLL, have both the EXE and the extension DLL link against the Lua DLL, then there'd only ever be 1 version of the static data as nature intended.
>>
>> You can compile Lua twice:
>>   - as DLL and link Lua C modules against that DLL,
>>   - then compile Lua statically as a part of your main executable.
>
> Well, and the above mentioned DLL should be a proxy DLL. See here[1] (also look at the links at the bottom).

Interesting, thanks! It hadn't occurred to me to go down that route to avoid the duplication of static data in the first place - I'll definitely take a look through the various LuaProxyDlls to see which can be most easily shoehorned into my toolchain :)

Assuming I were to continue down the (increasingly unwise-looking) route of having the extensions also statically linking in lua.lib, rather than using a proxy DLL, does anyone have a view on the validity of passing lua_States between multiple copies of the Lua code? I only ask because my project unfortunately has a rat's nest of dependencies, subprojects and generated build files which might make it quite difficult to make the extension DLLs link against a proxy DLL instead of the lua.lib indirectly used elsewhere. The fun of adding new dependencies into a project whose makefiles/buildscripts/etc are approximately the same size as the entire Lua source...

Cheers,

Tom