Using Lua in dynamic modules with static linking

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Lua in dynamic modules with static linking

Chris Smith
Hi all,

I have read all that I can find on building and linking Lua libraries, but I’m still a bit unsure if what I want to do will work.  I’d appreciate it if someone can clarify things for me.  Here is my situation:

1. I have an application that uses Lua.
2. The application needs to dynamically load extension modules (note these are _not_ Lua extension modules, but for the application itself).
3. The extension module also uses Lua for its own purposes.
4. The use of Lua in the application and module is completely isolated — no sharing of any sort.

I think the best way of doing this is to statically link both the application and the module against the Lua libraries.  I understand that means the Lua code will be duplicated in both application and module, but that is actually a good thing for me since it means the application and module can have mismatched Lua libraries.  Is this correct, or will I encounter problems?  I need this to work on OS X and Windows platforms.

Many thanks,
Chris

Chris Smith <[hidden email]>



Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

William Ahern
On Tue, Feb 26, 2019 at 12:42:02PM +0000, Chris Smith wrote:

> Hi all,
>
> I have read all that I can find on building and linking Lua libraries, but
> I’m still a bit unsure if what I want to do will work. I’d appreciate it
> if someone can clarify things for me. Here is my situation:
>
> 1. I have an application that uses Lua.
> 2. The application needs to dynamically load extension modules (note these
> are _not_ Lua extension modules, but for the application itself).
> 3. The extension module also uses Lua for its own purposes.
> 4. The use of Lua in the application and module is completely isolated —
> no sharing of any sort.
>
> I think the best way of doing this is to statically link both the
> application and the module against the Lua libraries. I understand that
> means the Lua code will be duplicated in both application and module, but
> that is actually a good thing for me since it means the application and
> module can have mismatched Lua libraries. Is this correct, or will I
> encounter problems? I need this to work on OS X and Windows platforms.

I've done this on OS X (Mach-O) and Linux (ELF). Only the extension module
needs to be statically linked to Lua. The main application can be
dynamically linked or even just be a stock lua binary. But there are two
caveats as static linking and static compiling are ambiguous phrases on
modern systems--the details matter.

1) What really matters is that the extension module's Lua symbols cannot be
visible. It's not sufficient for symbol definition to exist in the same
shared object. If they are visible then at runtime the dynamic linker will
link the main application's Lua symbols (technically, the first global
symbol loaded from any shared object, such as liblua.so) into the extension
module's call sites rather than the ones defined in the module. This
behavior is what makes LD_PRELOAD work, but in most situations it's the
opposite behavior of what you'd expect. There are at least two (maybe three)
ways to hide the symbols:

1.a) #include the Lua source into the same C (or C++) source file as the
extension module and set LUA_API to "static". This is the obvious,
standard's compliant approach. However, this can be unwieldy (especially for
C++) and not great for incremental builds.

1.b) Use GCC's or clang's -fvisibility=hidden flag when building Lua. (Or
try setting LUA_API to '__attribute__((visibility ("hudden")))'). This has
the same effect as 1.a (truly "static" linking) but you can still build and
cross-link separate .o object files.

1.c) I think Mach-O has the ability to tweak symbol precedence to favor
symbols defined in same shared object. I haven't explored this, though.

Whichever way you choose confirm using the nm utility. On OS X you can do
`nm -g extension-module.so` to see the list of global symbols (defined and
undefined). You shouldn't see any Lua symbols.

2) The extension module's Lua runtime won't be able support dynamic Lua
modules--its Lua symbols won't be visible, but even if they were they would
have lower precedence than other Lua symbols previously loaded. You'll need
compile any Lua modules directly into the extension module and explicitly
load them--e.g. with luaL_requiref.

I'm not very familiar with Visual Studio so don't know how to do things
there. If Visual Studio doesn't support something like -fvisibility=hidden
you could try clang.

Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Chris Smith

> On 26 Feb 2019, at 19:45, William Ahern <[hidden email]> wrote:
>
>> On Tue, Feb 26, 2019 at 12:42:02PM +0000, Chris Smith wrote:
<snip>

>> I think the best way of doing this is to statically link both the
>> application and the module against the Lua libraries. I understand that
>> means the Lua code will be duplicated in both application and module, but
>> that is actually a good thing for me since it means the application and
>> module can have mismatched Lua libraries. Is this correct, or will I
>> encounter problems? I need this to work on OS X and Windows platforms.
>
> I've done this on OS X (Mach-O) and Linux (ELF). Only the extension module
> needs to be statically linked to Lua. The main application can be
> dynamically linked or even just be a stock lua binary. But there are two
> caveats as static linking and static compiling are ambiguous phrases on
> modern systems--the details matter.

If I understand correctly, the two caveats you detail only apply because the main application is dynamically linked to the Lua library? If I statically linked both application and module to their respective Lua libraries then I wouldn’t expect any issues, correct?

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Andrew Gierth
In reply to this post by William Ahern
>>>>> "William" == William Ahern <[hidden email]> writes:

 William> There are at least two (maybe three) ways to hide the symbols:

This is an issue we've had to deal with in PostgreSQL, so we have
working examples of methods to deal with it on a fairly wide range of
platforms. (In our case, we have some functions that have "frontend" and
"backend" versions with the same name, where the "backend" version is
linked into the postgres server binary, but that binary sometimes needs
to load dynamic modules that contain frontend code and which have been
linked against the frontend version.)

We actually went so far as to include an explicit "link canary" file
that we can test to ensure that functions have not been mis-linked to
the wrong versions.

You can see the compiler options PostgreSQL uses for building shared
libraries (including dynamically loaded modules) for all our platforms
except Windows here:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/Makefile.shlib

(for Windows there's a bunch of scripts under src/tools/msvc)

 William> 1.a) #include the Lua source into the same C (or C++) source
 William> file as the extension module and set LUA_API to "static". This
 William> is the obvious, standard's compliant approach. However, this
 William> can be unwieldy (especially for C++) and not great for
 William> incremental builds.

This would have been impractical for us.

 William> 1.b) Use GCC's or clang's -fvisibility=hidden flag when
 William> building Lua. (Or try setting LUA_API to
 William> '__attribute__((visibility ("hudden")))'). This has the same
 William> effect as 1.a (truly "static" linking) but you can still build
 William> and cross-link separate .o object files.

A better variation on this is to link your dynamic libraries with an
explicit exports / version-script file that defines an exported symbol
API (optionally with a version) and hides all other symbols.

We currently use export files on AIX , MacOS, *BSD, Linux, and Windows.
(On Windows, and I believe also on AIX, the export file is required.) We
would likely use it on Solaris(-derivatives) and other ELF platforms if
anyone cared enough about them to contribute a tested makefile patch.

For Linux and *BSD the version script can just look something like:

{ global:
    func1;
    func2;
    func3;
  local: *;
};

and link with -Wl,--version-script=filename

For MacOS it's a plain list of symbols with _ prepended, and the compile
option is -exported_symbols_list filename

 William> 1.c) I think Mach-O has the ability to tweak symbol precedence
 William> to favor symbols defined in same shared object. I haven't
 William> explored this, though.

I don't know about MacOS, but a fair range of platforms support the
-Bsymbolic linker option, which means (when linking a shared library)
"resolve symbols within the library to their definitions now, rather
than at runtime". This is the option we use on HP-UX and Solaris.

 William> I'm not very familiar with Visual Studio so don't know how to
 William> do things there. If Visual Studio doesn't support something
 William> like -fvisibility=hidden you could try clang.

Windows DLL symbol resolution is not like ELF; one of the big
differences is that DLLs export only the symbols you explicitly tell
them to export (in the .DEF file) and nothing else.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Andrew Gierth
In reply to this post by Chris Smith
>>>>> "Chris" == Chris Smith <[hidden email]> writes:

 Chris> If I understand correctly, the two caveats you detail only apply
 Chris> because the main application is dynamically linked to the Lua
 Chris> library? If I statically linked both application and module to
 Chris> their respective Lua libraries then I wouldn’t expect any
 Chris> issues, correct?

Incorrect at least on ELF platforms (I don't know about MacOS), and this
is exactly the problem that bit us in PostgreSQL.

Our scenario went like this: libpgcommon.a (a static library) comes in
multiple versions: libpgcommon.a is for the frontend, libpgcommon_srv.a
is for the backend. They define the same functions but with different
implementations.

The main application, "postgres", is linked statically to
libpgcommon_srv.a. "postgres" can (on request) dynamically load the
module "postgres_fdw.so" with dlopen; postgres_fdw.so is dynamically
linked against libpq.so, which is statically linked against
libpgcommon.a.

What we found is that on platforms where we weren't using a version
script to hide symbols, libpq.so's calls to functions in libpgcommon
were actually being resolved not to the ones statically linked into
libpq.so, but to the copies from the "postgres" binary which of course
came from libpgcommon_srv.a; we had crashes as a result. See e.g.
https://www.postgresql.org/message-id/flat/153626613985.23143.4743626885618266803%40wrigleys.postgresql.org

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Viacheslav Usov
In reply to this post by Chris Smith
On Tue, Feb 26, 2019 at 1:42 PM Chris Smith <[hidden email]> wrote:

> I think the best way of doing this is to statically link both the application and the module against the Lua libraries.

As said by the others, there is a way to make this approach work. What you should also consider is the eventual need to use non-pure-Lua libraries. When both the main executable and the dynamic modules link Lua statically, they will have to link those libraries statically, too. Or, if you find a robust way to link Lua statically yet load non-pure-Lua libraries dynamically, I would very much appreciate having a look at your achievement :)

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Chris Smith
In reply to this post by Andrew Gierth
> On 27 Feb 2019, at 01:05, Andrew Gierth <[hidden email]> wrote:
>
>>>>>> "Chris" == Chris Smith <[hidden email]> writes:
>
> Chris> If I understand correctly, the two caveats you detail only apply
> Chris> because the main application is dynamically linked to the Lua
> Chris> library? If I statically linked both application and module to
> Chris> their respective Lua libraries then I wouldn’t expect any
> Chris> issues, correct?
>
> Incorrect at least on ELF platforms (I don't know about MacOS), and this
> is exactly the problem that bit us in PostgreSQL.
>
> Our scenario went like this: libpgcommon.a (a static library) comes in
> multiple versions: libpgcommon.a is for the frontend, libpgcommon_srv.a
> is for the backend. They define the same functions but with different
> implementations.
>
> The main application, "postgres", is linked statically to
> libpgcommon_srv.a. "postgres" can (on request) dynamically load the
> module "postgres_fdw.so" with dlopen; postgres_fdw.so is dynamically
> linked against libpq.so, which is statically linked against
> libpgcommon.a.
>
> What we found is that on platforms where we weren't using a version
> script to hide symbols, libpq.so's calls to functions in libpgcommon
> were actually being resolved not to the ones statically linked into
> libpq.so, but to the copies from the "postgres" binary which of course
> came from libpgcommon_srv.a; we had crashes as a result. See e.g.
> https://www.postgresql.org/message-id/flat/153626613985.23143.4743626885618266803%40wrigleys.postgresql.org


I admit I didn’t fully believe this, but I wrote a small test and mostly confirmed that you are right.  As I write this I realise that I didn’t recreate your situation exactly in my test and missed one level of indirection; my test is effectively this:

  (app —static—> lib_a) —dlopen—> (module —static—> lib_b)

Where lib_a and lib_b have identical function interfaces.  In this situation, for me, it works perfectly: lib_a provides the functions for app and lib_b provides the functions for module.  However, if I replace ‘dlopen’ with ’shared’, then lib_a provides everything.  (On OS X, shared also works perfectly.). Fortunately, the working dlopen scenario is what I need, so this may work for me (if my test isn’t just a fluke).

I feel like I’m missing something though, because this should mean that shared libraries are almost unusable.  Apps should be going haywire all over the place due to clashing function names.

Regards,
Chris

Chris Smith <[hidden email]>


Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Chris Smith
In reply to this post by Viacheslav Usov
On 27 Feb 2019, at 09:31, Viacheslav Usov <[hidden email]> wrote:

On Tue, Feb 26, 2019 at 1:42 PM Chris Smith <[hidden email]> wrote:

> I think the best way of doing this is to statically link both the application and the module against the Lua libraries.

As said by the others, there is a way to make this approach work. What you should also consider is the eventual need to use non-pure-Lua libraries. When both the main executable and the dynamic modules link Lua statically, they will have to link those libraries statically, too. Or, if you find a robust way to link Lua statically yet load non-pure-Lua libraries dynamically, I would very much appreciate having a look at your achievement :)

For my use case, Lua will never have to load binary libraries.  If something can’t be done in pure Lua then my embedding application will implement it and inject a C function into the environment.

However, considering the problem anyway, I assume the issue is that if Lua is linked statically, then a binary module won’t have access to the Lua API?  (There will be no reverse bindings, for want of a better term?)  In that case, my first inclination would be to use traditional ‘interface’ semantics: pass a structure to the module on initialisation that contains the Lua state and pointers to the Lua API functions.  That would provide a cleaner separation between module and application and allow for backwards compatibility if you update the Lua version used by application.  This does imply that you have control over the modules as well, though — it wouldn’t work for a random third-party module.

Regards,
Chris

Chris Smith <[hidden email]>


Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

Viacheslav Usov
On Wed, Feb 27, 2019 at 6:04 PM Chris Smith <[hidden email]> wrote:

> In that case, my first inclination would be to use traditional ‘interface’ semantics: pass a structure to the module on initialisation that contains the Lua state and pointers to the Lua API functions.

This much is clear. Moreover, this can be done with some pre-processor magic so that both the host and loadable libs could be just rebuilt as if against the stock Lua library, with the host ending up linked statically while the libs linking dynamically to the hosted implementation. I have discovered a truly remarkable proof of this theorem which this margin is too small to contain.

Cheers,
V.
 
Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

William Ahern
In reply to this post by Viacheslav Usov
On Wed, Feb 27, 2019 at 10:31:17AM +0100, Viacheslav Usov wrote:

> On Tue, Feb 26, 2019 at 1:42 PM Chris Smith <[hidden email]> wrote:
>
> > I think the best way of doing this is to statically link both the
> application and the module against the Lua libraries.
>
> As said by the others, there is a way to make this approach work. What you
> should also consider is the eventual need to use non-pure-Lua libraries.
> When both the main executable and the dynamic modules link Lua statically,
> they will have to link those libraries statically, too. Or, if you find a
> robust way to link Lua statically yet load non-pure-Lua libraries
> dynamically, I would very much appreciate having a look at your achievement
> :)

Solaris and Linux/glibc have dlmopen, which loads a shared object in a
unique symbol namespace. This permits loading a module with its own exported
Lua symbols (which could have been dynamically loaded as a shared library
dependency), and it could dynamically load more modules, including Lua
modules. I think you'd need to ensure that code within this namespace
consistently used dlmopen with the correct identifier, perhaps by patching
Lua or interposing dlopen.

I've never tried dlmopen so can't testify to its reliability.

Reply | Threaded
Open this post in threaded view
|

Re: Using Lua in dynamic modules with static linking

William Ahern
In reply to this post by Chris Smith
On Wed, Feb 27, 2019 at 05:04:19PM +0000, Chris Smith wrote:

> > On 27 Feb 2019, at 09:31, Viacheslav Usov <[hidden email]> wrote:
> >
> > On Tue, Feb 26, 2019 at 1:42 PM Chris Smith <[hidden email] <mailto:[hidden email]>> wrote:
> >
> > > I think the best way of doing this is to statically link both the application and the module against the Lua libraries.
> >
> > As said by the others, there is a way to make this approach work. What you should also consider is the eventual need to use non-pure-Lua libraries. When both the main executable and the dynamic modules link Lua statically, they will have to link those libraries statically, too. Or, if you find a robust way to link Lua statically yet load non-pure-Lua libraries dynamically, I would very much appreciate having a look at your achievement :)
>
>
> For my use case, Lua will never have to load binary libraries.  If something can’t be done in pure Lua then my embedding application will implement it and inject a C function into the environment.
>
> However, considering the problem anyway, I assume the issue is that if Lua
> is linked statically, then a binary module won’t have access to the Lua
> API? (There will be no reverse bindings, for want of a better term?)  In
> that case, my first inclination would be to use traditional ‘interface’
> semantics: pass a structure to the module on initialisation that contains
> the Lua state and pointers to the Lua API functions. That would provide a
> cleaner separation between module and application and allow for backwards
> compatibility if you update the Lua version used by application. This does
> imply that you have control over the modules as well, though — it wouldn’t
> work for a random third-party module.

And if you have control over the module itself it would be easier to just
tweak the build and statically link it. The Debian Lua policy requires all
binary Lua module packages to build .a archives precisely so you can
statically link Lua modules.

Unfortunately, too many Lua modules unintentionally export their internal
routines (i.e. don't use static scope or don't provide the ability to
control visibility with macros like FOO_API and FOO_LOCAL). Some
distributions' Lua packages also miscompile liblua.a (e.g. Red Hat's) so
that it's useless for use in a multiple Lua library context. So you're stuck
building from source, patching the code or using linker scripts to control
scope.

If you write Lua C modules (or any C module or library), it's good practice
to prefix FOO_API (or FOO_PUBLIC) to your public interfaces and FOO_LOCAL to
non-static scoped internal interfaces. Even on systems that support a linker
script to enumerate exported interfaces, it still provides useful
flexibility to downstream packagers and builders.

See my post elsethread regarding the dlmopen alternative.