Statically linked executables

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Statically linked executables

orenbenkiki
What is the recommended way to address the following use case:

* I am creating an executable which uses Lua to provide some scripting support.
* I am using `luaL_openlibs` to load the Lua standard libraries.
* I am not loading any other ("non-standard") C-modules.
* I do load pure some Lua modules though.
* I am statically linking this executable, completely avoiding the use of any shared libraries.

Simply adding `-static` in the linking flags, together with `-llua` and `-lm` doesn't work, since `luaL_openlibs()` calls `dlopen`...

Linking with `-static` and `-ldl` works, but gives a linker warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Which makes sense, I guess; if I load a dynamic library it will dynamically load glibc, and things will get messy if that glibc is different from the static glibc linked to the rest of the executable.

At any rate, even if this linker error was not an issue, the goal is to create a completely static executable, without any dynamic dependencies, so how to ensure `dlopen` works is not the issue I'm trying to solve.

The question is, instead, how do I avoid the call of `dlopen` in the first place?

I have seen some old threads discussing this issue (or, at least, something very close to it).
One direction was building Lua in a non-standard way (tweaking the Makefile, tweaking the source code...). For example:

    make clean ansi CFLAGS='-ULUA_DL_DLOPEN'

I haven't investigated this direction yet, because it implies the project can no longer be built with the "standard" Lua installation on the system. That means that just listing Lua as a dependency would not be sufficient for allowing the project to compile.

An obvious workaround is to include a complete Lua copy inside the project's source tree - that would be ~1M of additional sources...

Is this the recommended practice for this use case, or is there a better way?
Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

Luiz Henrique de Figueiredo
>     make clean ansi CFLAGS='-ULUA_DL_DLOPEN'

For Lua 5.3 this would simply be
        make clean generic

This is the same as compiling Lua with no flags (though the Makefile in
the tarball adds -DLUA_COMPAT_5_2, which you may not want).

If you want POSIX stuff do
        make clean posix

This is the same as compiling Lua with -DLUA_USE_POSIX.
It does not include dlopen.

Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

Roberto Ierusalimschy
In reply to this post by orenbenkiki
> [...]
>
> Simply adding `-static` in the linking flags, together with `-llua` and
> `-lm` doesn't work, since `luaL_openlibs()` calls `dlopen`...
>
> [...]
>
> The question is, instead, how do I avoid the call of `dlopen` in the first
> place?

Just a small correction, for the records: Lua never *calls* 'dlopen' by
itself. In particular, 'luaL_openlibs' does not call 'dlopen'. Lua
only calls 'dlopen' if the script calls 'require' (and 'require' finds
an appropriate file in the C path) or if the script calls
'package.loadlib'. (Of course, that does not elliminate the fact that
Lua has to be linked to the dlopen library with -ldl, therefore
causing the problems you are relating.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

orenbenkiki
> Just a small correction, for the records: Lua never *calls* 'dlopen' by
> itself. In particular, 'luaL_openlibs' does not call 'dlopen'. Lua
> only calls 'dlopen' if the script calls 'require' (and 'require' finds
> an appropriate file in the C path) or if the script calls
'> package.loadlib'. (Of course, that does not elliminate the fact that
> Lua has to be linked to the dlopen library with -ldl, therefore
> causing the problems you are relating.)

Interesting...

Does this mean that "in theory" I can ignore the linker warning, because, given I load only the standard libraries (and some pure Lua modules), `dlopen` will never actually be invoked? That would be scary even if it works...

Alternatively, I could replace `-ldl` with providing my own `dlopen`, which would `assert` that it shouldn't have been called, indicating that`require` is trying to load a strange C module, and kill the program (acceptable in this scenario). Presumably that would fix the linker error and be "safe". Does this workaround make sense?

Thanks,

Oren.
Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

Ross Berteig
In reply to this post by Roberto Ierusalimschy


On 6/27/2017 12:13 PM, Roberto Ierusalimschy wrote:

>> [...]
>>
>> Simply adding `-static` in the linking flags, together with `-llua` and
>> `-lm` doesn't work, since `luaL_openlibs()` calls `dlopen`...
>>
>> [...]
>>
>> The question is, instead, how do I avoid the call of `dlopen` in the first
>> place?
> Just a small correction, for the records: Lua never *calls* 'dlopen' by
> itself. In particular, 'luaL_openlibs' does not call 'dlopen'. Lua
> only calls 'dlopen' if the script calls 'require' (and 'require' finds
> an appropriate file in the C path) or if the script calls
> 'package.loadlib'. (Of course, that does not elliminate the fact that
> Lua has to be linked to the dlopen library with -ldl, therefore
> causing the problems you are relating.)
>
> -- Roberto
>

You could always provide a set of stubbed entry points instead of -ldl.
If your stubbed version of dlopen() returns NULL then require will give
up on all dynamically loaded C modules, but will otherwise continue to
work with pure Lua modules. You might want to also empty (or
significantly reduce) package.cpath.

Something like this (untested) might be enough:

#include <dlfcn.h>

void *dlopen(const char *filename, int flag) { return NULL; }
char *dlerror(void) { return "libdl not implemented"; }
void *dlsym(void *handle, const char *symbol) { return NULL; }
int dlclose(void *handle) { return 0;

Note that you can provide C modules that are also statically linked into
your program by listing them in package.preload where require will look
for them before attempting to use dlopen() to get them from a DLL or .so.

--
Ross Berteig                               [hidden email]
Cheshire Engineering Corp.           http://www.CheshireEng.com/
+1 626 303 1602


Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

orenbenkiki
I just created stubs similar to the above (I also added printing logging nice error messages, just in case...) and it works fine so far.

Stubbing `-ldl` would seem to be the recommended way to provide a fully statically linked executable with embedded Lua, then.

Many thanks!

Oren.
Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

Roberto Ierusalimschy
In reply to this post by orenbenkiki
> Alternatively, I could replace `-ldl` with providing my own `dlopen`, which
> would `assert` that it shouldn't have been called, indicating that`require`
> is trying to load a strange C module, and kill the program (acceptable in
> this scenario). Presumably that would fix the linker error and be "safe".
> Does this workaround make sense?

Yes. This is more or less what Lua itself does if you compile it without
defining LUA_USE_DLOPEN.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Statically linked executables

William Ahern
In reply to this post by orenbenkiki
On Tue, Jun 27, 2017 at 12:47:23PM -0700, orenbenkiki wrote:
> I just created stubs similar to the above (I also added printing logging nice
> error messages, just in case...) and it works fine so far.
>
> Stubbing `-ldl` would seem to be the recommended way to provide a fully
> statically linked executable with embedded Lua, then.
>
> Many thanks!

Recommended in so far as you have no control over how Lua is compiled.
Presumably you're using a system-managed Lua that was compiled with dynamic
module support. But if you control how Lua is compiled then the recommended
method would be as Luiz and Roberto describe.

FWIW, IMHO, most Linux distributions do a horrible job of packaging Lua.
Debian only recently recognized Lua 5.1, 5.2, and 5.3 as wholly distinct
languages, finally treating them similarly to Python 2 and 3. So Lua 5.1 is
installed as liblua5.1.so and liblua5.1.a with headers under lua5.1/, and
dependent Lua 5.1 modules require liblua5.1 or liblua5.1-dev. Moreover,
Debian Stretch changed the packaging policy so that installing a Lua library
package like lua-foo will install modules for Lua 5.1, 5.2, and 5.3 (if
possible), which brings them full circle--they both recognize that Lua 5.1,
5.2, and 5.3 are distinct, but actually attempt to address packaging
confusion of those unfamiliar with the details of Lua versioning.

Red Hat- and Fedora-based distributions still treat the Lua 5.x series as a
single language, and largely for that reason still require third-party
packages for Lua 5.2 and Lua 5.3. By their reasoning Lua 5.2 or 5.3 would
necessarily replace Lua 5.1 (i.e. the lua and lua-devel packages are
implicitly Lua 5.x), which they don't want to do because too many apps rely
on the Lua 5.1 API and won't be upgraded. The circular reasoning is
mind-boggling. But in any event they're committed to the notion (AFAICT,
primarily out of ignorance) that there should be a single lua package, and
so IMO Red Hat and Fedora distributions have a fundamentally broken Lua
packaging policy.

Also, inexplicably, on Red Hat- and Fedora-based distributions /usr/bin/lua
has an RPATH set to /usr/lib64, which breaks search ordering viz-a-viz
LD_LIBRARY_PATH. The proper way to add an explicit library search directory
for an interpreter is to set RUNPATH (not RPATH), as RUNPATH doesn't
override LD_LIBRARY_PATH. Both Perl and Python uses RUNPATH, even on Red Hat
and Fedora. This is more evidence regarding the brokeness of Red Hat's Lua
packaging, especially given how odd it is to hardcode /usr/lib64 in the
interpreter at all. Perl and Python set RUNPATH in order to make visible
non-standard directories (e.g. /usr/lib64/perl5/CORE). They don't have a
RUNPATH to /usr/lib64.

NetBSD arguably made the same mistake when importing Lua into their base
system as "-llua". Though at least they've moved to Lua 5.3. Perhaps they
knew what they were doing. Given that NetBSD base is one monolithic project
and that they already made the leap to 5.3 without changing library names it
might have been a knowing, intentional, and perhaps defensible choice.