How to find out which modules a binary Lua .so declares?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to find out which modules a binary Lua .so declares?

Hisham
So, here's an interesting problem: since multiple Lua modules can be
stored in a single .so file (looking at you, LuaSec!), one can't
really tell which Lua modules a rock contains by scanning the
filesystem.

This is currently causing `luarocks show` display incomplete data, and
has also caused bugs because other LR commands were naively assuming a
filesystem-to-modules correspondence. I do keep a list of module files
in rock_manifest, so those bugs are fixable.

But for luarocks.loader (a package searcher that resolves the right
version of a module based on LuaRocks metadata, so that if you require
a module A version 1 and it depends on module B version 3, it will
pick the right one even if you have multiple B's installed in your
system), the only way LuaRocks can resolve a require()d module is if
it knows which rock it belongs to. So missing this data means that
luarocks.loader is currently unable to resolve the LuaSec binary
modules.

I guess the only way to build complete module data will be to use nm
on Unix (and dumpbin on Windows I guess), at `luarocks build` time,
and search for "luaopen_*" symbols.

But then how to know if luaopen_foo_bar_bla means
require("foo_bar_bla"), or require("foo.bar.bla"), or
require("foo.bar_bla"), or require("foo_bar.bla")?

AFAICT, the right answer is "all of the above", so my current plan is
to "normalize" on one of those (say, dots) when storing the metadata
(i.e., "luaopen_foo_bar" means "foo.bar"), and have luarocks.loader
"normalize" the require() input when searching for .so files, in case
a non-"normalized" .lua filename cannot be found. Does that make
sense?

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: How to find out which modules a binary Lua .so declares?

Marc Balmer


> Am 31.08.2018 um 13:29 schrieb Hisham <[hidden email]>:
>
> So, here's an interesting problem: since multiple Lua modules can be
> stored in a single .so file (looking at you, LuaSec!), one can't
> really tell which Lua modules a rock contains by scanning the
> filesystem.
>
> This is currently causing `luarocks show` display incomplete data, and
> has also caused bugs because other LR commands were naively assuming a
> filesystem-to-modules correspondence. I do keep a list of module files
> in rock_manifest, so those bugs are fixable.
>
> But for luarocks.loader (a package searcher that resolves the right
> version of a module based on LuaRocks metadata, so that if you require
> a module A version 1 and it depends on module B version 3, it will
> pick the right one even if you have multiple B's installed in your
> system), the only way LuaRocks can resolve a require()d module is if
> it knows which rock it belongs to. So missing this data means that
> luarocks.loader is currently unable to resolve the LuaSec binary
> modules.
>
> I guess the only way to build complete module data will be to use nm
> on Unix (and dumpbin on Windows I guess), at `luarocks build` time,
> and search for "luaopen_*" symbols.
>
> But then how to know if luaopen_foo_bar_bla means
> require("foo_bar_bla"), or require("foo.bar.bla"), or
> require("foo.bar_bla"), or require("foo_bar.bla")?
>
> AFAICT, the right answer is "all of the above", so my current plan is
> to "normalize" on one of those (say, dots) when storing the metadata
> (i.e., "luaopen_foo_bar" means "foo.bar"), and have luarocks.loader
> "normalize" the require() input when searching for .so files, in case
> a non-"normalized" .lua filename cannot be found. Does that make
> sense?

On Linux, you could use dl_iterate_phdr.  But note that this is not portable.

- Marc


Reply | Threaded
Open this post in threaded view
|

Re: How to find out which modules a binary Lua .so declares?

Dirk Laurie-2
In reply to this post by Hisham
Op Vr., 31 Aug. 2018 om 13:30 het Hisham <[hidden email]> geskryf:

>
> So, here's an interesting problem: since multiple Lua modules can be
> stored in a single .so file (looking at you, LuaSec!), one can't
> really tell which Lua modules a rock contains by scanning the
> filesystem.
>
> This is currently causing `luarocks show` display incomplete data, and
> has also caused bugs because other LR commands were naively assuming a
> filesystem-to-modules correspondence. I do keep a list of module files
> in rock_manifest, so those bugs are fixable.
>
> But for luarocks.loader (a package searcher that resolves the right
> version of a module based on LuaRocks metadata, so that if you require
> a module A version 1 and it depends on module B version 3, it will
> pick the right one even if you have multiple B's installed in your
> system), the only way LuaRocks can resolve a require()d module is if
> it knows which rock it belongs to. So missing this data means that
> luarocks.loader is currently unable to resolve the LuaSec binary
> modules.
>
> I guess the only way to build complete module data will be to use nm
> on Unix (and dumpbin on Windows I guess), at `luarocks build` time,
> and search for "luaopen_*" symbols.
>
> But then how to know if luaopen_foo_bar_bla means
> require("foo_bar_bla"), or require("foo.bar.bla"), or
> require("foo.bar_bla"), or require("foo_bar.bla")?
>
> AFAICT, the right answer is "all of the above", so my current plan is
> to "normalize" on one of those (say, dots) when storing the metadata
> (i.e., "luaopen_foo_bar" means "foo.bar"), and have luarocks.loader
> "normalize" the require() input when searching for .so files, in case
> a non-"normalized" .lua filename cannot be found. Does that make
> sense?

Would it be cheating to run a little Lua program that requires the
"main" module and then snoops around in package.loaded and/or
debug.getregistry() to see what else came with it?

I find some code I wrote too long ago to understand fully now,
with a comment "These functions rely on undocumented features
of Lua 5.2.x (x>0) and Lua 5.3.0." So I suppose it is a dodgy
thing to do :-(

Reply | Threaded
Open this post in threaded view
|

Re: How to find out which modules a binary Lua .so declares?

Hisham
On Fri, 31 Aug 2018 at 14:22, Dirk Laurie <[hidden email]> wrote:

>
> Op Vr., 31 Aug. 2018 om 13:30 het Hisham <[hidden email]> geskryf:
> >
> > So, here's an interesting problem: since multiple Lua modules can be
> > stored in a single .so file (looking at you, LuaSec!), one can't
> > really tell which Lua modules a rock contains by scanning the
> > filesystem.
> >
> > This is currently causing `luarocks show` display incomplete data, and
> > has also caused bugs because other LR commands were naively assuming a
> > filesystem-to-modules correspondence. I do keep a list of module files
> > in rock_manifest, so those bugs are fixable.
> >
> > But for luarocks.loader (a package searcher that resolves the right
> > version of a module based on LuaRocks metadata, so that if you require
> > a module A version 1 and it depends on module B version 3, it will
> > pick the right one even if you have multiple B's installed in your
> > system), the only way LuaRocks can resolve a require()d module is if
> > it knows which rock it belongs to. So missing this data means that
> > luarocks.loader is currently unable to resolve the LuaSec binary
> > modules.
> >
> > I guess the only way to build complete module data will be to use nm
> > on Unix (and dumpbin on Windows I guess), at `luarocks build` time,
> > and search for "luaopen_*" symbols.
> >
> > But then how to know if luaopen_foo_bar_bla means
> > require("foo_bar_bla"), or require("foo.bar.bla"), or
> > require("foo.bar_bla"), or require("foo_bar.bla")?
> >
> > AFAICT, the right answer is "all of the above", so my current plan is
> > to "normalize" on one of those (say, dots) when storing the metadata
> > (i.e., "luaopen_foo_bar" means "foo.bar"), and have luarocks.loader
> > "normalize" the require() input when searching for .so files, in case
> > a non-"normalized" .lua filename cannot be found. Does that make
> > sense?
>
> Would it be cheating to run a little Lua program that requires the
> "main" module and then snoops around in package.loaded and/or
> debug.getregistry() to see what else came with it?

It would, as there is no general way to know what is the "main" Lua
module — in LuaSec, for example there's ssl.so which exports ssl.core
and others, but require("ssl") loads ssl.lua, and there's no guarantee
it will luaopen all modules at startup. Also, requiring modules may
cause side-effects, so it's not something that LuaRocks can blindly
do.

At this point, if we were to ask the rock author for help to identify
the main module or something to that effect, we could just follow
Enrique's suggestion here:

"Honestly, I would just add a "list your modules here if they don't
match the filesystem" field to the rockspec and from that moment on
consider it a bug by the libraries that don't use it when they need
to."
-- https://twitter.com/otikik/status/1035489554621517824

This is sounding more and more like a good idea, since doing it
automagically via catching symbols would still wouldn't resolve the
"how do I translate a luaopen_foo_bar name to the intended module
name", and I haven't bumped into that many modules that do this (I
think cqueues also bundles multiple modules into a single .so), and it
doesn't sound like an unreasonable thing to ask from rockspec authors.

One slight complication is that binary module filenames change
according to systems (ie, ssl.so vs ssl.dll) so it would need some
agnostic notation, like

exports = {
   ["ssl"] = { "ssl.core", "ssl.config", "ssl.context", "ssl.x509" },
   ["parentdir.binarymod"] = { "parentdir.binarymod.foo",
"parentdir.binarymod.bar" },
}

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: How to find out which modules a binary Lua .so declares?

Pierre Chapuis
On Fri, Aug 31, 2018, at 14:38, Hisham wrote:

> At this point, if we were to ask the rock author for help to identify
> the main module or something to that effect, we could just follow
> Enrique's suggestion here:
>
> "Honestly, I would just add a "list your modules here if they don't
> match the filesystem" field to the rockspec and from that moment on
> consider it a bug by the libraries that don't use it when they need
> to."
> -- https://twitter.com/otikik/status/1035489554621517824

I agree. Let us not live with magic just to simplify the configuration files
for a few modules which, if they have this issue, are probably already
pretty complex.

--
Pierre Chapuis