Modules with standalone main program

classic Classic list List threaded Threaded
60 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Albert Chan
On Feb 24, 2018, at 3:30 PM, Andrew Gierth <[hidden email]> wrote:

> albertmcchan> I now know why require "mod" set package.loaded.mod to
> albertmcchan> some userdata
>
> 5.1 does this, but 5.2+ do not.
>
> --
> Andrew.

I just read require implementation (in lua) for lua 5.1 and 5.2

https://github.com/pygy/require.lua/blob/master/require.lua

lua 5.2 does not use sentinel, and does less error checking.
From inside mod.lua, there is no way to know if it were "required"

I guess this is a lua 5.1 "feature":

-- lua 5.1 test for require "mod" inside mod.lua

if package.loaded.mod then return mod_func end



Reply | Threaded
Open this post in threaded view
|

Renaming Lua modules (was Re: Modules with standalone main program)

Sean Conner
In reply to this post by Soni "They/Them" L.
It was thus said that the Great Soni They/Them L. once stated:
>
> Why does nobody support renaming modules, it's 2018 :(

  Please rename a module written in C and get it to run, without
recompilation.  Then report your findings back here.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Paul K-2
In reply to this post by Dirk Laurie-2
Hi Dirk,

> I.e. "doflile" is treated just like "require'.
> I want "require" to return and all other ways of running the file to continue.

It wasn't clear from your question that you want to distinguish
between require and dofile calls, but it shouldn't be too difficult to
add:

if pcall(debug.getlocal, 4, 1) then
  if select('#', ...) == 0 then
    print("dofile package")
  else
    print("require package")
  end
else
  print("main script")
end

Paul.

On Sat, Feb 24, 2018 at 12:37 PM, Dirk Laurie <[hidden email]> wrote:

> 2018-02-24 21:46 GMT+02:00 Paul K <[hidden email]>:
>>> Unfortunately, the conclusion was that there seems to be no fail-proof method.
>
> I don't need a fail-proof method. I need a method that works for a
> one-file module of which I know the name by which it will be required.
>
>> I already suggested a method earlier that works with Lua 5.1+ and
>> handles command line parameters as well as require/dofile calls:
>>
>> if pcall(debug.getlocal, 4, 1) then
>>   print("package")
>> else
>>   print("main script")
>> end
>
> It does not do what I want.
>
> $ lua
> Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
>> dofile"lmodules/mod.lua"
> package
>
> I.e. "doflile" is treated just like "require'.
>
> I want "require" to return and all other ways of running the file to continue.
>

Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Paul K-2
In reply to this post by Albert Chan
>> if pcall(debug.getlocal, 4, 1) then
> can you elaborate what 4 and 1 mean ?

It checks if there is anything in the 1st variable at the 4th level,
which would be the caller of the current module (if required) or
nothing in the case of the main script.

> how does it know we are testing mod.lua being required (or not) ?
> and not testing for something else ?
> ("mod" is never mentioned in above test)

It doesn't care about the name of the module; it only checks the stack.

> the stand-alone program might also require some other modules, which
> also might require others ... does above work all the time ?

Yes.

Paul.

Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Albert Chan
In reply to this post by Paul K-2
On Feb 24, 2018, at 6:03 PM, Paul K <[hidden email]> wrote:

> Hi Dirk,
>
>> I.e. "doflile" is treated just like "require'.
>> I want "require" to return and all other ways of running the file to continue.
>
> It wasn't clear from your question that you want to distinguish
> between require and dofile calls, but it shouldn't be too difficult to
> add:
>
> if pcall(debug.getlocal, 4, 1) then
>  if select('#', ...) == 0 then
>    print("dofile package")
>  else
>    print("require package")
>  end
> else
>  print("main script")
> end
>
> Paul.

why does dofile package have zero argument ?

without actually testing for require "mod.lua", it seems kinda leap of faith.

P.S. just noticed a bug in require, it will accept false from mod.lua
       as valid value, defeating the purpose of package.loaded cache table

--> fix: patched require to use false as sentinel, with a guarantee
     that package.loaded.mod always eval to true (after require)

--> new test for require "mod", invalid false as sentinel

if package.loaded.mod == false then return mod_func end
Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Soni "They/Them" L.
In reply to this post by Sean Conner


On 2018-02-24 07:19 PM, Sean Conner wrote:
> It was thus said that the Great Soni They/Them L. once stated:
>> Why does nobody support renaming modules, it's 2018 :(
>    Please rename a module written in C and get it to run, without
> recompilation.  Then report your findings back here.
>
>    -spc
>
>

Am I allowed to recompile Lua, or monkeypatch require? The default
behaviour is very much crap for C modules and I've been talking about it
for months now.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Sean Conner
It was thus said that the Great Soni They/Them L. once stated:

>
> On 2018-02-24 07:19 PM, Sean Conner wrote:
> >It was thus said that the Great Soni They/Them L. once stated:
> >>Why does nobody support renaming modules, it's 2018 :(
> >   Please rename a module written in C and get it to run, without
> >recompilation.  Then report your findings back here.
> >
> >   -spc
> >
> >
>
> Am I allowed to recompile Lua, or monkeypatch require? The default
> behaviour is very much crap for C modules and I've been talking about it
> for months now.

  Why can't you just rename the modules?

  Hint:  the thread starting here: http://lua-users.org/lists/lua-l/2013-06/msg00464.html

  -spc (You have?)




Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Roberto Ierusalimschy
In reply to this post by Tony Papadimitriou
> Unfortunately, the conclusion was that there seems to be no
> fail-proof method.
>
> This is a very important feature but there is zero interest from the
> Lua team to provide a reliable method.

How do you know that?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Soni "They/Them" L.
In reply to this post by Sean Conner


On 2018-02-25 01:26 AM, Sean Conner wrote:

> It was thus said that the Great Soni They/Them L. once stated:
>> On 2018-02-24 07:19 PM, Sean Conner wrote:
>>> It was thus said that the Great Soni They/Them L. once stated:
>>>> Why does nobody support renaming modules, it's 2018 :(
>>>    Please rename a module written in C and get it to run, without
>>> recompilation.  Then report your findings back here.
>>>
>>>    -spc
>>>
>>>
>> Am I allowed to recompile Lua, or monkeypatch require? The default
>> behaviour is very much crap for C modules and I've been talking about it
>> for months now.
>    Why can't you just rename the modules?
>
>    Hint:  the thread starting here: http://lua-users.org/lists/lua-l/2013-06/msg00464.html
>
>    -spc (You have?)
>
>
>
>

Yes, Lua likes to hardcode C module names and doesn't call them with
their modname + path.

My idea is for a module loading system that looks for "luaopen_" symbol.
then "luaopen_submod" and so on.

When requiring something that leads to an .so, it looks for "luaopen_"
if the full path matched, "luaopen_submod" if it only matched up to the
last, and so on.

And then I'd give it the modname and the path to SO, and maybe also the
partial modname used to find the SO.

It would be a much better system that Lua doesn't support. It's so nice
being able to pack libs with your scripts (and re-export any of those
libs if your API relies on them).

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Albert Chan
In reply to this post by Paul K-2
On Feb 24, 2018, at 6:03 PM, Paul K <[hidden email]> wrote:

> Hi Dirk,
>
>> I.e. "doflile" is treated just like "require'.
>> I want "require" to return and all other ways of running the file to continue.
>
> It wasn't clear from your question that you want to distinguish
> between require and dofile calls, but it shouldn't be too difficult to
> add:
>
> if pcall(debug.getlocal, 4, 1) then
>  if select('#', ...) == 0 then
>    print("dofile package")
>  else
>    print("require package")
>  end
> else
>  print("main script")
> end
>
> Paul.

figured out why dofile have zero argument.

what if later versions of dofile allow optional command-line args ?
what if user use this form: loadfile "mod.lua" "arg" ?

it will then goto require branch.


Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Sean Conner
In reply to this post by Soni "They/Them" L.
It was thus said that the Great Soni They/Them L. once stated:
> On 2018-02-25 01:26 AM, Sean Conner wrote:

> >It was thus said that the Great Soni They/Them L. once stated:
> >>On 2018-02-24 07:19 PM, Sean Conner wrote:
> >>>It was thus said that the Great Soni They/Them L. once stated:
> >>>>Why does nobody support renaming modules, it's 2018 :(
> >>>   Please rename a module written in C and get it to run, without
> >>>recompilation.  Then report your findings back here.
> >>>
> >>>   -spc
> >>>
> >>>
> >>Am I allowed to recompile Lua, or monkeypatch require? The default
> >>behaviour is very much crap for C modules and I've been talking about it
> >>for months now.
> >   Why can't you just rename the modules?
> >
> >   Hint:  the thread starting here:
> >   http://lua-users.org/lists/lua-l/2013-06/msg00464.html
> >
> >   -spc (You have?)
> >
> >
> >
> >
>
> Yes, Lua likes to hardcode C module names and doesn't call them with
> their modname + path.

  I don't think Roberto "likes to hardcode C module names":

> Modules are not always linked dynamically (see, for instance, the
> standard Lua libraries). With a single name, one would not be able to
> statically link a set of modules to a program.
>
> (Moreover, I am not sure your experience number [5] is portable. IIRC,
> we had problems in the past with some systems that signalled name
> conflicts even with dynamic libraries.)
>
> -- Roberto

and the footnote:

> [5] Years ago I wrote a program that would load modules at run time, and
> each module exports the same symbol.  

  And if you read the documentation under "package.searchers":

> My idea is for a module loading system that looks for "luaopen_" symbol.
> then "luaopen_submod" and so on.
>
> When requiring something that leads to an .so, it looks for "luaopen_"
> if the full path matched, "luaopen_submod" if it only matched up to the
> last, and so on.
>
> And then I'd give it the modname and the path to SO, and maybe also the
> partial modname used to find the SO.
>
> It would be a much better system that Lua doesn't support.

  I direct you to the documentation to package.searchers:

        When looking for a module, require calls each of these searchers in
        ascending order, with the module name (the argument given to
        require) as its sole parameter.

        ...

        The fourth searcher tries an all-in-one loader. It searches the C
        path for a library for the root name of the given module. For
        instance, when requiring a.b.c, it will search for a C library for
        a. If found, it looks into it for an open function for the
        submodule; in our example, that would be luaopen_a_b_c. With this
        facility, a package can pack several C submodules into one single
        library, with each submodule keeping its original open function.

  It does appear that Lua 5.3 also passes in the path (as the second Lua
parameter) to each searcher, so either the documenation is wrong, or it's an
undocumted feature.

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Soni "They/Them" L.


On 2018-02-25 04:51 PM, Sean Conner wrote:

> It was thus said that the Great Soni They/Them L. once stated:
>> On 2018-02-25 01:26 AM, Sean Conner wrote:
>>> It was thus said that the Great Soni They/Them L. once stated:
>>>> On 2018-02-24 07:19 PM, Sean Conner wrote:
>>>>> It was thus said that the Great Soni They/Them L. once stated:
>>>>>> Why does nobody support renaming modules, it's 2018 :(
>>>>>    Please rename a module written in C and get it to run, without
>>>>> recompilation.  Then report your findings back here.
>>>>>
>>>>>    -spc
>>>>>
>>>>>
>>>> Am I allowed to recompile Lua, or monkeypatch require? The default
>>>> behaviour is very much crap for C modules and I've been talking about it
>>>> for months now.
>>>    Why can't you just rename the modules?
>>>
>>>    Hint:  the thread starting here:
>>>    http://lua-users.org/lists/lua-l/2013-06/msg00464.html
>>>
>>>    -spc (You have?)
>>>
>>>
>>>
>>>
>> Yes, Lua likes to hardcode C module names and doesn't call them with
>> their modname + path.
>    I don't think Roberto "likes to hardcode C module names":
>
>> Modules are not always linked dynamically (see, for instance, the
>> standard Lua libraries). With a single name, one would not be able to
>> statically link a set of modules to a program.

Yes, static linking is a case where the shared object is "", so the
resulting submodule path is always the full name.

E.g. "" . "hello.world" = "luaopen_hello_world" (this is always the case
for the stdlib), "hello" . "world" = "luaopen_world", "hello.world" . ""
= "luaopen_", etc.

Are there cases of static linking where you can't just change the symbol
names?

>>
>> (Moreover, I am not sure your experience number [5] is portable. IIRC,
>> we had problems in the past with some systems that signalled name
>> conflicts even with dynamic libraries.)
>>
>> -- Roberto
> and the footnote:
>
>> [5] Years ago I wrote a program that would load modules at run time, and
>> each module exports the same symbol.

Since C modules are already architecture- and OS-specific, it seems
reasonable to have it one way in general, but support alternatives for
OSes that can handle it better.

>    And if you read the documentation under "package.searchers":
>
>> My idea is for a module loading system that looks for "luaopen_" symbol.
>> then "luaopen_submod" and so on.
>>
>> When requiring something that leads to an .so, it looks for "luaopen_"
>> if the full path matched, "luaopen_submod" if it only matched up to the
>> last, and so on.
>>
>> And then I'd give it the modname and the path to SO, and maybe also the
>> partial modname used to find the SO.
>>
>> It would be a much better system that Lua doesn't support.
>    I direct you to the documentation to package.searchers:
>
> When looking for a module, require calls each of these searchers in
> ascending order, with the module name (the argument given to
> require) as its sole parameter.
>
> ...
>
> The fourth searcher tries an all-in-one loader. It searches the C
> path for a library for the root name of the given module. For
> instance, when requiring a.b.c, it will search for a C library for
> a. If found, it looks into it for an open function for the
> submodule; in our example, that would be luaopen_a_b_c. With this
> facility, a package can pack several C submodules into one single
> library, with each submodule keeping its original open function.
>
>    It does appear that Lua 5.3 also passes in the path (as the second Lua
> parameter) to each searcher, so either the documenation is wrong, or it's an
> undocumted feature.

$ lua
Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
 > package.preload.hello = print
 > require "hello"
hello    nil
true
 > io.open("world.lua", "w"):write("print (...)"):close()
true
 > require "world"
world    ./world.lua
true
 > -- can't test a C module here but w/e, what happens if you print the
C stack from a C module?

>    -spc
>

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Tony Papadimitriou
In reply to this post by Roberto Ierusalimschy
Based on previous discussion responses when I had requested this capability!
I may have misunderstood, of course!

-----Original Message-----
From: Roberto Ierusalimschy
Sent: Sunday, February 25, 2018 2:24 PM
To: Lua mailing list
Subject: Re: Modules with standalone main program

> Unfortunately, the conclusion was that there seems to be no
> fail-proof method.
>
> This is a very important feature but there is zero interest from the
> Lua team to provide a reliable method.

How do you know that?

-- Roberto


Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Sean Conner
In reply to this post by Soni "They/Them" L.
It was thus said that the Great Soni They/Them L. once stated:
> On 2018-02-25 04:51 PM, Sean Conner wrote:
>
> Are there cases of static linking where you can't just change the symbol
> names?

  Well, you can change the name in the source code (more portable, probably
the "easiest" way).  Changing the name after you have object code is ... I
don't know.  If you can, it's most likely very system dependent.  

> >   It does appear that Lua 5.3 also passes in the path (as the second Lua
> >parameter) to each searcher, so either the documenation is wrong, or it's
> >an undocumted feature.
>
> $ lua
> Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
> > package.preload.hello = print
> > require "hello"
> hello    nil

  And if you read the entire section you would see this behavior documented.
I didn't quote the entire section (I apologize if I gave the impression I
did).

> true
> > io.open("world.lua", "w"):write("print (...)"):close()
> true
> > require "world"
> world    ./world.lua
> true
> > -- can't test a C module here but w/e, what happens if you print the
> C stack from a C module?

  I'll assume you mean the Lua stack from a C lua function.  So here you go:

        #include <stdio.h>
        #include <lua.h>
        #include <lauxlib.h>
       
        int luaopen_cmod(lua_State *L)
        {
          int max = lua_gettop(L);
          int i;
       
          for (i = 1 ; i <= max ; i++)
          {
            char const *name;
            char const *type;
            int         ri;
       
            lua_getglobal(L,"tostring");
            lua_pushvalue(L,i);
            lua_call(L,1,1);
       
            name = lua_tostring(L,-1);
            type = luaL_typename(L,i);
            ri   = i - max - 1;
       
            printf("Stack: %d %d - %s %s\n",i,ri,type,name);
            lua_pop(L,1);
          }
       
          lua_pushboolean(L,1);
          return 1;
        }

        [spc]lucy:/tmp>lua-53
        Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
        > require "cmod"
        Stack: 1 -2 - string cmod
        Stack: 2 -1 - string ./cmod.so
        true
        >

  Answers your question?

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Soni "They/Them" L.


On 2018-02-25 10:58 PM, Sean Conner wrote:
> It was thus said that the Great Soni They/Them L. once stated:
>> On 2018-02-25 04:51 PM, Sean Conner wrote:
>>
>> Are there cases of static linking where you can't just change the symbol
>> names?
>    Well, you can change the name in the source code (more portable, probably
> the "easiest" way).  Changing the name after you have object code is ... I
> don't know.  If you can, it's most likely very system dependent.
>

I assume static linking relies on dlopen(NULL, ...)/the global symbol
table. This means the symbol must be exported anyway, which does make it
trivial to rename it, as far as I know.

I assume all other cases would have to be directly embedded in the Lua
source somehow. (I believe the procedure for this is described
somewhere, something to do with modifying luaL_openlibs?)

>>>    It does appear that Lua 5.3 also passes in the path (as the second Lua
>>> parameter) to each searcher, so either the documenation is wrong, or it's
>>> an undocumted feature.
>> $ lua
>> Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
>>> package.preload.hello = print
>>> require "hello"
>> hello    nil
>    And if you read the entire section you would see this behavior documented.
> I didn't quote the entire section (I apologize if I gave the impression I
> did).
>
>> true
>>> io.open("world.lua", "w"):write("print (...)"):close()
>> true
>>> require "world"
>> world    ./world.lua
>> true
>>> -- can't test a C module here but w/e, what happens if you print the
>> C stack from a C module?
>    I'll assume you mean the Lua stack from a C lua function.  So here you go:
>
> #include <stdio.h>
> #include <lua.h>
> #include <lauxlib.h>
>
> int luaopen_cmod(lua_State *L)
> {
>  int max = lua_gettop(L);
>  int i;
>
>  for (i = 1 ; i <= max ; i++)
>  {
>    char const *name;
>    char const *type;
>    int         ri;
>
>    lua_getglobal(L,"tostring");
>    lua_pushvalue(L,i);
>    lua_call(L,1,1);
>
>    name = lua_tostring(L,-1);
>    type = luaL_typename(L,i);
>    ri   = i - max - 1;
>
>    printf("Stack: %d %d - %s %s\n",i,ri,type,name);
>    lua_pop(L,1);
>  }
>
>  lua_pushboolean(L,1);
>  return 1;
> }
>
> [spc]lucy:/tmp>lua-53
> Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
> > require "cmod"
> Stack: 1 -2 - string cmod
> Stack: 2 -1 - string ./cmod.so
> true
> >
>
>    Answers your question?
>
>    -spc
>
>

Interesting. So C modules could be renameable if they supported what I
proposed.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Sean Conner
It was thus said that the Great Soni They/Them L. once stated:

> On 2018-02-25 10:58 PM, Sean Conner wrote:
> >It was thus said that the Great Soni They/Them L. once stated:
> >>On 2018-02-25 04:51 PM, Sean Conner wrote:
> >>
> >>Are there cases of static linking where you can't just change the symbol
> >>names?
> >   Well, you can change the name in the source code (more portable,
> >   probably
> >the "easiest" way).  Changing the name after you have object code is ... I
> >don't know.  If you can, it's most likely very system dependent.
> >
>
> I assume static linking relies on dlopen(NULL, ...)

  Static linking has *nothing* to do with dlopen().  That's an entirely
different process.  Okay ...

  Static linking.  I'll ignore dynamic linking for this example. You have
two C files:

        ---[ c1.c ]---

        extern int C();

        int A()       { ... }
        int B( C(); ) { ... }

        ---[ c2.c ]---

        extern int A();
        extern int B();

        int C( A(); A(); ) { ... }
        int main()         { ... }

  You compile each separately and end up with object files.  Each object
file has something like this:

        ---[ c1.o ]---

        DEFINED
                A 100
                B 200

        UNDEFINED
                C 210

        ---[ c2.o ]---

        DEFINED
                C 100
                main 200

        UNDEFINED
                A 110
                A 130

  For the DEFINED sections, this list contains the name, and the offset into
the program portion it defines.  So we have function A() lives at offset 100
in the program portion of c1.o, B() lives at offset 200 in the program
portion.  Note how c2.o has different names but the same offsets (they
aren't addresses *yet*).  

  The UNDEFINED section lists symbols not defined in the source but are
referenced.  For c1.o, we have function C().  The offset here is where we
need to store the address of C() when linking.  In c2.0, we call A() twice,
and thus, need the address twice, at offsets 110 and 130 (in c2.o).

  A *linker* will take these two object files and stitch (or *HINT HINT*
LINK) them together.  During the linking phase, the offsets become actual
addresses and everything is patched up.  So we might end up with the
following layout:

        address comment
        500 start of function A
        700 start of function B
        710 address of function C stored here as part of CALL instruction
        1000 start of function C
        1010 address of function A stored here as part of CALL instruction
        1030 address of function A stored here as part of CALL instruction
        1200 address of main

  Everything is defined, nothing needs to be loaded, and the program runs.  

  Okay, dynamic linking, part I.  We'll modify things up a bit in the C
code:

        ---[ c1.c ]---

        extern int printf();
        extern int C();

        int A( printf(); ) { ... }
        int B( C(); )      { ... }

        ---[ c2.c ]---

        extern int getpid();
        extern int A();
        extern int B();

        int C( A(); A(); )         { ... }
        int main( getpid(); C(); ) { ... }

Compilation to object files we get:

        ---[ c1.o ]---

        DEFINED
                A 100
                B 200

        UNDEFINED
                printf 110
                C 210

        ---[ c2.o ]---

        DEFINED
                C 100
                main 200

        UNDEFINED
                getpid 110
                A 120
                A 150

  The link phase happens.  When the linker goes through the libc library, it
notices that it's a dynamic link library and instead of including the object
code into the final executable, it just places references with some
addtional information in the executable that looks a lot like what we had in
object files:

        DYNAMIC_LIB
                libc We need this library at run time

        UNDEFINED
                printf 510 need address of printf stored here as part of CALL instruction
                getpid 1010 need address of getpid stored here as part of CALL instruction

  At *run time*, the kernel will locate the proper location of the library
libc, and then patch up the executable with the locations of the required
functions.  User code *does not call dlopen* for this.  This happens behind
the scenes (and yes, I've simplified this quite a bit but this is the gist
of what happens).

  Okay, dynamic linking, part II.  This is when the usercode calls dlopen().
dlopen() will locate the proper library and ensure it's in memory.  The code
then calls dlsym() to return the locations of symbols out of the loaded
dynamic library.  This is what Lua uses to load Lua modules written in C.

> /the global symbol
> table. This means the symbol must be exported anyway, which does make it
> trivial to rename it, as far as I know.

  How?  The name comes from the source code and is embedded into the object
file (and possibly the executable if it's part of a dynamic library loaded
at runtime).  

> I assume all other cases would have to be directly embedded in the Lua
> source somehow. (I believe the procedure for this is described
> somewhere, something to do with modifying luaL_openlibs?)

  Nope.  I describe one way here:

        http://boston.conman.org/2013/03/23.1

which involves adding some new entries to the package.searchers array (I
update package.loaders but that's because I'm using Lua 5.1 in that
particular project).  Another way when statically linking in C based Lua
modules is to populate the luaopen_() calls in the package.preload array.

> Interesting. So C modules could be renameable if they supported what I
> proposed.

  Again, how?

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Paul K-2
In reply to this post by Albert Chan
> what if later versions of dofile allow optional command-line args ?

We'll cross this bridge when we get there. For now this logic should
work with all Lua 5.1+ versions.

Paul.

Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Dirk Laurie-2
2018-02-26 5:43 GMT+02:00 Paul K <[hidden email]>:
>> what if later versions of dofile allow optional command-line args ?
>
> We'll cross this bridge when we get there. For now this logic should
> work with all Lua 5.1+ versions.

Why is it better to use debug.getlocal than "'..."?

Reply | Threaded
Open this post in threaded view
|

Re: Renaming Lua modules (was Re: Modules with standalone main program)

Soni "They/Them" L.
In reply to this post by Sean Conner


On 2018-02-26 12:17 AM, Sean Conner wrote:

> It was thus said that the Great Soni They/Them L. once stated:
>> On 2018-02-25 10:58 PM, Sean Conner wrote:
>>> It was thus said that the Great Soni They/Them L. once stated:
>>>> On 2018-02-25 04:51 PM, Sean Conner wrote:
>>>>
>>>> Are there cases of static linking where you can't just change the symbol
>>>> names?
>>>    Well, you can change the name in the source code (more portable,
>>>    probably
>>> the "easiest" way).  Changing the name after you have object code is ... I
>>> don't know.  If you can, it's most likely very system dependent.
>>>
>> I assume static linking relies on dlopen(NULL, ...)
>    Static linking has *nothing* to do with dlopen().  That's an entirely
> different process.  Okay ...
>
>    Static linking.  I'll ignore dynamic linking for this example. You have
> two C files:
>
> ---[ c1.c ]---
>
> extern int C();
>
> int A()       { ... }
> int B( C(); ) { ... }
>
> ---[ c2.c ]---
>
> extern int A();
> extern int B();
>
> int C( A(); A(); ) { ... }
> int main()         { ... }
>
>    You compile each separately and end up with object files.  Each object
> file has something like this:
>
> ---[ c1.o ]---
>
> DEFINED
> A 100
> B 200
>
> UNDEFINED
> C 210
>
> ---[ c2.o ]---
>
> DEFINED
> C 100
> main 200
>
> UNDEFINED
> A 110
> A 130
>
>    For the DEFINED sections, this list contains the name, and the offset into
> the program portion it defines.  So we have function A() lives at offset 100
> in the program portion of c1.o, B() lives at offset 200 in the program
> portion.  Note how c2.o has different names but the same offsets (they
> aren't addresses *yet*).
>
>    The UNDEFINED section lists symbols not defined in the source but are
> referenced.  For c1.o, we have function C().  The offset here is where we
> need to store the address of C() when linking.  In c2.0, we call A() twice,
> and thus, need the address twice, at offsets 110 and 130 (in c2.o).
>
>    A *linker* will take these two object files and stitch (or *HINT HINT*
> LINK) them together.  During the linking phase, the offsets become actual
> addresses and everything is patched up.  So we might end up with the
> following layout:
>
> address comment
> 500 start of function A
> 700 start of function B
> 710 address of function C stored here as part of CALL instruction
> 1000 start of function C
> 1010 address of function A stored here as part of CALL instruction
> 1030 address of function A stored here as part of CALL instruction
> 1200 address of main
>
>    Everything is defined, nothing needs to be loaded, and the program runs.
>
>    Okay, dynamic linking, part I.  We'll modify things up a bit in the C
> code:
>
> ---[ c1.c ]---
>
> extern int printf();
> extern int C();
>
> int A( printf(); ) { ... }
> int B( C(); )      { ... }
>
> ---[ c2.c ]---
>
> extern int getpid();
> extern int A();
> extern int B();
>
> int C( A(); A(); )         { ... }
> int main( getpid(); C(); ) { ... }
>
> Compilation to object files we get:
>
> ---[ c1.o ]---
>
> DEFINED
> A 100
> B 200
>
> UNDEFINED
> printf 110
> C 210
>
> ---[ c2.o ]---
>
> DEFINED
> C 100
> main 200
>
> UNDEFINED
> getpid 110
> A 120
> A 150
>
>    The link phase happens.  When the linker goes through the libc library, it
> notices that it's a dynamic link library and instead of including the object
> code into the final executable, it just places references with some
> addtional information in the executable that looks a lot like what we had in
> object files:
>
> DYNAMIC_LIB
> libc We need this library at run time
>
> UNDEFINED
> printf 510 need address of printf stored here as part of CALL instruction
> getpid 1010 need address of getpid stored here as part of CALL instruction
>
>    At *run time*, the kernel will locate the proper location of the library
> libc, and then patch up the executable with the locations of the required
> functions.  User code *does not call dlopen* for this.  This happens behind
> the scenes (and yes, I've simplified this quite a bit but this is the gist
> of what happens).
>
>    Okay, dynamic linking, part II.  This is when the usercode calls dlopen().
> dlopen() will locate the proper library and ensure it's in memory.  The code
> then calls dlsym() to return the locations of symbols out of the loaded
> dynamic library.  This is what Lua uses to load Lua modules written in C.
>
>> /the global symbol
>> table. This means the symbol must be exported anyway, which does make it
>> trivial to rename it, as far as I know.
>    How?  The name comes from the source code and is embedded into the object
> file (and possibly the executable if it's part of a dynamic library loaded
> at runtime).

dlopen(NULL, ...) is a thing. Read about it. It means the symbols need
to be exported during linking, even if the linking is static. It just
means you have dynamic lookup of "static"-ish symbols. It's complicated.

If you're turning them straight into a statically linked library, you
can still do it but I'm not sure how. (I mean the linker has to find the
symbols somewhere, right?)

>
>> I assume all other cases would have to be directly embedded in the Lua
>> source somehow. (I believe the procedure for this is described
>> somewhere, something to do with modifying luaL_openlibs?)
>    Nope.  I describe one way here:
>
> http://boston.conman.org/2013/03/23.1
>
> which involves adding some new entries to the package.searchers array (I
> update package.loaders but that's because I'm using Lua 5.1 in that
> particular project).  Another way when statically linking in C based Lua
> modules is to populate the luaopen_() calls in the package.preload array.

Assuming you're turning pre-built shared objects into statically linked
libraries, this is still trivial.

>
>> Interesting. So C modules could be renameable if they supported what I
>> proposed.
>    Again, how?

Because they can rely on the passed-in names rather than relying on
hardcoded names.

>    -spc
>

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Modules with standalone main program

Albert Chan
In reply to this post by Dirk Laurie-2
On Feb 26, 2018, at 2:35 AM, Dirk Laurie <[hidden email]> wrote:

> 2018-02-26 5:43 GMT+02:00 Paul K <[hidden email]>:
>>> what if later versions of dofile allow optional command-line args ?
>>
>> We'll cross this bridge when we get there. For now this logic should
>> work with all Lua 5.1+ versions.
>
> Why is it better to use debug.getlocal than "'..."?
>

require "mod"
lua mod.lua mod
loadfile "mod.lua" "mod"

ALL of above, ... = "mod", so ... is a bad test for require vs standalone

my solution is to patch require, using false as sentinel, like this:

function require(name)                          -- pseudo-code for loadlib.c ll_require()
  local cache = package.loaded[name]
  if cache ~= nil then return cache end  -- return even if false, avoid infinite loops
  package.loaded[name] = false            -- signal loading "mod" INSIDE require()
  -- do actual require "mod"
  return package.loaded[name]             -- assert bool(return) == true ("mod" loaded)
end

function package.loading(name)           -- direct test for require "mod"
  return package.loaded[name] == false
end

so, inside mod.lua, I can test for require "mod" like this:

if package.loading "mod" then return mod_func end


123