Non-uniqueness of module names

classic Classic list List threaded Threaded
53 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Roberto Ierusalimschy
> The letter of the law is now "avoid creating names that start with an
> underscore followed by one or more uppercase letters".

Note that the manual says to avoid *creating* such names, not using
them. For instance, everybody can use (and many people do) _ENV for
their onw variables in their programs, to do what it was created for.

If the Lua team "creates" the name _VERSION for module identification,
everybody could use it for that purpose. (I fully agree with Hisham that
it is not a good idea to use that name unless we have that first step.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Soni "They/Them" L.


On 2019-04-21 5:20 p.m., Roberto Ierusalimschy wrote:

>> The letter of the law is now "avoid creating names that start with an
>> underscore followed by one or more uppercase letters".
> Note that the manual says to avoid *creating* such names, not using
> them. For instance, everybody can use (and many people do) _ENV for
> their onw variables in their programs, to do what it was created for.
>
> If the Lua team "creates" the name _VERSION for module identification,
> everybody could use it for that purpose. (I fully agree with Hisham that
> it is not a good idea to use that name unless we have that first step.)
>
> -- Roberto
>

when can we expect to see math._VERSION? ;)

Reply | Threaded
Open this post in threaded view
|

Bug in Lua 5.3? (was Re: Non-uniqueness of module names)

Sean Conner
In reply to this post by Coda Highland
It was thus said that the Great Coda Highland once stated:
> On Sat, Apr 20, 2019 at 5:33 PM Sean Conner <[hidden email]> wrote:
> > It was thus said that the Great Coda Highland once stated:
> >
> > > the import name always matches the directory containing the module. The
> >
> >   Again, pretty much the same as Lua.
>
> Pretty much, but not entirely. The fact that some modules don't do this is
> one of the reasons this thread has come up at all.

  That's more an issue for Lua 5.1 than 5.2+.
>
> > > import scheme also does aggressive namespacing so you have to go out of
> > > your way to do stuff that might screw up some other module's transitive
> > > dependencies.
> >
> >   Okay, so how does *this* work then?
>
> Mostly, ironclad module scoping. Each file has its own global scope, and

  ...

> Duplicated dependencies don't necessarily even get mapped onto each other.
> If they're not the same module (with compatibility determined by the
> importer's explicitly-requested versioning information) then you can
> actually end up with two copies of the module loaded in, and since they're
> different modules with isolated namespaces they don't interfere with each
> other. It's not the BEST solution to dependency hell because it leads to
> bloating and bugfixes that don't get applied consistently, but it does mean
> that a transitive dependency is a little less likely to screw you over.

  So basically, version numbers are checked when loading modules.  I thought
Lua could do this for C-based modules.  From the manual:

        package.config

                A string describing some compile-time configurations for
                packages. This string is a sequence of lines:

                        ...
                       
                        The fifth line is a mark to ignore all text after it
                        when building the luaopen_ function name. Default is '-'.

                ...

                The name of this C function is the string "luaopen_"
                concatenated with a copy of the module name where each dot
                is replaced by an underscore.  Moreover, if the module name
                has a hyphen, its suffix after (and including) the first
                hyphen is removed. For instance, if the module name is
                a.b.c-v2.1, the function name will be luaopen_a_b_c.

So in trying this, I created the following:

prog.lua

        local a = require "a"
        local b = require "b"

        a.foo()
        b.foo()

a.lua

        local mod = require "mod-1.2.3"

        return {
          foo = function()
            print('This is module a')
            mod.foo()
          end
        }

b.lua

        local mod = require "mod-2.0.1"
       
        return {
          foo = function()
            print('This is module b')
            mod.foo()
          end
        }

mod-v1.2.3.c

        #include <lua.h>
       
        static int foo(lua_State *L)
        {
          lua_getglobal(L,"print");
          lua_pushliteral(L,"This is module mod-1.2.3");
          lua_call(L,1,0);
          return 0;
        }
       
        int luaopen_mod(lua_State *L)
        {
          lua_createtable(L,0,1);
          lua_pushcfunction(L,foo);
          lua_setfield(L,-2,"foo");
          return 1;
        }

mod-2.0.1.c

        #include <lua.h>
       
        static int foo(lua_State *L)
        {
          lua_getglobal(L,"print");
          lua_pushliteral(L,"This is module mod-2.0.1");
          lua_call(L,1,0);
          return 0;
        }
       
        int luaopen_mod(lua_State *L)
        {
          lua_createtable(L,0,1);
          lua_pushcfunction(L,foo);
          lua_setfield(L,-2,"foo");
          return 1;
        }

What I expected to see was:

        This is module a
        This is module mod-1.2.3
        This is module b
        This is module mod-2.0.1

But what I got was:

lua: ./a.lua:2: module 'mod-v1.2.3' not found:
        no field package.preload['mod-v1.2.3']
        no file '/home/spc/.luarocks/share/lua/5.3/mod-v1/2/3.lua'
        no file '/home/spc/.luarocks/share/lua/5.3/mod-v1/2/3/init.lua'
        no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1/2/3.lua'
        no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1/2/3/init.lua'
        no file '/usr/local/share/lua/5.3/mod-v1/2/3.lua'
        no file '/usr/local/share/lua/5.3/mod-v1/2/3/init.lua'
        no file '/usr/local/lib/lua/5.3/mod-v1/2/3.lua'
        no file '/usr/local/lib/lua/5.3/mod-v1/2/3/init.lua'
        no file './mod-v1/2/3.lua'
        no file './mod-v1/2/3/init.lua'
        no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1/2/3.so'
        no file '/usr/local/lib/lua/5.3/mod-v1/2/3.so'
        no file '/usr/local/lib/lua/5.3/loadall.so'
        no file './mod-v1/2/3.so'
        no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1.so'
        no file '/usr/local/lib/lua/5.3/mod-v1.so'
        no file '/usr/local/lib/lua/5.3/loadall.so'
        no file './mod-v1.so'
stack traceback:
        [C]: in function 'require'
        ./a.lua:2: in main chunk
        [C]: in function 'require'
        prog.lua:3: in main chunk
        [C]: in ?

  Am I doing something wrong?  Is Lua doing something wrong?  You can tell I
was confused because one module is "mod-v1.2.3" and the other one is
"mod-2.0.1".  In thinking about this further, even if this did work, it
wouldn't necessarily work on all systems for the same reason the exported
function from C modules isn't just "luaopen()" (because some (IMHO)
braindead systems can't deal with loading dynanic linked object with the
same exports).

  -spc (Confused by all this)

Reply | Threaded
Open this post in threaded view
|

Re: Bug in Lua 5.3? (was Re: Non-uniqueness of module names)

Sean Conner
It was thus said that the Great Sean Conner once stated:

>
> a.lua
>
> local mod = require "mod-1.2.3"
>
> return {
>  foo = function()
>    print('This is module a')
>    mod.foo()
>  end
> }

  Oops.  I started writing the email before I completely finished my
testing.  The updated a.lua file should be:

        local mod = require "mod-v1.2.3"
       
        return {
          foo = function()
            print('This is module a')
            mod.foo()
          end
        }

  The results are still:

> lua: ./a.lua:2: module 'mod-v1.2.3' not found:
>         no field package.preload['mod-v1.2.3']
>         no file '/home/spc/.luarocks/share/lua/5.3/mod-v1/2/3.lua'
>         no file '/home/spc/.luarocks/share/lua/5.3/mod-v1/2/3/init.lua'
>         no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1/2/3.lua'
>         no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1/2/3/init.lua'
>         no file '/usr/local/share/lua/5.3/mod-v1/2/3.lua'
>         no file '/usr/local/share/lua/5.3/mod-v1/2/3/init.lua'
>         no file '/usr/local/lib/lua/5.3/mod-v1/2/3.lua'
>         no file '/usr/local/lib/lua/5.3/mod-v1/2/3/init.lua'
>         no file './mod-v1/2/3.lua'
>         no file './mod-v1/2/3/init.lua'
>         no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1/2/3.so'
>         no file '/usr/local/lib/lua/5.3/mod-v1/2/3.so'
>         no file '/usr/local/lib/lua/5.3/loadall.so'
>         no file './mod-v1/2/3.so'
>         no file '/home/spc/.luarocks/lib/lua/5.3/mod-v1.so'
>         no file '/usr/local/lib/lua/5.3/mod-v1.so'
>         no file '/usr/local/lib/lua/5.3/loadall.so'
>         no file './mod-v1.so'
> stack traceback:
>         [C]: in function 'require'
>         ./a.lua:2: in main chunk
>         [C]: in function 'require'
>         prog.lua:3: in main chunk
>         [C]: in ?

  Just thought I should clarify that.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Bug in Lua 5.3? (was Re: Non-uniqueness of module names)

Coda Highland
In reply to this post by Sean Conner
On Sun, Apr 21, 2019 at 4:18 PM Sean Conner <[hidden email]> wrote:
> Duplicated dependencies don't necessarily even get mapped onto each other.
> If they're not the same module (with compatibility determined by the
> importer's explicitly-requested versioning information) then you can
> actually end up with two copies of the module loaded in, and since they're
> different modules with isolated namespaces they don't interfere with each
> other. It's not the BEST solution to dependency hell because it leads to
> bloating and bugfixes that don't get applied consistently, but it does mean
> that a transitive dependency is a little less likely to screw you over.

  So basically, version numbers are checked when loading modules. 

Not even close. It would be nice if it were that simple.

No, version numbers are checked by the package manager when installing modules. It creates a directory tree where each module gets a copy (or symlink) of its dependencies inside its own definition, and then that module's imports are resolved within its own path before falling back to the configured path.

I can't exactly say I recommend this approach. </understatement>

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Bug in Lua 5.3? (was Re: Non-uniqueness of module names)

Sean Conner
In reply to this post by Sean Conner

  Okay, some more playing around, I think I got this working.
So I have mod-v1.2.3 and mod-2.0.1.  To start with, they can't have those
names, they need to be mod-v1.so and mod-2.so.  Then, to include these:

        mod123_1 = require "mod-v1"
        mod123_2 = require "mod-v1.2"
        mod123_3 = require "mod-v1.2.3"

        mod201_1 = require "mod-2"
        mod201_2 = require "mod-2.0"
        mod201_3 = require "mod-2.0.1"

Furthermore, from some investigation, the module is only found by the last
searcher, the *all-in-one* loader.  So mod123_1, mod123_2 and mod123_3 are
all the same module; same with mod201_1, mod201_2 and mod201_3.

  This is ... less than useful.

  This can work if I name the modules mod-1-2-3.so, mod-1-2-4.so (which I
added, testing this out) and mod-2-0-1.so.  So now I can do:

        local a = require "a" -- calls require "mod-1-2-3"
        local b = require "b" -- calls require "mod-1-2-4"
        local c = require "c" -- calls require "mod-2-0-1"
       
        a.foo()
        b.foo()
        c.foo()

        This is module a
        This is module mod-1.2.3 -- print statement out of date
        This is module b
        This is module mod-1.2.4 -- I didn't bother changing these
        This is module c
        This is module mod-2.0.1

  Man, is that unintuitive.  But it does work with Lua modules as well.  I
can rename a.lua, b.lua and c.lua to a-1-0-0.lua, a-1-1-0.lua and
a-2-0-0.lua, change the above to:

        local a = require "a-1-0-0"
        local b = require "a-1-1-0"
        local c = require "a-2-0-0"

and it still works.  Ugly though.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Chris Jones
In reply to this post by Roberto Ierusalimschy
Apologies for resurrecting this months-old thread, but I would like to chime in to support the idea of `require` returning a second result of the path+filename where the module was found. This would be incredibly helpful for me in a plugin loader I'm working on in Hammerspoon. For now I'll go with Sean's ideas further down the thread (thanks Sean!), but having `require` do it for free would be a very welcome convenience!

On Mon, 15 Apr 2019 at 14:45, Roberto Ierusalimschy <[hidden email]> wrote:
I am not sure you are aware that the module itself already gets
this information:

---- file temp.lua
print(...)

$ lua -l temp
    --> temp    ./temp.lua

Maybe 'require' could simply return this information as a second result?

-- Roberto



--
Cheers,

Chris
Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Roberto Ierusalimschy
> Apologies for resurrecting this months-old thread, but I would like to
> chime in to support the idea of `require` returning a second result of the
> path+filename where the module was found.

This is already implemented in Lua 5.4 alpha.
See https://www.lua.org/work/

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Chris Jones
Fantastic, thank you very much!

On Thu, 15 Aug 2019 at 23:43, Roberto Ierusalimschy <[hidden email]> wrote:
> Apologies for resurrecting this months-old thread, but I would like to
> chime in to support the idea of `require` returning a second result of the
> path+filename where the module was found.

This is already implemented in Lua 5.4 alpha.
See https://www.lua.org/work/

-- Roberto



--
Cheers,

Chris
Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Paul Ducklin
In reply to this post by Roberto Ierusalimschy
Did not realise that - very useful to know!

(Seems that if the module comes from package.loaded, the second return value is nil - is that the only way nil can happen?)

I switched all my own work to 5.4 a while ago, mainly for the reason of a decent PRNG by default. Strange reason to choose, but it was mine. (I subsequently swapped out xoshiro256** for Jenkins-7,13,37 which was a simple change because they both have 4x64-bit internal state variables. Similar code in both cases but different design justifications by the creators.)

On 15 Aug 2019, at 22:43, Roberto Ierusalimschy <[hidden email]> wrote:

>> Apologies for resurrecting this months-old thread, but I would like to
>> chime in to support the idea of `require` returning a second result of the
>> path+filename where the module was found.
>
> This is already implemented in Lua 5.4 alpha.
> See https://www.lua.org/work/
>
> -- Roberto
>

Reply | Threaded
Open this post in threaded view
|

Re: RNGs (Was: Non-uniqueness of module names)

nobody
On 18/08/2019 23.39, Paul Ducklin wrote:
> I subsequently swapped out xoshiro256** for Jenkins-7,13,37 which
> was a simple change because they both have 4x64-bit internal state
> variables. Similar code in both cases but different design
> justifications by the creators.

I personally prefer Melissa O'Neill's PCG scheme, but I chose to use
that as a separate library.  (There's extra functionality beyond the
basic seed/random API – seeking, compound states, … and that also makes
it easier to have LOTS of independent states.)

Can you say why you chose to swap out the internal RNG instead of using
it as a library?  (Also, what's your reason to prefer that RNG?)

-- nobody

Reply | Threaded
Open this post in threaded view
|

Re: RNGs (Was: Non-uniqueness of module names)

Doug Currie
On Mon, Aug 19, 2019 at 6:02 PM nobody <[hidden email]> wrote:
On 18/08/2019 23.39, Paul Ducklin wrote:
> I subsequently swapped out xoshiro256** for Jenkins-7,13,37 ...

Can you say why you chose to swap out the internal RNG instead of using
it as a library?  (Also, what's your reason to prefer that RNG?)

Not speaking for Paul, but...

M. E. O'Neill's blog has very interesting posts on all three of these generators, and supports both of your preferences over xoshiro256** .


I also concur that a library implementation is a fine way to go.

e

 
Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Lorenzo Donati-3
In reply to this post by Roberto Ierusalimschy
On 15/08/2019 22:52, Roberto Ierusalimschy wrote:

>> Apologies for resurrecting this months-old thread, but I would like to
>> chime in to support the idea of `require` returning a second result of the
>> path+filename where the module was found.
>
> This is already implemented in Lua 5.4 alpha.
> See https://www.lua.org/work/
>
> -- Roberto
>
>

Is there a chance this enhancement will be backported to 5.3 branch?

It seems a very easy and useful change that doesn't break old code and
doesn't cause "paradigm" shifts.

It would be nice to have a 5.3.6 having that (and usual pending bug
fixes) until 5.4 is stable and finalized.

On a related note:

A long standing request from me for the interpreter on Windows was to
have a way to know the exact /absolute/ filesystem location of a running
script.

I know there are some tricks that do that, but they are not 100%
reliable. In some edge cases you could end up with a /relative/ path,
depending on how the interpreter was called.

I envision the interpreter defining a variable like say
_CURRENT_SCRIPT_PATH and _CURRENT_SCRIPT_DIR that hold the relevant
/absolute/ filesystem paths.

IIRC on Linux this is not so important/feasible, but on Windows there is
an API call (I can't remember it now off the top of my head) to get the
actual absolute path of the executable ("getModule"-something?!?)
 From that on the interpreter can deduce the actual path of the script
and so on.

Thanks!

Lorenzo


123