Non-uniqueness of module names

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Non-uniqueness of module names

Dirk Laurie-2
The Lua ecosystem will never be worth that name as long as there is no
way to address the module naming issue.

There are two modules available via LuaRocks, neither of which is
called `complex` but both of which install a module loadable (in
theory) by
`require "complex"`. I have been using one for years, the other (where
`complex` is only incidental support for a main module) only for a
week. When today I ran a program dependent on the first of these
modules, it failed, because the interface for the two modules is
different, and the second module, finding its code in a file called
complex.lua, is preferred over the first, which finds its code in a
file called complex.so.

Of course there are workarounds. I know them.

But life could be made easier for the poor programmer by addressing
two great defects:

In the Lua standard library, there is no easy way of detecting where
the module was found. One can repeat all the work using
package.searchpath to find out where the module should have been
found, but you can't check whether the module actually loaded was that
one.

In LuaRocks, there is no check, when you install a rock, that there
could be naming conflicts between modules supplied by that rock and
modules supllied by rocks already loaded.

One is really forced to check for signatures in installed modules, or
run a functionality check, or supply such a detailed package.path that
the desired module can't be missed. Any of these makes a mockery of
the supposed ease of use of the module system.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Dibyendu Majumdar
On Fri, 12 Apr 2019 at 14:50, Dirk Laurie <[hidden email]> wrote:
>
> The Lua ecosystem will never be worth that name as long as there is no
> way to address the module naming issue.

Hi Dirk,

It took me a while to understand this, but Lua doesn't care about how
you manage modules or if there is an eco system of modules. Basically
you are on your own, so you can fix it yourself.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Roberto Ierusalimschy
In reply to this post by Dirk Laurie-2
> [...]
>
> But life could be made easier for the poor programmer by addressing
> two great defects:
>
> In the Lua standard library, there is no easy way of detecting where
> the module was found. One can repeat all the work using
> package.searchpath to find out where the module should have been
> found, but you can't check whether the module actually loaded was that
> one.

What would be like such feature? Do you have something in mind?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Abhijit Nandy
So why aren't the paths from where the modules are loaded, simply kept in a table mapped to the module name?



On Fri, Apr 12, 2019 at 8:10 PM Roberto Ierusalimschy <[hidden email]> wrote:
> [...]
>
> But life could be made easier for the poor programmer by addressing
> two great defects:
>
> In the Lua standard library, there is no easy way of detecting where
> the module was found. One can repeat all the work using
> package.searchpath to find out where the module should have been
> found, but you can't check whether the module actually loaded was that
> one.

What would be like such feature? Do you have something in mind?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Luiz Henrique de Figueiredo
> So why aren't the paths from where the modules are loaded, simply kept in a table mapped to the module name?

How would that help scripts that need to run in several environments?
The paths are almost certainly not portable.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Dirk Laurie-2
Op Vr. 12 Apr. 2019 om 17:27 het Luiz Henrique de Figueiredo
<[hidden email]> geskryf:
>
> > So why aren't the paths from where the modules are loaded, simply kept in a table mapped to the module name?
>
> How would that help scripts that need to run in several environments?
> The paths are almost certainly not portable.

In my particular case, knowing whether the found file is Lua code is
already enough.

If package.found[modname] contains the result stored in *name after
 findloader(L, name);
(line 606, loadlib.c)
then my application could issue a message (and/or write it to a log file) saying

Module "complex" found as /usr/local/share/lua/5.3/complex.lua

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Sean Conner
In reply to this post by Abhijit Nandy
It was thus said that the Great Abhijit Nandy once stated:
> So why aren't the paths from where the modules are loaded, simply kept in a
> table mapped to the module name?

  Because not all Lua modules have a path.  Aside from the builtin modules
like io or table, it is possible to modify the module system to load modules
not on the filesystem.  At work, I have a custom Lua interpreter that embeds
all the Lua modules we need into a single Lua executable.  This includes
modules written in C and Lua [1].  There is no path for these modules.

  Then I played around with an idea to load Lua modules directly from a zip
file [2][3].  What's the path for these?  That of the zip file?  It doesn't
make sense.

  -spc

[1] http://boston.conman.org/2013/03/23.1

[2] http://boston.conman.org/2018/02/19.2
        https://github.com/spc476/LEM

[3] It kind of worked, in that I could load and run Lua modules written
        in Lua directly from the zip file, but not those written in C.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Sean Conner
In reply to this post by Dirk Laurie-2
It was thus said that the Great Dirk Laurie once stated:

> Op Vr. 12 Apr. 2019 om 17:27 het Luiz Henrique de Figueiredo
> <[hidden email]> geskryf:
> >
> > > So why aren't the paths from where the modules are loaded, simply kept in a table mapped to the module name?
> >
> > How would that help scripts that need to run in several environments?
> > The paths are almost certainly not portable.
>
> In my particular case, knowing whether the found file is Lua code is
> already enough.
>
> If package.found[modname] contains the result stored in *name after
>  findloader(L, name);
> (line 606, loadlib.c)
> then my application could issue a message (and/or write it to a log file) saying
>
> Module "complex" found as /usr/local/share/lua/5.3/complex.lua
  I have a script (attached) that when run from the command line will tell
you where a module is located:

[spc]lucy:~>luawhich io
package.loaded["io"]
[spc]lucy:~>luawhich org.conman.syslog
/home/spc/.luarocks/lib/lua/5.3/org/conman/syslog.so
/usr/local/lib/lua/5.3/org/conman/syslog.so
[spc]lucy:~>

  It even works to a degree with my custom Lua interpreter at work (which
contains all the modules we need in the executable):

[spc]dynamic-147:~>kslua luawhich org.conman.syslog
package.preload["org.conman.syslog"]
/usr/local/lib/lua/5.1/org/conman/syslog.so
[spc]dynamic-147:~>kslua luawhich sim
package.preload["sim"]
[spc]dynamic-147:~>

  Here we can see org.conman.syslog is found in package.preloaded[] (since
it's a) in my kslua program [1] and b) written in C).  I also include a
bunch of modules written in Lua but because of the whay they work and how
they're processed (they're comporessed [2] in the executable), I had to add
a searcher (loader in Lua 5.1)---this isn't known by my program though:

[spc]dynamic-147:~>kslua luawhich org.conman.date
/usr/local/share/lua/5.1/org/conman/date.lua
[spc]dynamic-147:~>

  This script could possibly be adapted as a module.

  -spc (It's an idea ... )

[1] Kitchen Sink Lua

[2] Using libz.  There's a few modules that are quite large.

luawhich (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Dirk Laurie-2
In reply to this post by Sean Conner
Op Vr. 12 Apr. 2019 om 21:38 het Sean Conner <[hidden email]> geskryf:
>
> It was thus said that the Great Abhijit Nandy once stated:
> > So why aren't the paths from where the modules are loaded, simply kept in a
> > table mapped to the module name?
>
>   Because not all Lua modules have a path.

All modules loaded by 'require' and not found in package.loaded have a
path. These are the ones of interest in situations where there may be
uncertainty as to where the package was found.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Sean Conner
It was thus said that the Great Dirk Laurie once stated:

> Op Vr. 12 Apr. 2019 om 21:38 het Sean Conner <[hidden email]> geskryf:
> >
> > It was thus said that the Great Abhijit Nandy once stated:
> > > So why aren't the paths from where the modules are loaded, simply kept in a
> > > table mapped to the module name?
> >
> >   Because not all Lua modules have a path.
>
> All modules loaded by 'require' and not found in package.loaded have a
> path. These are the ones of interest in situations where there may be
> uncertainty as to where the package was found.

  Except for those in package.preload.

  -spc (Which was the point of the email you replied to ... )

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Gabriel Bertilson
In reply to this post by Dirk Laurie-2
I've used a function like this, which returns the path of a module, or
else `nil` and the string containing the paths that were searched.

function get_path(module_name)
    local res, err = package.searchpath(module_name, package.path)
    if not res then
        local temp_err
        res, temp_err = package.searchpath(module_name, package.cpath)
        if not res then
            return nil, err .. temp_err
        end
    end
    return res
end

It could be used in a custom `require` function that would return the
module and the path, something like this:

local real_require = require
function require(module_name)
    local res = real_require(module_name)
    if res then
        return res, get_path(module_name)
    end
end

Not the most efficient because it searches the paths twice.

— Gabriel

— Gabriel



On Fri, Apr 12, 2019 at 8:50 AM Dirk Laurie <[hidden email]> wrote:

>
> The Lua ecosystem will never be worth that name as long as there is no
> way to address the module naming issue.
>
> There are two modules available via LuaRocks, neither of which is
> called `complex` but both of which install a module loadable (in
> theory) by
> `require "complex"`. I have been using one for years, the other (where
> `complex` is only incidental support for a main module) only for a
> week. When today I ran a program dependent on the first of these
> modules, it failed, because the interface for the two modules is
> different, and the second module, finding its code in a file called
> complex.lua, is preferred over the first, which finds its code in a
> file called complex.so.
>
> Of course there are workarounds. I know them.
>
> But life could be made easier for the poor programmer by addressing
> two great defects:
>
> In the Lua standard library, there is no easy way of detecting where
> the module was found. One can repeat all the work using
> package.searchpath to find out where the module should have been
> found, but you can't check whether the module actually loaded was that
> one.
>
> In LuaRocks, there is no check, when you install a rock, that there
> could be naming conflicts between modules supplied by that rock and
> modules supllied by rocks already loaded.
>
> One is really forced to check for signatures in installed modules, or
> run a functionality check, or supply such a detailed package.path that
> the desired module can't be missed. Any of these makes a mockery of
> the supposed ease of use of the module system.
>

Jim
Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Jim
In reply to this post by Dirk Laurie-2
On 4/12/19, Dirk Laurie <[hidden email]> wrote:
> Any of these makes a mockery of the supposed ease
> of use of the module system.

indeed.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Sean Conner
In reply to this post by Abhijit Nandy
It was thus said that the Great Abhijit Nandy once stated:
> So why aren't the paths from where the modules are loaded, simply kept in a
> table mapped to the module name?

  Let's assume this happens.  package.locations[] contains the path from
where a given module was loaded from.  What types of checks could one
realistically do given this information?  On my system, a module can appear
in one of these paths:

        /home/spc/.luarocks/share/lua/5.3/
        /home/spc/.luarocks/lib/lua/5.3/
        /usr/local/share/lua/5.3/
        /usr/local/lib/lua/5.3/
        ./

  For a program I'm using, yes, I can check to see if module complex was
loaded from "/home/spc/.luarocks/share/lua/5.3/complex.lua" but I can't
release it that way for others to use.

  Another issue I've encountered is with a module installed in multiple
locations.  Granted, this is something I do to myself.  I make a change to
one of my modules and install it from source where it's installed to
"/usr/local/share/lua/5.3/".  But I forgot that I installed it from LuaRocks
the last time I released it (to make sure it works via LuaRocks) and the
previous version is in "/home/spc/.luarocks/share/lua/5.3/", which is listed
first in package.path.  And I wonder why the new code isn't working ...

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Dirk Laurie-2
Op So. 14 Apr. 2019 om 22:32 het Sean Conner <[hidden email]> geskryf:
>
> It was thus said that the Great Abhijit Nandy once stated:
> > So why aren't the paths from where the modules are loaded, simply kept in a
> > table mapped to the module name?
>
>   Let's assume this happens.  package.locations[] contains the path from
> where a given module was loaded from.

The full filename, please. The distinction between module.so,
module.lua and module/init.lua is important.

> What types of checks could one
> realistically do given this information?  On my system, a module can appear
> in one of these paths:
>
>         /home/spc/.luarocks/share/lua/5.3/
>         /home/spc/.luarocks/lib/lua/5.3/
>         /usr/local/share/lua/5.3/
>         /usr/local/lib/lua/5.3/
>         ./
>
>   For a program I'm using, yes, I can check to see if module complex was
> loaded from "/home/spc/.luarocks/share/lua/5.3/complex.lua" but I can't
> release it that way for others to use.

It is not important to anticipate how the filename is used, only to
preserve the location where the module loader was found.

Simlar information is provided by several well respected software
products that satisfy a dependency from any of several sources.

* Un*x provides the command `which`.
* TeX displays on the console and writes to a log file every file that
it includes.
* Lua's debug.getinfo gives a field `source`.

Lua's default PATH and CPATH are much less ambitious than the paths of
Un*x or TeX, but the information is so basic that the question should
rather be why not than why.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Roberto Ierusalimschy
In reply to this post by Dirk Laurie-2
I am not sure you are aware that the module itself already gets
this information:

---- file temp.lua
print(...)

$ lua -l temp
    --> temp ./temp.lua

Maybe 'require' could simply return this information as a second result?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Gerhard Sittig-2
In reply to this post by Dirk Laurie-2
On Mon, 2019-04-15 at 09:14 +0200, Dirk Laurie wrote:
>
> Simlar information is provided by several well respected software
> products that satisfy a dependency from any of several sources.
>
> * Un*x provides the command `which`.

That's a nice example.

The which(1) utility only works for input which happens to be an
executable file somewhere in a path.  This need not match what
actually happens (or happened) when the argument to which(1) got
invoked or will get invoked.  Besides executables files there are
also functions and aliases and builtin commands.  Things
addressable by some path is only a subset of the problem space.

Execution of the which(1) command also happens at a different
point in time than the execution of the executable that you want
to gather information about.  So the result may not mean a thing,
or may be wrong and misleading.  This translates to looking up
the module's content and thus assumed behaviour and running the
module at different points in time.

There are other approaches which are equally unreliable.  Lookup
in proc(5) how the process got started, and the information won't
tell you much.  The result can have become obsolete, need not be
correct, need not be there at all, may return a different result
at the time you query the detail (or query again).  This is
similar to pasing to the module at runtime what was gathered at
load time -- need not mean a thing.

Some platforms supported by Lua may not even have the notion of
"a path" to a module.  You may assume a specific environment
which does not necessarily translate well everywhere.  There were
examples of builtin and preloaded modules.  There were examples
of modules loaded from volatile storage (ZIP archives, network
sources, pipes, strings, etc).


Assuming a specific use at the OP's, namely to tell different
modules of the same name apart, it's more reliable to test
versions, or even better test the availability of features or
their behaviour.  Very much what some Lua interpreter version
checks do.  Because the initial question was how to tell those
different modules apart, was it not?

Insisting in specific filenames (if they even exist) seems to be
just a workaround after the actual problem of telling module
behaviour apart was not available (or was not considered, or one
of several approaches was picked early without considering other
approaches).


> Lua's default PATH and CPATH are much less ambitious than the paths of
> Un*x or TeX, but the information is so basic that the question should
> rather be why not than why.

Hmm.  Wasn't "why not?" the wrong approach to "selling" a
suggested change to the Lua authors? :)  When you expect them to
consider a change or extension, they do want to know _why_ they
should bother adding and then maintaining that extension ...
There should be a real reason why a change should be made, not a
lack of (spoken, and agreed on) reasons to reject the suggestion.


virtually yours
Gerhard Sittig
--
     If you don't understand or are scared by any of the above
             ask your parents or an adult to help you.

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Luiz Henrique de Figueiredo
> Assuming a specific use at the OP's, namely to tell different
> modules of the same name apart, it's more reliable to test
> versions

My modules contain a "version" field that can be used for that.
If only module writers could agree to have that field and on what is
inside it...

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Sean Conner
In reply to this post by Dirk Laurie-2
It was thus said that the Great Dirk Laurie once stated:
>
> It is not important to anticipate how the filename is used, only to
> preserve the location where the module loader was found.

  Sigh.

  Did it not occur to you that you could write your own require() function
and get what you want?  Even if it's just a proof-of-concept and a way to
see if it breaks any existing code.  It's not hard.  Here's a version that
will store the location information in package.where (as well as which
searcher in package.searchers[] returned the information, since not all
searchers return path information), in addition to returning the path name
as a second return value (per Roberto's suggestion):

-- ***********************************************************************
--
--    A replacement require() function.
--    Copyright (C) 2019 by Sean Conner.
--
--    This program is free software: you can redistribute it and/or modify
--    it under the terms of the GNU Affero General Public License as
--    published by the Free Software Foundation, either version 3 of the
--    License, or (at your option) any later version.
--
--    This program is distributed in the hope that it will be useful,
--    but WITHOUT ANY WARRANTY; without even the implied warranty of
--    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
--    GNU Affero General Public License for more details.
--
--    You should have received a copy of the GNU Affero General Public License
--    along with this program.  If not, see <https://www.gnu.org/licenses/>.
--
--    Comments, questions and criticisms can be sent to: [hidden email]
--
-- ***********************************************************************

package.where = {}

function require(modname)
  if package.loaded[modname] then
    return package.loaded[modname]
  end
 
  local err = {}

  for i,search in ipairs(package.searchers) do
    local modf,param = search(modname)
    if type(modf) == 'string' then
      table.insert(err,modf)
    elseif type(modf) == 'function' then
      local rc = modf(modname,param)
      package.loaded[modname] = package.loaded[modname] or rc or true
      package.where [modname] = { i , param }
      return package.loaded[modname],param
    end
  end
 
  error(table.concat(err,'\n'),2)
end

  -spc (Why the onerous license?  Because this is what you get when you
        don't do your own homework!)

Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Sean Conner
In reply to this post by Luiz Henrique de Figueiredo
It was thus said that the Great Luiz Henrique de Figueiredo once stated:
> > Assuming a specific use at the OP's, namely to tell different
> > modules of the same name apart, it's more reliable to test
> > versions
>
> My modules contain a "version" field that can be used for that.
> If only module writers could agree to have that field and on what is
> inside it...

  Perhaps the version number that matches the LuaRocks version?  As an
example, for the rockspec "org.conman.syslog-2.1.3-3.rockspec", the version
inside would be _VERSION = "2.1.3".

  That's fine for modules that return a table of functions (which is
probably 90% of all modules).  But not all modules need be a table (all the
modules under org.conman.parsers are LPEG expressions).

  -spc




Reply | Threaded
Open this post in threaded view
|

Re: Non-uniqueness of module names

Soni "They/Them" L.
In reply to this post by Sean Conner


On 2019-04-15 6:08 p.m., Sean Conner wrote:

> It was thus said that the Great Dirk Laurie once stated:
>> It is not important to anticipate how the filename is used, only to
>> preserve the location where the module loader was found.
>    Sigh.
>
>    Did it not occur to you that you could write your own require() function
> and get what you want?  Even if it's just a proof-of-concept and a way to
> see if it breaks any existing code.  It's not hard.  Here's a version that
> will store the location information in package.where (as well as which
> searcher in package.searchers[] returned the information, since not all
> searchers return path information), in addition to returning the path name
> as a second return value (per Roberto's suggestion):
>
> -- ***********************************************************************
> --
> --    A replacement require() function.
> --    Copyright (C) 2019 by Sean Conner.
> --
> --    This program is free software: you can redistribute it and/or modify
> --    it under the terms of the GNU Affero General Public License as
> --    published by the Free Software Foundation, either version 3 of the
> --    License, or (at your option) any later version.
> --
> --    This program is distributed in the hope that it will be useful,
> --    but WITHOUT ANY WARRANTY; without even the implied warranty of
> --    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> --    GNU Affero General Public License for more details.
> --
> --    You should have received a copy of the GNU Affero General Public License
> --    along with this program.  If not, see <https://www.gnu.org/licenses/>.
> --
> --    Comments, questions and criticisms can be sent to: [hidden email]
> --
> -- ***********************************************************************
>
> package.where = {}
>
> function require(modname)
>    if package.loaded[modname] then
>      return package.loaded[modname]
>    end
>    
>    local err = {}
>
>    for i,search in ipairs(package.searchers) do
>      local modf,param = search(modname)
>      if type(modf) == 'string' then
>        table.insert(err,modf)
>      elseif type(modf) == 'function' then
>        local rc = modf(modname,param)
>        package.loaded[modname] = package.loaded[modname] or rc or true
>        package.where [modname] = { i , param }
>        return package.loaded[modname],param
>      end
>    end
>    
>    error(table.concat(err,'\n'),2)
> end

Nice! Altho I think you should use table.concat(err, '') instead. IIRC
this is how vanilla require does it.

This probably doesn't enjoy changing package.searchers at runtime, tho.
I'd put the whole search function in the where entry, and use named
indexes. This isn't a hot path - or shouldn't be - anyway.

>
>    -spc (Why the onerous license?  Because this is what you get when you
> don't do your own homework!)
>

I might use this at some point, just because.

123