A major problem with the Lua ecosystem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
62 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

A major problem with the Lua ecosystem

Dirk Laurie-2
2018-02-01 0:08 GMT+02:00 Dibyendu Majumdar <[hidden email]>:

> On 31 January 2018 at 22:03, Paul K <[hidden email]> wrote:
>>> require "paths"
>>> paths.require "libtorch"
>>
>>> The paths.lua script is generated from:
>>> https://github.com/torch/torch7/blob/master/paths.lua.in
>>> I must be missing something here.
>>
>> I think it's a different paths module. I'd expect it to be this one:
>> https://github.com/torch/paths
>>
>
> Ah okay thanks! Strange that there is 'paths.lua' in the Lua script folder.

The problem has two parts:

1. The name of the module is often a common word like "paths"
which may easily be non-unique.
2. The name under which the module is contributed to LuaRocks
is not the same as the name that comes in "require".

The Lua team has done what they could: "require" can specify
subpaths, which so far seem mostly to be exploited by writers
of large module collections.

What we as a community shoud do is to agree on a standard
directory tree to act as containers. just one or two levels.
Then we can say "paths = require 'torch.paths'" and make
sure we get the right one.

For this idea to work with "lua -l" , we also need the patch
to preload libraries with customized global names
   http://lua-users.org/wiki/LuaPowerPatches
so that

   lua -l paths=torch.paths

is possible.

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Pierre Chapuis
On Thu, Feb 1, 2018, at 09:22, Dirk Laurie wrote:

> For this idea to work with "lua -l" , we also need the patch
> to preload libraries with customized global names
>    http://lua-users.org/wiki/LuaPowerPatches
> so that
>
>    lua -l paths=torch.paths
>
> is possible.

Is there a major advantage compared to this, which works today?

    lua -e "paths=require'torch.paths'"

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

云风 Cloud Wu
In reply to this post by Dirk Laurie-2
Dirk Laurie <[hidden email]>于2018年2月1日周四 下午4:22写道:
2018-02-01 0:08 GMT+02:00 Dibyendu Majumdar <[hidden email]>:
> On 31 January 2018 at 22:03, Paul K <[hidden email]> wrote:
>>> require "paths"
>>> paths.require "libtorch"
>>
>>> The paths.lua script is generated from:
>>> https://github.com/torch/torch7/blob/master/paths.lua.in
>>> I must be missing something here.
>>
>> I think it's a different paths module. I'd expect it to be this one:
>> https://github.com/torch/paths
>>
>
> Ah okay thanks! Strange that there is 'paths.lua' in the Lua script folder.

The problem has two parts:

1. The name of the module is often a common word like "paths"
which may easily be non-unique.
2. The name under which the module is contributed to LuaRocks
is not the same as the name that comes in "require".

The Lua team has done what they could: "require" can specify
subpaths, which so far seem mostly to be exploited by writers
of large module collections.

What we as a community shoud do is to agree on a standard
directory tree to act as containers. just one or two levels.
Then we can say "paths = require 'torch.paths'" and make
sure we get the right one.

For this idea to work with "lua -l" , we also need the patch
to preload libraries with customized global names
   http://lua-users.org/wiki/LuaPowerPatches
so that

   lua -l paths=torch.paths

is possible.


I'd like the lua package system can support relative path like python.

So I use a simple function `import` to replace official `require` in my project:

local loaded = package.loaded
local searchpath = package.searchpath

function import(modname)
if modname then
local prefix = modname:match "(.*%.).*$" or (modname .. ".")
return function(name)
local fullname = prefix .. name
local m = loaded[fullname] or loaded[name]
if m then
return m
end
if searchpath(fullname, package.path) then
return require(fullname)
else
return require(name)
end
end
else
return require
end
end

And then put this line in the first line of every sub-module :

local require = import and import(...) or require


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Dirk Laurie-2
In reply to this post by Pierre Chapuis
2018-02-01 11:42 GMT+02:00 Pierre Chapuis <[hidden email]>:

> On Thu, Feb 1, 2018, at 09:22, Dirk Laurie wrote:
>
>> For this idea to work with "lua -l" , we also need the patch
>> to preload libraries with customized global names
>>    http://lua-users.org/wiki/LuaPowerPatches
>> so that
>>
>>    lua -l paths=torch.paths
>>
>> is possible.
>
> Is there a major advantage compared to this, which works today?
>
>     lua -e "paths=require'torch.paths'"

No more major than being able to write "obj:method(...)"
instead of "obj.method(obj,...)", "function foo(...)" instead of
"foo = function(...)", "tbl.key" instead of "tbl['key']" etc.

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Dibyendu Majumdar
In reply to this post by Pierre Chapuis

Hi

Sorry I should know the answer to this but I don't. Does Lua support dotted module names... i.e. map to directory path?

Thanks and regards
Dibyendu ‎

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:
>
> Hi
>
> Sorry I should know the answer to this but I don't. Does Lua support dotted module names... i.e. map to directory path?

  You mean something like:

        net = require "org.conman.net" --?

If so, then yes, Lua does.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Tim Hill


> On Feb 1, 2018, at 8:41 AM, Sean Conner <[hidden email]> wrote:
>
> It was thus said that the Great Dibyendu Majumdar once stated:
>>
>> Hi
>>
>> Sorry I should know the answer to this but I don't. Does Lua support dotted module names... i.e. map to directory path?
>
>  You mean something like:
>
> net = require "org.conman.net" --?
>
> If so, then yes, Lua does.
>
>  -spc
>
>

This cleanly solves the external namespace, assuming someone can promote some form of standardization around whatever convention is agreed (LuaRocks team???). But it leaves open that global variable used for the assignment target. If I control the whole corpus of Lua code in a project, then I can control the internal global namespace, but if I’m using 3rd party Lua code which ITSELF using require(), how to I handle namespace collisions on these globals?

This would seem to recommend a coding style where a loadable Lua module use locals only for require():

local net = require(“org.timhill.network”)


—Tim


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Sean Conner
It was thus said that the Great Tim Hill once stated:

> > On Feb 1, 2018, at 8:41 AM, Sean Conner <[hidden email]> wrote:
> > It was thus said that the Great Dibyendu Majumdar once stated:
> >>
> >> Sorry I should know the answer to this but I don't. Does Lua support
> >> dotted module names... i.e. map to directory path?
> >
> >  You mean something like:
> >
> > net = require "org.conman.net" --?
> >
> > If so, then yes, Lua does.
>
> This cleanly solves the external namespace, assuming someone can promote
> some form of standardization around whatever convention is agreed
> (LuaRocks team???). But it leaves open that global variable used for the
> assignment target. If I control the whole corpus of Lua code in a project,
> then I can control the internal global namespace, but if I’m using 3rd
> party Lua code which ITSELF using require(), how to I handle namespace
> collisions on these globals?

  That's really only an issue with lua 5.1, which *will always* create a
global when loading a module.  Lua 5.2+ will no longer create the global and
it's up to the programmer to cache the value from require() into some
variable.

  So yes, this also works:

        local net = require "org.conman.net"

Lua 5.1 will still create a global "org", inside which is "conman" and
inside that "net".  Lua 5.2+ won't do that.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Sean Conner
In reply to this post by Dirk Laurie-2
It was thus said that the Great Dirk Laurie once stated:
>
> The problem has two parts:
>
> 1. The name of the module is often a common word like "paths"
> which may easily be non-unique.
> 2. The name under which the module is contributed to LuaRocks
> is not the same as the name that comes in "require".

  The worst example of #2 that which I've found is Lua XML.  The LuaRocks
module is "luaxml".  The file is "LuaXml", which creates (at least under Lua
5.1 which I'm still using) a global "xml".  It's a real mess.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Soni "They/Them" L.
In reply to this post by Tim Hill


On 2018-02-01 06:34 PM, Tim Hill wrote:

>
>> On Feb 1, 2018, at 8:41 AM, Sean Conner <[hidden email]> wrote:
>>
>> It was thus said that the Great Dibyendu Majumdar once stated:
>>> Hi
>>>
>>> Sorry I should know the answer to this but I don't. Does Lua support dotted module names... i.e. map to directory path?
>>   You mean something like:
>>
>> net = require "org.conman.net" --?
>>
>> If so, then yes, Lua does.
>>
>>   -spc
>>
>>
> This cleanly solves the external namespace, assuming someone can promote some form of standardization around whatever convention is agreed (LuaRocks team???). But it leaves open that global variable used for the assignment target. If I control the whole corpus of Lua code in a project, then I can control the internal global namespace, but if I’m using 3rd party Lua code which ITSELF using require(), how to I handle namespace collisions on these globals?
>
> This would seem to recommend a coding style where a loadable Lua module use locals only for require():
>
> local net = require(“org.timhill.network”)
> …
>
> —Tim
>
>

I don't like using centralized authorities (DNS) for this stuff.

Why not use UUIDs?

local net = require "YOUR_UUID.network"

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Paige DePol
Soni They/Them L. <[hidden email]> wrote:

> I don't like using centralized authorities (DNS) for this stuff.
>
> Why not use UUIDs?
>
> local net = require "YOUR_UUID.network"

UUID Identifier:

local mod = require "5ae29a42-0795-11e8-ba89-0ed5f89f718b"

Versus:

local mod = require "net.fizzypop.socket"

I definitely prefer the domain style identifier over the UUID
style ones. Also, it's not like you have to register domain
lookalike identifiers with a central registrar or anything.

It is just a convenient way to create a hierarchy for modules
created by the same person or organisation. The domain style
identifiers are like UUIDs, just ones that carry meaning.

For example, it is way more apparent what "net.fizzypop.socket"
will do... without having to look it up somewhere how would
you know what "5ae29a42-0795-11e8-ba89-0ed5f89f718b" does?

Actually, if anything, you would almost need to create some
sort of central registry to manage the UUIDs used by people
and organisations and to map UUIDs to some logical names.

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Peter Aronoff
Paige DePol <[hidden email]> wrote:

> Soni They/Them L. <[hidden email]> wrote:
>
> > I don't like using centralized authorities (DNS) for this stuff.
> >
> > Why not use UUIDs?
> >
> > local net = require "YOUR_UUID.network"
>
> UUID Identifier:
>
> local mod = require "5ae29a42-0795-11e8-ba89-0ed5f89f718b"
>
> Versus:
>
> local mod = require "net.fizzypop.socket"

In fairness to Soni, shouldn’t that be

local mod = require "5ae29a42-0795-11e8-ba89-0ed5f89f718b.socket"

> For example, it is way more apparent what "net.fizzypop.socket" will
> do... without having to look it up somewhere how would you know what
> "5ae29a42-0795-11e8-ba89-0ed5f89f718b" does?

See above: this is a draw: the socket part gives equal information in both.

That said, I also prefer the domain-style.

P

--
We have not been faced with the need to satisfy someone else's
requirements, and for this freedom we are grateful.
    Dennis Ritchie and Ken Thompson, The UNIX Time-Sharing System

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Dibyendu Majumdar
In reply to this post by Dirk Laurie-2
On 1 February 2018 at 08:22, Dirk Laurie <[hidden email]> wrote:

> 2018-02-01 0:08 GMT+02:00 Dibyendu Majumdar <[hidden email]>:
>> On 31 January 2018 at 22:03, Paul K <[hidden email]> wrote:
>>>> require "paths"
>>>> paths.require "libtorch"
>>>
>>>> The paths.lua script is generated from:
>>>> https://github.com/torch/torch7/blob/master/paths.lua.in
>>>> I must be missing something here.
>>>
>>> I think it's a different paths module. I'd expect it to be this one:
>>> https://github.com/torch/paths
>>>
>>
>> Ah okay thanks! Strange that there is 'paths.lua' in the Lua script folder.
>
> The problem has two parts:
>
> 1. The name of the module is often a common word like "paths"
> which may easily be non-unique.
> 2. The name under which the module is contributed to LuaRocks
> is not the same as the name that comes in "require".
>
> The Lua team has done what they could: "require" can specify
> subpaths, which so far seem mostly to be exploited by writers
> of large module collections.
>
> What we as a community shoud do is to agree on a standard
> directory tree to act as containers. just one or two levels.
> Then we can say "paths = require 'torch.paths'" and make
> sure we get the right one.
>

I didn't even realize that Lua supported require 'a.b.c' and that this
is translated to a filesystem path, maybe because I haven't seen it
being used.

Will certainly make sure that all modules in the Ravi distro follow a
supplier.modue (e.g. torch.paths) pattern - can't imagine why people
don't do this. It is a no brainer to me.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Sean Conner
In reply to this post by Paige DePol
It was thus said that the Great Paige DePol once stated:
>
> Actually, if anything, you would almost need to create some
> sort of central registry to manage the UUIDs used by people
> and organisations and to map UUIDs to some logical names.

  Not really.  The generally used format for UUID is version 4 [1] is
randomly generated, and it's considered unlikely that randomly generated
UUIDs will clash (it's possible, but very unlikely).  And most modern Unix
systems (at least Linux and Mac OS-X, both of which I tested) come with a
command (uuidgen) that generates UUIDs.

  There are other versions as well---version 1 (not generally recommended
because of security concerns [2]) is well guarenteed to be unique since it's
based off the time of generation, and versions 3 and 5 use various hash
algorithms to generate UUIDs.

  This really depends upon a organization (or person) consistently using the
same UUID for each module released.

  -spc

[1] RFC-4122

[2] It embeds the MAC address of the generating node inside the UUID.

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Paige DePol
In reply to this post by Peter Aronoff
Peter Aronoff <[hidden email]> wrote:

> In fairness to Soni, shouldn’t that be
>
> local mod = require "5ae29a42-0795-11e8-ba89-0ed5f89f718b.socket"

Ah yes, the original identifier was:

> local net = require(“org.timhill.network”)

Where Soni's suggestion was:

> local net = require "YOUR_UUID.network"

I left off 'network' by accident in my followup as I was comparing the
two styles of identifier; UUID vs Domain.

If appending words to a UUID to make it have meaning is required then
I fail to see how that is any different than just using a domain style
identifier in the first place. At least the domain style identifier has
the extra bonus of carrying information about the module author.

~Paige


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Zach Villers
In reply to this post by Dirk Laurie-2
> The problem has two parts:
 
> 1. The name of the module is often a common word like "paths"
> which may easily be non-unique.
> 2. The name under which the module is contributed to LuaRocks
> is not the same as the name that comes in "require".
>
> The Lua team has done what they could: "require" can specify
> subpaths, which so far seem mostly to be exploited by writers
> of large module collections.
>

I'm really new to Lua and programming in general. Would it be possible to implement something like python's virtualenv for Lua? I guess I'm just asking for *nix-likes (I don't know windows). I think virtualenv creates almost a chroot for the python interpreter. I have a link for a video to do research, but again, am pretty lost at this point. When I'm trying to learn a new module, it's difficult to understand if I don't have $LUA_PATH set correctly or if I'm calling the module wrong, or something else.

http://pyvideo.org/pycon-us-2011/pycon-2011--reverse-engineering-ian-bicking--39-s.html

Thanks
--
  Zach Villers
  [hidden email]
 

Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Paige DePol
In reply to this post by Sean Conner
Sean Conner <[hidden email]> wrote:

> It was thus said that the Great Paige DePol once stated:
>> Actually, if anything, you would almost need to create some
>> sort of central registry to manage the UUIDs used by people
>> and organisations and to map UUIDs to some logical names.
>
>  Not really.  The generally used format for UUID is version 4 [1] is
> randomly generated, and it's considered unlikely that randomly generated
> UUIDs will clash (it's possible, but very unlikely).  And most modern Unix
> systems (at least Linux and Mac OS-X, both of which I tested) come with a
> command (uuidgen) that generates UUIDs.
>
>  There are other versions as well---version 1 (not generally recommended
> because of security concerns [2]) is well guarenteed to be unique since it's
> based off the time of generation, and versions 3 and 5 use various hash
> algorithms to generate UUIDs.
>
>  This really depends upon a organization (or person) consistently using the
> same UUID for each module released.
>
>  -spc
>
> [1] RFC-4122
>
> [2] It embeds the MAC address of the generating node inside the UUID.

After doing some reading I can see that there are both completely random
UUIDs, as well as ones that are generated from namespaces. It was more the
namespace ones I was thinking about, vs the totally random 128-bit ones.

My point about a registry was that if everyone was using UUIDs then we might
want some sort of registry so we'd know which UUID namespaces belonged to
which person or company.

I guess I just fail to see how using UUIDs instead of easy to read domain
style identifiers would make things any easier or useful is all.

~Paige



Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Soni "They/Them" L.


On 2018-02-01 08:30 PM, Paige DePol wrote:

> Sean Conner <[hidden email]> wrote:
>
>> It was thus said that the Great Paige DePol once stated:
>>> Actually, if anything, you would almost need to create some
>>> sort of central registry to manage the UUIDs used by people
>>> and organisations and to map UUIDs to some logical names.
>>   Not really.  The generally used format for UUID is version 4 [1] is
>> randomly generated, and it's considered unlikely that randomly generated
>> UUIDs will clash (it's possible, but very unlikely).  And most modern Unix
>> systems (at least Linux and Mac OS-X, both of which I tested) come with a
>> command (uuidgen) that generates UUIDs.
>>
>>   There are other versions as well---version 1 (not generally recommended
>> because of security concerns [2]) is well guarenteed to be unique since it's
>> based off the time of generation, and versions 3 and 5 use various hash
>> algorithms to generate UUIDs.
>>
>>   This really depends upon a organization (or person) consistently using the
>> same UUID for each module released.
>>
>>   -spc
>>
>> [1] RFC-4122
>>
>> [2] It embeds the MAC address of the generating node inside the UUID.
> After doing some reading I can see that there are both completely random
> UUIDs, as well as ones that are generated from namespaces. It was more the
> namespace ones I was thinking about, vs the totally random 128-bit ones.
>
> My point about a registry was that if everyone was using UUIDs then we might
> want some sort of registry so we'd know which UUID namespaces belonged to
> which person or company.
>
> I guess I just fail to see how using UUIDs instead of easy to read domain
> style identifiers would make things any easier or useful is all.

This is code we're talking about. I don't think code should rely on a
centralized service like DNS. Code should be decentralized.

Just make a random UUID, publish it somewhere, perhaps with a notation
"lua-package:UUID", and have it indexed by search engines. This also
doesn't depend on DNS and the code/UUID could be hosted on e.g. a Tor
hidden service (which would make it a bit tricky to look up, but at
least it wouldn't be centralized).

>
> ~Paige
>
>
>

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Gregg Reynolds-2
In reply to this post by Soni "They/Them" L.


On Feb 1, 2018 3:14 PM, "Soni "They/Them" L." <[hidden email]> wrote:
...
I don't like using centralized authorities (DNS) for this stuff.

Why not use UUIDs?

local net = require "YOUR_UUID.network"

Same reason we do not use random strings for var names: the reader is a human.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.



Reply | Threaded
Open this post in threaded view
|

Re: A major problem with the Lua ecosystem

Gregg Reynolds-2
In reply to this post by Soni "They/Them" L.


On Feb 1, 2018 4:34 PM, "Soni "They/Them" L." <[hidden email]> wrote:


This is code we're talking about. I don't think code should rely on a centralized service like DNS. Code should be decentralized.

Waiting for your DNS replacement...
1234