package.searcher modification for lua modules

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

package.searcher modification for lua modules

aryajur
Hi,
      Currently if package.path has a path like:

C:\a\b\c\..\?\d\?.lua

Then it require("mod") searches for

C:\a\b\c\..\mod\d\mod.lua  -> C:\a\b\mod\d\mod.lua

This is good.

require("mod.submod") searches for

C:\a\b\c\..\mod\submod\d\mod\submod.lua

It would be much better I think if it searched:

C:\a\b\c\..\mod\d\mod\submod.lua

It can be achieved by adding a custom searcher in the package.searchers but wouldn't it support a better hierarchy of code if the existing searcher did that? 

Thanks,
Milind

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Alexey Melnichuk-2
Здравствуйте, Milind.

Вы писали 21 сентября 2015 г., 23:44:04:

> Hi,
>       Currently if package.path has a path like:

> C:\a\b\c\..\?\d\?.lua

> Then it require("mod") searches for

> C:\a\b\c\..\mod\d\mod.lua  -> C:\a\b\mod\d\mod.lua

> This is good.

> require("mod.submod") searches for

> C:\a\b\c\..\mod\submod\d\mod\submod.lua

> It would be much better I think if it searched:

> C:\a\b\c\..\mod\d\mod\submod.lua

> It can be achieved by adding a custom searcher in the
> package.searchers but wouldn't it support a better hierarchy of code
> if the existing searcher did that?

> Thanks,
> Milind

I   use   just   that.   I   use  lua_init  env  variable  to  set  my
pakage.loaders/searchrs.   I   use   strings   for   pakage.path   like
`*\share\?.lua;lua*\share\?.lua;lua-*\share\?.lua`.  Where  `*` is lib
name  which calculate as first part in require argument. This does not
works  for  some modules like `mime` from luasocket or `re` from lpeg.
for  such  modules  i  have table to associate them with library. Also
my loadrs update %PATH% env variable for current process. So I have
separate directory for library which contain `share` dir with all
modules,  `test`  dir  with  tests,  `doc`  dir with documentation and
`path`  dir with external deps (e.g. libuv.dll for lluv or zmq.dll for
lzmq). `path` adds to process env %path% variable just before loading lua
module.  So  you  can use specific version for lua module. E.g. I have
several  openssl  in  my  %PATH%  but  lua module have to use specific
version.


--
С уважением,
 Alexey                          mailto:[hidden email]


---
Это сообщение проверено на вирусы антивирусом Avast.
https://www.avast.com/antivirus


Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Sean Conner
In reply to this post by aryajur
It was thus said that the Great Milind Gupta once stated:
> Hi,
>       Currently if package.path has a path like:
>
> C:\a\b\c\..\?\d\?.lua

  Why add the parent directory?  This just translates to

        C:/a/b/?/d/?.lua [1]

> Then it require("mod") searches for
>
> C:\a\b\c\..\mod\d\mod.lua  -> C:\a\b\mod\d\mod.lua
>
> This is good.

  What is the extra directory buying you?  I fail to see the benefit of an
extra directory.

> require("mod.submod") searches for
>
> C:\a\b\c\..\mod\submod\d\mod\submod.lua
>
> It would be much better I think if it searched:
>
> C:\a\b\c\..\mod\d\mod\submod.lua

  What about a module named "org.conman.syslog"?

  -spc

[1] Contrary to what Windows people may believe, Windows can support
        paths with '/'.  No, really!  It can.  It has since MS-DOS 2.0
        (1982).  It's only the shell, COMMAND.COM that can't [2]

[2] And it can if you call a MS-DOS function to change the option
        character from '/' to '-'.  

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

aryajur



  Why add the parent directory?  This just translates to

        C:/a/b/?/d/?.lua [1]

Yes it does. The c directory comes in because in luaconf the path is defined relative to the lua executable directory for example. so C:/a/b/c is the directory path of the lua executable which gets substituted when the lua executable is started and ../?/d/?.lua is the path that was placed relative to it. So package.path shows the path like that. 

> Then it require("mod") searches for
>
> C:\a\b\c\..\mod\d\mod.lua  -> C:\a\b\mod\d\mod.lua
>
> This is good.

  What is the extra directory buying you?  I fail to see the benefit of an
extra directory.

> require("mod.submod") searches for
>
> C:\a\b\c\..\mod\submod\d\mod\submod.lua
>
> It would be much better I think if it searched:
>
> C:\a\b\c\..\mod\d\mod\submod.lua

  What about a module named "org.conman.syslog"?

The thought is that sub modules would reside inside the sub directories where the main module resides. So for org.conman.syslog, org is the main module and then syslog is the submodule so it should translate to:
C:\a\b\org\d\org\conman\syslog.lua

This allows for example just pull the whole directory structure of many github projects and have them available immediately for example the following directory structure:

project1
   - src   (project1.lua)
        - project1    (submod1.lua)
project2
   - src   (project2.lua)
project3
   - src   (project3.lua)
 lua53  (lua.exe)

if I have ../?/src/?.lua in my package.path then all these work:
require("project1")
require("project2")
require("project3")

But require("project1.submod1") does not work. 
Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

aryajur
In reply to this post by Alexey Melnichuk-2
I   use   just   that.   I   use  lua_init  env  variable  to  set  my
pakage.loaders/searchrs.   I   use   strings   for   pakage.path   like
`*\share\?.lua;lua*\share\?.lua;lua-*\share\?.lua`.  Where  `*` is lib
name  which calculate as first part in require argument. This does not
works  for  some modules like `mime` from luasocket or `re` from lpeg.
for  such  modules  i  have table to associate them with library. Also
my loadrs update %PATH% env variable for current process. So I have
separate directory for library which contain `share` dir with all
modules,  `test`  dir  with  tests,  `doc`  dir with documentation and
`path`  dir with external deps (e.g. libuv.dll for lluv or zmq.dll for
lzmq). `path` adds to process env %path% variable just before loading lua
module.  So  you  can use specific version for lua module. E.g. I have
several  openssl  in  my  %PATH%  but  lua module have to use specific
version.

Thank you! I didn't know about LUA_INIT. Thank you for sharing your methodology. I will setup my environment similarly.
Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

aryajur


On Tue, Sep 22, 2015 at 12:40 AM, Milind Gupta <[hidden email]> wrote:
I   use   just   that.   I   use  lua_init  env  variable  to  set  my
pakage.loaders/searchrs.   I   use   strings   for   pakage.path   like
`*\share\?.lua;lua*\share\?.lua;lua-*\share\?.lua`.  Where  `*` is lib
name  which calculate as first part in require argument. This does not
works  for  some modules like `mime` from luasocket or `re` from lpeg.
for  such  modules  i  have table to associate them with library. Also
my loadrs update %PATH% env variable for current process. So I have
separate directory for library which contain `share` dir with all
modules,  `test`  dir  with  tests,  `doc`  dir with documentation and
`path`  dir with external deps (e.g. libuv.dll for lluv or zmq.dll for
lzmq). `path` adds to process env %path% variable just before loading lua
module.  So  you  can use specific version for lua module. E.g. I have
several  openssl  in  my  %PATH%  but  lua module have to use specific
version.

One thing I see is that LUA_INIT is checked by the lua.exe program not by core Lua. So if you create a lua thread with llthreads or create a new lua_state which executes some lua code then that will not see the LUA_INIT updates automatically. Or am I missing something? 
Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Alexey Melnichuk-2

> One thing I see is that LUA_INIT is checked by the lua.exe program
> not by core Lua. So if you create a lua thread with llthreads or
> create a new lua_state which executes some lua code then that will
> not see the LUA_INIT updates automatically. Or am I missing something?

No.   You  are  right.  This  is  why  I  add  support  LUA_INIT to my
llthread2.ex module[1]. But in general you need do it by yourself.

[1] https://github.com/moteus/lua-llthreads2/blob/master/src/lua/llthreads2/ex.lua#L66-L91


--
С уважением,
 Alexey                          mailto:[hidden email]


---
Это сообщение проверено на вирусы антивирусом Avast.
https://www.avast.com/antivirus


Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Sean Conner
In reply to this post by aryajur
It was thus said that the Great Milind Gupta once stated:

> >
> >
> >   Why add the parent directory?  This just translates to
> >
> >         C:/a/b/?/d/?.lua [1]
> >
>
> Yes it does. The c directory comes in because in luaconf the path is
> defined relative to the lua executable directory for example. so C:/a/b/c
> is the directory path of the lua executable which gets substituted when the
> lua executable is started and ../?/d/?.lua is the path that was placed
> relative to it. So package.path shows the path like that.
>
> >
> > > Then it require("mod") searches for
> > >
> > > C:\a\b\c\..\mod\d\mod.lua  -> C:\a\b\mod\d\mod.lua
> > >
> > > This is good.
> >
> >   What is the extra directory buying you?  I fail to see the benefit of an
> > extra directory.
> >
> > > require("mod.submod") searches for
> > >
> > > C:\a\b\c\..\mod\submod\d\mod\submod.lua
> > >
> > > It would be much better I think if it searched:
> > >
> > > C:\a\b\c\..\mod\d\mod\submod.lua
> >
> >   What about a module named "org.conman.syslog"?
> >
>
> The thought is that sub modules would reside inside the sub directories
> where the main module resides. So for org.conman.syslog, org is the main
> module and then syslog is the submodule so it should translate to:
> C:\a\b\org\d\org\conman\syslog.lua

  !!!!

  Why does everybody assume "org" is the "main" module when it's not?  It's
not "org" with "conman.syslog" as a submodule, it's "org.conman.syslog".
The first two levels, "org.conman" just namespace my modules.

  Geeze, I might have to make "org.flummux.syslog" just to prove the point!

> This allows for example just pull the whole directory structure of many
> github projects and have them available immediately for example the
> following directory structure:
>
> project1
>    - src   (project1.lua)
>         - project1    (submod1.lua)
> project2
>    - src   (project2.lua)
> project3
>    - src   (project3.lua)
>  lua53  (lua.exe)
>
> if I have ../?/src/?.lua in my package.path then all these work:
> require("project1")
> require("project2")
> require("project3")
>
> But require("project1.submod1") does not work.

  That's an interesting solution, but how many github projects/Lua modules
will that work with?  

  -spc (I must be the odd-man out ... )


Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Jerome Vuarand
In reply to this post by Sean Conner
2015-09-22 8:02 GMT+01:00 Sean Conner <[hidden email]>:
> [1]     Contrary to what Windows people may believe, Windows can support
>         paths with '/'.  No, really!  It can.  It has since MS-DOS 2.0
>         (1982).  It's only the shell, COMMAND.COM that can't [2]
>
> [2]     And it can if you call a MS-DOS function to change the option
>         character from '/' to '-'.

That's not accurate. The cmd.exe shell can deal with slashes in paths
even without replacing the options prefix, for example:

cd /D C:/
cd /Users/yourusername

Where slashes break things is in a myriad of obscure failure points
scattered in the programs (MS & 3rd-party) and even some win32 APIs.
So yeah, Windows supports forward slashes just as well as Unix
supports spaces in paths: with many pitfalls.

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Coda Highland
On Tue, Sep 22, 2015 at 7:29 AM, Jerome Vuarand
<[hidden email]> wrote:

> 2015-09-22 8:02 GMT+01:00 Sean Conner <[hidden email]>:
>> [1]     Contrary to what Windows people may believe, Windows can support
>>         paths with '/'.  No, really!  It can.  It has since MS-DOS 2.0
>>         (1982).  It's only the shell, COMMAND.COM that can't [2]
>>
>> [2]     And it can if you call a MS-DOS function to change the option
>>         character from '/' to '-'.
>
> That's not accurate. The cmd.exe shell can deal with slashes in paths
> even without replacing the options prefix, for example:
>
> cd /D C:/
> cd /Users/yourusername
>
> Where slashes break things is in a myriad of obscure failure points
> scattered in the programs (MS & 3rd-party) and even some win32 APIs.
> So yeah, Windows supports forward slashes just as well as Unix
> supports spaces in paths: with many pitfalls.
>

cmd.exe is not COMMAND.COM.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Hisham
In reply to this post by Sean Conner
On 22 September 2015 at 05:00, Sean Conner <[hidden email]> wrote:
>   Why does everybody assume "org" is the "main" module when it's not?  It's
> not "org" with "conman.syslog" as a submodule, it's "org.conman.syslog".
> The first two levels, "org.conman" just namespace my modules.
>
>   Geeze, I might have to make "org.flummux.syslog" just to prove the point!

>   -spc (I must be the odd-man out ... )

Well, I am with you: note that a certain Lua project has a bunch of
modules with names such as "luarocks.fetch", "luarocks.fs.unix.tools",
and so on, and there is no main module called "luarocks".

-- Hisham (let's make some T-shirts "namespacing is not a crime")
http://hisham.hm/

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

aryajur


>   Why does everybody assume "org" is the "main" module when it's not?  It's
> not "org" with "conman.syslog" as a submodule, it's "org.conman.syslog".
> The first two levels, "org.conman" just namespace my modules.
>
>   Geeze, I might have to make "org.flummux.syslog" just to prove the point!

>   -spc (I must be the odd-man out ... )

Well, I am with you: note that a certain Lua project has a bunch of
modules with names such as "luarocks.fetch", "luarocks.fs.unix.tools",
and so on, and there is no main module called "luarocks".

-- Hisham (let's make some T-shirts "namespacing is not a crime")
http://hisham.hm/

I don't think namespacing is a crime :) . But even if it is a namespace I would have thought that I would want to place things with 1 namespace in a particular location. Currently with org.conman.syslog and say org.conman.xx and with a lua path C:\a\b\c\..\?\d\?.lua It would try to find xx and syslog in 2 different directories. I would think it doesn't matter if you have it as a namespace or a sub module the intention was that things inside a particular redirection reside location. Maybe I am missing some other cases that would fail.

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

aryajur
In reply to this post by Sean Conner


  That's an interesting solution, but how many github projects/Lua modules
will that work with?

  -spc (I must be the odd-man out ... )



It works with quite a few. Usually people follow some sort of standard in all their own projects at least so whatever sub directories they choose you could have it in your package.path definition and it would work with a bunch of projects.

But I think this is just how I applied it and it seemed a '?' in between should not replicate the whole namespace or submodule hierarchy since it leads to entirely different directories for the final file which is a sub module for the same main module or probably a module for the same namespace.

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Rena
In reply to this post by aryajur

On Sep 22, 2015 11:43 AM, "Milind Gupta" <[hidden email]> wrote:
>
>
>>
>> >   Why does everybody assume "org" is the "main" module when it's not?  It's
>> > not "org" with "conman.syslog" as a submodule, it's "org.conman.syslog".
>> > The first two levels, "org.conman" just namespace my modules.
>> >
>> >   Geeze, I might have to make "org.flummux.syslog" just to prove the point!
>>
>> >   -spc (I must be the odd-man out ... )
>>
>> Well, I am with you: note that a certain Lua project has a bunch of
>> modules with names such as "luarocks.fetch", "luarocks.fs.unix.tools",
>> and so on, and there is no main module called "luarocks".
>>
>> -- Hisham (let's make some T-shirts "namespacing is not a crime")
>> http://hisham.hm/
>
>
> I don't think namespacing is a crime :) . But even if it is a namespace I would have thought that I would want to place things with 1 namespace in a particular location. Currently with org.conman.syslog and say org.conman.xx and with a lua path C:\a\b\c\..\?\d\?.lua It would try to find xx and syslog in 2 different directories. I would think it doesn't matter if you have it as a namespace or a sub module the intention was that things inside a particular redirection reside location. Maybe I am missing some other cases that would fail.
>
It sounds like what you're saying is require("syslog") should succeed whether you have org.conman.syslog or foo.bar.syslog or even just syslog?

Reply | Threaded
Open this post in threaded view
|

Re: package.searcher modification for lua modules

Sean Conner
It was thus said that the Great Rena once stated:

> On Sep 22, 2015 11:43 AM, "Milind Gupta" <[hidden email]> wrote:
> >
> >
> >>
> >> >   Why does everybody assume "org" is the "main" module when it's not?
> It's
> >> > not "org" with "conman.syslog" as a submodule, it's
> "org.conman.syslog".
> >> > The first two levels, "org.conman" just namespace my modules.
> >> >
> >> >   Geeze, I might have to make "org.flummux.syslog" just to prove the
> point!
> >>
> >> >   -spc (I must be the odd-man out ... )
> >>
> >> Well, I am with you: note that a certain Lua project has a bunch of
> >> modules with names such as "luarocks.fetch", "luarocks.fs.unix.tools",
> >> and so on, and there is no main module called "luarocks".
> >>
> >> -- Hisham (let's make some T-shirts "namespacing is not a crime")
> >> http://hisham.hm/
> >
> >
> > I don't think namespacing is a crime :) . But even if it is a namespace I
> > would have thought that I would want to place things with 1 namespace in a
> > particular location. Currently with org.conman.syslog and say org.conman.xx
> > and with a lua path C:\a\b\c\..\?\d\?.lua It would try to find xx and
> > syslog in 2 different directories. I would think it doesn't matter if you
> > have it as a namespace or a sub module the intention was that things inside
> > a particular redirection reside location. Maybe I am missing some other
> > cases that would fail.
> >
> It sounds like what you're saying is require("syslog") should succeed
> whether you have org.conman.syslog or foo.bar.syslog or even just syslog?

 What he's saying is that given the following lua path:

        C:/a/b/c/../?/d/?.lua

  that

        require "org.conman.syslog"

would look for

        C:/a/b/c/../org/conman/syslog/d/org/conman/syslog.lua [1]

because most modules are just <name>, not <name>.<subname>.<subsubname>.

  It also appears that he has several modules, foo, bar, baz, that all have
a similar directory structure:

        foo/
        foo/src
        foo/src/foo.lua

        bar/
        bar/src
        bar/src/bar.lua

        baz/
        baz/src
        baz/src/baz.lua

And so an easy eay to reference each module is to change the lua path:

        ?/src/?.oua

instead of installing the modules into /usr/local/share/lua/5.x/ (or
whereever).  My modules, org.conman.*, and Hisham's modules, luarocks.*,
break under that scheme.  But that he (Milind) would like a searcher that
expands the paths like:

        c:/a/b/c/../org/d/org/conman/syslog.lua

which *could* work for Hisham's modules, but definitely not for mine [2].

  -spc (and just to mess with people, I wouldn't mind seeing Luarocks adapt
        the org.luarocks namespace 8-)

[1] Even though org.conman.syslog is a C based module, so it would have
        a .so extension.

[2] To test my modules, I have to install them first, as the layout I
        have is:

                orgconman/lua/table.lua -- this is org.conman.table
                orgconman/lua/cc.lua -- this is org.coman.cc

                orgconman/src/syslog.c -- this is org.conman.syslog
                orgconman/src/signal.c -- this is org.conman.signal