Relative Requires in Lua

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Relative Requires in Lua

Soni "They/Them" L.
Disclaimer: If this sounds like a blog post, that's because this is a
blog post. (Should blog posts be tagged somehow? Say, [BP] or something?)

Lua's most overlooked feature is the require function. People assume it
just loads a file and returns whatever it loaded. But it does a lot more
than that.

For one, it implements a cache. You can require the same module over and
over and it'll always return the same value. While this doesn't get
overlooked often, I've seen people implement a cache on top of the
standard require()... or attempt to reload a module with the standard
require() .-.
However, you can override this by fiddling with the contents of
package.loaded.

For two, Lua's module system fully supports relative requires. This is
one of the most overlooked features of Lua! I see people doing `local
submod = require "thismod.submod"` all the time. That's not how you do it!

The proper way to do relative requires is:

thismod/init.lua
-----
local modname = ...
local submod = require(modname .. ".submod")
return {submod=submod}
-----

thismod/submod.lua
-----
return {
   helloworld = function() print("Hello World!") end
}
-----

Then, if you require "thismod", you can call
thismod.submod.helloworld(). However, if you rename the thismod
directory, and require the new directory name, it'll still work! Unlike
when using hardcoded module names.

This "relative require" pattern is never mentioned in Programming in
Lua, 3rd edition, which's a bit of a shame. It seems most Lua users
don't know about this - even those who have worked with Lua extensively.

Don't make a searcher that supports `require"./name"`, as that'll just
get cached as "./name". Make a searcher that supports
`require(modname..".name")` instead, where `modname` comes from `...`.

This might be a new pattern to you, but it's a very useful pattern. I
use it all the time with my oversized modules, precisely because it
supports renaming.

There are a few other things require does that I'm not gonna go into.
This isn't supposed to be a complete tutorial on modules - unlike the
modules section of PiL (which I think should include this, however I
don't have PiL 4 to check). But this is the basic idea of relative require.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Relative Requires in Lua

Egor Skriptunoff-2
On Thu, Oct 13, 2016 at 10:28 PM, Soni L. <[hidden email]> wrote:

The proper way to do relative requires is:

thismod/init.lua
-----
local modname = ...
local submod = require(modname .. ".submod")
return {submod=submod}
-----

thismod/submod.lua
-----
return {
  helloworld = function() print("Hello World!") end
}
-----

Thanks, it is quite useful.

BTW, there is an idea how to improve require-related logic
require() should be rewritten so that each call to require() would add current module's directory to the beginning of package.path.
This "extended" package.path will affect only nested invocations of require().
Actually, it would be a stack of directories (synchronized with stack of "require" invocations).

This way you can simply omit concatenation of module names:

thismod/init.lua
-----
local submod = require("submod")
return {submod=submod}
-----

The module "submod" should be searched in the following places in the following order:
1) thismod/submod.lua
2) thismod/submod/init.lua
3) all standard paths (replacing "?" with "submod" in original package.path)
Reply | Threaded
Open this post in threaded view
|

Re: Relative Requires in Lua

Sean Conner
It was thus said that the Great Egor Skriptunoff once stated:

> On Thu, Oct 13, 2016 at 10:28 PM, Soni L. <[hidden email]> wrote:
>
> > The proper way to do relative requires is:
> >
> > thismod/init.lua
> > -----
> > local modname = ...
> > local submod = require(modname .. ".submod")
> > return {submod=submod}
> > -----
> >
> > thismod/submod.lua
> > -----
> > return {
> >   helloworld = function() print("Hello World!") end
> > }
> > -----
> >
> > Thanks, it is quite useful.

  This won't work with Lua modules written in C.  There was a thread about
this three years ago:

        http://lua-users.org/lists/lua-l/2013-06/threads.html#00464

> BTW, there is an idea how to improve require-related logic
> require() should be rewritten so that each call to require() would add
> current module's directory to the beginning of package.path.
> This "extended" package.path will affect only nested invocations of
> require().
> Actually, it would be a stack of directories (synchronized with stack of
> "require" invocations).

  First of all, all this will do is end up lengthening package.path with
repetitious paths already in package.path unless you take pains to write
code to check if the path already exists in package.path.  

  Second, Lua has no concept of "directories" (or "folders" for that
matter).  All Lua does (using the second, third and fourth default
searchers; loaders in Lua 5.1) is replace the module name in a pattern and
attempt to open a file with the resulting name.

> This way you can simply omit concatenation of module names:
>
> thismod/init.lua
> -----
> local submod = require("submod")
> return {submod=submod}
> -----
>
> The module "submod" should be searched in the following places in the
> following order:
> 1) thismod/submod.lua
> 2) thismod/submod/init.lua
> 3) all standard paths (replacing "?" with "submod" in original package.path)

  Third, this still leaves the first searcher, which simply scans
package.preload[] for the given module name, so if you are trying to load
"custom.sub1.foo", then that will attemp to look for

        package.preload['custom.sub1.foo']

  Then there are the custom loaders supplied by the user.  I have one at
work that will look for a module (written in Lua) internally (it's
compressed and compiled into the executable [1])---no concept of a
"directory" or "folder" there at all.

  -spc (So how is this supposed to work again?)

[1] I go over a bit of the details here:
        http://boston.conman.org/2013/03/23.1

Reply | Threaded
Open this post in threaded view
|

Re: Relative Requires in Lua

Soni "They/Them" L.


On 15/10/16 03:35 AM, Sean Conner wrote:

> It was thus said that the Great Egor Skriptunoff once stated:
>> On Thu, Oct 13, 2016 at 10:28 PM, Soni L. <[hidden email]> wrote:
>>
>>> The proper way to do relative requires is:
>>>
>>> thismod/init.lua
>>> -----
>>> local modname = ...
>>> local submod = require(modname .. ".submod")
>>> return {submod=submod}
>>> -----
>>>
>>> thismod/submod.lua
>>> -----
>>> return {
>>>    helloworld = function() print("Hello World!") end
>>> }
>>> -----
>>>
>>> Thanks, it is quite useful.
>    This won't work with Lua modules written in C.  There was a thread about
> this three years ago:
>
> http://lua-users.org/lists/lua-l/2013-06/threads.html#00464

But does C count as Lua?

>> BTW, there is an idea how to improve require-related logic
>> require() should be rewritten so that each call to require() would add
>> current module's directory to the beginning of package.path.
>> This "extended" package.path will affect only nested invocations of
>> require().
>> Actually, it would be a stack of directories (synchronized with stack of
>> "require" invocations).
>    First of all, all this will do is end up lengthening package.path with
> repetitious paths already in package.path unless you take pains to write
> code to check if the path already exists in package.path.
>
>    Second, Lua has no concept of "directories" (or "folders" for that
> matter).  All Lua does (using the second, third and fourth default
> searchers; loaders in Lua 5.1) is replace the module name in a pattern and
> attempt to open a file with the resulting name.
>
>> This way you can simply omit concatenation of module names:
>>
>> thismod/init.lua
>> -----
>> local submod = require("submod")
>> return {submod=submod}
>> -----
>>
>> The module "submod" should be searched in the following places in the
>> following order:
>> 1) thismod/submod.lua
>> 2) thismod/submod/init.lua
>> 3) all standard paths (replacing "?" with "submod" in original package.path)
>    Third, this still leaves the first searcher, which simply scans
> package.preload[] for the given module name, so if you are trying to load
> "custom.sub1.foo", then that will attemp to look for
>
> package.preload['custom.sub1.foo']
>
>    Then there are the custom loaders supplied by the user.  I have one at
> work that will look for a module (written in Lua) internally (it's
> compressed and compiled into the executable [1])---no concept of a
> "directory" or "folder" there at all.
>
>    -spc (So how is this supposed to work again?)
>
> [1] I go over a bit of the details here:
> http://boston.conman.org/2013/03/23.1
>

I made the OP because of Luvit ( https://luvit.io/ ). The people trying
to overcomplicate 'require' don't understand what's going on with Luvit
right now.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Relative Requires in Lua

Philipp Janda
In reply to this post by Soni "They/Them" L.
Am 13.10.2016 um 21:28 schröbte Soni L.:

>
> For two, Lua's module system fully supports relative requires. This is
> one of the most overlooked features of Lua! I see people doing `local
> submod = require "thismod.submod"` all the time. That's not how you do it!
>
> The proper way to do relative requires is:
>
> thismod/init.lua
> -----
> local modname = ...
> local submod = require(modname .. ".submod")
> return {submod=submod}
> -----
>
> thismod/submod.lua
> -----
> return {
>   helloworld = function() print("Hello World!") end
> }
> -----
>
> Then, if you require "thismod", you can call
> thismod.submod.helloworld(). However, if you rename the thismod
> directory, and require the new directory name, it'll still work! Unlike
> when using hardcoded module names.

I think relative `require` is an anti-pattern for the following reasons:

1.  The module name is part of the public interface, so you shouldn't
rename it at all (or at least bump the major version number afterwards).

2.  If you move a module `a.x` to `a.s.x`, all relative `require`s in
this module *and of* this module break. With absolute `require` only the
`require`s *of* this module need to be fixed.

3.  It only works for internal modules that are never `require`d from
the outside or else the renaming feature stops working anyway.

4.  It leads to bloat. For modules like

         return { -- file a.lua
           b = require( ... .. ".b" ),
           c = require( ... .. ".c" ),
           -- ...
         }

     `local a = require("a")` will load all submodules even if only a
few of those modules are actually needed.

5.  The interface gets uglier as well:

         local a = require( "a" )
         a.b.u.x.func()

     compared to

         local x = require( "a.b.u.x" )
         x.func()

6.   You can run into `require` cycles if module `a.b.u.x` needs
functions from module `a.b` but should also be part of it.

Philipp




Reply | Threaded
Open this post in threaded view
|

Re: Relative Requires in Lua

Soni "They/Them" L.


On 15/10/16 08:00 AM, Philipp Janda wrote:

> Am 13.10.2016 um 21:28 schröbte Soni L.:
>>
>> For two, Lua's module system fully supports relative requires. This is
>> one of the most overlooked features of Lua! I see people doing `local
>> submod = require "thismod.submod"` all the time. That's not how you
>> do it!
>>
>> The proper way to do relative requires is:
>>
>> thismod/init.lua
>> -----
>> local modname = ...
>> local submod = require(modname .. ".submod")
>> return {submod=submod}
>> -----
>>
>> thismod/submod.lua
>> -----
>> return {
>>   helloworld = function() print("Hello World!") end
>> }
>> -----
>>
>> Then, if you require "thismod", you can call
>> thismod.submod.helloworld(). However, if you rename the thismod
>> directory, and require the new directory name, it'll still work! Unlike
>> when using hardcoded module names.
>
> I think relative `require` is an anti-pattern for the following reasons:
>
> 1.  The module name is part of the public interface, so you shouldn't
> rename it at all (or at least bump the major version number afterwards).
>
> 2.  If you move a module `a.x` to `a.s.x`, all relative `require`s in
> this module *and of* this module break. With absolute `require` only
> the `require`s *of* this module need to be fixed.
>
> 3.  It only works for internal modules that are never `require`d from
> the outside or else the renaming feature stops working anyway.
>
> 4.  It leads to bloat. For modules like
>
>         return { -- file a.lua
>           b = require( ... .. ".b" ),
>           c = require( ... .. ".c" ),
>           -- ...
>         }
>
>     `local a = require("a")` will load all submodules even if only a
> few of those modules are actually needed.
>
> 5.  The interface gets uglier as well:
>
>         local a = require( "a" )
>         a.b.u.x.func()
>
>     compared to
>
>         local x = require( "a.b.u.x" )
>         x.func()
>
> 6.   You can run into `require` cycles if module `a.b.u.x` needs
> functions from module `a.b` but should also be part of it.
>
> Philipp
>
>
>
>

This is not an antipattern. You're just using it wrong.

See also: https://github.com/luvit/luvit/pull/932

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Relative Requires in Lua

Martin
In reply to this post by Soni "They/Them" L.
Some time ago I've implemented sort of relative require() for my
own code, named request(). It operates on relative paths, converts
them to absolute paths for require() and calls it.

My need was to load code from sibling directory, for example
in deep_path/a/code.lua I wish to load code from deep_path/b/code.lua.
It may be done as request('^.b.code'). Limitation is that this call
only works when we have somewhere current path prefix, which is true
at first loading module.

So in main code you type request('deep_path.a.code').

This stores prefix 'deep_path.a' and calls require('deep_path.a.code').

request() in a/code.lua then knows current prefix and unites it
with '^.b', so current prefix becomes 'deep_path.b'.
And then calls require('deep_path.b.code').

You may see implementation in
https://github.com/martin-eden/workshop/blob/master/base.lua