modules, require, magic

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
160 messages Options
12345678
Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Petite Abeille

On Oct 19, 2011, at 9:08 PM, Philipp Janda wrote:

> What happens if the name passed to require and the name passed to module differ?

Then hopefully you know what you are doing... or you have a typo...

> Wouldn't require just return true

In case of name mismatch, require raises an exception...

E.g.:

local foo = require( 'baz' )

 module 'baz' not found

> and the actual module ends up somewhere hidden in package.loaded?

Modules are named. They will end up under their assigned name.

> It seems to me that the name parameter to module might be problematic if you don't set globals.

How so? This is not meant to prevent people from doing silly mistakes. Just to name their modules.

>
> Maybe you shouldn't unconditionally replace all non-table values from package.loaded in case some other module placed a function or a string there (which could happen if "require-name" and "module-name" differed).

Ah, yes... there is a sort of marker put in package.loaded by require to sort out circular dependencies during loading... I don't recall the details, but module circa 5.1 has all this sorted out... e.g.:

http://www.lua.org/source/5.1/loadlib.c.html#ll_module
http://www.lua.org/source/5.1/loadlib.c.html#ll_require

> The proposed module-function doesn't mess with globals anymore, but it does mess with package.loaded

Well... "mess" might not be the right term... it simply sets it... just as module 5.1 does... module & require work both in terms of package.loaded in tandem...

> and therefore should make sure that it does not disrupt other legitimate uses of package.loaded.

Sure.





Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Patrick Donnelly
In reply to this post by Roberto Ierusalimschy
On Wed, Oct 19, 2011 at 1:26 PM, Roberto Ierusalimschy
<[hidden email]> wrote:

>>
>> On Oct 19, 2011, at 5:32 PM, Roberto Ierusalimschy wrote:
>>
>> > If you really want 'module', wouldn't be simpler to use
>> >
>> > _ENV = module(...)
>> >
>> > with a simplified version of 'module'?
>>
>> By simplified, do you mean a 5.2 version not doing any of the setfenv() mambo jambo of the 5.1 implementation, but otherwise equivalent in functionality?
>>
>> With the environment switching as an explicit _ENV assignment?
>
> For instance (also without the global setting). Something like this:
>
> function module (name, ...)
>  local env = package.loaded[name]
>  if env == nil then
>    env = {}
>    package.loaded[name] = env
>  end
>  env.NAME = name
>  env.M = env
>  for _, f in ipairs{...} do
>    f(env)
>  end
>  return env
> end

[Assuming the above function were put in Lua 5.2,] package.seeall has
questionable usefulness even-more-so now that modules are not put in
the global environment. I would also second the syntactic sugar
addition of something like:

local abs, floor in require "math"

or similar. That way we can nicely document dependencies without
taking dozens of lines of code to do it.

--
- Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Roberto Ierusalimschy
In reply to this post by Hisham Muhammad
> Certainly an improvement from handling the module table explicitly
> (and no segfaults this time). But having basically the same function
> being provided by the loader[1] would obviate the need for explicitly
> mentioning _ENV.

I think explicitly mentioning _ENV is good. If you are working with
a different global environment, why hide that information? Moreover,
a modified 'load' function forces everyone to live with its magic.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Matthew Wild
On 19 October 2011 21:24, Roberto Ierusalimschy <[hidden email]> wrote:
>> Certainly an improvement from handling the module table explicitly
>> (and no segfaults this time). But having basically the same function
>> being provided by the loader[1] would obviate the need for explicitly
>> mentioning _ENV.
>
> I think explicitly mentioning _ENV is good. If you are working with
> a different global environment, why hide that information? Moreover,
> a modified 'load' function forces everyone to live with its magic.
>

Agreed. Also if someone doesn't want to change the environment, but
just use the other features of module() they can simply

   local mymodule = module("mymodule")

   function mymodule.myfunction()
   end

You can even name this table 'exports', for those missing Node.js :)

Regards,
Matthew

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Javier Guerra Giraldez
On Wed, Oct 19, 2011 at 3:37 PM, Matthew Wild <[hidden email]> wrote:
> Agreed. Also if someone doesn't want to change the environment, but
> just use the other features of module() they can simply
>
>   local mymodule = module("mymodule")
>
>   function mymodule.myfunction()
>   end

that's exactly how i'd use that new module()

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Roberto Ierusalimschy
In reply to this post by Hisham Muhammad
> To my surprise, this causes a segmentation fault in Lua 5.2 beta
> (confirmed by Fabio). Commenting the "local mod" in simple.lua:1 makes
> the segfault go away but it still doesn't work as I expected (I
> probably don't understand _ENV completely).

The bug has nothing to do with _ENV. The following trivial program
segfaults:

-----------------------------
local a
function t()
  local mod = 1
  a, mod = 1
end

t()
-----------------------------

The parser generates a wrong opcode (refering to 'a' as upvalue 1 instead
of 0), but I did not have time yet to find why.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Hisham Muhammad
In reply to this post by Roberto Ierusalimschy
On Wed, Oct 19, 2011 at 6:24 PM, Roberto Ierusalimschy
<[hidden email]> wrote:
>> Certainly an improvement from handling the module table explicitly
>> (and no segfaults this time). But having basically the same function
>> being provided by the loader[1] would obviate the need for explicitly
>> mentioning _ENV.
>
> I think explicitly mentioning _ENV is good. If you are working with
> a different global environment, why hide that information?

Fair. I could argue otherwise, but that's indeed more a matter of
taste, which I respect.

One remaining issue lies in the visibility of globals:

David Manura wrote:

>> Hisham wrote:
>> The module() function gave us an easy answer for the question "how do
>> I write a module" that mirror other languages that offer constructs for modularity.
>> [...] If module() is broken beyond repair and there is no clean way to add a
>> library function to do that, I'd dare to say that this is something
>> important enough that a language construct would be justifiable
>
> OTOH, there are multiple ways to use this function: The somewhat
> canonical "easy way" involves package.seeall, and maybe we should all
> just accept that, in the way in another life I just accept Perl 5.
> Josh's principle I think is important though: "[it's about] making it
> easier to do the right thing. With module it's easy to get things
> wrong".

Many of the complaints against module() are actually against
package.seeall. The issues of exposing unrelated globals through a
module and inheriting dependencies are caused by it. The module()
implementation you suggested leaves this question open, so my
suggestion in that front would be something along the lines of:

---------- variation on Roberto's module
function module (name, ...)
 local env = package.loaded[name]
 if env == nil then
   env = {}
   package.loaded[name] = env
 end
 env._NAME = name
 env._M = env
 for _, f in ipairs{...} do
   env = f(env) -- <=== allow f to redefine env
 end
 return env
end

local lua_libs = {}
for k,v in pairs(package.loaded._G) do
 lua_libs[k] = v
end

function package.seebase(mod)
   local env = {}
   setmetatable(env, {
      __index = function(t,k)
         return rawget(mod, k) or rawget(lua_libs, k)
      end,
      __newindex = mod
   })
   return env
end
----------

This proposed package.seebase exposes Lua standard libraries to the
module without exporting them. (I only had to make one slight change
to the module() function you posted; if that is doable without this
change I'd be happy to be corrected.) I think package.seebase brings
the benefits of package.seeall without its two big drawbacks mentioned
above.

I'd even argue that this should be the default behavior for module()
when called with a single argument, in line with David Manura's
concerns that the canonical "easy way" should help the user get things
right.

> Moreover,
> a modified 'load' function forces everyone to live with its magic.

Not necessarily; Fabio's original loader did this but loader5
restricted the magic to the users of module().

-- Hisham
http://hisham.hm/ - http://luarocks.org/

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Javier Guerra Giraldez
On Wed, Oct 19, 2011 at 4:15 PM, Hisham <[hidden email]> wrote:
>   env = f(env) -- <=== allow f to redefine env

A: a small bug, this returned `env` isn't put back into packages.loaded

B: your 'f' can add a metatable to env and it will hold, no need to
return it back.

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Philipp Janda
In reply to this post by Petite Abeille
On 19.10.2011 21:36, Petite Abeille wrote:

>
> On Oct 19, 2011, at 9:08 PM, Philipp Janda wrote:
>
>> What happens if the name passed to require and the name passed to module differ?
>
> Then hopefully you know what you are doing... or you have a typo...
>
>> Wouldn't require just return true
>
> In case of name mismatch, require raises an exception...
>
> E.g.:
>
> local foo = require( 'baz' )
>
>   module 'baz' not found

require raises an exception if "require-name" and "file-name" don't
match, but it cannot handle a mismatch between "require-name" and
"module-name", because it doesn't know "module-name".

E.g.:
$ mkdir extern
$ cat > extern/mod.lua
local print = print
_ENV = module( "mod" )

print( (...), _NAME )

function echo( ... )
   print( ... )
end
^D

$ cat > main.lua
function module( aName, ... )
   local aModule = package.loaded[ aName ]
   if type( aModule ) ~= 'table' then
     aModule = {}
     aModule._M = aModule
     aModule._NAME = aName
     aModule._PACKAGE = aName:sub( 1, aName:len() - (
aName:reverse():find( '.', 1, true ) or 0 ) )
     package.loaded[ aName ] = aModule
     for anIndex = 1, select( '#', ... ) do
       select( anIndex, ... )( aModule )
     end
   end
   return aModule
end

local mod = require( "extern.mod" )
mod.echo( "hello world" )
^D

$ lua5.2 main.lua
extern.mod mod
lua5.2: main.lua:17: attempt to index local 'mod' (a boolean value)
stack traceback:
        main.lua:17: in main chunk
        [C]: in ?


>
>> and the actual module ends up somewhere hidden in package.loaded?
>
> Modules are named. They will end up under their assigned name.

Yes, they are named by require and put in package.loaded under
"require-name" and they are named again by module and if "module-name"
is different put in package.loaded again under "module-name".


>
>> It seems to me that the name parameter to module might be problematic if you don't set globals.
>
> How so? This is not meant to prevent people from doing silly mistakes. Just to name their modules.

No, it enables people to make silly mistakes (s.a). It's a bad way to
just name a module, because it forces users to edit the name if they
want to move your module to a subdirectory -- why not use a comment instead.

>
>>
>> Maybe you shouldn't unconditionally replace all non-table values from package.loaded in case some other module placed a function or a string there (which could happen if "require-name" and "module-name" differed).
>
> Ah, yes... there is a sort of marker put in package.loaded by require to sort out circular dependencies during loading... I don't recall the details, but module circa 5.1 has all this sorted out... e.g.:
>
> http://www.lua.org/source/5.1/loadlib.c.html#ll_module
> http://www.lua.org/source/5.1/loadlib.c.html#ll_require

Not what I meant, but good point as well. What I meant is that *I* put
functions in package.loaded, and I would hate it if some module
overwrites my modules by doing module( "samename" ).

>
>> The proposed module-function doesn't mess with globals anymore, but it does mess with package.loaded
>
> Well... "mess" might not be the right term... it simply sets it... just as module 5.1 does... module&  require work both in terms of package.loaded in tandem...

You are right, module 5.1 would also overwrite non-table modules. I
didn't know that. What it should do is: create new table if
package.loaded[ x ] is nil, reuse the table if package.loaded[ x ] is a
table and raise an error otherwise.

>
>> and therefore should make sure that it does not disrupt other legitimate uses of package.loaded.
>
> Sure.
>

Good :-)


Philipp


Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Hisham Muhammad
In reply to this post by Javier Guerra Giraldez
2011/10/19 Javier Guerra Giraldez <[hidden email]>:
> On Wed, Oct 19, 2011 at 4:15 PM, Hisham <[hidden email]> wrote:
>>   env = f(env) -- <=== allow f to redefine env
>
> A: a small bug, this returned `env` isn't put back into packages.loaded

That was intentional: the returned environment sees Lua globals, but
the module in packages.loaded contains only the functions exported by
the module. This is done to avoid the problem with package.seeall
where you end up accessing every global through the module.

> B: your 'f' can add a metatable to env and it will hold, no need to
> return it back.

Yes, the point was to produce a different table.

-- Hisham
http://hisham.hm/ - http://luarocks.org/

Reply | Threaded
Open this post in threaded view
|

syntactic sugar for imports and exports

Tony Finch
In reply to this post by Tony Finch
Further thoughts on the use+mention problem, where we have to repeat a
name to use it as a variable and to mention it as a table index or
function argument. The main examples are:

  local name = name            -- keep name in scope after altering _ENV
  local name = module.name     -- import unqualified name from module
  local name = require "name"  -- module linkage
  { name = name }              -- export variable into table

The usual proposal to improve the first two is as follows, but I'm trying
to find a more general solution.

  local name in _ENV
  local name in module

I call the first three "projection" from a table or function into
variables and the last one "injection" from variables into a table.

It seems to be helpful to think of both projection and injection as
special kinds of assignment. This emphasizes their duality, and it
suggests a better syntax for injection.

So the syntax for projection might be like

  foo, bar, zig <- tbl

where "foo, bar, zig" is a list of variable names and "tbl" is an
expresion. The "<-" is pronounced "from" as in "assigned from" or
"imported from". This is equivalent to

  foo, bar, zig = tbl.foo, tbl.bar, tbl.zig

except that tbl is evaluated once. A projection statement starting with
"local" has the obvious translation:

  local foo, bar, zig <- tbl
  local foo, bar, zig = tbl.foo, tbl.bar, tbl.zig

Injection might be like

  tbl -< foo, bar, zig

which is equivalent to

  tbl.foo, tbl.bar, tbl.zig = foo, bar, zig

with the same conditions as before. Injection needs a different operator
so you can tell the difference between "x <- y" and "y -< x". The "-<"
symbol is the syringe operator.

I think on balance it doesn't make sense to support projecting from
functions directly. This avoids muddling the concepts of indexing and
calling, but you can still use the __index metamethod if you want. For
example,

  -- syntactic sugar for require

  library = setmetatable(package.loaded, {
              __index = function (_, name)
                          return require(name)
                        end
            })

  -- get math and io modules from the library

  local math, io <- library

Tony.
--
f.anthony.n.finch  <[hidden email]>  http://dotat.at/
Dover, Wight, Portland, Plymouth: Northwest 4 or 5, increasing 6 at times,
backing west 3 later. Slight or moderate, occasionally rough in Plymouth at
first. Showers. Good.

Reply | Threaded
Open this post in threaded view
|

Re: syntactic sugar for imports and exports

Daurnimator
I agree about your 'projection' bit; even if I do hate the operator
The injection not so much;
I usually just return a table at the end of my modules:


-- My module

local function foo() end

local function bar() end

return {
    init_foo = foo ;
    bar = bar
}

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Mark Hamburg
In reply to this post by Hisham Muhammad
Going down this path, what might actually be nice is to have the modifiers (e.g.,package.seeall) take and return both an environment and a module. They start out equal but they can move apart. So, package.seeall builds an environment table that or's together the module and the environment in place before the call to module and sets it to write into the module table. package.writeprotect could replace the module with a proxy table referencing the module table rewiring _M appropriately. But note that those functions need to be specified in the executed order, so maybe this should all just be standard behavior.

        function makemodule( name, oldenv, writeprotect )

                -- yada, yada look for an existing module, etc

                local M = { }
                local ENV = M
                local MM = M

                if writeprotect then
                        MM = setmetatable( { }, {
                                __index = M,
                                __newindex = function( _, k, v )
                                        error( string.format( "Attempt to write field %s of module %s", tostring( k ), tostring( name ) ) -- level #?
                                end
                        } )
                end

                M._M = MM
                -- set M._NAME and M._PACKAGE as appropriate

                if oldenv then
                        ENV = setmetatable( { }, {
                                __index = function( _, k )
                                        local v = M[ k ]
                                        if v == nil then -- Don't ignore v == false
                                                v = oldenv[ k ]
                                        end
                                        return v
                                end,
                                __newindex = M -- Does this need to take extra care about what happens if we set the field to nil?
                        } )
                end

                package.loaded[ name ] = MM

                return ENV, M

        end

5.2 could use this as:

        _ENV = makemodule( name, _ENV, writeprotect )

5.1 would presumably want to wrap it with code that fetched the environment. 5.2 could do this too though actually 5.2 makes it easier to work with the environment.

One distinct downside from a performance standpoint, however, is that global lookups get expensive because they need to fail in the proxy and then invoke the metamethod code. So, I still don't know that I see the value in fiddling with the environment.

Mark


Reply | Threaded
Open this post in threaded view
|

Re: syntactic sugar for imports and exports

Petite Abeille
In reply to this post by Tony Finch

On Oct 20, 2011, at 12:40 AM, Tony Finch wrote:

> The usual proposal to improve the first two is as follows, but I'm trying
> to find a more general solution.

What about something along these lines:

local function import( aName, ... )
  local aModule = require( aName )
  local aFunction = debug.getinfo( 2, 'f' ).func
  local _, anEnvironement = debug.getupvalue( aFunction, 1 )

  for anIndex = 1, select( '#', ... ) do
    local aName = select( anIndex, ... )
    anEnvironement[ aName ] = aModule[ aName ]
  end
end

import( 'math', 'min', 'max' )

print( min )
print( max )

> function: 0x10001af60
> function: 0x10001aee0


Reply | Threaded
Open this post in threaded view
|

Re: syntactic sugar for imports and exports

Mark Hamburg
On Oct 19, 2011, at 4:30 PM, Petite Abeille wrote:

>
> On Oct 20, 2011, at 12:40 AM, Tony Finch wrote:
>
>> The usual proposal to improve the first two is as follows, but I'm trying
>> to find a more general solution.
>
> What about something along these lines:
>
> local function import( aName, ... )
>  local aModule = require( aName )
>  local aFunction = debug.getinfo( 2, 'f' ).func
>  local _, anEnvironement = debug.getupvalue( aFunction, 1 )
>
>  for anIndex = 1, select( '#', ... ) do
>    local aName = select( anIndex, ... )
>    anEnvironement[ aName ] = aModule[ aName ]
>  end
> end
>
> import( 'math', 'min', 'max' )
>
> print( min )
> print( max )
>
>> function: 0x10001af60
>> function: 0x10001aee0

That populates the environment but what one probably really wants are locals.

Mark



Reply | Threaded
Open this post in threaded view
|

Re: syntactic sugar for imports and exports

Daurnimator
In reply to this post by Petite Abeille
On 20 October 2011 10:30, Petite Abeille <[hidden email]> wrote:

>
> What about something along these lines:
>
> local function import( aName, ... )
>  local aModule = require( aName )
>  local aFunction = debug.getinfo( 2, 'f' ).func
>  local _, anEnvironement = debug.getupvalue( aFunction, 1 )
>
>  for anIndex = 1, select( '#', ... ) do
>    local aName = select( anIndex, ... )
>    anEnvironement[ aName ] = aModule[ aName ]
>  end
> end
>
> import( 'math', 'min', 'max' )
>
> print( min )
> print( max )
>
>> function: 0x10001af60
>> function: 0x10001aee0
>
>

That isn't equivalent; we like to have the functions as locals:

local max , min = math.min , math.max

you can't make a function of it, because you can't create/add locals
in the current point in lua evolution.

Reply | Threaded
Open this post in threaded view
|

Re: syntactic sugar for imports and exports

Josh Simmons
On Thu, Oct 20, 2011 at 11:02 AM, Daurnimator <[hidden email]> wrote:

> On 20 October 2011 10:30, Petite Abeille <[hidden email]> wrote:
>>
>> What about something along these lines:
>>
>> local function import( aName, ... )
>>  local aModule = require( aName )
>>  local aFunction = debug.getinfo( 2, 'f' ).func
>>  local _, anEnvironement = debug.getupvalue( aFunction, 1 )
>>
>>  for anIndex = 1, select( '#', ... ) do
>>    local aName = select( anIndex, ... )
>>    anEnvironement[ aName ] = aModule[ aName ]
>>  end
>> end
>>
>> import( 'math', 'min', 'max' )
>>
>> print( min )
>> print( max )
>>
>>> function: 0x10001af60
>>> function: 0x10001aee0
>>
>>
>
> That isn't equivalent; we like to have the functions as locals:
>
> local max , min = math.min , math.max
>
> you can't make a function of it, because you can't create/add locals
> in the current point in lua evolution.
>
>

I will note that this has been suggested and implemented before at least once.

http://lua-users.org/lists/lua-l//2005-09/msg00219.html

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

James Graves-2
In reply to this post by Hisham Muhammad
 Hisham <[hidden email]> wrote:

I think that was in the right direction. Geoff Leyland and Steve
Donovan provided valuable feedback, here's an improved version:

-------- begin loader5.lua ( https://gist.github.com/1298652 )
local lua_libs = {}
for k,v in pairs(package.loaded._G) do
  lua_libs[k] = v
end

local function loader(name, modpath)
 local mod, env = {}, {}
 env.module = function (name)
   mod._NAME = name
   setmetatable(env, { __index = function(t,k) return rawget(mod, k)
or rawget(lua_libs, k) end, __newindex = mod })
   return env
 end
 local fh = assert(io.open(modpath, 'rb'))
 local source = fh:read'*a'
 fh:close()
 local ret = assert(load(source, modpath, 'bt', env))(name)
 return ret or mod
end

-- replace Lua loader
package.searchers[2] = function (name)
 local modpath, msg = package.searchpath(name, package.path)
 if modpath then
   return loader, modpath
 else
   return nil, msg
 end
end
-------- end loader5.lua

In this version, module definitions are not frozen (as per Geoff
Leyland's concern) and no environment overhead is added to modules
that choose not to use module() (as per Steve Donovan's concern).

It may be more complex/magic than the Lua 5.1 implementation of
module() (is this revision as well?), but it does what we all wanted
module() to do (declare a module and allow declaration of public and
private entries through the environment), while giving users
convenience equivalent to that people use package.seeall (another
popular construct) for, without the widely discussed drawbacks of Lua
5.1 module().

I tried this out.  I like it.  A lot.  It seems quite clear and simple to use.  Are there any other downsides?

Thanks,

James

Reply | Threaded
Open this post in threaded view
|

Re: modules, require, magic

Roberto Ierusalimschy
In reply to this post by Hisham Muhammad
> > Moreover,
> > a modified 'load' function forces everyone to live with its magic.
>
> Not necessarily; Fabio's original loader did this but loader5
> restricted the magic to the users of module().

Maybe I am missing something, but I got the impression that, with
loader5 (as posted to the list), I get an empty environment if I do not
call 'module'.

  local function loader(name, modpath)
    local mod, env = {}, {}
   env.module = function (name)  ...   -- not called
   ...
   local ret = assert(load(source, modpath, 'bt', env))(name)
   ...
  end

Moreover, it seems that loader5 has some other drawbacks. (It needs to
read the whole module as a string; all accesses to globals go all the
time through function calls.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: syntactic sugar for imports and exports

Petite Abeille
In reply to this post by Mark Hamburg

On Oct 20, 2011, at 2:00 AM, Mark Hamburg wrote:

> That populates the environment but what one probably really wants are locals.

Sure. On the other hand, it has the same net effect: fold one namespace into another. Without having to change the entire language.


12345678