Require

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Require

D Burgess-2
Was this a really dumb question or have I missed something 
on the list?
-------------
What (if any) plans are in place for Lua 5.1 and require()?

Will venv be incorporated into Lua 5.1 baselib?

Diegos LuaSocket and the Kepler project seem to have 
conceded the failings of the existing (Lua5.0) require().

Does this mean we get a new require in Lua 5.1?

regards
David B




Reply | Threaded
Open this post in threaded view
|

Re: Require

Diego Nehab-3
Hi,

I definitely wasn't a dumb question. I have been bothering Roberto with
this topic for some time now (sometimes off-line), and I think that soon
he will converge into a nice solution (i.e., a new require() with friends).

Lua is used in many different ways and the tough problem is to reach a
unified solution that is general enough to satisfy various embedding and
scripting needs, but closed enough to enforce some kind of uniformity.
And, of course, simple to implement, to use as a developer, and as a end
user.

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: Require

Steven Elkins
On Sat, 24 Jul 2004 00:46:45 -0400 (EDT), Diego Nehab
<[hidden email]> wrote:

> I definitely wasn't a dumb question. I have been bothering Roberto with
> this topic for some time now (sometimes off-line), and I think that soon
> he will converge into a nice solution (i.e., a new require() with friends).

I hope Roberto's solution leaves room for the technique at the very end of

    http://www.lua.org/pil/15.4.html

since I've become addicted to it.

Reply | Threaded
Open this post in threaded view
|

Re: Require

Luiz Henrique de Figueiredo
In reply to this post by Diego Nehab-3
> Lua is used in many different ways and the tough problem is to reach a
> unified solution that is general enough to satisfy various embedding and
> scripting needs, but closed enough to enforce some kind of uniformity.

Another way to read this is that "require" (in its various incarnations
discussed here) is trying to set policy not just provide a mechanism.
This goes against the goals of Lua. But if we want to have reliable Lua
installations, then some kind of policy is needed.

Perhaps "require" and friends would be better housed in a separate library...
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: Require

D Burgess-2
In reply to this post by D Burgess-2
On 2004-07-24 at 20:56:11 Luiz Henrique de Figueiredo <[hidden email]> wrote:

>But if we want to have reliable Lua
>installations, then some kind of policy is needed.
Absolutely agree with this.

DB			




Reply | Threaded
Open this post in threaded view
|

Re: Require

Diego Nehab-3
In reply to this post by Luiz Henrique de Figueiredo
Hi,

> Perhaps "require" and friends would be better housed in a separate
> library...

I agree with that, as long as it it a "separate library" that comes
standard with Lua, like lauxlib. Maybe we could have another round of
open discussion on the topic and then you guys could take the input,
use the fact this is not realy a democracy, and come up with a
solution for everyone.  :o)

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: Require

D Burgess-2
In reply to this post by D Burgess-2
>
>> Perhaps "require" and friends would be better housed in a separate
>> library...
>
>I agree with that, as long as it it a "separate library" that comes
>standard with Lua, like lauxlib. 
>
I agree with Diego



Reply | Threaded
Open this post in threaded view
|

Re: Require

Mark Hamburg-4
In reply to this post by D Burgess-2
FYI...

On my project we've implemented a modified version of require based around a
notion of code packaging. In the case of MacOS X, these are bundles. Every
bundle gets it's own environment that inherits from the global environment.
That way multiple scripts and C code in the same bundle can see each other,
but they don't stomp the rest of the environment.

We've also built an import mechanism in which one issues a call:

    import "name"

This looks up "name" in a dictionary and returns the value that's there if
it already exists and invokes code to go find something to load if it
doesn't. The loaded code then exports one or more named values (generally
tables). This relies on having metadata associated with loadable entities
indicating what names they can supply.

"require" is mostly used as part of loading pieces of an individual module
while "import" is used to invoke the loading of modules.

Mark


Reply | Threaded
Open this post in threaded view
|

Re: Require

Rici Lake-2
That is very pretty.

I assume when you say "exports one or more named values" that you mean that it exports them to the dictionary import() uses? In that
case, you could even make import the dictionary, instead of a
function, and define an __index metamethod.

That would make the implementation something like this, leaving out
the interesting parts (like the bundle-finder and the bundle-runner):

import = setmetatable({}, {__index =
  function(self, name)
    local bundle, bundlename = findBundleForName(name)
    setfenv(bundle, setmetatable({}, {__index = getfenv(0)})
    local exports = bundle()
    for k, v in exports do
      if rawget(self, k) == nil then self[k] = v end
    end
    local rv = rawget(self, name)
    if rv == nil then
      error(bundlename .. " did not export " .. name)
    end
    return rv
  end})

-------

local showRect = import.showRect

-------


On 24-Jul-04, at 7:36 PM, Mark Hamburg wrote:

FYI...

On my project we've implemented a modified version of require based around a notion of code packaging. In the case of MacOS X, these are bundles. Every bundle gets it's own environment that inherits from the global environment. That way multiple scripts and C code in the same bundle can see each other,
but they don't stomp the rest of the environment.

We've also built an import mechanism in which one issues a call:

    import "name"

This looks up "name" in a dictionary and returns the value that's there if
it already exists and invokes code to go find something to load if it
doesn't. The loaded code then exports one or more named values (generally tables). This relies on having metadata associated with loadable entities
indicating what names they can supply.

"require" is mostly used as part of loading pieces of an individual module
while "import" is used to invoke the loading of modules.

Mark


Reply | Threaded
Open this post in threaded view
|

Re: Require

Diego Nehab-3
In reply to this post by Steven Elkins
Hi,

> I hope Roberto's solution leaves room for the technique at the very end of
>
>     http://www.lua.org/pil/15.4.html
>
> since I've become addicted to it.

I have been pushing for the automatic locals/globals way to define which
functions get exported. The ideas I heard so far are indeed compatible
with that last method.

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

RE: Require

Vijay Aswadhati-2
In reply to this post by Mark Hamburg-4
> We've also built an import mechanism in which one issues a call:
>
>    import "name"
>

Cool. Does your "import" feature handle hierarchy? 
As in Java: 
<code>
import java.io.InputStream;
</code>

Assuming the "bundle" has InputStream among other resources.

--v.a



Reply | Threaded
Open this post in threaded view
|

Re: Require

Mark Hamburg-4
In reply to this post by Rici Lake-2
Yes, export puts values in the dictionary used by import.

I thought about making import just use the __index metamethod but was left
feeling fuzzy on what to do in the event of an error. If it really acts just
like a table, then it should just return nil if the value isn't present.
True an __index metamethod can throw an exception, but that's a departure
from what the syntax would lead one to expect.

I like your implementation. It's a bit simpler than mine. On the other hand,
the complexity in mine may be from the fact that I take explicit steps to
detect and disallow cyclic import.

I've been thinking of extending my import function to allow specifying a
series of names to extract from the exported value:

    local _, addObserver, removeObserver, notifyObservers =
        import( "com.baymoon.Observations", "add", "remove", "notify" )

Missing names would trigger an exception at import time.

Mark

on 7/24/04 5:56 PM, Rici Lake at [hidden email] wrote:

> That is very pretty.
> 
> I assume when you say "exports one or more named values" that you mean
> that it exports them to the dictionary import() uses? In that
> case, you could even make import the dictionary, instead of a
> function, and define an __index metamethod.
> 
> That would make the implementation something like this, leaving out
> the interesting parts (like the bundle-finder and the bundle-runner):
> 
> import = setmetatable({}, {__index =
>  function(self, name)
>    local bundle, bundlename = findBundleForName(name)
>    setfenv(bundle, setmetatable({}, {__index = getfenv(0)})
>    local exports = bundle()
>    for k, v in exports do
>      if rawget(self, k) == nil then self[k] = v end
>    end
>    local rv = rawget(self, name)
>    if rv == nil then
>      error(bundlename .. " did not export " .. name)
>    end
>    return rv
>  end})
> 
> -------
> 
> local showRect = import.showRect


Reply | Threaded
Open this post in threaded view
|

Re: Require

Mark Hamburg-4
In reply to this post by Vijay Aswadhati-2
It uses arbitrary strings as identifiers, so:

    import "lua.io.InputStream"

Speaking of which, bundles are basically directories (MacOS X defines them
more specifically, but for these purposes directories will do). We include a
special "info.lua" file in the directory that says something like:

info.lua:

    return {
        exports = {
            [ "com.baymoon.Stack" ] = "collections.lua",
            [ "com.baymoon.Queue" ] = "collections.lua",
                -- two exports from one file
            [ "com.baymoon.Matrix" ] = "C:load_matrix"
                -- assumes a single library DLL in which we
                -- will find the symbol load_matrix
        }
    }

Bundles can load piecemeal. If we import stacks or queues but not matrices,
then load_matrix won't get called.

If while loading a Lua file, it wants to make use of other code, for
example, we might have some shared utilities that were used by multiple
files in the bundle but not to be exported themselves, we make use of the
bundle-specific require. Thus, collections.lua can say:

    require "utilities.lua"

Or

    require "C:load_utilities"

This will use something much like the existing require mechanism to see that
the code runs exactly once and will run the appropriate code from within the
bundle. It does not go through the mapping provided by info.lua but just
uses filenames.

I'm not sure keeping the name "require" was entirely the right thing to do
but my replacement seemed more or less in keeping with the spirit of the
existing code.

Mark

on 7/25/04 3:56 AM, Vijay Aswadhati at [hidden email] wrote:

> 
>> We've also built an import mechanism in which one issues a call:
>> 
>>    import "name"
>> 
> 
> Cool. Does your "import" feature handle hierarchy?
> As in Java: 
> <code>
> import java.io.InputStream;
> </code>
> 
> Assuming the "bundle" has InputStream among other resources.
> 
> --v.a
> 
> 


Reply | Threaded
Open this post in threaded view
|

Re: Require

Diego Nehab-3
In reply to this post by D Burgess-2
Hi,

Here are the main ideas I have been trying to sell. I have thoughts on how
to sort out some of the arising problems, but the important thing is to make
sure the ideas are good.  The two main goals I advocate are the following:

    Client scripts should be as portable across distributions as possible
        local socket = require"socket" -- should work if LuaSocket is there

    Library code whould be independent of packaging
        Lua code can be on filesystem, in loh, in resource.
        C code can be statically linked or in shared library

I believe the best way to define which symbols get exported into the
namespace is the local/global paradigm, implemented with setfenv.

    For purists, it *is* possible to prevent the global environment from
    being visible inside the namespace, so I only see advantages.

I also believe it is worth it to add cycle awareness to the require()
function. It's just one line of code anyways.

I like the idea of hierarchical names, such as require("socket.http").
I don't care if the namespace is actually exported into a global
socket.http, as long as it is *also* returned by require().

The biggest challenge in the is how to map a name, such as
"socket.http", into a function (loaded Lua chunk or a C entrypoint)
to be called. There doesn't appear to be a way to make everyone happy.
I believe the best way to do it is to provide some standard that works
for mainstream OSs (those that have directories, at least) *and*
provide an easy way to override the mapping function, preserving the rest
of the functionality.

I think that it will be hard to get require() to deal well with both C and
Lua libraries (very hard to find a mapping function that does a good job).
I really believe we need a function that behaves somewhat like require(),
but targetted at loading C libraries. Still, the user should only care
about the require() function. It's the library developer that needs to care
about the other function.

I think every C library should have a Lua loader module. These loaders can
be really simple and efficient and add a lot to the flexibility of the
packaging system. Require() would only deal with those loaders. The loader
than has to find the C code to load, and here is where we need the
equivalent of require(), but for C code.

It is especially important to allow libraries that have both C and Lua
parts to share the same namespace during the export process.

The developer always knows what he is writing, so I think the attempt use
require() for everything doesn't help, but is instead harmful (can't have
same name for C part and Lua part, forcing artificial aliases).

These are my $2... :o)

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: Require

Rici Lake-2
In reply to this post by Mark Hamburg-4

On 25-Jul-04, at 12:33 PM, Mark Hamburg wrote:

I thought about making import just use the __index metamethod but was left feeling fuzzy on what to do in the event of an error. If it really acts just like a table, then it should just return nil if the value isn't present.

Yes, that is a good argument. Of course, later attempts to use the
result as a table would generate a run-time error, but it would be
nicer to be more precise. I was actually assuming that it was ok
to throw an exception from a table lookup; an alternative would be
to return nil and ask careful programmers to type:

local Observations = assert(import.Observations)

which would also allow the programmer to try something else, or probe for optional plugins. (True for the function version, as well, of course.)

If you are expecting bundle names to be hierarchical, then the function is probably better, but they could certain both exist simultaneously.

the complexity in mine may be from the fact that I take explicit steps to detect and disallow cyclic import.

There was quite a discussion about cyclic import at some point. How does one handle mutually dependent packages? It is not always easy to eliminate this. Indeed, to my mind this is the semantic difference between "require" and "import": require defines a dependency (this package won't work without the other one) while import defines a need:(I'm about to use this package, I need it now.) Of course, that is just me, and the particular words are not important, but there do seem to be two use cases.

I've been thinking of extending my import function to allow specifying a series of names to extract from the exported value:

Perhaps this little function is actually useful:

function ensure(tab, ...)
  for i = 1, args.n do
    if tab[args[i]] then args[i] = tab[args[i]]
                    else error("Table is missing key '"
                               .. tostring(args[i])
                               .."'")
    end
  end
  return unpack(args)
end

local addObserver, removeObserver, notifyObservers =
        ensure(import "com.baymoon.Observations",
               "add", "remove", "notify" )


But, actually, I would personally go for lazy name loading instead; i.e. the import function makes a table and attaches loader routines to each name, without actually running the loader routines.

I do feel strongly that such loader routines should exist, for reasons I have ranted about sufficiently before. :)

R.


Reply | Threaded
Open this post in threaded view
|

Re: Require

Asko Kauppi-3
In reply to this post by Diego Nehab-3

And also multiple layers of scripts sharing the same namespace (extending it).

This would allow for 'derivation' of libraries (with extra features) easily. Or should such derivatives always have a separate name (even if they're backwards compatible?).

I've added "version awareness" in LuaX (which has 'USES' function that can take an optional version info, meaning I need i.e. luaSocket 2.0 but nothing earlier).

-ak


Also, each module contains an _info table, with some descriptive details:

> USES 'lib.paragui'
USES 'lib.paragui': /Users/asko/TestBox/Sources/luax-perforce/Modules/Paragui/ paragui_wrap.lua

> dump(lib.paragui._info)
PLATFORMS :     'Win32, Linux, OS X'
AUTHOR :        '[hidden email]'
NOTES : 'Also the end application should carry the FreeType copyright message in its about box or similar.'
LICENSE :       'LGPL+BSD'
COPYRIGHT : 'Portions copyright (c) 1996-2002 The FreeType Project (www.freetype.org). All rights reserved.'
DEPENDENCIES :  'paragui (v.1.1.7) by Alexander Pipelka
sdl (v.1.2) by Sam Lantinga
freetype2 by David Turner et.al.
'
MODULE :        'ParaGui 1.1.7+'




25.7.2004 kello 22:53, Diego Nehab kirjoitti:

 It is especially important to allow libraries that have both C and Lua
parts to share the same namespace during the export process.


Reply | Threaded
Open this post in threaded view
|

Re: Require

Jamie Webb-3
In reply to this post by Diego Nehab-3
On Sun, Jul 25, 2004 at 03:53:05PM -0400, Diego Nehab wrote:
>     For purists, it *is* possible to prevent the global environment from
>     being visible inside the namespace, so I only see advantages.
 
It's not just about purity: there are security issues. Untrusted code
should be able to use libraries, and not be able to get at restricted
functions through the library namespace, or be able to redefine the
functions called by the library.

> The biggest challenge in the is how to map a name, such as
> "socket.http", into a function (loaded Lua chunk or a C entrypoint)
> to be called. There doesn't appear to be a way to make everyone happy.
> I believe the best way to do it is to provide some standard that works
> for mainstream OSs (those that have directories, at least) *and*
> provide an easy way to override the mapping function, preserving the rest
> of the functionality.

Presumably there will be a global table of mappings, which may be created in
advance from C, in order to deal with static libraries. If the require
function just looks in that table and expects the function to be
there, then the default filesystem-based loader could be the __index
metamethod for that table.

Anyone requiring different functionality could replace __index, either
retaining the default to make a chain, or replacing it entirely. That
would mean exposing a primitive_require or something, which would take
a loaded chunk and do the trickery necessary to turn it into a
namespace-constructor.

-- Jamie Webb

Reply | Threaded
Open this post in threaded view
|

Re: Require

Edgar Toernig
In reply to this post by Diego Nehab-3
Diego Nehab wrote:
>
> Here are the main ideas I have been trying to sell. I have thoughts on how
> to sort out some of the arising problems, but the important thing is to make
> sure the ideas are good.  The two main goals I advocate are the following:
> 
>     Client scripts should be as portable across distributions as possible
>         local socket = require"socket" -- should work if LuaSocket is there

Fine.

>     Library code whould be independent of packaging
>         Lua code can be on filesystem, in loh, in resource.
>         C code can be statically linked or in shared library

Fine.

> I believe the best way to define which symbols get exported into the
> namespace is the local/global paradigm, implemented with setfenv.

Hmm... I don't like the setfenv idea.  Let the library decide where
it puts its stuff.  "Clean" libraries won't do nasty things but still
be able to replace/add globals.  IMHO, a recommendation on how "clean"
libraries should be implemented would suffice.  Don't force developers
to a specific model.  (Btw, I'm still hoping that setfenv gets nuked
in 5.1 *g*)

> I also believe it is worth it to add cycle awareness to the require()
> function. It's just one line of code anyways.

Agree.

> I like the idea of hierarchical names, such as require("socket.http").
> I don't care if the namespace is actually exported into a global
> socket.http, as long as it is *also* returned by require().

Fine.  Let require just cache the first result (possibly nil) of the
init function.  Nothing more.  Whether a library actually returns a
namespace table or not should be irrelevant to require.  It's the
policy of the library.

> The biggest challenge in the is how to map a name, such as
> "socket.http", into a function (loaded Lua chunk or a C entrypoint)
> to be called. There doesn't appear to be a way to make everyone happy.
> I believe the best way to do it is to provide some standard that works
> for mainstream OSs (those that have directories, at least) *and*
> provide an easy way to override the mapping function, preserving the rest
> of the functionality.

Right, that's a difficult part.

> I think that it will be hard to get require() to deal well with both C and
> Lua libraries (very hard to find a mapping function that does a good job).
> I really believe we need a function that behaves somewhat like require(),
> but targetted at loading C libraries.

Hmm... why?  require is just a front-end.  Sure, one has to decide at one
point which actual loader is invoked but shouldn't it be at a lower level?

An extendable loader system that is able to determine both location and
type of the data and invoked via loadfile() would be marvelous.

> I think every C library should have a Lua loader module. These loaders can
> be really simple and efficient and add a lot to the flexibility of the
> packaging system. Require() would only deal with those loaders. The loader
> than has to find the C code to load, and here is where we need the
> equivalent of require(), but for C code.

To have the possibility to wrap any library with a Lua script is IMHO
crucial.  But force it?  No!  Just think about statically linked
libraries...

> It is especially important to allow libraries that have both C and Lua
> parts to share the same namespace during the export process.

The less policy you put into require the easier it is for the library
developer to take care of that *g*

> These are my $2... :o)

And my 2 Euros ;-)

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Require

Rici Lake-2

On 25-Jul-04, at 9:08 PM, Edgar Toernig wrote:

(Btw, I'm still hoping that setfenv gets nuked
in 5.1 *g*)

Agreed.

A version which only worked on Lua functions (not C functions and not call stack pseudo-indices) would be manageable.

Actually, I agree with most of what et wrote, except that I personally favour the style where the package-definition (C or Lua) gets passed a table into which it is expected to put its definitions; Wim's LTN-11 works very nicely for me.

Rici


Reply | Threaded
Open this post in threaded view
|

Re: Require

Diego Nehab-3
In reply to this post by Edgar Toernig
Hi,

> > I believe the best way to define which symbols get exported into the
> > namespace is the local/global paradigm, implemented with setfenv.
>
> Hmm... I don't like the setfenv idea.  Let the library decide where
> it puts its stuff.  "Clean" libraries won't do nasty things but still
> be able to replace/add globals.  IMHO, a recommendation on how "clean"
> libraries should be implemented would suffice.  Don't force developers
> to a specific model.  (Btw, I'm still hoping that setfenv gets nuked
> in 5.1 *g*)

We still need a standard so that require() obtains the namespace to be
cached and return. I like the convention of returning whatever the chunk
returns.

However, I believe the most natural way of telling Lua which names
should be exported is the use of local/global names, and the best way to
do this is to have a function which libraries that choose to do so can call to
set thigns up. Roberto was thinking about a package(...) function for
this purpose (this is where the setfenv magic would happen).

In such a context, even libraries that choose to call package() can
still replace/add globals (_G.print = nil).

Libraries that don't do it could still return just the namespace they
created by hand.

> > I like the idea of hierarchical names, such as require("socket.http").
> > I don't care if the namespace is actually exported into a global
> > socket.http, as long as it is *also* returned by require().
>
> Fine.  Let require just cache the first result (possibly nil) of the
> init function.  Nothing more.  Whether a library actually returns a
> namespace table or not should be irrelevant to require.  It's the
> policy of the library.

Agree.

> > I think that it will be hard to get require() to deal well with both C and
> > Lua libraries (very hard to find a mapping function that does a good job).
> > I really believe we need a function that behaves somewhat like require(),
> > but targetted at loading C libraries.
>
> Hmm... why?  require is just a front-end.  Sure, one has to decide at one
> point which actual loader is invoked but shouldn't it be at a lower level?

This is for extra freedom. LuaSocket, for instance, has a C part and a
Lua part. The Lua part, which is loaded by require(), wants to load
the C part. The C part might be a shared library lying somewhere, but it
might be linked statically. To ensure it will always work, I need
someone else to map names to C entrypoints, just like require() maps names
to Lua chunks.

It shouldn't be up to the library developer to come up with the mapping.
It should be up to the guy that is creating the instalation. This new
function would be part of the instalation, not part of LuaSocket.

> An extendable loader system that is able to determine both location and
> type of the data and invoked via loadfile() would be marvelous.

*If* a C function and a Lua chunk could be treated exactly the same way,
I would agree.

> To have the possibility to wrap any library with a Lua script is IMHO
> crucial.  But force it?  No!  Just think about statically linked
> libraries...

What is the big deal? Just stick the loaded chunk somewhere the mapping
function can find. It doesn't matter if it's a Lua file in the filesystem,
a statically linked LOH or Lua source inside a resource. What matters is
that it *can* be any of these without changes to the library code.

It's up to the mapping function to find the loaders, and it's up to
whoever is creating a distribution to come up with a mapping function
and decide if it's going to be a LOH, a lua file etc.

> > It is especially important to allow libraries that have both C and Lua
> > parts to share the same namespace during the export process.
>
> The less policy you put into require the easier it is for the library
> developer to take care of that *g*

And harder for someone who is putting 25 different libraries into a
distribution to figure out how to do it.

[]s,
Diego.

12