package proposal

classic Classic list List threaded Threaded
50 messages Options
123
Reply | Threaded
Open this post in threaded view
|

package proposal

Roberto Ierusalimschy
This is a proposal for a more standard package usage in Lua.  It
includes some changes in the `require' function, some changes in the
`luaL_openlib' function, and a new function `module'. We already got
previous comments about this proposal from Diego Nehab (LuaSocket),
Andre Carregal (Kepler) and Tomas Gorham (Kepler).

The following is a description of the new system.  At the end of this
text is an implementation for the functions `require' and `module' in
Lua, which serves as a more precise specification.


* A *package* is a collection of modules distributed together.
Each *module* lives in single file, which contains a single module.
(Of course, a package may comprise a single module.)

* Modules can be loaded in (at least) three forms:
pre-loaded (that is, the module is loaded in the Lua state
during C initialization code, like the standard libraries);
pre-linked (the module code is linked with the Lua executable,
but must be loaded explicitly);
and dynamically (the module code is found through a search in
some path, and then linked and loaded).


* Given a statement <<require"mod">>,
require first checks whether it is already loaded,
looking for a field "mod" in a control table `package.loaded'.
(So, pre-loaded modules must register themselves in the package.loaded table.)

Otherwise, `require' checks for a function in package.preload["mod"];
if found, it calls that function to load the module.
(So, pre-linked modules must register their open functions in
the package.preload table.)

Otherwise, require looks for a file "mod" in the path package.cpath
(typically it will be something like "./?.so;/usr/local/lib/lua/5.0/?.so"
or "./?.dll;C:windows/?.dll").
If found, it tries to call function "luaopen_mod" on that
library to open it.
Otherwise, it looks for "mod" in the path package.path (old LUA_PATH),
loads it with "loadfile" and runs it.

* It is expected that a module "mod" will put all its stuff in a
global table "mod". Moreover, it will arrange for the
call <<require"mod">> to return that table, so that users can
write

   local xl = require"mod"

and use local xl to access the module.


* Module names can be hierarquical (e.g., "socket.smtp").
A collection of modules with a common root forms a package.
Such hierarchy affects only module names.
The fact that two modules A and B are in the same package
does not create any special relation between A and B.
A module named "mod.lili" should be in a file "mod/lili" in
the path.  Such module should be loaded in global mod.lili.
(Like most strings in this proposal,
the "/" separator can be changed in the luaconfig file.
A system may choose to use a "_" as separator and therefore keep all
modules in a flat directory.)


* The `module' function provides an easy way to create Lua modules
that conform to what is expected from a module.
To write a module, the programmer simply writes

  module(...)     -- ... is the vararg expression

at the start of the chunk and then declares
all module stuff as global values.
The `module' function creates a global with the appropriate name
if it does not exist and sets it as the metatable of
the new module.
It also sets that global as the value of package.loaded[given-name],
so that the corresponding `require' call will return the module.
It also arranges for the module to inherit from _G,
so that it can access (but not change) global values directly.

The `module' function also defines two names in the new namespace:
_NAME has the module name.
_PACK has the package name, which a module can use to require its
own relatives (e.g., <<require _PACK.."mysiblingname">>).


* The `luaL_openlib' function will be changed to put the table
wherein it opens the library in package.loaded[libname].
Therefore, whatever the way the library was open,
a later call to `require' will detect that it is already
open and will return it.
(The standard libraries are a particular example: you can
write <<local s = require"string">> to use the string library.)


* Because everything is written on top of regular Lua stuff (tables,
environments, metatables, etc.) it is easy to bend any rule when there
is a good reason. For instance, if a module exports a single function,
it can return it directly, instead of putting it into a table; a file
may export its stuff into someone else's namespace; more complicated C
modules may be imported through an auxiliary Lua file that calls
loadlib; etc.



--------------------------------------------------------------

package = {}
package.path = LUA_PATH or os.getenv("LUA_PATH") or
             ("./?.lua;" ..
              "/usr/local/share/lua/5.0/?.lua;" ..
              "/usr/local/share/lua/5.0/?/init.lua" )
 
package.cpath = os.getenv("LUA_CPATH") or
             "./?.so;" ..
             "/usr/local/lib/lua/5.0/?.so;" ..
             "/usr/local/lib/lua/5.0/lib?.so"

package.loaded = {}

package.preload = {}


--
-- looks for a file `name' in given path
--
local function search (path, name)
  for c in string.gfind(path, "[^;]+") do
    c = string.gsub(c, "%?", name)
    local f = io.open(c)
    if f then   -- file exist?
      f:close()
      return c
    end
  end
  return nil    -- file not found
end


--
-- new require
--
function _G.require (name)
  if not package.loaded[name] then
    package.loaded[name] = true
    local f = package.preload[name]
    if not f then
      local filename = string.gsub(name, "%.", "/")
      local fullname = search(package.cpath, filename)
      if fullname then
        local openfunc = "luaopen_" .. string.gsub(name, "%.", "")
        f = assert(loadlib(fullname, openfunc))
      else
        fullname = search(package.path, filename)
        if not fullname then
          error("cannot find "..name.." in path "..package.path, 2)
        end
        f = assert(loadfile(fullname))
      end
    end
    local res = f(name)
    if res then package.loaded[name] = res end
  end
  return package.loaded[name]
end


--
-- auxiliar function to read "nested globals"
--
local function getfield (t, f)
  for w in string.gfind(f, "[%w_]+") do
    if not t then return nil end
    t = t[w]
  end
  return t
end


--
-- auxiliar function to write "nested globals"
--
local function setfield (t, f, v)
  for w in string.gfind(f, "([%w_]+)%.") do
    t[w] = t[w] or {}   -- create table if absent
    t = t[w]            -- get the table
  end
  local w = string.gsub(f, "[%w_]+%.", "")   -- get last field name
  t[w] = v            -- do the assignment
end


--
-- new module function
--
function _G.module (name)
  local ns = getfield(_G, name)         -- search for namespace
  if not ns then
    ns = {}                             -- create new namespace
    setmetatable(ns, {__index = _G})
    setfield(_G, name, ns)
    ns._NAME = name
    ns._PACK = string.gsub(name, "[^.]*$", "")
  end
  package.loaded[name] = ns
  setfenv(2, ns)
end


--------------------------------------------------

-- Roberto

Reply | Threaded
Open this post in threaded view
|

RE: package proposal

Kevin Baca-2
This is very similar to something I'm doing - which means I like it very
much.

One thing I'll note - it seems that it's possible to have multiple
modules in a single file simply by calling module() multiple times.

I don't think this is a bad thing, but it violates the requirement of
one module per file.  Perhaps that requirement should be removed?

-Kevin


> -----Original Message-----
> From: [hidden email] 
> [[hidden email]] On Behalf Of 
> Roberto Ierusalimschy
> Sent: Friday, September 17, 2004 9:50 AM
> To: Lua discussion list
> Subject: package proposal
> 
> 
> 
> This is a proposal for a more standard package usage in Lua.  It
> includes some changes in the `require' function, some changes in the
> `luaL_openlib' function, and a new function `module'. We already got
> previous comments about this proposal from Diego Nehab (LuaSocket),
> Andre Carregal (Kepler) and Tomas Gorham (Kepler).
> 
> The following is a description of the new system.  At the end of this
> text is an implementation for the functions `require' and `module' in
> Lua, which serves as a more precise specification.
> 
> 
> * A *package* is a collection of modules distributed together.
> Each *module* lives in single file, which contains a single module.
> (Of course, a package may comprise a single module.)
> 
> * Modules can be loaded in (at least) three forms:
> pre-loaded (that is, the module is loaded in the Lua state
> during C initialization code, like the standard libraries);
> pre-linked (the module code is linked with the Lua executable,
> but must be loaded explicitly);
> and dynamically (the module code is found through a search in
> some path, and then linked and loaded).
> 
> 
> * Given a statement <<require"mod">>,
> require first checks whether it is already loaded,
> looking for a field "mod" in a control table `package.loaded'.
> (So, pre-loaded modules must register themselves in the 
> package.loaded table.)
> 
> Otherwise, `require' checks for a function in package.preload["mod"];
> if found, it calls that function to load the module.
> (So, pre-linked modules must register their open functions in
> the package.preload table.)
> 
> Otherwise, require looks for a file "mod" in the path package.cpath
> (typically it will be something like 
> "./?.so;/usr/local/lib/lua/5.0/?.so"
> or "./?.dll;C:windows/?.dll").
> If found, it tries to call function "luaopen_mod" on that
> library to open it.
> Otherwise, it looks for "mod" in the path package.path (old LUA_PATH),
> loads it with "loadfile" and runs it.
> 
> * It is expected that a module "mod" will put all its stuff in a
> global table "mod". Moreover, it will arrange for the
> call <<require"mod">> to return that table, so that users can
> write
> 
>    local xl = require"mod"
> 
> and use local xl to access the module.
> 
> 
> * Module names can be hierarquical (e.g., "socket.smtp").
> A collection of modules with a common root forms a package.
> Such hierarchy affects only module names.
> The fact that two modules A and B are in the same package
> does not create any special relation between A and B.
> A module named "mod.lili" should be in a file "mod/lili" in
> the path.  Such module should be loaded in global mod.lili.
> (Like most strings in this proposal,
> the "/" separator can be changed in the luaconfig file.
> A system may choose to use a "_" as separator and therefore keep all
> modules in a flat directory.)
> 
> 
> * The `module' function provides an easy way to create Lua modules
> that conform to what is expected from a module.
> To write a module, the programmer simply writes
> 
>   module(...)     -- ... is the vararg expression
> 
> at the start of the chunk and then declares
> all module stuff as global values.
> The `module' function creates a global with the appropriate name
> if it does not exist and sets it as the metatable of
> the new module.
> It also sets that global as the value of package.loaded[given-name],
> so that the corresponding `require' call will return the module.
> It also arranges for the module to inherit from _G,
> so that it can access (but not change) global values directly.
> 
> The `module' function also defines two names in the new namespace:
> _NAME has the module name.
> _PACK has the package name, which a module can use to require its
> own relatives (e.g., <<require _PACK.."mysiblingname">>).
> 
> 
> * The `luaL_openlib' function will be changed to put the table
> wherein it opens the library in package.loaded[libname].
> Therefore, whatever the way the library was open,
> a later call to `require' will detect that it is already
> open and will return it.
> (The standard libraries are a particular example: you can
> write <<local s = require"string">> to use the string library.)
> 
> 
> * Because everything is written on top of regular Lua stuff (tables,
> environments, metatables, etc.) it is easy to bend any rule when there
> is a good reason. For instance, if a module exports a single function,
> it can return it directly, instead of putting it into a table; a file
> may export its stuff into someone else's namespace; more complicated C
> modules may be imported through an auxiliary Lua file that calls
> loadlib; etc.
> 
> 
> 
> --------------------------------------------------------------
> 
> package = {}
> package.path = LUA_PATH or os.getenv("LUA_PATH") or
>              ("./?.lua;" ..
>               "/usr/local/share/lua/5.0/?.lua;" ..
>               "/usr/local/share/lua/5.0/?/init.lua" )
>  
> package.cpath = os.getenv("LUA_CPATH") or
>              "./?.so;" ..
>              "/usr/local/lib/lua/5.0/?.so;" ..
>              "/usr/local/lib/lua/5.0/lib?.so"
> 
> package.loaded = {}
> 
> package.preload = {}
> 
> 
> --
> -- looks for a file `name' in given path
> --
> local function search (path, name)
>   for c in string.gfind(path, "[^;]+") do
>     c = string.gsub(c, "%?", name)
>     local f = io.open(c)
>     if f then   -- file exist?
>       f:close()
>       return c
>     end
>   end
>   return nil    -- file not found
> end
> 
> 
> --
> -- new require
> --
> function _G.require (name)
>   if not package.loaded[name] then
>     package.loaded[name] = true
>     local f = package.preload[name]
>     if not f then
>       local filename = string.gsub(name, "%.", "/")
>       local fullname = search(package.cpath, filename)
>       if fullname then
>         local openfunc = "luaopen_" .. string.gsub(name, "%.", "")
>         f = assert(loadlib(fullname, openfunc))
>       else
>         fullname = search(package.path, filename)
>         if not fullname then
>           error("cannot find "..name.." in path "..package.path, 2)
>         end
>         f = assert(loadfile(fullname))
>       end
>     end
>     local res = f(name)
>     if res then package.loaded[name] = res end
>   end
>   return package.loaded[name]
> end
> 
> 
> --
> -- auxiliar function to read "nested globals"
> --
> local function getfield (t, f)
>   for w in string.gfind(f, "[%w_]+") do
>     if not t then return nil end
>     t = t[w]
>   end
>   return t
> end
> 
> 
> --
> -- auxiliar function to write "nested globals"
> --
> local function setfield (t, f, v)
>   for w in string.gfind(f, "([%w_]+)%.") do
>     t[w] = t[w] or {}   -- create table if absent
>     t = t[w]            -- get the table
>   end
>   local w = string.gsub(f, "[%w_]+%.", "")   -- get last field name
>   t[w] = v            -- do the assignment
> end
> 
> 
> --
> -- new module function
> --
> function _G.module (name)
>   local ns = getfield(_G, name)         -- search for namespace
>   if not ns then
>     ns = {}                             -- create new namespace
>     setmetatable(ns, {__index = _G})
>     setfield(_G, name, ns)
>     ns._NAME = name
>     ns._PACK = string.gsub(name, "[^.]*$", "")
>   end
>   package.loaded[name] = ns
>   setfenv(2, ns)
> end
> 
> 
> --------------------------------------------------
> 
> -- Roberto
> 


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Asko Kauppi-3
In reply to this post by Roberto Ierusalimschy

I was going to say the "[^;]+" pattern won't work but seems you've corrected that since 5.0.2. :)

Another detail: does this mean we can reassign the loop var in future Lua? ("for c", then "c=")

-ak


17.9.2004 kello 19:49, Roberto Ierusalimschy kirjoitti:

 local function search (path, name)
  for c in string.gfind(path, "[^;]+") do
    c = string.gsub(c, "%?", name)


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Asko Kauppi-3
In reply to this post by Roberto Ierusalimschy

One question:

Is there a way to "try" require, that is to find out whether a module would exist, without causing a major alert. Of course, pcall() can be used for this.. is that the way?

I'd use it for code where extra abilities can be enabled (s.a. ftp upload if sockets are around) but even without them, the code can do some stuff.

-ak


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

D Burgess-4
In reply to this post by Roberto Ierusalimschy
Thanks Roberto.
question:

Q 1) If I have mylib.lua and mylib.dll, then  "require'mylib'" will
load the DLL first. (I like this). What was the thinking behing this?

Q 2) If CPATH is empty then the mylib.lua would be loaded rather than mylib.dll.
Methinks that for Win32 (at least) that an empty or non-existent CPATH
should result in loadlib("mylib", ...). This will by default search
the executable
directory, the current directory, system directories and then whatever is in
the PATH environment variable.

So
>package.cpath = os.getenv("LUA_CPATH") or
>            "./?.so;" ..
>            "/usr/local/lib/lua/5.0/?.so;" ..
>            "/usr/local/lib/lua/5.0/lib?.so"

>            "/usr/local/lib/lua/5.0/?.so;" ..
>            "/usr/local/lib/lua/5.0/lib?.so"
is sort of redundant because  the first place loadlib looks is where lua.exe
exists.
>            "./?.so;" ..
is sort of redundant because cwd  is the second place loadlib looks.
In general loading from cwd is

hence I think this should become 

package.cpath = os.getenv("LUA_CPATH") or ""

and 
the load sequence should be

1. if LUA_CPATH exists or is non blank use your specified search strategy.

2. loadlib("module")

Did I miss something  here?

Q 3) Luiz's library name convention of namespace = 'mylib' and
the file name of 'lmylib.so' and the open function of
'luaopen_mylib' is changed to use 'mylib.so'. 
Again what was the reason for this change?

Also for Win32 methinks it would be better to try loadlib() rather than do the
fairly slow "fopen()" existence check.

DB

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Wim Couwenberg-4
In reply to this post by Roberto Ierusalimschy
I use a 'split' function (see below) to iterate over
possible locations.  It accepts nested (possibly
empty) branches between parentheses.  Also, it treats
`|' and `;' as equivalent separators.

Here's an example:

split("(.|wim(/aap|/noot))/?(|.lua;.lc);mod/?.(mod|mc)",
print)

output:

./?
./?.lua
./?.lc
wim/aap/?
wim/aap/?.lua
wim/aap/?.lc
wim/noot/?
wim/noot/?.lua
wim/noot/?.lc
mod/?.mod
mod/?.mc

Here's the implementation of split:

local split

do

local cache = setmetatable({}, {__mode = "v"})

local function topsplit(path, fun)
  local open, close = string.find(path, "%b()")
  local pos = string.find(path, "[|;]")
  local first = 1
  while pos do
    if open and pos > open and pos < close then
      pos = close + 1
      open, close = string.find(path, "%b()", pos)
    else
      fun(string.sub(path, first, pos - 1))
      pos = pos + 1
      first = pos
    end
    pos = string.find(path, "[|;]", pos)
  end
  fun(string.sub(path, first))
end

local function subsplit(part, fun)
  local open, close = string.find(part, "%b()")
  if open then
    local pre = string.sub(part, 1, open - 1)
    local post = string.sub(part, close + 1)
    split(string.sub(part, open + 1, close - 1),
function(part)
      local head = pre .. part
      subsplit(post, function(tail)
        fun(head .. tail)
      end)
    end)
  else
    fun(part)
  end
end

function split(path, fun)
  local c = cache[path]
  if c then
    for i, p in ipairs(c) do fun(p) end
  else
    c = {}

    local function add(part)
      table.insert(c, part)
      fun(part)
    end

    topsplit(path, function(part)
      subsplit(part, add)
    end)

    cache[path] = c
  end
end

end


--
Wim


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Reuben Thomas-5
In reply to this post by Roberto Ierusalimschy
On Fri, 17 Sep 2004, Roberto Ierusalimschy wrote:

This is a proposal for a more standard package usage in Lua.

Sorely needed, and good to see it's being done. I like the way that pre-compiled and Lua modules are treated uniformly. This is important.

* A *package* is a collection of modules distributed together.
Each *module* lives in single file, which contains a single module.
(Of course, a package may comprise a single module.)

I'm skeptical of these definitions. I like the namespace rules, but I'm not sure why packages need to exist (are package-relative module names really useful?), nor what "distributed together" means. In particular, packages seem to be ocnfused with namespaces: why shouldn't independently-distributed modules providing related functionality be grouped into a package (and even require one another)?

The "one file = one module" rule is arbitrary and wrong. Most languages with a decent module system don't have it. Modules are effectively types (= namespaces), not physical units of code. Of course, there's nothing wrong with demanding that each module be loaded by a single file.

I think, all in all, this proposal suffers from feature bloat. Yes, the current various namespace/module schemes need to be accommodated, and yes, a credible standard is needed so we can stop worrying about how to integrate code from different sources. But this won't be done by being too complex.

The CPAN model might offer some lessons here: from a code writing (as opposed to packaging) point of view, it's almost trivial to write a new CPAN module, and yet it provides an effective way for programmers to contribute to the massive pool of Perl it provides.

--
http://rrt.sc3d.org/ | Crews help mock terror casualties (BBC)

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Thatcher Ulrich
In reply to this post by D Burgess-4
On Sep 18, 2004 at 10:42 +1000, David Burgess wrote:
> 
> Q 1) If I have mylib.lua and mylib.dll, then "require'mylib'" will
> load the DLL first. (I like this). What was the thinking behing
> this?

This is kind of a nitpick, but that's actually the only feature of the
proposal I don't like at first glance.  It's not a huge problem, but
for the pattern of a Lua wrapper around a C module, it means the DLL
must have a different base name than the Lua wrapper.  E.g.:

SDL.lua:
----
module"SDL"

assert(require "gluahost")
assert(gluahost.load("SDL_dll"))	-- different name!

function self_test()
	 -- ... etc ...
end
-- ... possibly other helper stuff ...
----

I think it would be more convenient for this case if .lua files got
loaded first, so the binary file could share the same base name as the
wrapper.  This would also allow modules to be extended after-the-fact
using a Lua wrapper.

On the other hand, the DLL name can always be mangled.

Overall, I'm hugely in favor of the proposal.

-- 
Thatcher Ulrich
http://tulrich.com

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Asko Kauppi-3

19.9.2004 kello 00:27, Thatcher Ulrich kirjoitti:

 On the other hand, the DLL name can always be mangled.

I'm not sure it can.. Remember, the initialisation function also has the moudle name in it. I'm not sure I like that very much, wouldn't a specific function name (s.a. 'open_module()') be easier, and it would allow free renaming of the file later.


Overall, I'm hugely in favor of the proposal.

Me, too.  It's about time. :)

-ak


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

David Given
Asko Kauppi wrote:
[...]
I'm not sure it can.. Remember, the initialisation function also has the moudle name in it. I'm not sure I like that very much, wouldn't a specific function name (s.a. 'open_module()') be easier, and it would allow free renaming of the file later.

This doesn't allow you to link all the modules together with the Lua runtime to form a static executable, because they'll all be exporting the same name. Not a particularly big issue, but it does add another layer of complexity.

--
[insert interesting .sig here]


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Remo Dentato
In reply to this post by Asko Kauppi-3
> > Overall, I'm hugely in favor of the proposal.
> Me, too.  It's about time. :)

I think that missing a complete packaging system so far is the only thing
that prevented Lua to become the most spread scripting language.

My 2c.
                            R.D


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Asko Kauppi-3
In reply to this post by David Given

Good point. :)

19.9.2004 kello 05:36, David Given kirjoitti:

 Asko Kauppi wrote:
[...]
I'm not sure it can.. Remember, the initialisation function also has the moudle name in it. I'm not sure I like that very much, wouldn't a specific function name (s.a. 'open_module()') be easier, and it would allow free renaming of the file later.

This doesn't allow you to link all the modules together with the Lua runtime to form a static executable, because they'll all be exporting the same name. Not a particularly big issue, but it does add another layer of complexity.

--
[insert interesting .sig here]



Reply | Threaded
Open this post in threaded view
|

Weak tables (was Re: package proposal)

Mark Hamburg-4
In reply to this post by Wim Couwenberg-4
on 9/18/04 7:17 AM, Wim Couwenberg at [hidden email] wrote:

> local cache = setmetatable({}, {__mode = "v"})

Would it make sense to provide standard globals for the various metatables
for weak tables? Or are they a small enough fraction of the average heap
that sharing them doesn't make much difference?

Mark


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Peter Loveday-2
In reply to this post by Asko Kauppi-3
I agree this is far overdue, and I hope this will be included as a standard system.

Lets face it, the only way anything ever gets agreed on here is if it comes from the Lua authors, the current state of modules etc is testimony to why we need this badly.

Don't forget consistant Lua dll/so naming while you're at it; binary modules need to link against the same lua runtime.

Love, Light and Peace,
- Peter Loveday
Director of Development, eyeon Software


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Roberto Ierusalimschy
In reply to this post by Kevin Baca-2
> One thing I'll note - it seems that it's possible to have multiple
> modules in a single file simply by calling module() multiple times.

More or less. You can *define* multiple modules in a single file, but
then you cannot import them individually.


> I don't think this is a bad thing, but it violates the requirement of
> one module per file.  Perhaps that requirement should be removed?

That is not a requirement, but more a "rule for common usage". As
written later in the proposal, all those rules can be broken if there is
a good reason.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Roberto Ierusalimschy
In reply to this post by Asko Kauppi-3
> Another detail:  does this mean we can reassign the loop var in future 
> Lua?  ("for c", then "c=")

No. It only means I wrote it wrong :)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Roberto Ierusalimschy
In reply to this post by Asko Kauppi-3
> Is there a way to "try" require, that is to find out whether a module 
> would exist, without causing a major alert.  Of course, pcall() can be 
> used for this.. is that the way?

Yes. This is an example of the "all is regular Lua" rule. We try to
offer the most common usage patterns; other usages can be programmed
with regular Lua stuff.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Mark Hamburg-4
In reply to this post by Roberto Ierusalimschy
I'm still pondering Roberto's proposal. It's certainly an improvement over
what is available right now.

For consideration in the discussion, I'm also going to throw out a summary
of the properties of our homebrew package system. I think I've enumerated
some of this before, but not in this detail. The code right now isn't
cleanly severable enough to offer up, but we might be able to work on that
given sufficient interest. (The interest needs to be real since it will
require talking to lawyers...)

* We group things into "bundles". These more or less map to MacOS X bundles,
but are essentially just directories.

* A bundle can offer to supply one or more "namespaces". It does so via a
special info.lua file which is a Lua file describing the properties of the
package. The idea is that one can scan and even cache info.lua files. For
example, an Info.lua file might look like the following:

    return {
        exports = {
            "namespace1" = "filename1",
            "namespace2" = "filename2"
        }
    }

* Namespace names are essentially arbitrary strings. No hierarchical package
structuring is assumed though it can be provided by using names
appropriately.

* Namespaces are imported via 'import "namespace"'. This can return an
arbitrary Lua value.

* import looks to see whether the namespace is already loaded and if not it
finds a bundle which exports that namespace and loads and executse the
appopriate file from within the bundle. We could easily handle multiple
types of file by trying various extensions on the file name. I don't recall
whether we do or not in the present implementation.

* We handle native code by assuming a specific block of code associated with
a bundle. The export above then becomes
"namespace" = "C:function_name"'. This allows multiple entrypoints to native
code. This could easily be modified to allow named native code files as well
though I think it would still be good to retain the ability to name
functions within them.

* Code within a bundle gets executed in an environment that references the
bundle so that other resources such as images can be found. We do this via a
new "global" _B defined in the environment for the bundle code.

* We define a bundle-specific version of require so that a bundle can load
itself piecewise without exposing those pieces to the rest of the world. It
might be nice to make this more uniform by for example redefining the
namespace import function within a bundle to see the private namespaces.

* When a namespace provider function runs, it can actually export multiple
namespaces. In particular, it can export the same value under multiple names
so that one can export the same thing as "myNamespace" and "myNamespace_v2".
This way, code that needs version 2 can explicitly import it but the
exporting code doesn't have to build separate structures.

* We do not add namespaces to the global namespace on the basis that this
potentially encourages the bad habit of relying on some other piece of code
to have done the import.

If adding to the globals was really useful, at the expense of not supporting
hierarchical names, one could define a metamethod on the globals table to
make import automatic. We have not done so.

* We disallow cyclic import.

* import provides an extended form:

        import( "namespace", "entry1", "entry2", ... )

This returns not just the namespace but looks up the values in the result
returning them as extra results and asserts if they are not defined. import
also asserts if it can't  obtain the namespace, though I could see
redefining it to return nil. For example:

    local _, addObserver, notifyObservers =
            import( "observations", "add", "notify" )

----

One nice thing about the info.lua files is that they make it easy to add
support for other things like looking for all of the file format filters
simply by having those get reported as extra entries.

Mark


Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Roberto Ierusalimschy
In reply to this post by D Burgess-4
> Q 1) If I have mylib.lua and mylib.dll, then  "require'mylib'" will
> load the DLL first. (I like this). What was the thinking behing this?

Currently, LUA_PATH may have a last fixed component, which will be
called whenever Lua cannot find a module. If we search the LUA_PATH
first, we will loose that facility.


> Q 2) If CPATH is empty then the mylib.lua would be loaded rather
> than mylib.dll.  Methinks that for Win32 (at least) that an empty or
> non-existent CPATH should result in loadlib("mylib", ...).

Why cpath would be empty or non-existent? Lua always initializes it to a
convenient default.


> Q 3) Luiz's library name convention of namespace = 'mylib' and the file
> name of 'lmylib.so' and the open function of 'luaopen_mylib' is changed
> to use 'mylib.so'.  Again what was the reason for this change?

I followed the guidelines in

   http://oss.digirati.com.br/packaging_lua/third-party.html

that were discussed in this list. Anyway, you only have to change
the cpath to add that `l' to library names (".../l?.dll").


-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: package proposal

Roberto Ierusalimschy
In reply to this post by Reuben Thomas-5
> why shouldn't independently-distributed modules providing related
> functionality be grouped into a package (and even require one another)?

What about "A *package* is a collection of interrelated modules"?


> The "one file = one module" rule is arbitrary and wrong. Most
> languages with a decent module system don't have it.

Python has it hardcoded. Perl allows a package to be defined in several
files, but there is no standard way to load such a package. (I am not
sure whether they qualify as "languages with a decent module system",
but there are few complains about their module systems.) In Lua, too,
you will be able to spread a module through several files (and even to
define several modules in one file), but there is no standard way to
import such modules.


> I think, all in all, this proposal suffers from feature bloat. [...]
> The CPAN model might offer some lessons here:

Which feature in this proposal are not present in Perl? (Ok, the stuff
about C modules and preloaded modules are not present in Perl, but Perl
almost do not use C modules.) Which specific feature do you think we
should remove?

-- Roberto

123