bundled packages

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

bundled packages

Adrian Sietsma
Hi.

As a frequent user of luasocket on windows, I have built a bundled dll, with
all the lua code as resources. Neat and tidy, with a small change to loadlib
(look for resources after lua, before luaopen_). This simplifies packaging,
and can be implemented as a seperate library that hooks into the package
loaders (it just needs copies of some of loadlib's static fn's).
I don't know if such a scheme translates readily to other os's : it seems a
good fit for windows : each lua script is embedded in the dll as RCDATA with
the name "luamodule_a.b.c". The only limitation is that module names are
case-insensitive.

My problem is this : I wish to combine 3 packages (socket, mime and ltn12)
into a single dll. To do this, I've added a table "alias" to package, so if
package.alias["mime"] = "socket", the c loader will look for mime in
socket.dll, if it can't find it in mime.dll. This again can be implemented
outside loadlib, at the cost of some code duplication.

Any problems with this approach ? It would certainly simplify distribution
on windows.

Adrian
Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Diego Nehab-3
Hi,

> My problem is this : I wish to combine 3 packages (socket, mime and ltn12)
> into a single dll. To do this, I've added a table "alias" to package, so if
> package.alias["mime"] = "socket", the c loader will look for mime in
> socket.dll, if it can't find it in mime.dll. This again can be implemented
> outside loadlib, at the cost of some code duplication.
>
> Any problems with this approach ? It would certainly simplify distribution on
> windows.

I would suggest two alternatives that don't require changing any C code.

1) using package.preload

Let's say you have packages a, b, c, inside some DLL d. You can do
something like this

     -- untested code follows

     package.preload.a = rc_loader("a", "d")
     package.preload.b = rc_loader("b", "d")
     package.preload.b = rc_loader("c", "d")

     -- returns a function that when invoked will load the appropriate
     -- chunk from the resource and call it
     function rc_loader(module, bundle)
         return function()
             local f = rc_chunk(module, bundle)
             f()
         end
     end

     function rc_chunk(module, bundle)
         -- somehow obtain the chunk from the resource in the DLL
         -- perhaps calling require(bundle) helps find it?
     end


2) Alternatively, create a new loader

     -- untested code follows

     function rc_loader(bundle)
         return function(name)
             -- somehow obtain the chunk from the resource in the DLL
             -- perhaps calling require(bundle) helps find it?
             return f
         end
     end

     table.insert(package.loaders, rc_loader("d"))

Hope I didn't mess up any details.

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Adrian Sietsma
Diego Nehab wrote:

>
> I would suggest two alternatives that don't require changing any C code.
>
> 1) using package.preload
>
> Let's say you have packages a, b, c, inside some DLL d. You can do
> something like this
>
>     -- untested code follows
>
>     package.preload.a = rc_loader("a", "d")
>     package.preload.b = rc_loader("b", "d")
>     package.preload.b = rc_loader("c", "d")
>
>     -- returns a function that when invoked will load the appropriate
>     -- chunk from the resource and call it
>     function rc_loader(module, bundle)
>         return function()
>             local f = rc_chunk(module, bundle)
>             f()
>         end
>     end
>
>     function rc_chunk(module, bundle)
>         -- somehow obtain the chunk from the resource in the DLL
>         -- perhaps calling require(bundle) helps find it?
>     end
you have described by lua prototype almost exactly :)
or for lazy loading:
package.preload.a = function(x) return assert(rcloader("a"))(x) end

> 2) Alternatively, create a new loader
>
>     -- untested code follows
>
>     function rc_loader(bundle)
>         return function(name)
>             -- somehow obtain the chunk from the resource in the DLL
>             -- perhaps calling require(bundle) helps find it?
>             return f
>         end
>     end
>
>     table.insert(package.loaders, rc_loader("d"))

that is almost exactly how i set up the resource search, to look for
"socket.ftp" as a resource in socket.dll.

>
> Hope I didn't mess up any details.
>
> Regards,
> Diego.
>
Thanks - that is very similar to my initial prototype; if i require
"socket", it sets up the preloader for "mime", "ltn12", etc.

I'm playing with "package.alias" so i can set up a mapping (eg. via
luainit), so that require"ltn12" knows to look in socket.dll, without having
to require (and load) socket.

ps no c code need be modified; require"rcloadlib" will insert the new loader
etc., it just needs duplicates of several loadlib internal fn's, like
findfile() etc.

Adrian
Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Diego Nehab-3
Hi,

> I'm playing with "package.alias" so i can set up a mapping (eg. via luainit),
> so that require"ltn12" knows to look in socket.dll, without having to require
> (and load) socket.

How does that work? Is package.alias specific to RC or is it generic
enough to say: Hey, the entrypoint for package "foo" is actually in the
library of package "bar". Does that even make sense? It would for C, but
not for Lua.

Why don't you like the preload solution? You wouldn't need the
package.alias table.

> ps no c code need be modified; require"rcloadlib" will insert the new loader
> etc., it just needs duplicates of several loadlib internal fn's, like
> findfile() etc.

Perhaps findfile should be exported by the package table? Lua Authors?
I actually suggested this some time ago, off-line I believe.

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Adrian Sietsma
Diego Nehab wrote:

> Hi,
>
>> I'm playing with "package.alias" so i can set up a mapping (eg. via
>> luainit), so that require"ltn12" knows to look in socket.dll, without
>> having to require (and load) socket.
>
>
> How does that work? Is package.alias specific to RC or is it generic
> enough to say: Hey, the entrypoint for package "foo" is actually in the
> library of package "bar".
That is true for C and RC functions.
Does that even make sense? It would for C, but
> not for Lua.
I agree, it makes no sense for Lua.
There may be a case for different lua search strategies, but I'm not making
it here.

In my implementation, package.alias is only used if the cRoot and RC loaders
cannot find a package; in that case it effectively re-tries cRoot and RC
with package.alias.name as the filename, again using cpath. Apart from
anything else, it allows me to rename dll's that have name clashes with
other windows software :
package.alias.socket="luasocket"
VS
package.preload["socket"]=RCload("full_path_to_luasocket_dll","socket").
package.preload["socket.core"]=loadlib("full_path_to_luasocket_dll","socket.core").
package.preload["socket.url"]=...

I have played games with the preload table, but to work in all cases (eg
require a.b.c), the preload table needs to be filled in for all sub-modules.
I am trying out the alias approach (which can still be an add-on loader) so
that package.alias.mime=socket will resolve correctly for require"mime.xyz",
without needing a whole slew of entries.

>
> ....
>
> Perhaps findfile should be exported by the package table? Lua Authors?
> I actually suggested this some time ago, off-line I believe.
That would save me replicating it, and make add-on loaders easier to implement.

Adrian.
Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Diego Nehab-3
Hi,

> In my implementation, package.alias is only used if the cRoot and RC loaders
> cannot find a package; in that case it effectively re-tries cRoot and RC with
> package.alias.name as the filename, again using cpath.

What does the RC loader do? The same as the C loader, but trying to get
the chunk from a resource name that is a function of the subpackage
name?

> I have played games with the preload table, but to work in all cases (eg
> require a.b.c), the preload table needs to be filled in for all sub-modules.
> I am trying out the alias approach (which can still be an add-on loader) so
> that package.alias.mime=socket will resolve correctly for require"mime.xyz",
> without needing a whole slew of entries.

How about playing tricks with the metatable of package.preload? You
should be able use pattern matching to resolve names.

[]s,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Adrian Sietsma
Diego Nehab wrote:

> Hi,
>
>> In my implementation, package.alias is only used if the cRoot and RC
>> loaders cannot find a package; in that case it effectively re-tries
>> cRoot and RC with package.alias.name as the filename, again using cpath.
>
>
> What does the RC loader do? The same as the C loader, but trying to get
> the chunk from a resource name that is a function of the subpackage
> name?
Yes. It looks for a resource called luamodule_a.b.c in the file(s) returned
from cpath.

>
>> I have played games with the preload table, but to work in all cases
>> (eg require a.b.c), the preload table needs to be filled in for all
>> sub-modules. I am trying out the alias approach (which can still be an
>> add-on loader) so that package.alias.mime=socket will resolve
>> correctly for require"mime.xyz", without needing a whole slew of entries.
>
>
> How about playing tricks with the metatable of package.preload? You
> should be able use pattern matching to resolve names.
>
Yes, I thought of that, but the alias approach is a lot simpler.

Adrian