bundled packages

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

bundled packages

Adrian Sietsma
Hi.

As a frequent user of luasocket on windows, I have built a bundled dll, with all the lua code as resources. Neat and tidy, with a small change to loadlib (look for resources after lua, before luaopen_). This simplifies packaging, and can be implemented as a seperate library that hooks into the package loaders (it just needs copies of some of loadlib's static fn's). I don't know if such a scheme translates readily to other os's : it seems a good fit for windows : each lua script is embedded in the dll as RCDATA with the name "luamodule_a.b.c". The only limitation is that module names are case-insensitive.

My problem is this : I wish to combine 3 packages (socket, mime and ltn12) into a single dll. To do this, I've added a table "alias" to package, so if package.alias["mime"] = "socket", the c loader will look for mime in socket.dll, if it can't find it in mime.dll. This again can be implemented outside loadlib, at the cost of some code duplication.

Any problems with this approach ? It would certainly simplify distribution on windows.

Adrian

Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Diego Nehab-3
Hi,

My problem is this : I wish to combine 3 packages (socket, mime and ltn12) into a single dll. To do this, I've added a table "alias" to package, so if package.alias["mime"] = "socket", the c loader will look for mime in socket.dll, if it can't find it in mime.dll. This again can be implemented outside loadlib, at the cost of some code duplication.

Any problems with this approach ? It would certainly simplify distribution on windows.

I would suggest two alternatives that don't require changing any C code.

1) using package.preload

Let's say you have packages a, b, c, inside some DLL d. You can do
something like this

    -- untested code follows

    package.preload.a = rc_loader("a", "d")
    package.preload.b = rc_loader("b", "d")
    package.preload.b = rc_loader("c", "d")

    -- returns a function that when invoked will load the appropriate
    -- chunk from the resource and call it
    function rc_loader(module, bundle)
        return function()
            local f = rc_chunk(module, bundle)
            f()
        end
    end

    function rc_chunk(module, bundle)
        -- somehow obtain the chunk from the resource in the DLL
        -- perhaps calling require(bundle) helps find it?
    end


2) Alternatively, create a new loader

    -- untested code follows

    function rc_loader(bundle)
        return function(name)
            -- somehow obtain the chunk from the resource in the DLL
            -- perhaps calling require(bundle) helps find it?
            return f
        end
    end

    table.insert(package.loaders, rc_loader("d"))

Hope I didn't mess up any details.

Regards,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Adrian Sietsma
Diego Nehab wrote:

I would suggest two alternatives that don't require changing any C code.

1) using package.preload

Let's say you have packages a, b, c, inside some DLL d. You can do
something like this

    -- untested code follows

    package.preload.a = rc_loader("a", "d")
    package.preload.b = rc_loader("b", "d")
    package.preload.b = rc_loader("c", "d")

    -- returns a function that when invoked will load the appropriate
    -- chunk from the resource and call it
    function rc_loader(module, bundle)
        return function()
            local f = rc_chunk(module, bundle)
            f()
        end
    end

    function rc_chunk(module, bundle)
        -- somehow obtain the chunk from the resource in the DLL
        -- perhaps calling require(bundle) helps find it?
    end
you have described by lua prototype almost exactly :)
or for lazy loading:
package.preload.a = function(x) return assert(rcloader("a"))(x) end

2) Alternatively, create a new loader

    -- untested code follows

    function rc_loader(bundle)
        return function(name)
            -- somehow obtain the chunk from the resource in the DLL
            -- perhaps calling require(bundle) helps find it?
            return f
        end
    end

    table.insert(package.loaders, rc_loader("d"))

that is almost exactly how i set up the resource search, to look for "socket.ftp" as a resource in socket.dll.


Hope I didn't mess up any details.

Regards,
Diego.

Thanks - that is very similar to my initial prototype; if i require "socket", it sets up the preloader for "mime", "ltn12", etc.

I'm playing with "package.alias" so i can set up a mapping (eg. via luainit), so that require"ltn12" knows to look in socket.dll, without having to require (and load) socket.

ps no c code need be modified; require"rcloadlib" will insert the new loader etc., it just needs duplicates of several loadlib internal fn's, like findfile() etc.

Adrian

Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Diego Nehab-3
Hi,

I'm playing with "package.alias" so i can set up a mapping (eg. via luainit), so that require"ltn12" knows to look in socket.dll, without having to require (and load) socket.

How does that work? Is package.alias specific to RC or is it generic
enough to say: Hey, the entrypoint for package "foo" is actually in the
library of package "bar". Does that even make sense? It would for C, but
not for Lua.

Why don't you like the preload solution? You wouldn't need the
package.alias table.

ps no c code need be modified; require"rcloadlib" will insert the new loader etc., it just needs duplicates of several loadlib internal fn's, like findfile() etc.

Perhaps findfile should be exported by the package table? Lua Authors?
I actually suggested this some time ago, off-line I believe.

Regards,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Adrian Sietsma
Diego Nehab wrote:
Hi,

I'm playing with "package.alias" so i can set up a mapping (eg. via luainit), so that require"ltn12" knows to look in socket.dll, without having to require (and load) socket.


How does that work? Is package.alias specific to RC or is it generic
enough to say: Hey, the entrypoint for package "foo" is actually in the
library of package "bar".
That is true for C and RC functions.
Does that even make sense? It would for C, but
not for Lua.
I agree, it makes no sense for Lua.
There may be a case for different lua search strategies, but I'm not making it here.

In my implementation, package.alias is only used if the cRoot and RC loaders cannot find a package; in that case it effectively re-tries cRoot and RC with package.alias.name as the filename, again using cpath. Apart from anything else, it allows me to rename dll's that have name clashes with other windows software :
package.alias.socket="luasocket"
VS
package.preload["socket"]=RCload("full_path_to_luasocket_dll","socket").
package.preload["socket.core"]=loadlib("full_path_to_luasocket_dll","socket.core").
package.preload["socket.url"]=...

I have played games with the preload table, but to work in all cases (eg require a.b.c), the preload table needs to be filled in for all sub-modules. I am trying out the alias approach (which can still be an add-on loader) so that package.alias.mime=socket will resolve correctly for require"mime.xyz", without needing a whole slew of entries.


....

Perhaps findfile should be exported by the package table? Lua Authors?
I actually suggested this some time ago, off-line I believe.
That would save me replicating it, and make add-on loaders easier to implement.

Adrian.

Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Diego Nehab-3
Hi,

In my implementation, package.alias is only used if the cRoot and RC loaders cannot find a package; in that case it effectively re-tries cRoot and RC with package.alias.name as the filename, again using cpath.

What does the RC loader do? The same as the C loader, but trying to get
the chunk from a resource name that is a function of the subpackage
name?

I have played games with the preload table, but to work in all cases (eg require a.b.c), the preload table needs to be filled in for all sub-modules. I am trying out the alias approach (which can still be an add-on loader) so that package.alias.mime=socket will resolve correctly for require"mime.xyz", without needing a whole slew of entries.

How about playing tricks with the metatable of package.preload? You
should be able use pattern matching to resolve names.

[]s,
Diego.

Reply | Threaded
Open this post in threaded view
|

Re: bundled packages

Adrian Sietsma
Diego Nehab wrote:
Hi,

In my implementation, package.alias is only used if the cRoot and RC loaders cannot find a package; in that case it effectively re-tries cRoot and RC with package.alias.name as the filename, again using cpath.


What does the RC loader do? The same as the C loader, but trying to get
the chunk from a resource name that is a function of the subpackage
name?
Yes. It looks for a resource called luamodule_a.b.c in the file(s) returned from cpath.


I have played games with the preload table, but to work in all cases (eg require a.b.c), the preload table needs to be filled in for all sub-modules. I am trying out the alias approach (which can still be an add-on loader) so that package.alias.mime=socket will resolve correctly for require"mime.xyz", without needing a whole slew of entries.


How about playing tricks with the metatable of package.preload? You
should be able use pattern matching to resolve names.

Yes, I thought of that, but the alias approach is a lot simpler.

Adrian