[ANN] mergelua

classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[ANN] mergelua

Sebastien Lai
Hello readers of lua-l,

I'm happy to announce a little utility of mine, called 'mergelua':
https://github.com/beelzebub/mergelua

This tool allows merging of (pure, noncompiled) Lua modules into a
single collective file.
It can be used to create packages of your Lua modules, similar to what
golang does, except that mergelua creates a single file.
Mergelua also features a local override of the 'require' function to
search the local moduletable first, and then execute the global
require() if the module does not exist in mergelua.

Synopsis:

    merge.lua <outputfilename>  <modulename>=<modulepath>
<anothermodule>=<anotherpath>

Example:

    merge.lua collection.lua oop=../oop.lua/oop.lua
configfile=../configfile/configfile.lua

There are probably a few things that might be tweaked:
   + minifying of Lua code (removing comments, newlines, etc)
   + support of bytecode-compiled chunks (unfortunately, I have yet to
find out how to store the data)
   + storing of .so / .dll files (same problem as above)

Please tell me what you think!

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Petite Abeille

On Oct 15, 2011, at 11:22 PM, Sebastien Lai wrote:

>   + support of bytecode-compiled chunks (unfortunately, I have yet to
> find out how to store the data)

Don't bother. This is what luac does out-of-the-box.

http://www.lua.org/manual/5.1/luac.html

Usage example:

http://dev.alt.textdrive.com/browser/HTTP/Makefile


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Sebastien Lai
On Sat, Oct 15, 2011 at 11:44 PM, Petite Abeille
<[hidden email]> wrote:
> Don't bother. This is what luac does out-of-the-box.

I'm aware of what luac does.
Though that will only work if the modules in question declare global
variables, but not when the Module returns a table (or any other
value, for that matter).

Mergelua however, honors both ways.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Petite Abeille

On Oct 15, 2011, at 11:54 PM, Sebastien Lai wrote:

> Though that will only work if the modules in question declare global
> variables,

In 5.1, modules are registered in package.loaded. Their incidental existence in _G, while unfortunate, is inconsequential.

> but not when the Module returns a table (or any other
> value, for that matter).

True, as this would interrupt the evaluation of the script.

This is why removing module from 5.2 is a boneheaded decision in my opinion.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Patrick Mc(avery-2
In reply to this post by Sebastien Lai
On 11-10-15 05:22 PM, Sebastien Lai wrote:

> Hello readers of lua-l,
>
> I'm happy to announce a little utility of mine, called 'mergelua':
> https://github.com/beelzebub/mergelua
>
> This tool allows merging of (pure, noncompiled) Lua modules into a
> single collective file.
> It can be used to create packages of your Lua modules, similar to what
> golang does, except that mergelua creates a single file.
> Mergelua also features a local override of the 'require' function to
> search the local moduletable first, and then execute the global
> require() if the module does not exist in mergelua.
>
> Synopsis:
>
>      merge.lua<outputfilename>   <modulename>=<modulepath>
> <anothermodule>=<anotherpath>
>
> Example:
>
>      merge.lua collection.lua oop=../oop.lua/oop.lua
> configfile=../configfile/configfile.lua
>
> There are probably a few things that might be tweaked:
>     + minifying of Lua code (removing comments, newlines, etc)
>     + support of bytecode-compiled chunks (unfortunately, I have yet to
> find out how to store the data)
>     + storing of .so / .dll files (same problem as above)
>
> Please tell me what you think!
>
>
Hi Sabastien

I've been a bit rattled with the endless discussions of locals vs
globals. I usually make a file copying script and concatenate the
various files into one. I find it simpler and I figure it is easier to
define scope with do - end blocks the the module mechanism. There isn't
a whole lot of documentation for mergelua, is this essential what it
does? I set up a configuration file and call mergelua on it?

Thanks for sharing your project with everyone-Patrick

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Patrick Mc(avery-2
In reply to this post by Petite Abeille

> This is why removing module from 5.2 is a boneheaded decision in my opinion.
Hi Petite Abeille

Your a super bright guy but do you have to phrase things with way? Lua
is a gift to us, we've paid nothing for it-Patrick



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Sebastien Lai
In reply to this post by Patrick Mc(avery-2
On Sun, Oct 16, 2011 at 12:05 AM, Patrick Mc(avery
<[hidden email]> wrote:
> Hi Sabastien
>
> I've been a bit rattled with the endless discussions of locals vs globals. I
> usually make a file copying script and concatenate the various files into
> one. I find it simpler and I figure it is easier to define scope with do -
Before I wrote merge.lua, that's what I did as well- but it wasn't
very nice, nor portable (/bin/bash anyone?)

>  There isn't a whole lot of
> documentation for mergelua, is this essential what it does? I set up a
> configuration file and call mergelua on it?
>

If you look at the merge.lua code[1], you'll notice that it is indeed
spartanic, doing only what is necessary to do. All in all, it's an
arguably nicer way of doing what could already be done with echo, cat,
and more cat :-)
The first post explains the usage quite well actually, mostly because
there really isn't much more to it.


[1] https://github.com/beelzebub/mergelua/blob/master/merge.lua

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Sebastien Lai
>> On Sun, Oct 16, 2011 at 12:05 AM, Patrick Mc(avery
>>  There isn't a whole lot of
>> documentation for mergelua, is this essential what it does? I set up a
>> configuration file and call mergelua on it?
>>
Apologies - I forgot to add to my previous post that 'configfile.lua'
is actually my inifile parser[1], not a configuration file passed to
merge.lua.

[1] https://github.com/beelzebub/configfile

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Petite Abeille
In reply to this post by Patrick Mc(avery-2

On Oct 16, 2011, at 12:09 AM, Patrick Mc(avery <[hidden email]> wrote:

>> This is why removing module from 5.2 is a boneheaded decision in my opinion.
> Your a super bright guy but do you have to phrase things with way?

Let me try to sugarcoat it for you: "This is why removing module from 5.2 is a shortsighted decision in my opinion."  Better? Wonder what Bill Maher would make of such weak phrasing :P

Still, it's a, hmmm, ^%*&$&*^%$%&^&*^*&%$%, err, regrettable mistake either way.




Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

David Manura
In reply to this post by Sebastien Lai
On Sat, Oct 15, 2011 at 5:22 PM, Sebastien Lai <[hidden email]> wrote:

> https://github.com/beelzebub/mergelua
> This tool allows merging of (pure, noncompiled) Lua modules into a
> single collective file. [...] Mergelua also features a local override of
> the 'require' function to search the local moduletable first
> [...]
> There are probably a few things that might be tweaked:
>   + minifying of Lua code (removing comments, newlines, etc)
>   + support of bytecode-compiled chunks (unfortunately, I have yet to
> find out how to store the data)
>   + storing of .so / .dll files (same problem as above)

I believe it's preferred to utilize package.preload rather than
override require.  There are a number of utilities like this linked on
the bottom of [1], notably precompiler/preloader, Squish, and
extensions to luac.  Squish handles minifying and compiling byte code
prior to bundling.  Bundling .so/.dll's as well is something some have
also mentioned, like my recent post on the LuaRocks mailing list that
made a comparison to similar things that have been done in Perl PAR
and Java JAR [2].  Your best bet for loading the .so/.dll's from your
bundle is to extract and cache them to a temporary directory (I'm not
aware of it being trivial to do otherwise) and load them into memory
by adjusting package.cpath or calling package.loadlib directly.

I haven't fully analyzed the available solutions though.  My suspicion
has been that string.dump is not as general as it could be [3,4], but
that has been moot on LuaJit which didn't add string.dump support
until recently.  Personally, I just use some combination of scripts
that work for me but are not entirely general.  A real world test of
some bundling tool like this would be to take all the modules in a
distribution (e.g. LuaForWindows/LuaRocks/LuaDist-Batteries), which
are written by many different authors, bundle them into a single file
with your tool, and run all the tests on those packages in the bundle.

I don't see how Lua 5.2-beta's treatment of module negatively weighs
into bundling.  A fundamental part of modularity indeed is
composability, and elimination of side-effects and obscure corner
cases in the module loader function (e.g. even making it pure [5])
furthers that goal.

[1] http://lua-users.org/wiki/BinToCee
[2] http://www.mail-archive.com/luarocks-developers@.../msg00196.html
[3] http://lua-users.org/lists/lua-l/2010-11/msg00898.html
[4] http://lua-users.org/lists/lua-l/2011-07/msg00082.html
[5] http://en.wikipedia.org/wiki/Pure_function

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Sebastien Lai
On Sun, Oct 16, 2011 at 3:40 AM, David Manura <[hidden email]> wrote:
> I believe it's preferred to utilize package.preload rather than
> override require.

I've been trying to insert package.preloade[<modulename>] = <etc> in
different ways, but it did not quite work for me.

So far my version with require(), which is only __locally__
overridden, appears to work a tiny wee bit more flawless than you
suggest. :-P

Anyway, I did remove my require() hook[1], and instead, inserted the
modules directly into `package.loaded`. It'll still honor the local
namespace, though. Who could possibly know why package.preload doesn't
work? :-P

[1] https://github.com/beelzebub/mergelua/blob/master/merge.lua#L114

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

KHMan
On 10/16/2011 8:20 PM, Sebastien Lai wrote:

> On Sun, Oct 16, 2011 at 3:40 AM, David Manura<[hidden email]>  wrote:
>> I believe it's preferred to utilize package.preload rather than
>> override require.
>
> I've been trying to insert package.preloade[<modulename>] =<etc>  in
> different ways, but it did not quite work for me.
>[snip]
> Anyway, I did remove my require() hook[1], and instead, inserted the
> modules directly into `package.loaded`. It'll still honor the local
> namespace, though. Who could possibly know why package.preload doesn't
> work? :-P

Code a short test case, then post it to the list and some of us
will poke it with a stick...

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Petite Abeille
In reply to this post by David Manura

On Oct 16, 2011, at 3:40 AM, David Manura wrote:

> I don't see how Lua 5.2-beta's treatment of module negatively weighs
> into bundling.

Perhaps a case of tunnel vision?

In 5.1, one can easily bundle multiple modules per Lua file, or concatenate multiple modules into one file, or package the entire enchilada in one binary file, courtesy of luac, without any fuss.

Not anymore in 5.2 without some contrived Goldberg concoctions (see above).

This is not progress by any measure of it, but rather a big step backward.

To add insult to injury, 5.2 introduces such frivolous function as table.pack -for no other obvious reason but to keep piece and harmony with unpack- while dropping the entire module functionality. Bad karma.

In summary: keep the 5.1 module functionality in 5.2 -minus the mess with the global namespace- as a useful, and meaningful, counterpart to require.  And be merry.






Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Roberto Ierusalimschy
> In 5.1, one can easily bundle multiple modules per Lua file, or
> concatenate multiple modules into one file, or package the entire
> enchilada in one binary file, courtesy of luac, without any fuss.

In Lua 5.2, I think you can easily write something like the following,
for multiple modules in one file:

do   -- Module 1
  local _ENV = mymodule("mod1")
  ...
end

do   -- Module 2
  local _ENV = mymodule("mod2")
  ...
end

where 'mymodule' is a somewhat trivial function.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Petite Abeille

On Oct 16, 2011, at 7:24 PM, Roberto Ierusalimschy wrote:

>> In 5.1, one can easily bundle multiple modules per Lua file, or
>> concatenate multiple modules into one file, or package the entire
>> enchilada in one binary file, courtesy of luac, without any fuss.
>
> In Lua 5.2, I think you can easily write something like the following,
> for multiple modules in one file:
>
> do   -- Module 1
>  local _ENV = mymodule("mod1")
>  ...
> end
>
> do   -- Module 2
>  local _ENV = mymodule("mod2")
>  ...
> end
>
> where 'mymodule' is a somewhat trivial function.

As oppose to:

module( 'mod1' )

...

module( 'mod2' )

...

This is a clear regression.

Trivial or not, module complements require and provides valuable functionalities (in contrast  to, say, table.pack).

At the risk of repeating myself verbatim [1]:

While part of the the behavior of lauxlib.luaL_pushmodule might be controversial (mangling global variables), the other functionalities of loadlib.ll_module are useful in and by themselves. Namely, (1) register module with _LOADED (lauxlib.luaL_pushmodule), (2) initialize module's _M, _NAME and _PACKAGE (loadlib.modinit), (3) execute options (lauxlib.dooptions), and (4) set the _ENV of the caller to be the module (lauxlib.set_env).

Keep these functionalities. Get rid of luaL_pushmodules' lua_pushglobaltable if you must.


[1] http://lua-users.org/lists/lua-l/2010-08/msg00310.html



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Matthew Wild
On 16 October 2011 18:51, Petite Abeille <[hidden email]> wrote:

>
> On Oct 16, 2011, at 7:24 PM, Roberto Ierusalimschy wrote:
>
>>> In 5.1, one can easily bundle multiple modules per Lua file, or
>>> concatenate multiple modules into one file, or package the entire
>>> enchilada in one binary file, courtesy of luac, without any fuss.
>>
>> In Lua 5.2, I think you can easily write something like the following,
>> for multiple modules in one file:
>>
>> do   -- Module 1
>>  local _ENV = mymodule("mod1")
>>  ...
>> end
>>
>> do   -- Module 2
>>  local _ENV = mymodule("mod2")
>>  ...
>> end
>>
>> where 'mymodule' is a somewhat trivial function.
>
> As oppose to:
>
> module( 'mod1' )
>
> ...
>
> module( 'mod2' )

... Error: attempt to call global 'module' (a nil value).

module() is too magic in my opinion.

Regards,
Matthew

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Petite Abeille

On Oct 16, 2011, at 8:05 PM, Matthew Wild wrote:

> ... Error: attempt to call global 'module' (a nil value).

Come on, you know better than that :)

> module() is too magic in my opinion.

Again, we had this discussion over one year ago and I think we agreed to disagree on that one [1].

Creating a closed, self-contained environment is a feature of module. That you dislike its specific implementation (setfenv), is a matter of taste, not functionality.

[1] http://lua-users.org/lists/lua-l/2010-08/msg00314.html
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Dirk Laurie-2
In reply to this post by Petite Abeille
2011/10/16 Petite Abeille <[hidden email]>:

> To add insult to injury, 5.2 introduces such frivolous function as table.pack -

Frivolous?

Now:

    local pack = table.unpack

Before:

    local function pack(...) local a = {...}; a.n=#a; return a end

OK. Frivolous.

Dirk

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Roberto Ierusalimschy
In reply to this post by Petite Abeille
> > In Lua 5.2, I think you can easily write something like the following,
> > for multiple modules in one file:
> >
> > do   -- Module 1
> >  local _ENV = mymodule("mod1")
> >  ...
> > end
> >
> > do   -- Module 2
> >  local _ENV = mymodule("mod2")
> >  ...
> > end
> >
> > where 'mymodule' is a somewhat trivial function.
>
> As oppose to:
>
> module( 'mod1' )
>
> ...
>
> module( 'mod2' )
>
> ...
>
> This is a clear regression.

In my view this is clear progress.

Modules do not interfere with each other: module 1 does not change the
environment for module 2 (as happened by mistake in your example),
module 2 does not see locals declared in module 1, etc. Each module is
sintactically distinct, with a clear boundary around each. They are
much more self contained.

-- Roberto


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] mergelua

Eduardo Ochs
In reply to this post by Dirk Laurie-2


On Sun, Oct 16, 2011 at 4:27 PM, Dirk Laurie <[hidden email]> wrote:
2011/10/16 Petite Abeille <[hidden email]>:

> To add insult to injury, 5.2 introduces such frivolous function as table.pack -

Frivolous?

Now:

   local pack = table.unpack

Before:

   local function pack(...) local a = {...}; a.n=#a; return a end

OK. Frivolous.

Dirk

Your definition is equivalent to

  local pack = function (...) return {n=#{...}, ...} end

but the right definition is:

  local pack = function (...) return {n=select("#", ...), ...} end

the "n" field of the top one is somewhat hard to predict when there
are nils among the arguments.

123