FAQ? combining all modules into a single bytecode file

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

FAQ? combining all modules into a single bytecode file

Christian Lindig-3

I assume this is a FAQ but could not find anything with Google: I'd like to combine all *.lua files of an application into a single bytecode file (using luac) which I would pass to Lua at startup. I tried this but observing Lua on Linux with strace(1) revealed that all or most *.lua files were still loaded. I assumed that the order of the modules matters and tried the order as I observed it through strace(1) but this did not help. Is there a recipe for compiling everything into a single bytecode file?

The application I am looking to compile is the Nanoki wiki which is implemented as several modules which are loaded using "require" plus some C libraries. Obviously the C libraries would still be loaded from the bytecode. However, my question is completely independent of Nanoki.

-- Christian



Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Luiz Henrique de Figueiredo
> Is there a recipe for compiling everything into a single bytecode file?

"luac *.lua" does that but probably not in the order you need. Moreover,
if you use require in your modules, then you need to add a fake require
at the start.

Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Jerome Vuarand
In reply to this post by Christian Lindig-3
2008/7/31 Christian Lindig <[hidden email]>:
> I assume this is a FAQ but could not find anything with Google: I'd like to
> combine all *.lua files of an application into a single bytecode file (using
> luac) which I would pass to Lua at startup. I tried this but observing Lua
> on Linux with strace(1) revealed that all or most *.lua files were still
> loaded. I assumed that the order of the modules matters and tried the order
> as I observed it through strace(1) but this did not help. Is there a recipe
> for compiling everything into a single bytecode file?
>
> The application I am looking to compile is the Nanoki wiki which is
> implemented as several modules which are loaded using "require" plus some C
> libraries. Obviously the C libraries would still be loaded from the
> bytecode. However, my question is completely independent of Nanoki.

This thread in the mailing list archive solves a similar problem:

http://lua-users.org/lists/lua-l/2008-06/msg00326.html

You can try the following script to generate a bytecode file of all
the listed modules (see the beginning of the script). Then in your
final program, just call dofile("preload.lbc") before using (through
'require' as usual) any of your modules.

local modules = {
	"foo",
	"foo.bar",
	"foo.baz",
}

local preload = ""
for _,modname in ipairs(modules) do
	preload = preload .. 'function package.preload["'..modname..'"](...)' .. '\n'
	preload = preload .. io.open(modname:gsub("%.", "/")..".lua"):read"*a" .. '\n'
	preload = preload .. 'end' .. '\n'
end

local output = io.open("preload.lua", "w")
output:write(preload)
output:close()

os.execute("luac -o preload.lbc preload.lua")

Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Christian Lindig-3

On Jul 31, 2008, at 1:42 PM, Jerome Vuarand wrote:
You can try the following script to generate a bytecode file of all
the listed modules (see the beginning of the script). Then in your
final program, just call dofile("preload.lbc") before using (through
'require' as usual) any of your modules.

Thanks for this idea. Wouldn't it make sense to implement this in luac as an option? Today, luac compiles multiple modules by prepending code that calls each chunk (i.e. module). As a consequence, embedded 'requires' are executed before all modules are executed. Is this behavior still useful in the presence of 'module' and 'require'? I think it would be better to defer these calls using the construction you suggested: for each module, call a hook that either executes the module (same behavior as now) but also could be used to add the module to package.preload. In the latter case execution would be triggered by 'require', which would be after all modules are entered.

In general, I find compiling all Lua code into a single bytecode file very attractive for deployment. On a desktop system the bytecode file would be loaded by lua(1), in an embedded system the bytecode could be embedded as a string into a C application.

-- Christian



Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

David Manura
Christian Lindig writes:
> On Jul 31, 2008, at 1:42 PM, Jerome Vuarand wrote:
> > You can try the following script to generate a bytecode file of all
> > the listed modules (see the beginning of the script). Then in your
> > final program, just call dofile("preload.lbc") before using (through
> > 'require' as usual) any of your modules.
> 
> Thanks for this idea. Wouldn't it make sense to implement this in  
> luac as an option? ...

Check out Pre-Loader, which was announced on this list a few weeks ago:

  http://loop.luaforge.net/release/preload.html
  http://lua-users.org/lists/lua-l/2008-07/msg00195.html
  http://lua-users.org/wiki/BinToCee



Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Tony Finch
In reply to this post by Christian Lindig-3
On Thu, 31 Jul 2008, Christian Lindig wrote:
>
> Today, luac compiles multiple modules by prepending code that calls each
> chunk (i.e. module). As a consequence, embedded 'requires' are executed
> before all modules are executed. Is this behavior still useful in the
> presence of 'module' and 'require'?

It works OK for me if I present the modules to luac sorted so that all
require calls occur after the corresponding module call.

Tony.
-- 
f.anthony.n.finch  <[hidden email]>  http://dotat.at/
BISCAY: SOUTHWESTERLY 5 OR 6, OCCASIONALLY 7 LATER. ROUGH. RAIN OR SHOWERS.
MODERATE OR GOOD.

Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Jerome Vuarand
2008/8/1 Tony Finch <[hidden email]>:
> On Thu, 31 Jul 2008, Christian Lindig wrote:
>>
>> Today, luac compiles multiple modules by prepending code that calls each
>> chunk (i.e. module). As a consequence, embedded 'requires' are executed
>> before all modules are executed. Is this behavior still useful in the
>> presence of 'module' and 'require'?
>
> It works OK for me if I present the modules to luac sorted so that all
> require calls occur after the corresponding module call.

That's why it's easier to only preload the modules. You can preload
them in any order, the actual loading will happen when necessary, when
first 'require'd.

Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Petite Abeille
In reply to this post by Luiz Henrique de Figueiredo

On Jul 31, 2008, at 1:42 PM, Luiz Henrique de Figueiredo wrote:

Is there a recipe for compiling everything into a single bytecode file?

"luac *.lua" does that but probably not in the order you need. Moreover, if you use require in your modules, then you need to add a fake require
at the start.

Once one has sorted out the dependencies order, then luac does indeed do The Right Thing (SM) :)

In the case of Nanoki, this looks something like this:

% luac -o Nanoki.luac Cache.lua Config.lua DB.lua Data.lua Diff.lua File.lua Finder.lua HTTP.lua HTTPExtra.lua URL.lua IPLocation.lua MIME.lua NaturalComparator.lua TCPServer.lua Template.lua Token.lua Trie.lua Unidecode.lua Wiki.lua WikiAbout.lua WikiContent.lua WikiContentFileService.lua WikiService.lua WikiContentService.lua WikiControlService.lua WikiDAV.lua WikiDate.lua WikiDateService.lua WikiDateNavigation.lua WikiDiffService.lua WikiFeed.lua WikiFileService.lua WikiFinder.lua WikiIndex.lua WikiIndexService.lua WikiIndexNavigation.lua WikiMainService.lua WikiMessage.lua WikiPath.lua WikiPreview.lua WikiRecent.lua WikiRecentService.lua WikiRevisionService.lua WikiSearch.lua WikiEditorService.lua WikiSearchService.lua WikiVersionService.lua XML.lua base64.lua markdown.lua Nanoki.lua

Which results in one 'Nanoki.luac' file of around 468 KB.

One can then run this all inclusive file to start Nanoki:

% lua Nanoki.luac . 0 1080

Cheers,

--
PA.
http://alt.textdrive.com/nanoki/






Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

steve donovan
On Sun, Aug 3, 2008 at 9:04 PM, Petite Abeille <[hidden email]> wrote:
> Once one has sorted out the dependencies order, then luac does indeed do The
> Right Thing (SM) :)

And then you can use lhf's most excellent srlua:

http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/

This will take a script (or bytecode file) and embed it into a Lua interpreter.

steve d.

Reply | Threaded
Open this post in threaded view
|

Re: FAQ? combining all modules into a single bytecode file

Luiz Henrique de Figueiredo
In reply to this post by Christian Lindig-3
> Wouldn't it make sense to implement [support for require] in luac?

I've written a version of luac in Lua that suports require:
	http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/5.1/luac.lua

It understands command lines such as:
	lua luac.lua x.lua y.lua -L a.lua b.lua c.lua

This creates a precompiled chunk that preloads a.lua, b.lua, c.lua and
then runs x.lua and y.lua, which may require a, b, c.

The list of modules to run and the list of modules to preload are both
optional.

Since this is a Lua program, it can be modified to suit other needs,
such as returning all values returned by x.lua and y.lua, or sending all
run-time command line args to them, etc.

For added fun, the program has no comments :-) [but you may wish to
print the fake main program that it creates to see how it works]

All feedback welcome. Enjoy.
--lhf