Bundling Lua code in your (Mach-O) executable.

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Bundling Lua code in your (Mach-O) executable.

David Jones-2
I recently did some work on Lua MIDI, http://sourceforge.net/projects/ luamidi/ , and part of work was bundling the Lua code directly into the executable.

Part of the build process for Lua MIDI builds an executable and stitches a .lua file into the executable so that the executable is standalone. Part of the application is written in Lua but the .lua files do not need to copied separately; the entire application can be distributed as a single binary file.

There are plenty of ways of doing this (for example translate the .lua files into .c files that have the contents in a C string), I think mostly the reason why I chose this way is that I happened to be reading about the Mach-O loader whilst I was needing this feature.

Because I use the Mach-O loader interface this technique only works on OS X; though the general principles will be similar on other operating systems, most of the work is in the platform-specific details. Since the work was non-trivial I thought I'd jot down a few notes in case anyone else wanted to do something similar

The technique works by creating a Mach-O Segment Searcher, MOSS. MOSS has the following features:

- Include one extra C file, the MOSS implementation, and call one function to enable MOSS in your application.
- require'foo' will search for foo in the Mach-O executable.
- Embed Lua code in your Mach-O executable at link time by adding a few options ("just link-and-go"). - External foo.lua file will override internal foo, easing the editing and development of pure Lua modules whilst the application is running (no relink required).

Anyone who wants the gory details (and code!) i urged to look at code/ lmoss.h and code/lmoss.c in the recent Lua MIDI release: http:// sourceforge.net/project/showfiles.php?group_id=131184 . I didn't really think it was worth unbundling.

moss_loader is the function installed in the package.loaders array. It allows modules to be found in the Mach-O executable. moss_loader works by grovelling over the structures in the Mach-O executable header, using the not-very-well-documented structures in <mach-o/ loader.h>. Here's a typical extract of the code (this finds the command segment called "Lua", it is used when MOSS is initialised):

  const struct load_command *command;
  const struct segment_command *segment;
  int i;

  command = (void *)(moss_header + 1);
  for(i=0; i<moss_header->ncmds; ++i) {
    if(command->cmd != LC_SEGMENT) {
      goto nextCommand;
    }
    segment = (void *)command;
    if(memcmp(segment->segname, "Lua", 4) != 0) {
      goto nextCommand;
    }
    /* Found a segment named "Lua". */
    moss_section = (void *)(segment + 1);
    moss_segment = segment;
    return;

nextCommand:
    command = (void *)(((char *)command) + command->cmdsize);
  }

Later, when require is called, the moss_loader function will search this segment for a section of the required name.

Conveniently the Mach-O file structure allows for a named segment to contain several named sections. So each Lua file can go in a section of its own name (foo.lua goes in section foo, say), and all those sections go in a single segment called "Lua".

Here's how you include a file in a named segment at link time (further reading in ld(1)):

gcc -o lua lua.o liblua.a -lm -arch ppc -arch i386 -g -sectcreate Lua lmidi ../../code/lmidi.lua ../../code/lmidi.o ../../code/lmoss.o - lreadline -framework CoreMIDI -framework CoreFoundation -framework CoreAudio -framework Carbon

Notice the -sectcreate option, "-sectcreate Lua lmidi ../../code/ lmidi.lua" includes the contents of the file "lmidi.lua" in the section named "lmidi" in the segment named "Lua". MOSS can now find this Lua file if you use "require'lmidi'".

The "just link-and-go" feature means that in principle an entire application consisting of Lua and C code can be built with one invocation of the compiler:

cc -sectcreate Lua foo foo.lua [repeat-for-each-Lua file] *.c

Cheers,
 drj

Reply | Threaded
Open this post in threaded view
|

Re: Bundling Lua code in your (ELF) executable.

Tony Finch
On Mon, 3 Mar 2008, David Jones wrote:
>
> There are plenty of ways of doing this (for example translate the .lua files
> into .c files that have the contents in a C string), I think mostly the reason
> why I chose this way is that I happened to be reading about the Mach-O loader
> whilst I was needing this feature.
>
> Because I use the Mach-O loader interface this technique only works on OS X;
> though the general principles will be similar on other operating systems, most
> of the work is in the platform-specific details.  Since the work was
> non-trivial I thought I'd jot down a few notes in case anyone else wanted to
> do something similar

Interesting. The easy linking is really nice.

On an ELF system with gcc and GNU binutils, you can turn a (source or
compiled) lua script into a linkable object as follows. The key magic is
the -b option which specifies the input file format, in this case, any old
binary data.

	ld -b binary -r -o MYSCRIPT.o MYSCRIPT.lua

When you link your program you must specify -export-dynamic so that the
binary file's symbols are available to dynamic objects, including the
run-time linker.

	gcc -export-dynamic -o MYPROG MYPROG.c MYSCRIPT.o

Then MYPROG can find the location of MYSCRIPT in memory using dlsym().

	void *start = dlsym(NULL, "_binary_MYSCRIPT_lua_start");
	void *end = dlsym(NULL, "_binary_MYSCRIPT_lua_end");
	luaL_loadbuffer(L, start, end-start, "MYSCRIPT");

You can also turn MYSCRIPT into a shared object using:

	ld -shared -o MYSCRIPT.so MYSCRIPT.o

Then MYPROG can use dlopen() and pass the resulting handle into the
dlsym() calls in the loadbuffer() code, or you can use the LD_PRELOAD
environment variable to link MYPROG and MYSCRIPT together at run-time.

Tony.
-- 
f.a.n.finch  <[hidden email]>  http://dotat.at/
FISHER: NORTHERLY 5 OR 6, INCREASING 6 TO GALE 8. ROUGH BECOMING VERY ROUGH.
WINTRY SHOWERS. MODERATE OR GOOD.