luac - output into ONE bytecode file?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

luac - output into ONE bytecode file?

Marius Kjeldahl
First of all, congrats to the designers and implementors of lua. I've been looking for a language where I can write portable code for various embedded platforms, and so far lua is by far the best language I have found for this. Compared to what is being offered on most handhelds/cellulars, lua looks like a "real" programming language, and the table abstraction looks usable for 90% of what I need (and the prerelease of the lua book showed me how to implement a simple queue in a few lines of lua). With some luck, lua _might_ actually be able to get the job done where Java has failed so far (handhelds).

Anyway, I'm targeting handhelds for one of my projects (removing the parser, executing raw bytecodes), and having to use files is of course difficult to do "cross-platform" on handhelds. While I stuff all my data inside strings, I have not found an elegant way to output single bytecode files. I'm using dofile("module1"), dofile("module2") etc in my main file, and currently these are being compiled into their own bytecode files which the main program will read and execute when run.

To avoid having multiple bytecode files (and thereby a distribution problem) I guess I can stuff all the source into one source file (replacing the dofile("module") stuff with the actual source for each module) and compile this one file, but ideally there should be an easier way to do this. Having an option to feed luac for instance that says "stuff it ALL into one file" would help a lot.

Before I possibly embark on this project myself, I though I would ask in case there is an easy way to do this today, or somebody else has already done this.

Thanks,

Marius


Reply | Threaded
Open this post in threaded view
|

RE: luac - output into ONE bytecode file?

Ashley Fryer-2

> -----Original Message-----
> From: Marius Kjeldahl
> Sent: Monday, April 16, 2001 3:06 PM

> Having an option to feed luac for instance that says
> "stuff it ALL into one file" would help a lot.

Why not just concatenate your source text files into a single source text
file, and then run luac over that?  That should be trivial to automate with
a shell script.

Regards,
ashley




Reply | Threaded
Open this post in threaded view
|

RE: luac - output into ONE bytecode file?

Luiz Henrique de Figueiredo
In reply to this post by Marius Kjeldahl
>> Having an option to feed luac for instance that says
>> "stuff it ALL into one file" would help a lot.
>
>Why not just concatenate your source text files into a single source text
>file, and then run luac over that?

luac already accepts several inputs files and outputs a single bytecode file
whose execution is equivalent to the execution of the input files.
That's the default behavior. Or are the usage message and man page unclear??
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: luac - output into ONE bytecode file?

Marius Kjeldahl
In reply to this post by Marius Kjeldahl
Luiz Henrique de Figueiredo wrote:

And it seems if it encounters a
dofile("filename") and filename is already bytecompiled in the same
file, it does not try to access any external file.


No, this is not true, but you may redefine "dofile" so that it is.
See a message about this posted to lua-l last week.
--lhf

That is strange. In my main source file I have:

dofile("moduleX")
dofile("moduleY")

function abc ()

end

and when I bytecompile the main source file together with moduleX and Y and move the bytecompile'd file together with the host executable to another machine which ONLY has the one bytecompiled file, everything seems to work fine.

I guess it might work for the dofile's at the "top level" then, but possible not for others (which I am not using at the moment). Do you think this is correct?

Marius


Reply | Threaded
Open this post in threaded view
|

A fast array implementation

Joshua Jensen
Has anyone done a non-hash array implementation for the Lua 4.0 compiler?
Something on the order of (I'm making up syntax):

array myArray[10]  -- All 10 elements set to nil by default
myArray[10] = 5      -- Array index 10 becomes the number 5.

I know that using the [] for array access would be ambiguous... is it a
table or an array?  But for the sake of this example, please bear with me.

To avoid initializing all 10 elements to nil first and then filling them in,
we could:

array myArray[10] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }

Why do this at all when the tables handle it fine?  It's all about speed.
Lua is becoming more and more popular in the gaming arena, and I know I look
for every optimization strategy I can come up with for it (because I want to
use Lua EVERYWHERE!!! :)  But really, it seems like an array implementation
would be a great addition to Lua.

I'm curious to hear other people's insights.

Thanks,
Joshua Jensen
Author, Workspace Whiz - A Visual Studio Add-in
http://workspacewhiz.com/


Reply | Threaded
Open this post in threaded view
|

RE: luac - output into ONE bytecode file?

Samson-3
In reply to this post by Marius Kjeldahl
Chunks are fairly simple, can't you just concatenate the binary files
together. As long as there are no symbol clashes it should be ok.

-----Original Message-----
From: Luiz Henrique de Figueiredo [[hidden email]]
Sent: Monday, April 16, 2001 11:54 PM
To: Multiple recipients of list
Subject: RE: luac - output into ONE bytecode file?


>> Having an option to feed luac for instance that says
>> "stuff it ALL into one file" would help a lot.
>
>Why not just concatenate your source text files into a single source text
>file, and then run luac over that?

luac already accepts several inputs files and outputs a single bytecode file
whose execution is equivalent to the execution of the input files.
That's the default behavior. Or are the usage message and man page unclear??
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: luac - output into ONE bytecode file?

Luiz Henrique de Figueiredo
In reply to this post by Marius Kjeldahl
>That is strange. In my main source file I have:
>
>dofile("moduleX")
>dofile("moduleY")
>
>function abc ()
>
>end
>
>and when I bytecompile the main source file together with moduleX and Y 
>and move the bytecompile'd file together with the host executable to 
>another machine which ONLY has the one bytecompiled file, everything 
>seems to work fine.

I think it's just because you're not testing whether dofile succeeds (it will
fail in your case...).
--lhf

Reply | Threaded
Open this post in threaded view
|

RE: luac - output into ONE bytecode file?

Luiz Henrique de Figueiredo
In reply to this post by Marius Kjeldahl
>Chunks are fairly simple, can't you just concatenate the binary files
>together. As long as there are no symbol clashes it should be ok.

This was true in 3.2, but starting in 4.0, binary files can have only one
precompiled chunk. But this does not mean that a binary file cannot combine
several files; luac does exactly that: if you do "luac a b c d" then luac
generates a fake main program that does the equivalent of "dofile'a'; dofile'b'
dofile'c'; dofile'd'. Try luac -l.
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: A fast array implementation

Ben Cooley
In reply to this post by Joshua Jensen
> Has anyone done a non-hash array implementation for the Lua 4.0 compiler?
> Something on the order of (I'm making up syntax):
>
> array myArray[10]  -- All 10 elements set to nil by default
> myArray[10] = 5      -- Array index 10 becomes the number 5.
>
> I know that using the [] for array access would be ambiguous... is it a
> table or an array?  But for the sake of this example, please bear with me.

If you look at the table implementation in lua, you'll be pleasantly
suprised to discover that for numeric indexes, it actually is an array (it
doesn't use hash values because it can't).  It's really amazingly clever.

What lua is really lacking is a sorted dictionary or tree.  It's really
impossible to implement any sort of sorted table.




Reply | Threaded
Open this post in threaded view
|

Re: A fast array implementation

Joshua Jensen
From: "Ben Cooley" <[hidden email]>
Sent: Wednesday, April 18, 2001 2:03 AM
> > Has anyone done a non-hash array implementation for the Lua 4.0
compiler?
> > Something on the order of (I'm making up syntax):
> >
> > array myArray[10]  -- All 10 elements set to nil by default
> > myArray[10] = 5      -- Array index 10 becomes the number 5.
> >
> > I know that using the [] for array access would be ambiguous... is it a
> > table or an array?  But for the sake of this example, please bear with
me.
>
> If you look at the table implementation in lua, you'll be pleasantly
> suprised to discover that for numeric indexes, it actually is an array (it
> doesn't use hash values because it can't).  It's really amazingly clever.

I did look at the table implementation in Lua.  lua_rawgeti() calls
luaH_getnum().  luaH_getnum() sure looks like it accesses a hash table to
me, complete with a hash on the index being requested and a run of the
bucket's linked list checking for a number type and then checking if the key
matches...

Josh
----------
Author, Workspace Whiz! - A Visual Studio Add-in
http://workspacewhiz.com/



Reply | Threaded
Open this post in threaded view
|

Re: A fast array implementation

Edgar Toernig
Joshua Jensen wrote:
> 
> I did look at the table implementation in Lua.  lua_rawgeti() calls
> luaH_getnum().  luaH_getnum() sure looks like it accesses a hash table to
> me, complete with a hash on the index being requested and a run of the
> bucket's linked list checking for a number type and then checking if the key
> matches...

Just look a little bit closer.  If the table only holds integer valued keys
from a to b (without missing elements) there are _no_ collisions.  So the
loop in luaH_getnum is never taken.  The hash has become an array.  Over-
writing already present keys is similar fast.  The only slower case is
growing the table (rehash log2(n) times).  But similar work would be necessary
for dynamic arrays too.

IMHO the _only_ benefit of arrays would be that there's no need to store the
keys.  They would take only half the memory...

Ciao, ET.


Reply | Threaded
Open this post in threaded view
|

Re: A fast array implementation

Joshua Jensen
From: "Edgar Toernig" <[hidden email]>
> Joshua Jensen wrote:
> >
> > I did look at the table implementation in Lua.  lua_rawgeti() calls
> > luaH_getnum().  luaH_getnum() sure looks like it accesses a hash table
to
> > me, complete with a hash on the index being requested and a run of the
> > bucket's linked list checking for a number type and then checking if the
key
> > matches...
>
> Just look a little bit closer.  If the table only holds integer valued
keys
> from a to b (without missing elements) there are _no_ collisions.  So the
> loop in luaH_getnum is never taken.  The hash has become an array.  Over-
> writing already present keys is similar fast.  The only slower case is
> growing the table (rehash log2(n) times).  But similar work would be
necessary
> for dynamic arrays too.
>
> IMHO the _only_ benefit of arrays would be that there's no need to store
the
> keys.  They would take only half the memory...

I strongly disagree here.  While I can't fully confirm your "no collisions"
response, I can confirm there are no collisions for small arrays.  And even
so, a straight retrieval of a value at a certain index still requires
multiple function calls, comparisons, etc.  An insertion is even more
costly.

By providing an array type, Lua can preallocate an array of TObjects (or
whatever they are) based on the size specified by the user.  To insert an
object into that array is as simple as array[index] = TObject.  Retrieval is
equally as simple.  Lua will run far quicker when using indexed access and
make it perform even better in the benchmarks.

Thanks,
Josh
Author, Workspace Whiz! - A Visual Studio Add-in
http://workspacewhiz.com/



Reply | Threaded
Open this post in threaded view
|

Re: A fast array implementation

Russell Y. Webb
> By providing an array type, Lua can preallocate an array of TObjects (or
> whatever they are) based on the size specified by the user.  To insert an
> object into that array is as simple as array[index] = TObject.  Retrieval is
> equally as simple.  Lua will run far quicker when using indexed access and
> make it perform even better in the benchmarks.

Can't an array type be defined from C?  It might not have all the best
syntactic integration with lua, but having a arraylib.c to add to lua would
be great and if everyone used it then it could become standard.  I'd use it
in some cases, but don't see any reason to go adding it to the standard
language now.

Russ


Reply | Threaded
Open this post in threaded view
|

Re: A fast array implementation

Joshua Jensen
For some reason this didn't make it to the list earlier...

From: "Russell Y. Webb" <[hidden email]>
> > By providing an array type, Lua can preallocate an array of TObjects (or
> > whatever they are) based on the size specified by the user.  To insert
an
> > object into that array is as simple as array[index] = TObject.
Retrieval is
> > equally as simple.  Lua will run far quicker when using indexed access
and
> > make it perform even better in the benchmarks.
>
> Can't an array type be defined from C?  It might not have all the best
> syntactic integration with lua, but having a arraylib.c to add to lua
would
> be great and if everyone used it then it could become standard.  I'd use
it
> in some cases, but don't see any reason to go adding it to the standard
> language now.

Yes, an array type could be defined from C.  To access it, as you
mentioned, would involve C callbacks.  I would wager the overhead for
making the call to the C callback would be larger than the overhead of
the existing hashing mechanism.

A better solution would be actual VM instructions... ARRAYCREATE,
ARRAYGET, and ARRAYSET should be fine.  ARRAYCREATE would create a fixed
size array, based on the size pushed to the stack.  ARRAYGET/SET would
simply take the array pointer and index on the stack and retrieve/set
the value.  Shouldn't be too much slower than C.

A mechanism like this would be vastly superior in speed to a table for
array-style access.

Thanks,
Josh