Wanted: Lua source code cruncher

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Wanted: Lua source code cruncher

David Given
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(I asked about this a while ago, and there wasn't one, but given the new
interest in Lua metaprogramming I thought it was worth asking again.)

I've got this build tool of which you have heard me speak before, Prime
Mover. It ships a small but reasonably verbose Lua program as part of
its binary. As it makes no sense to ship comments in a binary, I'd
rather like to make this smaller.

With C, I can drastically reduce the size of the source code by using a
cruncher utility: this reads in the C source and emits an equivalent but
smaller source file by removing whitespace, comments, renaming locals to
smaller versions, etc.

Is there a Lua version of such a tool?

It's actually a reasonably simple problem; you just read the source file
into an AST, perform some simple manipulations (renaming locals), and
then you write it out again. If anyone has a Lua reader/writer utility,
it ought to be trivial to adapt it to do this. Does anyone?

- --
ââââ ïïïïïïïïïïïïïï âââââ http://www.cowlark.com âââââ
â "Wizards get cranky, / Dark days dawn, / Riders smell manky, / The
â road goes on. / Omens are lowering, / Elves go West; / The Shire needs
â scouring, / You may as well quest." - John M. Ford
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH0+CDf9E0noFvlzgRAm8jAKDJOC6Cv7enL6fa84NHt+LNVtGjPQCfRie0
2KBSes2tma+Zi8t4kNJ/CMA=
=MEJB
-----END PGP SIGNATURE-----


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Luiz Henrique de Figueiredo
> With C, I can drastically reduce the size of the source code by using a
> cruncher utility: this reads in the C source and emits an equivalent but
> smaller source file by removing whitespace, comments, renaming locals to
> smaller versions, etc.

My lstrip does all that, except rename locals; but it's in C.

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

David Given
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luiz Henrique de Figueiredo wrote:
>> With C, I can drastically reduce the size of the source code by using a
>> cruncher utility: this reads in the C source and emits an equivalent but
>> smaller source file by removing whitespace, comments, renaming locals to
>> smaller versions, etc.
> 
> My lstrip does all that, except rename locals; but it's in C.

Unfortunately lstrip isn't suitable for me, but looking it up on Google
led me to LuaSrcDiet, which does pretty much what I want (...except
locals); so thanks anyway.

Runing LuaSrcDiet on pm reduces it down to 58%, which isn't bad.
Unfortunately, once this is bolted on to the Lua interpreter source
itself, this turns into a real reduction of the resulting binary to 95%
- --- which is barely worth measuring.

Local renaming would improve that no end, as I tend to use fairly long
variable names, but I suspect what I really need is a better C cruncher.
It ought to be possible to Huffman encode all symbol names by creative
use of #define. Hmm...

- --
ââââ ïïïïïïïïïïïïïï âââââ http://www.cowlark.com âââââ
â "Wizards get cranky, / Dark days dawn, / Riders smell manky, / The
â road goes on. / Omens are lowering, / Elves go West; / The Shire needs
â scouring, / You may as well quest." - John M. Ford
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH1AA0f9E0noFvlzgRAsBtAKDezTZ46xIgrrfGfWGMEeYmGJD+1ACg1VKs
DG3a2dLis7SW/9eI/dbEavc=
=U4uS
-----END PGP SIGNATURE-----


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Chris Marrin

On Mar 9, 2008, at 8:20 AM, David Given wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luiz Henrique de Figueiredo wrote:
With C, I can drastically reduce the size of the source code by using a cruncher utility: this reads in the C source and emits an equivalent but smaller source file by removing whitespace, comments, renaming locals to
smaller versions, etc.

My lstrip does all that, except rename locals; but it's in C.

Unfortunately lstrip isn't suitable for me, but looking it up on Google
led me to LuaSrcDiet, which does pretty much what I want (...except
locals); so thanks anyway.

Runing LuaSrcDiet on pm reduces it down to 58%, which isn't bad.
Unfortunately, once this is bolted on to the Lua interpreter source
itself, this turns into a real reduction of the resulting binary to 95%
- --- which is barely worth measuring.

Local renaming would improve that no end, as I tend to use fairly long
variable names, but I suspect what I really need is a better C cruncher.
It ought to be possible to Huffman encode all symbol names by creative
use of #define. Hmm...


I'm not sure of the nature of your app, but maybe it would be easier to just compress your Lua source using gzip and then use gzio or something to uncompress it? Gzip does a great job of compressing things like long variable names which repeat throughout the text. You may still want to get rid of comments to get maximum compression, though.

-----
~Chris
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Louis Mamakos
In reply to this post by David Given
why not just use luac?  Perhaps that doesn't meet the original requirements somehow.



-----Original Message-----
From: Luiz Henrique de Figueiredo <[hidden email]>
Date: Sunday, Mar 9, 2008 10:08 am
Subject: Re: Wanted: Lua source code cruncher
To: Lua list <[hidden email]>Reply-To: Lua list <[hidden email]>

> With C, I can drastically reduce the size of the source code by using a
 cruncher utility: this reads in the C source and emits an equivalent but
 smaller source file by removing whitespace, comments, renaming locals to
 smaller versions, etc.

My lstrip does all that, except rename locals; but it's in C.




Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Ico Doornekamp

* On 2008-03-09 Louis Mamakos <[hidden email]> wrote  :

> > > With C, I can drastically reduce the size of the source code by using a
> > > cruncher utility: this reads in the C source and emits an equivalent but
> > > smaller source file by removing whitespace, comments, renaming locals to
> > > smaller versions, etc.
> > 
> > My lstrip does all that, except rename locals; but it's in C.
>
> why not just use luac?  Perhaps that doesn't meet the original requirements somehow.

I believe the goal of the original poster is to reduce code size. My
experience is that compiled lua is not always that much smaller then the
original source (about 85%), and somtimes even bigger. 

It is trivial to alter the lua sourcecode to allow transparent loading
of gzipped files by changing fopen/fread/getc into gzopen/gzread/gzetc
etc and linking to zlib. I have a patch for this lying around if anybody
is interested, this allows for reading of gzipped lua source- or bytecode.

-- 
:wq
^X^Cy^K^X^C^C^C^C

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

KHMan
In reply to this post by David Given
David Given wrote:
> Luiz Henrique de Figueiredo wrote:
>>> With C, I can drastically reduce the size of the source code by using a
>>> cruncher utility: this reads in the C source and emits an equivalent but
>>> smaller source file by removing whitespace, comments, renaming locals to
>>> smaller versions, etc.
>> My lstrip does all that, except rename locals; but it's in C.
> 
> Unfortunately lstrip isn't suitable for me, but looking it up on Google
> led me to LuaSrcDiet, which does pretty much what I want (...except
> locals); so thanks anyway.
> 
> Runing LuaSrcDiet on pm reduces it down to 58%, which isn't bad.
> Unfortunately, once this is bolted on to the Lua interpreter source
> itself, this turns into a real reduction of the resulting binary to 95%
> - --- which is barely worth measuring.

LuaSrcDiet just whacks whitespace and comments; it doesn't really
change anything else so you should get the same binary chunk. I
did LuaSrcDiet as a fun bit of scripting, so updating it is pretty
low-priority. It's true that the biggest win to implement would be
optimizing the local names.

> [snip]
> variable names, but I suspect what I really need is a better C cruncher.
> It ought to be possible to Huffman encode all symbol names by creative
> use of #define. Hmm...

-- 
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Rob Kendrick
In reply to this post by Louis Mamakos
On Sun, 2008-03-09 at 12:12 -0400, Louis Mamakos wrote:
> why not just use luac?  Perhaps that doesn't meet the original requirements somehow.

Indeed it does not - the output of luac is not portable between
machines.  What David is trying to do is produce something that builds
on as many platforms as possible while having as little overhead as can
be achieved.

B.


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

KHMan
In reply to this post by Chris Marrin
Chris Marrin wrote:
> 
> On Mar 9, 2008, at 8:20 AM, David Given wrote:
> [snip]
>>>> With C, I can drastically reduce the size of the source code by using a
>>>> cruncher utility: this reads in the C source and emits an equivalent
>>>> but
>>>> smaller source file by removing whitespace, comments, renaming
>>>> locals to
>>>> smaller versions, etc.
>>>
>>> My lstrip does all that, except rename locals; but it's in C.
>> [snip]
> 
> I'm not sure of the nature of your app, but maybe it would be easier to
> just compress your Lua source using gzip and then use gzio or something
> to uncompress it? Gzip does a great job of compressing things like long
> variable names which repeat throughout the text. You may still want to
> get rid of comments to get maximum compression, though.

For source code, comment and whitespace removal is a pretty big
win. To improve further by renaming locals will help less, but
AFAIK nobody's written a tool to do that yet. I think it is a
limited 'win'. To illustrate, say we have two versions:

A:	i = i + 1
	i = i + 1

B:	foobar = foobar + 1
	foobar = foobar + 1

B is longer by 20 bytes. For version A, 'i' will be one literal
code, the second 'i' will be one literal code, and the entire line
2 will be one match code. For version B, 'foobar' will be
literals, the 'foobar ' will be one match code, and line 2 will
still be one match code. Roughly, compressed B is longer than
compressed A by 4 literals and 1 match, say about 6 bytes in
total, less with compressed literals. I suspect the overall effect
of renaming locals for compressed source code is just a few percent.

zlib is great for general-purpose compression, as Ico has pointed
out. To get an equivalent extra few percent of savings, a solid
archive (concatenation of files, then compressed) will do much
better than separate files. Solid bzip2 or lzma will give even
better results. It is likely that the effect of renaming locals
will be smaller than say, the improvement due to solid archiving,
or switching to bzip2, etc. At this point, it is not a real
necessity and becomes more of a benchmarking exercise.

For binary chunks, all the size_t and integers will give a lot of
00 00 00 octets, so an LZ-based compressor might end up with lower
average match lengths and more matches, thus possibly poorer
compression. VM instructions, due to field alignment, also benefit
less from compression of their constituent octets.

So all of this becomes very subjective and very much dependent on
the characteristics of particular sources... For some old Yueliang
code, I did some testing long ago, and zip compression ratio is as
follows:

			Size	Zipped	Ratio
Original sources	130162	 29546	22.7%
with LuaSrcDiet		 48308	 13841	28.7%
luac			108174	 32238	29.8%
luac -s			 64930	 21867	33.7%

One can also tokenize the source like old Basic interpreters, but
again, that is more of an academic exercise, since we can easily
get good or better compression by choosing a better compressor.

-- 
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Chris Marrin

On Mar 9, 2008, at 12:10 PM, KHMan wrote:

Chris Marrin wrote:

On Mar 9, 2008, at 8:20 AM, David Given wrote:
[snip]
With C, I can drastically reduce the size of the source code by using a cruncher utility: this reads in the C source and emits an equivalent
but
smaller source file by removing whitespace, comments, renaming
locals to
smaller versions, etc.

My lstrip does all that, except rename locals; but it's in C.
[snip]

I'm not sure of the nature of your app, but maybe it would be easier to just compress your Lua source using gzip and then use gzio or something to uncompress it? Gzip does a great job of compressing things like long variable names which repeat throughout the text. You may still want to
get rid of comments to get maximum compression, though.

For source code, comment and whitespace removal is a pretty big
win. To improve further by renaming locals will help less, but
AFAIK nobody's written a tool to do that yet. I think it is a
limited 'win'. To illustrate, say we have two versions:

A:	i = i + 1
	i = i + 1

B:	foobar = foobar + 1
	foobar = foobar + 1

B is longer by 20 bytes. For version A, 'i' will be one literal
code, the second 'i' will be one literal code, and the entire line
2 will be one match code. For version B, 'foobar' will be
literals, the 'foobar ' will be one match code, and line 2 will
still be one match code. Roughly, compressed B is longer than
compressed A by 4 literals and 1 match, say about 6 bytes in
total, less with compressed literals. I suspect the overall effect
of renaming locals for compressed source code is just a few percent.


But it's worse than that. Changing variable names in a language like Lua is not really possible, because of runtime property lookup. Imagine this:

    local a = { }
    a.myLongPropertyName = 10
    local b = a["myLongPropertyName"]

That might not be the best style, but it is legal. And if you changed the property name, the above would fail, unless you changed the string to match. You MIGHT be able to get a preprocessor to catch the above case, but you couldn't catch them all. Consider this:

function getBase() return "my" end
...
local b = a[base().."LongPropertyName"]

finding all variants of the above is an impossible task.

-----
~Chris
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Luiz Henrique de Figueiredo
In reply to this post by KHMan
> It's true that the biggest win to implement would be optimizing the local names.

Not such a big win: local names are stored once once per scope.

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Miles Bader-2
In reply to this post by Chris Marrin
Chris Marrin <[hidden email]> writes:
> But it's worse than that. Changing variable names in a language like Lua
> is not really possible, because of runtime property lookup.

Er, they were talking about changing the names of _local_ variables,
which do not have the problem you describe...

-Miles

-- 
Christian, n. One who follows the teachings of Christ so long as they are not
inconsistent with a life of sin.

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Aaron Brown
In reply to this post by Luiz Henrique de Figueiredo
LHF wrote:

Not such a big win: local names are stored once once per
scope.

Not in source code, where a given local takes up the length
of its name in characters every time it's used.

--
Aaron
http://arundelo.com/


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Tony Finch
In reply to this post by David Given
On Sun, 9 Mar 2008, David Given wrote:
>
> I've got this build tool of which you have heard me speak before, Prime
> Mover. It ships a small but reasonably verbose Lua program as part of
> its binary. As it makes no sense to ship comments in a binary, I'd
> rather like to make this smaller.

Compile it with luac -s and compress the result. I've attached some
utility code for loading Lua chunks (source or compiled, since Lua doesn't
care) that have been compressed in zlib or gzip format. I also have a
little deflate utility for making zlib format files, which are a few bytes
shorter than gzip.

This is probably only a win if your app is dynamically linked to Lua,
since you'd need quite a lot of Lua code to get near the size of liblua.a
(200K on my machine).

Tony.
-- 
f.a.n.finch  <[hidden email]>  http://dotat.at/
LUNDY FASTNET: SOUTHWESTERLY 5 TO 7 INCREASING GALE 8 TO STORM 10 THEN VEERING
NORTHWESTERLY, OCCASIONALLY VIOLENT STORM 11 IN FASTNET, PERHAPS VIOLENT STORM
11 LATER IN LUNDY. ROUGH OR VERY ROUGH BECOMING HIGH OR VERY HIGH,
OCCASIONALLY PHENOMENAL LATER IN FASTNET. SHOWERS. GOOD BECOMING POOR.
/*
 * Load Lua chunks in gzip or zlib format.
 *
 * Written by Tony Finch <[hidden email]> <[hidden email]>
 * at the University of Cambridge Computing Service.
 * You may do anything with this, at your own risk.
 *
 * $Cambridge: users/fanf2/sdist/lunzip.h,v 1.3 2008/03/05 11:52:06 fanf2 Exp $
 */

/*
 * A lua_Reader for loading a Lua chunk from a buffer.
 *	void *lunbuf = lunbuf_data(buffer, size);
 *	ret = lua_load(L, lunbuf_reader, lunbuf, name);
 *	lunbuf_free(lunbuf);
 * is equivalent to
 *	luaL_loadbuffer(L, buffer, size, name);
 * is equivalent to
 *	lunbuf_load(L, buffer, size, name);
 */
void *lunbuf_data(const char *buffer, size_t size);
const char *lunbuf_reader(lua_State *L, void *opaque, size_t *size);
void lunbuf_free(void *opaque);

/*
 * A lua_Reader for loading a compressed Lua chunk. It acts as a
 * wrapper that inflates the data returned by another lua_Reader.
 * For example, see the source of lunzip_load().
 */
void *lunzip_data(lua_Reader reader, void *opaque);
const char *lunzip_reader(lua_State *L, void *opaque, size_t *size);
void lunzip_free(void *opaque);

/*
 * Wrappers in the manner of the luaL_load...() functions.
 */
int lunbuf_load(lua_State *L,
    const char *buffer, size_t size, const char *name);
int lunzip_load(lua_State *L,
    lua_Reader reader, void *opaque, const char *name);
int lunzip_loadbuffer(lua_State *L,
    const char *buffer, size_t size, const char *name);

/* eof */
/*
 * Load Lua chunks in gzip or zlib format.
 *
 * Written by Tony Finch <[hidden email]> <[hidden email]>
 * at the University of Cambridge Computing Service.
 * You may do anything with this, at your own risk.
 *
 * $Cambridge: users/fanf2/sdist/lunzip.c,v 1.3 2008/03/05 11:52:06 fanf2 Exp $
 */

#include <stdlib.h>
#include <zlib.h>
#include "lua.h"
#include "lunzip.h"

#define BUFFER 4096

/* lunbuf */

typedef struct lunbuf_state {
	const char *buffer;
	size_t size;
} lunbuf_state;

void *
lunbuf_data(const char *buffer, size_t size) {
	lunbuf_state *state = malloc(sizeof(*state));
	if(state == NULL) return(NULL);
	state->buffer = buffer;
	state->size = size;
	return(state);
}

const char *
lunbuf_reader(lua_State *L, void *opaque, size_t *size) {
	lunbuf_state *state = opaque;
	L = L;
	if(state->size == 0) return(NULL);
	*size = state->size;
	state->size = 0;
	return(state->buffer);
}

void
lunbuf_free(void *opaque) {
	free(opaque);
}

/* lunzip */

typedef struct lunzip_state {
	lua_Reader reader;
	void *opaque;
	z_stream z;
	union {
		unsigned char b[BUFFER];
		const char c[BUFFER];
	} buffer;
} lunzip_state;

void *
lunzip_data(lua_Reader reader, void *opaque) {
	int ret;
	lunzip_state *state = malloc(sizeof(*state));
	if(state == NULL) return(NULL);
	state->reader = reader;
	state->opaque = opaque;
	state->z.zalloc = NULL;
	state->z.zfree = NULL;
	state->z.opaque = NULL;
	state->z.next_in = NULL;
	state->z.avail_in = 0;
	ret = inflateInit2(&state->z, 47);
	if(ret == Z_OK) return(state);
	if(ret != Z_MEM_ERROR) abort(); /* programming error */
	free(state);
	return(NULL);
}

const char *
lunzip_reader(lua_State *L, void *opaque, size_t *size) {
	lunzip_state *state = opaque;
	int ret;
	if(state->z.avail_in == 0)
		state->z.next_in = state->reader(L, state->opaque,
		    &state->z.avail_in);
	if(state->z.next_in == NULL)
		state->z.avail_in = 0;
	state->z.next_out = state->buffer.b;
	state->z.avail_out = sizeof(state->buffer);
	ret = inflate(&state->z, Z_SYNC_FLUSH);
	*size = state->z.next_out - state->buffer.b;
	if(*size != 0) return(state->buffer.c);
	return(NULL);
}

void
lunzip_free(void *opaque) {
	lunzip_state *state = opaque;
	if(state == NULL) return;
	inflateEnd(&state->z);
	free(state);
}

/* canned loaders */

int
lunbuf_load(lua_State *L,
    const char *buffer, size_t size, const char *name) {
	int ret;
	void *lunbuf = lunbuf_data(buffer, size);
	if(lunbuf == NULL) return(LUA_ERRMEM);
	ret = lua_load(L, lunbuf_reader, lunbuf, name);
	lunbuf_free(lunbuf);
	return(ret);
}

int
lunzip_load(lua_State *L,
    lua_Reader reader, void *opaque, const char *name) {
	int ret;
	void *lunzip = lunzip_data(reader, opaque);
	if(lunzip == NULL) return(LUA_ERRMEM);
	ret = lua_load(L, lunzip_reader, lunzip, name);
	lunzip_free(lunzip);
	return(ret);
}

int
lunzip_loadbuffer(lua_State *L,
    const char *buffer, size_t size, const char *name) {
	int ret;
	void *lunbuf = lunbuf_data(buffer, size);
	if(lunbuf == NULL) return(LUA_ERRMEM);
	ret = lunzip_load(L, lunbuf_reader, lunbuf, name);
	lunbuf_free(lunbuf);
	return(ret);
}

/* eof */
/*
 * Compress a stream in zlib format (less rubric than gzip).
 *
 * Written by Tony Finch <[hidden email]> <[hidden email]>
 * at the University of Cambridge Computing Service.
 * You may do anything with this, at your own risk.
 *
 * $Cambridge: users/fanf2/sdist/deflate.c,v 1.1 2008/03/04 22:33:07 fanf2 Exp $
 */

#include <err.h>
#include <stdio.h>
#include <zlib.h>

#define BUFFER 4096

int main(void) {
	unsigned char in[BUFFER], out[BUFFER];
	unsigned char *inp = in;
	z_stream z;
	size_t n;
	int mode, ret;

	z.zalloc = NULL;
	z.zfree = NULL;
	z.opaque = NULL;
	ret = deflateInit(&z, 9);
	if(ret != Z_OK) errx(1, "init: %s (%d)", z.msg, ret);
	mode = Z_NO_FLUSH;
	z.avail_in = 0;
	for(;;) {
		if(z.avail_in == 0) z.next_in = inp = in;
		n = fread(inp, 1, sizeof(in)-(inp-in), stdin);
		if(n == 0 && feof(stdin)) mode = Z_FINISH;
		if(n == 0 && ferror(stdin)) err(1, "read");
		inp += n;
		z.avail_in += n;
		z.next_out = out;
		z.avail_out = sizeof(out);
		ret = deflate(&z, mode);
		if(ret != Z_OK && ret != Z_STREAM_END)
			errx(1, "decompress: %s (%d)", z.msg, ret);
		n = z.next_out - out;
		if(fwrite(out, 1, n, stdout) < n) err(1, "write");
		if(ret == Z_STREAM_END) break;
	}
	ret = deflateEnd(&z);
	if(ret != Z_OK) errx(1, "finish: %s (%d)", z.msg, ret);
	return(0);
}

/* eof */
/*
 * Convert a file to a C array so that it can be embedded in a program.
 *
 * Written by Tony Finch <[hidden email]> <[hidden email]>
 * at the University of Cambridge Computing Service.
 * You may do anything with this, at your own risk.
 *
 * $Cambridge: users/fanf2/sdist/data2c.c,v 1.4 2007/10/27 12:55:52 fanf2 Exp $
 */

#include <stdio.h>
#include <string.h>

int
main(int argc, char *argv[])
{
	if(argc != 2) {
		fprintf(stderr, "usage: data2c arrayname <data >data.c\n");
		return(1);
	}
	printf("/* auto-generated by data2c */\n");
	printf("static const unsigned %s[] = {\n", argv[1]);
	for(;;) {
		unsigned line[8], count;
		memset(line, 0, sizeof(line));
		count = fread(line, 1, sizeof(line), stdin);
		for(unsigned i = 0; sizeof(*line) * i < count; i++)
			printf(" %u,", line[i]);
		if(count < sizeof(line))
			break;
		else
			printf("\n");
	}
	if(ferror(stdin)) {
		perror("error reading input");
		return(1);
	} else {
		printf("\n};\n");
		return(0);
	}
}

/* eof */
Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Luiz Henrique de Figueiredo
In reply to this post by Aaron Brown
> > Not such a big win: local names are stored once once per scope.
> 
> Not in source code, where a given local takes up the length
> of its name in characters every time it's used.

You're right, of course.  I was thinking about precompiled programs.
Sorry for the noise.

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

KHMan
In reply to this post by Tony Finch
Tony Finch wrote:
> On Sun, 9 Mar 2008, David Given wrote:
>> I've got this build tool of which you have heard me speak before, Prime
>> Mover. It ships a small but reasonably verbose Lua program as part of
>> its binary. As it makes no sense to ship comments in a binary, I'd
>> rather like to make this smaller.
> 
> Compile it with luac -s and compress the result. I've attached some
> utility code for loading Lua chunks (source or compiled, since Lua doesn't
> care) that have been compressed in zlib or gzip format. I also have a
> little deflate utility for making zlib format files, which are a few bytes
> shorter than gzip.
> 
> This is probably only a win if your app is dynamically linked to Lua,
> since you'd need quite a lot of Lua code to get near the size of liblua.a
> (200K on my machine).

For completeness, yet another alternative that is simpler for
self-contained executables (more popular on Win32) is embedding
source or binary chunks and compressing it. With UPX, UCL-max is
about gzip-6, plus new versions of UPX has LZMA, suitable for
contemporary desktop machines. For the developer, no extra code
needs to be written.

-- 
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Rob Kendrick
On Mon, 2008-03-10 at 11:02 +0800, KHMan wrote:

> For completeness, yet another alternative that is simpler for
> self-contained executables (more popular on Win32) is embedding
> source or binary chunks and compressing it. With UPX, UCL-max is
> about gzip-6, plus new versions of UPX has LZMA, suitable for
> contemporary desktop machines. For the developer, no extra code
> needs to be written.

Indeed - I've tended to luac, and then gzip the program to embed it in
the rest of my application: although gzipping compressed source is
generally smaller still.

Of course, this doesn't help with David's problem; he needs it to be
portable between machines, so executables are out of the question, and
one can't reliably use gzip either, because it might not be installed.

B.


Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Asko Kauppi
In reply to this post by Aaron Brown

If someone really wants to do this; to replace local names by, say, 'a0', 'a1', and so on, it is not at all difficult.

About 20 lines of a luaSub filter will do it. Personally, I don't see the point in doing so.

-asko


Aaron Brown kirjoitti 9.3.2008 kello 23:22:

LHF wrote:

Not such a big win: local names are stored once once per
scope.

Not in source code, where a given local takes up the length
of its name in characters every time it's used.

--
Aaron
http://arundelo.com/



Reply | Threaded
Open this post in threaded view
|

Re: Wanted: Lua source code cruncher

Rob Kendrick
On Mon, 2008-03-10 at 12:50 +0200, Asko Kauppi wrote:
> If someone really wants to do this; to replace local names by, say,  
> 'a0', 'a1', and so on, it is not at all difficult.

The difficult part comes in selecting variable names that do not clash
with any globals the program may create and access at runtime.  I'm not
entirely sure the problem is solvable.

> About 20 lines of a luaSub filter will do it.  Personally, I don't see  
> the point in doing so.

It's advantageous if you're trying to make your source as small as
possible, which is handy for all sorts of applications.

B.



Reply | Threaded
Open this post in threaded view
|

RE: Wanted: Lua source code cruncher

Jerome Vuarand-2
In reply to this post by David Given
David Given wrote:
> Local renaming would improve that no end, as I tend to use fairly
> long variable names, but I suspect what I really need is a better C
> cruncher.  

If you try a local variable renaming method, keep in mind that you can
use all characters considered as alphabetic in the current locale, which
can be much more than 26 lowercase and 26 uppercase latin letters.
However, I don't know if there exists such an extended locale which is
as portable as you want PM to be.


12