FW: srlua extension to run with glued zip file

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

FW: srlua extension to run with glued zip file

Frank Bechmann
Sorry for the empty 1st post.

I've now a basic extension to srlua running which allows you to combine multiple (lua-) files in one zip file and than start it as a standalone executable.

I've included the source, a short readme, and a test configuration in the attached zip file. A more detailed usage info and info about the current limitations is given in  a comment in the source file srluaz.c at the top of the main() function. You will also need srlua from http://luaforge.net/projects/srlua/ to get it running (I use SCons for compilation, if you do the same I can send you the SConstruct file).

Note that it's not thoroughly tested and not tested under any Unix at all. Also, since this is my 1st C program for years be mercyful.

Any feedback welcome.

Regards, 
   Frank

________________________________________________________________
Verschicken Sie romantische, coole und witzige Bilder per SMS!
Jetzt neu bei WEB.DE FreeMail: http://freemail.web.de/?mc=021193

Attachment: mysrlua.zip
Description: Zip compressed data

Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

duck
>I've now a basic extension to srlua running which allows you to combine 
>multiple (lua-) files in one zip file and than start it as a standalone 
>executable.

Great idea (if the amount the Lua code -- and you can precompile it, of 
course, to save space -- compresses is larger than the size of the 
decompressor...).

Another way to do this on Windows would be to embed the "glued" codes
as resource entries (so they are inside the EXE proper) and then to UPX 
the whole file. (UPX is a run-time in-memory unpackaging system. Highly 
recommended to impress your friends when showing them how small your
Lua run-time is compared to their Perl or Python distro. The UPX 
decompressor is very small.)

Has anyone tried using UPX for this (and is there an analogous way to 
embed binary data into ELF binaries so that the original executable is 
unaffected but the embedded data in technically "inside" as far as 
exectuable packers are concerned)?


Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Steve Donovan-2
In reply to this post by Frank Bechmann
>>> [hidden email] 01/05/05 01:22PM >>>
Has anyone tried using UPX for this (and is there an analogous way to 
embed binary data into ELF binaries so that the original executable is

unaffected but the embedded data in technically "inside" as far as 
exectuable packers are concerned)?

Luiz did something exactly like this, based on reslib:  I quote:

"I've finished the first version of my library for reading resources
from
executable programs. Get it at
	http://www.tecgraf.puc-rio.br/~lhf/tmp/ear.tar.gz 

Just open it and type make. It should show you the list of resources
(files)
found inside the a.out that it runs."

steve d.



Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Luiz Henrique de Figueiredo
> is there an analogous way to embed binary data into ELF binaries so
> that the original executable is unaffected but the embedded data in
> technically "inside" as far as exectuable packers are concerned)?

Yes, ELF support PROGBITS sections for that. It's pretty simple to use,
once you know the correct magic, of course.
 
> Luiz did something exactly like this, based on reslib:  I quote:

Yes, but I've given up this ide for now because it wouldn't work for Windows
and because it's ok to glue arbitrary data to executables. (Would UPX choke
on these?) Hence, srlua is as simple as you get.

So, the tarball mentioned in the previous post is no longer there.

However, I've not given up the idea of embedding several files in an
executable as a virtual read-only filesystem, complete with directory trees.
I've found that the CPIO format is a simple as you get for this task. And it's
seekable so you don't need to expand the archive into a temporary directory.
 
If there's interest, I can get back to this.
--lhf

Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Frank Bechmann
In reply to this post by Frank Bechmann
> 
> However, I've not given up the idea of embedding several files in an
> executable as a virtual read-only filesystem, complete with directory trees.
> I've found that the CPIO format is a simple as you get for this task. And it's
> seekable so you don't need to expand the archive into a temporary directory.
>  

I also planned to use a VFS when I started, but given my current lua knowlegde this turned out to be a hard 1st step (and BTW: winzip also unpacks tgz files to your temp dir and doesn't remove them afterwards, and its version 9.x). From what I saw I had the fear that it would be necessary to 1st create an IO API for the C core prior to support a standalone exe based on a VFS. 

Surely I would be interested in a more clever solution, I already gained what I wanted to gain: learned something.

Regards, Frank
________________________________________________________________
Verschicken Sie romantische, coole und witzige Bilder per SMS!
Jetzt neu bei WEB.DE FreeMail: http://freemail.web.de/?mc=021193


Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Jay Carlson
In reply to this post by duck
duck wrote:
I've now a basic extension to srlua running which allows you to combine multiple (lua-) files in one zip file and than start it as a standalone executable.


Great idea (if the amount the Lua code -- and you can precompile it, of course, to save space -- compresses is larger than the size of the decompressor...).

Another way to do this on Windows would be to embed the "glued" codes
as resource entries (so they are inside the EXE proper) and then to UPX the whole file. (UPX is a run-time in-memory unpackaging system. Highly recommended to impress your friends when showing them how small your Lua run-time is compared to their Perl or Python distro. The UPX decompressor is very small.)

Has anyone tried using UPX for this (and is there an analogous way to embed binary data into ELF binaries so that the original executable is unaffected but the embedded data in technically "inside" as far as exectuable packers are concerned)?

I'm confused. Why don't you just use the linker to put in the data that you want? The most portable way to do this is just stuff everything into a const char[] and compile it. If you want to skip the octets->.c->.o overhead I suppose you could use objcopy to go directly from octets->.o, or use a linker script for more control.

If the goal is to avoid having the whole development environment including headers on the machine performing the append, then ship a partially linked version of the lua executable with a weak symbol that will be supplied by the .o containing the octets. The target machine then just needs objcopy and a linker.

If you can't depend on having a linker on the target machine, yeah, you're going to have to play around with libelf like lhf says. But just about everybody has a linker.

Jay


Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Frank Bechmann
In reply to this post by Frank Bechmann
> I'm confused.  Why don't you just use the linker to put in the data that 
> you want?  The most portable way to do this is just stuff everything 
> into a const char[] and compile it.  

1) Because with the srluaz approach I can develop as normal with various
lua sources in variuos directories, using 'require' and whatsoever. 
The other way I see no easy way to *transparently" make lua find and 
exeute the code in these char[]S (how would a require be resolved there?). 
If you have any idea how to achieve this, it is welcome.

2) Once I want to create a standalone "package" I just can use winzip to 
collect my sources, no "configuration" for what to include and no batch 
mechanism for its processing is required.

> If you want to skip the 
> octets->.c->.o overhead I suppose you could use objcopy to go directly 
> from octets->.o, or use a linker script for more control.

To be honest I didn't care much about size and startup speed. As I wrote in
another thread I'm mainly interested in creating small tool apps as
standalone executables in a Windows environment. 1 MB more and an
initial copy and unzip before the real program starts is really not an issue than.

> If the goal is to avoid having the whole development environment 
> including headers on the machine performing the append, then ship a 
> partially linked version of the lua executable with a weak symbol that 
> will be supplied by the .o containing the octets.  The target machine 
> then just needs objcopy and a linker.
> 
> If you can't depend on having a linker on the target machine, yeah, 
> you're going to have to play around with libelf like lhf says.  But just 
> about everybody has a linker.

No, that's not the problem. I have the development environment in my
hands and want to just make usage of the executables on the clients
as easy as possible.

Regards, Frank

__________________________________________________________
Mit WEB.DE FreePhone mit hoechster Qualitaet ab 0 Ct./Min.
weltweit telefonieren! http://freephone.web.de/?mc=021201


Reply | Threaded
Open this post in threaded view
|

RES: FW: srlua extension to run with glued zip file

André de Leiradella
In reply to this post by duck
>> I've now a basic extension to srlua running which allows you to
>> combine multiple (lua-) files in one zip file and than start it as a
>> standalone executable.
> 
> Great idea (if the amount the Lua code -- and you can precompile it,
> of 
> course, to save space -- compresses is larger than the size of the
> decompressor...).
> 
> Another way to do this on Windows would be to embed the "glued" codes
> as resource entries (so they are inside the EXE proper) and then to
> UPX the whole file. (UPX is a run-time in-memory unpackaging system.
> Highly  
> recommended to impress your friends when showing them how small your
> Lua run-time is compared to their Perl or Python distro. The UPX
> decompressor is very small.) 
> 
> Has anyone tried using UPX for this (and is there an analogous way to
> embed binary data into ELF binaries so that the original executable is
> unaffected but the embedded data in technically "inside" as far as
> exectuable packers are concerned)?

Not tried it, but as long as Windows resources are concerned UPX can
pack them:

# upx -h
...
Options for win32/pe & rtm32/pe:
  --compress-exports=0    do not compress the export section
  --compress-exports=1    compress the export section [default]
  --compress-icons=0      do not compress any icons
  --compress-icons=1      compress all but the first icon
  --compress-icons=2      compress all but the first icon directory
[default]
  --compress-resources=0  do not compress any resources at all
  --strip-relocs=0        do not strip relocations
  --strip-relocs=1        strip relocations [default]
...

I made a library some time ago called luareader
(http://www.geocities.com/andre_leiradella/#luareader) that can read
from FILE pointers, file descriptors and from memory. It allows
on-the-fly decompression of the streams using zlib or bzip2. In fact, it
has a reader that can decide if the underlying stream where data is
located is uncompressed, compressed with zlib or compressed with bzip2
and act accordingly.

It already has a reader that makes some substitutions to Lua source code
to make OOP in Lua easier, translating sequences of @id (field access),
@id(...) (method call) and @@id(...) (inherited method call) to
user-defined Lua code. But take care, this reader is a hack.

I have a unfinished 2.0 version of luareader that can process files
inside a tar file. The tar file can reside in the file system, in memory
or even inside another tar file.

Andre de Leiradella


Reply | Threaded
Open this post in threaded view
|

RES: FW: srlua extension to run with glued zip file

André de Leiradella
In reply to this post by Luiz Henrique de Figueiredo
> However, I've not given up the idea of embedding several files in an
> executable as a virtual read-only filesystem, complete with directory
> trees. I've found that the CPIO format is a simple as you get for
> this task. And it's seekable so you don't need to expand the archive
> into a temporary directory.    
> 
I made such a file system using ZIP files for a project that I'm no
longer working on. It was meant to be a Lua game framework using SDL.
The ZIP file was wrapped by SDL_Rwops streams and I used lua_load to
read from them.

Changing it to read from ZIP files located at the end of executables and
dropping the SDL_Rwops wrapper is easy...

Andre de Leiradella


Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Daniel Quintela
In reply to this post by Frank Bechmann
Did you look at Kurmi for packing methodology?
I think you can use it "as is" or take some ideas from it.

--DQ


Reply | Threaded
Open this post in threaded view
|

RES: FW: srlua extension to run with glued zip file

André de Leiradella
In reply to this post by Jay Carlson
> I'm confused.  Why don't you just use the linker to put in the data
> that you want?  The most portable way to do this is just stuff
> everything into a const char[] and compile it.  If you want to skip
> the octets->.c->.o overhead I suppose you could use objcopy to go
> directly from octets->.o, or use a linker script for more control.
> 
One problem with that is that your char[] array will seat in RAM even
after it has been compiled by Lua, which looks like a waste. Reading
from somewhere (a file, a chunk of data at the end of the executable or
a Windows resource, be them compressed or not) does not waste memory.

Andre de Leiradella


Reply | Threaded
Open this post in threaded view
|

RES: FW: srlua extension to run with glued zip file

André de Leiradella
In reply to this post by André de Leiradella
> Not tried it, but as long as Windows resources are concerned UPX can
> pack them: 
> 
> # upx -h
> ...
> Options for win32/pe & rtm32/pe:
>   --compress-exports=0    do not compress the export section
>   --compress-exports=1    compress the export section [default]
>   --compress-icons=0      do not compress any icons
>   --compress-icons=1      compress all but the first icon
>   --compress-icons=2      compress all but the first icon directory
> [default]
>   --compress-resources=0  do not compress any resources at all
>   --strip-relocs=0        do not strip relocations
>   --strip-relocs=1        strip relocations [default]
> ...
> 
Talking about that, does anyone know how UPX handle compressed
resources? Resources are loaded from disk as needed AFAIK, does UPX
decompress and put them in RAM? I don't think there is a hook in Windows
to allow on-the-fly decompression of resource data.

If UPX decompress everything in RAM and you want to save memory, it's
better to use zlib or bzip2 to decompress data that you compressed
beforehand (not using UPX) and have UPX compress only your code.

Andre de Leiradella


Reply | Threaded
Open this post in threaded view
|

Re: RES: FW: srlua extension to run with glued zip file

Jay Carlson
In reply to this post by André de Leiradella
André de Leiradella wrote:
I'm confused.  Why don't you just use the linker to put in the data
that you want?  The most portable way to do this is just stuff
everything into a const char[] and compile it.  If you want to skip
the octets->.c->.o overhead I suppose you could use objcopy to go
directly from octets->.o, or use a linker script for more control.


One problem with that is that your char[] array will seat in RAM even
after it has been compiled by Lua, which looks like a waste. Reading
from somewhere (a file, a chunk of data at the end of the executable or
a Windows resource, be them compressed or not) does not waste memory.

That's not true. On any modern desktop or server system, executables are demand-loaded. Anything read-only and only touched once, like that const char[], is an good candidate for the OS to just drop, in the same way that it drops caches of files. Files hang around in memory after you read them too...

This does depend on a little on OS cache tuning. Some kernels prioritize retaining mmap'd pages (like from executables) over retained caches of files. But it's only a matter of degree.

I suppose you could provide a stronger hint to the kernel by calling madvise, or munmap the data after you were done. You'd have to be careful how you set up the latter of course.

My first real C programming experiences were on the Amiga and MS-DOS where what you said is true. I continue to find the physical memory usage on demand paged operating systems to be unintuitive.

Jay

Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Frank Bechmann
In reply to this post by Frank Bechmann
Lua list <[hidden email]> schrieb am 06.01.05 01:19:51:
> 
> Did you look at Kurmi for packing methodology?
> I think you can use it "as is" or take some ideas from it.
> 
> --DQ
> 

thx, I will check it.

btw: I was about to ask for a link but tried to google before just to make sure and it was the 1st hit :) 
I would have bet a finnish death metal band would be on top.

__________________________________________________________
Mit WEB.DE FreePhone mit hoechster Qualitaet ab 0 Ct./Min.
weltweit telefonieren! http://freephone.web.de/?mc=021201


Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Philippe Lhoste
In reply to this post by Jay Carlson
Jay Carlson wrote:
I'm confused. Why don't you just use the linker to put in the data that you want? The most portable way to do this is just stuff everything into a const char[] and compile it. If you want to skip the octets->.c->.o overhead I suppose you could use objcopy to go directly from octets->.o, or use a linker script for more control.

If the goal is to avoid having the whole development environment including headers on the machine performing the append, then ship a partially linked version of the lua executable with a weak symbol that will be supplied by the .o containing the octets. The target machine then just needs objcopy and a linker.

If you can't depend on having a linker on the target machine, yeah, you're going to have to play around with libelf like lhf says. But just about everybody has a linker.

Not on Windows.

A typical use I saw with AutoIt, an tool to automate GUI tasks on Windows, is to have an exe (the runtime) and a tool to glue the user's code to the exe to make self-running scripts.

The target is end users, knowing (near to) nothing to programming (the syntax is very primitive, close to .bat/.cmd), which are unlikely to have any programming tool like a linker.

--
Philippe Lhoste
--  (near) Paris -- France
--  http://Phi.Lho.free.fr
--  --  --  --  --  --  --  --  --  --  --  --  --  --

Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Jay Carlson
Philippe Lhoste wrote:
Jay Carlson wrote:

I'm confused. Why don't you just use the linker to put in the data that you want? The most portable way to do this is just stuff everything into a const char[] and compile it. If you want to skip the octets->.c->.o overhead I suppose you could use objcopy to go directly from octets->.o, or use a linker script for more control.

If the goal is to avoid having the whole development environment including headers on the machine performing the append, then ship a partially linked version of the lua executable with a weak symbol that will be supplied by the .o containing the octets. The target machine then just needs objcopy and a linker.

If you can't depend on having a linker on the target machine, yeah, you're going to have to play around with libelf like lhf says. But just about everybody has a linker.


Not on Windows.

Right. I suppose you could package up mingw GNU ld into a point-and-click tool. But the context for this thread was "what can we do for ELF?"

Jay

Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

duck
In reply to this post by Luiz Henrique de Figueiredo
> > Luiz did something exactly like this, based on reslib:  I quote:
> 
> Yes, but I've given up this ide for now because it wouldn't work for Windows
> and because it's ok to glue arbitrary data to executables. (Would UPX choke
> on these?) Hence, srlua is as simple as you get.

UPX allows you to compress files with trailing data...it just keeps it at 
the end of the new, smaller EXE. Of course, if you have embedded a pointer
to the tail inside the EXE, it will be wrong for the compressed file's
tail (so don't do that :-)

> However, I've not given up the idea of embedding several files in an
> executable as a virtual read-only filesystem, complete with directory trees.
> I've found that the CPIO format is a simple as you get for this task. And it's
> seekable so you don't need to expand the archive into a temporary directory.

The glued data doesn't have to be seekable directly with lseek() -- you 
can allow a decompression layer in there.

BTW, not writing to disk is where it's at under Windows: the various Perl 
and Python packagers out there spew out shiggabytes of temporary guff. If 
the code doesn't terminate cleanly or timeously, it all gets left behind.
So a sysadmin trying to write an emergency malware cleanup util, or a
"ptach me quick" tool, for the company's logon script can end up dropping 
multi-MB of garbage at each logon on each user's PC. Ugly. 

> If there's interest, I can get back to this.

Great idea. 

Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

duck
In reply to this post by Jay Carlson
> >>If you can't depend on having a linker on the target machine, yeah, 
> >>you're going to have to play around with libelf like lhf says.  But 
> >>just about everybody has a linker.
> >
> >
> >Not on Windows.
> 
> Right.  I suppose you could package up mingw GNU ld into a 
> point-and-click tool.  But the context for this thread was "what can we 
> do for ELF?"

Wellllll, the question was mine: if you can bodge the binary data in 
resources in Windows, is there perchance a similar trick in ELF? (I'm not 
too fussed if there isn't.)

For a busy and malware-aggravated Windows admin, you have to agree that:

COPY /B UNITARYLUACORE.EXE+MYMALWARECOUNTERHACK.LUA FIXWORM.EXE

is a pretty tidy build command!


Reply | Threaded
Open this post in threaded view
|

Re: FW: srlua extension to run with glued zip file

Frank Bechmann
In reply to this post by Frank Bechmann
> 
> Wellllll, the question was mine: if you can bodge the binary data in 
> resources in Windows, is there perchance a similar trick in ELF? (I'm not 
> too fussed if there isn't.)
> 
> For a busy and malware-aggravated Windows admin, you have to agree that:
> 
> COPY /B UNITARYLUACORE.EXE+MYMALWARECOUNTERHACK.LUA FIXWORM.EXE
> 
> is a pretty tidy build command!
> 

Duh, and I hoped I got it: with this how should the UNITARYLUACORE.EXE know, where the code of the Lua file starts? This is the very sence of the glue executable from the srlua package, using it just means to type

GLUE SRLUA.EXE MYMALWARECOUNTERHACK.LUA FIXWORM.EXE

which seems also to be bearable.

Do I have to look at the source again?

________________________________________________________________
Verschicken Sie romantische, coole und witzige Bilder per SMS!
Jetzt neu bei WEB.DE FreeMail: http://freemail.web.de/?mc=021193


Reply | Threaded
Open this post in threaded view
|

RES: RES: FW: srlua extension to run with glued zip file

André de Leiradella
In reply to this post by Jay Carlson
> That's not true.  On any modern desktop or server system, executables
> are demand-loaded.  Anything read-only and only touched once, like
> that const char[], is an good candidate for the OS to just drop, in
> the same way that it drops caches of files.  Files hang around in
> memory after you read them too...
> 
> This does depend on a little on OS cache tuning.  Some kernels
> prioritize retaining mmap'd pages (like from executables) over
> retained caches of files.  But it's only a matter of degree.
> 
> I suppose you could provide a stronger hint to the kernel by calling
> madvise, or munmap the data after you were done.  You'd have to be
> careful how you set up the latter of course.
> 
> My first real C programming experiences were on the Amiga and MS-DOS
> where what you said is true.  I continue to find the physical memory
> usage on demand paged operating systems to be unintuitive.
> 
> Jay

I completely agree, but as you said it depends on OS cache tuning or on
giving the right hints to the kernel while trying *not* to munmap your
code or data that is still needed. That's why I prefer to play safe and
read temporary data (i.e. Lua scripts) from the file system at the end
of an executable or as a file by itself. Compressing this data is only
an atempt to make the app distro smaller.

Andre de Leiradella


12