Lua Without a FileSystem?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua Without a FileSystem?

Russell Haley
Hi, 

I've finally managed to get the pieces together to run Lua in the RTEMS operating system. A big shout out to whoever lparam is:


Anyway, now that I can load a hello world string and iterate a table, I was wondering about actually doing something with Lua. For my first attempt I would like to load penlight. I'm using the sparc erc32 (default) build with their emulator and it doesn't have a filesystem. errr... Help?

RTEMS is a little different because it's one big executable. I have a directory full of RTEMS libraries. Lua is just one library that I 'installed' in the library tree. To use Lua, I just link it into an example application and call their build system. Out the other end, I receive an EXE which contains the entire executable image - operating system and all at 5.3MB. (so cool).


I understand I can compile with luac, but I'm not sure how that gets included in the exe (in an *.a file?). I see srlua (thank you lua team) and was going to start playing with that, but thought I'd ask if there is anyone with experience before I go stumbling around the internet? 

To be sure, I have no idea what I'm even looking for. I've asked a similar question on the RTEMS mailing list and there is some information about loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be doing some digging on those items, but any advice would be grand. 

Cheers, 

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Philipp Janda
Am 10.04.2018 um 06:15 schröbte Russell Haley:
> Hi,

Hi!

>
> I've finally managed to get the pieces together to run Lua in the RTEMS
> operating system. A big shout out to whoever lparam is:
>
> https://github.com/lparam/rtems-liblua
>
> Anyway, now that I can load a hello world string and iterate a table, I was
> wondering about actually doing something with Lua. For my first attempt I
> would like to load penlight. I'm using the sparc erc32 (default) build with
> their emulator and it doesn't have a filesystem. errr... Help?
>
> To be sure, I have no idea what I'm even looking for. I've asked a similar
> question on the RTEMS mailing list and there is some information about
> loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be
> doing some digging on those items, but any advice would be grand.

I'd use one of the various amalgamation scripts, put the resulting
single big script into a char array, and load it from there. But a
searcher that loads from a tar would be cool (and probably also useful
for others) as well.

>
> Cheers,
>
> Russ
>

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Sean Conner
In reply to this post by Russell Haley
It was thus said that the Great Russell Haley once stated:

> Hi,
>
> I've finally managed to get the pieces together to run Lua in the RTEMS
> operating system. A big shout out to whoever lparam is:
>
> https://github.com/lparam/rtems-liblua
>
> Anyway, now that I can load a hello world string and iterate a table, I was
> wondering about actually doing something with Lua. For my first attempt I
> would like to load penlight. I'm using the sparc erc32 (default) build with
> their emulator and it doesn't have a filesystem. errr... Help?
>
> RTEMS is a little different because it's one big executable. I have a
> directory full of RTEMS libraries. Lua is just one library that I
> 'installed' in the library tree. To use Lua, I just link it into an example
> application and call their build system. Out the other end, I receive an
> EXE which contains the entire executable image - operating system and all
> at 5.3MB. (so cool).
>
>
> I understand I can compile with luac, but I'm not sure how that gets
> included in the exe (in an *.a file?). I see srlua (thank you lua team) and
> was going to start playing with that, but thought I'd ask if there is
> anyone with experience before I go stumbling around the internet?
>
> To be sure, I have no idea what I'm even looking for. I've asked a similar
> question on the RTEMS mailing list and there is some information about
> loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be
> doing some digging on those items, but any advice would be grand.

  I go into some detail in a few blog posts:

        http://boston.conman.org/2013/03/22.1
        http://boston.conman.org/2013/03/23.1

Over the years, some small details have changed since that write-up.  First,
for modules written in C, I add the luaopen_() C calls to the Lua array
package.preload[]:

        lua_getglobal(L,"package");
        lua_getfield(L,-1,"preload");
        luaL_register(L,NULL,preload); /* Yes, we're still using Lua 5.1 */

and the code is linked into the executable along with Lua.  The Lua modules
written in Lua are compiled into bytecode, compressed [1], converted to C (a
constant array) and then compiled into the final executable.  The Makefile
rule is:

        %.o : %.lua
                $(LUAC) -o $(*D)/$(*F).out $<
                $(BIN2C) -9 -o $(*D)/$(*F).l.c -t $(NS)$(*F) $(*D)/$(*F).out
                $(CC) $(CFLAGS) -c -o $@ $(*D)/$(*F).l.c
                $(RM) $(*D)/$(*F).out $(*D)/$(*F).l.c

  Pulling this apart, we run luac; the output goes to a file based upon the
name ("*D" is the leading portion of the directory; "*F" is the base
filename) with an extension of ".out".  Then I run a custom bin2c that will
compress (using libz, with the highest compression setting) the data and
spit out a C file.  The C file looks a bit like:

        char const c_module_name[] =
        {
          15,0,44,191, ...
        };
        size_t const c_module_name_size = 1234u;

  This is then compiled into an object file (to be later included in the
final executable) and the two temporary files (the ".out" and the ".l.c")
are deleted.  Lua is then extended with a custom module loader in
package.loaders[] that will locate the module based upon the name and
uncompress them (the code isn't that much different from what I presented,
except for the calls to libz for uncompression---it's the function
preloadlua_lualoader() in the 2013/03/23.1 entry listed above).

  I don't know how much you know about C and linking so this might be a bit
too much and confusing ...

  -spc (I should point out I'm compiling a custom Lua interpreter with a
        bunch of modules built in ... )

[1] I'm not sure why I originally compiled Lua to bytecode.  It's not
        really needed, and the text version compresses a bit better than the
        binary version.  There hasn't been a critical need to change this at
        work, so I haven't.  Your milage may vary.

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Sean Conner
In reply to this post by Philipp Janda
It was thus said that the Great Philipp Janda once stated:

> Am 10.04.2018 um 06:15 schröbte Russell Haley:
> >Hi,
>
> Hi!
>
> >
> >I've finally managed to get the pieces together to run Lua in the RTEMS
> >operating system. A big shout out to whoever lparam is:
> >
> >https://github.com/lparam/rtems-liblua
> >
> >Anyway, now that I can load a hello world string and iterate a table, I was
> >wondering about actually doing something with Lua. For my first attempt I
> >would like to load penlight. I'm using the sparc erc32 (default) build with
> >their emulator and it doesn't have a filesystem. errr... Help?
> >
> >To be sure, I have no idea what I'm even looking for. I've asked a similar
> >question on the RTEMS mailing list and there is some information about
> >loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be
> >doing some digging on those items, but any advice would be grand.
>
> I'd use one of the various amalgamation scripts, put the resulting
> single big script into a char array, and load it from there. But a
> searcher that loads from a tar would be cool (and probably also useful
> for others) as well.

  I've did a "proof-of-concept" for this, but with ZIP files [1][2].  For
modules written in Lua, they can be loaded directly from the file; but for
C-based modules, they have to be written to disk first before loading.

  But it sounds like Russell doesn't have much of a filesystem to work from
...

  -spc

[1] https://github.com/spc476/LEM

[2] http://boston.conman.org/2018/02/19.2

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Pierre Chapuis
On Tue, Apr 10, 2018, at 07:18, Sean Conner wrote:

> > I'd use one of the various amalgamation scripts, put the resulting
> > single big script into a char array, and load it from there. But a
> > searcher that loads from a tar would be cool (and probably also useful
> > for others) as well.
>
>   I've did a "proof-of-concept" for this, but with ZIP files [1][2].  For
> modules written in Lua, they can be loaded directly from the file; but for
> C-based modules, they have to be written to disk first before loading.

I use something like that at work as well. Sadly it's proprietary and
I don't have permission to share it.

It's not very complicated though. The tar code uses the library from
LuaRocks ([1]). For C modules, I agree they have to be written to disk.
In practice we don't do that, we compile and link them statically with
the executable.

[1] https://github.com/luarocks/luarocks/blob/master/src/luarocks/tools/tar.lua

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

pocomane
In reply to this post by Russell Haley
On Tue, Apr 10, 2018 at 6:15 AM, Russell Haley <[hidden email]> wrote:
> Hi, ...

I am used to allocate a static big array in my .c and fill it with the
content of the script. I have some code [1] but, probably, for your
scenario, you need to extract the binary injection tool from the main
executable. You can pre-amalgamate your lua sources with tools like
luasquish [2]. For the C modules, you can statically link them.

---

[1] https://github.com/pocomane/binject (please, ignore 'luancher', it
uses yet an old, messy, binject version)

[2] http://matthewwild.co.uk/projects/squish/home

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Frank Kastenholz-2
In reply to this post by Russell Haley
As others have said
Put your scripts in a set of big char arrays and load them from there.
It’s fairly straight forward and works perfectly well.

I suggest generating a hash of the scripts and checking it ... to protect against memory corruption.

We looked at saving compiled byte code vs not compiled. It didn’t seem like a big enough saving.

Compression saves space. But costs time and CPU to load. In our environment we were too cpu/time constrained so didn’t compress. 

And , also as others have said, I can’t go into more details :-(

Frank


On Apr 10, 2018, at 12:15 AM, Russell Haley <[hidden email]> wrote:

Hi, 

I've finally managed to get the pieces together to run Lua in the RTEMS operating system. A big shout out to whoever lparam is:


Anyway, now that I can load a hello world string and iterate a table, I was wondering about actually doing something with Lua. For my first attempt I would like to load penlight. I'm using the sparc erc32 (default) build with their emulator and it doesn't have a filesystem. errr... Help?

RTEMS is a little different because it's one big executable. I have a directory full of RTEMS libraries. Lua is just one library that I 'installed' in the library tree. To use Lua, I just link it into an example application and call their build system. Out the other end, I receive an EXE which contains the entire executable image - operating system and all at 5.3MB. (so cool).


I understand I can compile with luac, but I'm not sure how that gets included in the exe (in an *.a file?). I see srlua (thank you lua team) and was going to start playing with that, but thought I'd ask if there is anyone with experience before I go stumbling around the internet? 

To be sure, I have no idea what I'm even looking for. I've asked a similar question on the RTEMS mailing list and there is some information about loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be doing some digging on those items, but any advice would be grand. 

Cheers, 

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Luiz Henrique de Figueiredo
> We looked at saving compiled byte code vs not compiled. It didn’t seem like
> a big enough saving.

An alternative is to remove comments and whitespace from source code
using lstrip or other tools. See
http://lua-users.org/wiki/CompressionAndArchiving

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Francisco Olarte
In reply to this post by Frank Kastenholz-2
On Tue, Apr 10, 2018 at 1:07 PM, Frank Kastenholz
<[hidden email]> wrote:
> Put your scripts in a set of big char arrays and load them from there.
> It’s fairly straight forward and works perfectly well.
>
> I suggest generating a hash of the scripts and checking it ... to protect
> against memory corruption.

Don't forget to hash the checker and the loader too. They should have
been stored in the same place.

...
> Compression saves space. But costs time and CPU to load. In our environment
> we were too cpu/time constrained so didn’t compress.

Also, don't forget you'll need space for decompression ( probably
small ) and for the decompressor ( unless you already have one for
other purposes ).

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Oliver Kroth


Am 10.04.2018 um 18:33 schrieb Francisco Olarte:

> On Tue, Apr 10, 2018 at 1:07 PM, Frank Kastenholz
> <[hidden email]> wrote:
>> Put your scripts in a set of big char arrays and load them from there.
>> It’s fairly straight forward and works perfectly well.
>>
>> I suggest generating a hash of the scripts and checking it ... to protect
>> against memory corruption.
> Don't forget to hash the checker and the loader too. They should have
> been stored in the same place.
>
> ...
>> Compression saves space. But costs time and CPU to load. In our environment
>> we were too cpu/time constrained so didn’t compress.
> Also, don't forget you'll need space for decompression ( probably
> small ) and for the decompressor ( unless you already have one for
> other purposes ).
>
> Francisco Olarte.
>
the minifier of squish is quite powerful, and can be set to need no
additional space to expand the code.
In any case it needs to be considered that not compiled code needs
memory space for the source and the compiled code.

--
Oliver

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Russell Haley
In reply to this post by Philipp Janda


On Mon, Apr 9, 2018 at 9:52 PM, Philipp Janda <[hidden email]> wrote:
Am 10.04.2018 um 06:15 schröbte Russell Haley:
Hi,

Hi!


I've finally managed to get the pieces together to run Lua in the RTEMS
operating system. A big shout out to whoever lparam is:

https://github.com/lparam/rtems-liblua

Anyway, now that I can load a hello world string and iterate a table, I was
wondering about actually doing something with Lua. For my first attempt I
would like to load penlight. I'm using the sparc erc32 (default) build with
their emulator and it doesn't have a filesystem. errr... Help?

To be sure, I have no idea what I'm even looking for. I've asked a similar
question on the RTEMS mailing list and there is some information about
loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be
doing some digging on those items, but any advice would be grand.

I'd use one of the various amalgamation scripts, put the resulting single big script into a char array, and load it from there. But a searcher that loads from a tar would be cool (and probably also useful for others) as well.

Hi Phillipp, 

Thanks for the vote of confidence on the big-ole-char-array. I'll probably start with that while I try to absorb the technical madness that pours forth from Mr. Connor. 

Russ
 


Cheers,

Russ


Philipp




Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Russell Haley
In reply to this post by Francisco Olarte


On Tue, Apr 10, 2018 at 9:33 AM, Francisco Olarte <[hidden email]> wrote:
On Tue, Apr 10, 2018 at 1:07 PM, Frank Kastenholz
<[hidden email]> wrote:
> Put your scripts in a set of big char arrays and load them from there.
> It’s fairly straight forward and works perfectly well.
>
> I suggest generating a hash of the scripts and checking it ... to protect
> against memory corruption.

Don't forget to hash the checker and the loader too. They should have
been stored in the same place.

...
> Compression saves space. But costs time and CPU to load. In our environment
> we were too cpu/time constrained so didn’t compress.

Also, don't forget you'll need space for decompression ( probably
small ) and for the decompressor ( unless you already have one for
other purposes ).
 
Yes, there seems to be a tar utility available with the OS.  I haven't looked to far yet.

Thanks
Russ

Francisco Olarte.


Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Russell Haley
In reply to this post by Oliver Kroth


On Tue, Apr 10, 2018 at 9:45 AM, Oliver Kroth <[hidden email]> wrote:


Am 10.04.2018 um 18:33 schrieb Francisco Olarte:
On Tue, Apr 10, 2018 at 1:07 PM, Frank Kastenholz
<[hidden email]> wrote:
Put your scripts in a set of big char arrays and load them from there.
It’s fairly straight forward and works perfectly well.

I suggest generating a hash of the scripts and checking it ... to protect
against memory corruption.
Don't forget to hash the checker and the loader too. They should have
been stored in the same place.

...
Compression saves space. But costs time and CPU to load. In our environment
we were too cpu/time constrained so didn’t compress.
Also, don't forget you'll need space for decompression ( probably
small ) and for the decompressor ( unless you already have one for
other purposes ).

Francisco Olarte.

the minifier of squish is quite powerful, and can be set to need no additional space to expand the code.
In any case it needs to be considered that not compiled code needs memory space for the source and the compiled code.
Good point. Thanks.

Russ 


--
Oliver


Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Francisco Olarte
In reply to this post by Russell Haley
Russell:

On Wed, Apr 11, 2018 at 8:18 AM, Russell Haley <[hidden email]> wrote:
...
> Yes, there seems to be a tar utility available with the OS.  I haven't
> looked to far yet.

tar is the Tape ARchiver, it does not compress, just pack many files /
dire entries / other things into one big file ( it does other things,
but it forks to another program for compressing ).

In the unix world where it originates you normally tend to have a
program do one thing, to make the equivalent of a zip file you
normally first put everything into a single file with tar and then
compress the result, hence the classical .tar.gz extension with
whatever compressor you want ( I think this is callied solid mode in
the rar ( or maybe arj ) family. zip and its cousins tend to compress
each file and then pack the result. This makes it easier to extract
single files but does not compress as well, as you cannot use
inter-file redundancy. ).

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Russell Haley


On Wed, Apr 11, 2018 at 2:23 AM, Francisco Olarte <[hidden email]> wrote:
Russell:

On Wed, Apr 11, 2018 at 8:18 AM, Russell Haley <[hidden email]> wrote:
...
> Yes, there seems to be a tar utility available with the OS.  I haven't
> looked to far yet.

tar is the Tape ARchiver, it does not compress, just pack many files /
dire entries / other things into one big file ( it does other things,
but it forks to another program for compressing ).
(embarrassed) Thanks for specifying, I should have known that by now. I'll be sure to get clarity on that point.

In the unix world where it originates you normally tend to have a
program do one thing, to make the equivalent of a zip file you
normally first put everything into a single file with tar and then
compress the result, hence the classical .tar.gz extension with
whatever compressor you want ( I think this is callied solid mode in
the rar ( or maybe arj ) family. zip and its cousins tend to compress
each file and then pack the result. This makes it easier to extract
single files but does not compress as well, as you cannot use
inter-file redundancy. ).

Francisco Olarte.


Reply | Threaded
Open this post in threaded view
|

Re: Lua Without a FileSystem?

Russell Haley
In reply to this post by Sean Conner


On Mon, Apr 9, 2018 at 10:14 PM, Sean Conner <[hidden email]> wrote:
It was thus said that the Great Russell Haley once stated:
> Hi,
>
> I've finally managed to get the pieces together to run Lua in the RTEMS
> operating system. A big shout out to whoever lparam is:
>
> https://github.com/lparam/rtems-liblua
>
> Anyway, now that I can load a hello world string and iterate a table, I was
> wondering about actually doing something with Lua. For my first attempt I
> would like to load penlight. I'm using the sparc erc32 (default) build with
> their emulator and it doesn't have a filesystem. errr... Help?
>
> RTEMS is a little different because it's one big executable. I have a
> directory full of RTEMS libraries. Lua is just one library that I
> 'installed' in the library tree. To use Lua, I just link it into an example
> application and call their build system. Out the other end, I receive an
> EXE which contains the entire executable image - operating system and all
> at 5.3MB. (so cool).
>
>
> I understand I can compile with luac, but I'm not sure how that gets
> included in the exe (in an *.a file?). I see srlua (thank you lua team) and
> was going to start playing with that, but thought I'd ask if there is
> anyone with experience before I go stumbling around the internet?
>
> To be sure, I have no idea what I'm even looking for. I've asked a similar
> question on the RTEMS mailing list and there is some information about
> loading a tar based FS, as well as reading ELF binaries (?Really?). I'll be
> doing some digging on those items, but any advice would be grand.

  I go into some detail in a few blog posts:

        http://boston.conman.org/2013/03/22.1
        http://boston.conman.org/2013/03/23.1

Over the years, some small details have changed since that write-up.  First,
for modules written in C, I add the luaopen_() C calls to the Lua array
package.preload[]:

        lua_getglobal(L,"package");
        lua_getfield(L,-1,"preload");
        luaL_register(L,NULL,preload); /* Yes, we're still using Lua 5.1 */

and the code is linked into the executable along with Lua.  The Lua modules
written in Lua are compiled into bytecode, compressed [1], converted to C (a
constant array) and then compiled into the final executable.  The Makefile
rule is:

        %.o : %.lua
                $(LUAC) -o $(*D)/$(*F).out $<
                $(BIN2C) -9 -o $(*D)/$(*F).l.c -t $(NS)$(*F) $(*D)/$(*F).out
                $(CC) $(CFLAGS) -c -o $@ $(*D)/$(*F).l.c
                $(RM) $(*D)/$(*F).out $(*D)/$(*F).l.c

  Pulling this apart, we run luac; the output goes to a file based upon the
name ("*D" is the leading portion of the directory; "*F" is the base
filename) with an extension of ".out".  Then I run a custom bin2c that will
compress (using libz, with the highest compression setting) the data and
spit out a C file.  The C file looks a bit like:

        char const c_module_name[] =
        {
          15,0,44,191, ...
        };
        size_t const c_module_name_size = 1234u;

  This is then compiled into an object file (to be later included in the
final executable) and the two temporary files (the ".out" and the ".l.c")
are deleted.  Lua is then extended with a custom module loader in
package.loaders[] that will locate the module based upon the name and
uncompress them (the code isn't that much different from what I presented,
except for the calls to libz for uncompression---it's the function
preloadlua_lualoader() in the 2013/03/23.1 entry listed above).

  I don't know how much you know about C and linking so this might be a bit
too much and confusing ...

Brilliant, thanks so much for this. The finer details of the linking are a little opaque but this looks like a good solution. I think the RTEMS guys will like this.

Russ

  -spc (I should point out I'm compiling a custom Lua interpreter with a
        bunch of modules built in ... )

[1]     I'm not sure why I originally compiled Lua to bytecode.  It's not
        really needed, and the text version compresses a bit better than the
        binary version.  There hasn't been a critical need to change this at
        work, so I haven't.  Your milage may vary.