Execution of pre compiled lua code from memory mapped files

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Execution of pre compiled lua code from memory mapped files

Hans Riekehof
Hi everyone,

i am quite new to lua and i want to execute pre compiled lua code directly from a memory mapped file.  The background is this:

I am working in the embedded systems area and as you know memory consumption especially RAM usage is critical. In the system i am currently working on with a small team, there is the nice situation that we have a quite large flash. The uC in this system is capable of executing code directly from flash. From this flash the uC boots a small operating system which includes basic POSIX functionality like file operations etc.
On top of that we also included Lua as a script language based on a implementation from the eLua-project. In the eLua Version, which is Lua 5.1, there is a patch included which should save RAM usage.
I know now that the patch is not working as I expected because the pre compiled modules are still copied into the "package.loaded" table and therefore consume a lot of RAM.

The best thing would be, if we could execute pre compiled lua code from a memory mapped file. Than we could have a lot of pre compiled lua modules in the flash and work with them without wasting RAM. This feature would also be useful in Linux and Windows. In Linux you could use mmap() in windows there is an equivalent I don't know right now.
I would like to make a feature request on this for some next lua Version :) I think a generic implementation of this would be great in a lot of use cases. It could also be a feature you can activate or deactivate during the compilation of lua. I imagine something like this:

If you call mod = require 'mod' the loader can recognize if mod is in a pre-compiled file which is memory mapable. In Linux this would turn out in a call to mmap() in an embedded system you can
return a pointer with the starting address of the file in memory.
Than you can start using mod as you want but it is not copied into the lua state.

If there is another way or if you have other ideas how to save RAM in a lua sate please tell me.

Best regards,
Hans





Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

BogdanM
Hi,

On Wed, May 27, 2015 at 12:54 PM, Hans Riekehof <[hidden email]> wrote:
Hi everyone,

i am quite new to lua and i want to execute pre compiled lua code directly from a memory mapped file.  The background is this:

I am working in the embedded systems area and as you know memory consumption especially RAM usage is critical. In the system i am currently working on with a small team, there is the nice situation that we have a quite large flash. The uC in this system is capable of executing code directly from flash. From this flash the uC boots a small operating system which includes basic POSIX functionality like file operations etc.
On top of that we also included Lua as a script language based on a implementation from the eLua-project. In the eLua Version, which is Lua 5.1, there is a patch included which should save RAM usage.
I know now that the patch is not working as I expected because the pre compiled modules are still copied into the "package.loaded" table and therefore consume a lot of RAM.

The patch in eLua is designed to let you execute _bytecode_ directly from flash. Right now it doesn't go further than this, so in that respect the patch is doing what's supposed to do. If I'm reading your e-mail right, what you want is the whole module's memory layout to be saved to flash and loaded from there, which might not be doable.
In any case, it would be useful if you opened an issue in the eLua github repository to talk about this.

Thanks,
Bogdan
 

The best thing would be, if we could execute pre compiled lua code from a memory mapped file. Than we could have a lot of pre compiled lua modules in the flash and work with them without wasting RAM. This feature would also be useful in Linux and Windows. In Linux you could use mmap() in windows there is an equivalent I don't know right now.
I would like to make a feature request on this for some next lua Version :) I think a generic implementation of this would be great in a lot of use cases. It could also be a feature you can activate or deactivate during the compilation of lua. I imagine something like this:

If you call mod = require 'mod' the loader can recognize if mod is in a pre-compiled file which is memory mapable. In Linux this would turn out in a call to mmap() in an embedded system you can
return a pointer with the starting address of the file in memory.
Than you can start using mod as you want but it is not copied into the lua state.

If there is another way or if you have other ideas how to save RAM in a lua sate please tell me.

Best regards,
Hans






Reply | Threaded
Open this post in threaded view
|

AW: Execution of pre compiled lua code from memory mapped files

Hans Riekehof

Hi Bogdan,

 

you are right i will also open an issue for this in the elua github repo. But I still think that including memory mapping directly to the Lua language would be a nice thing. Not just for embedded systems.

 

Best,

Hans

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Dirk Laurie-2
2015-05-28 10:28 GMT+02:00 Hans Riekehof <[hidden email]>:

> But I still think that including memory mapping directly to the Lua language
> would be a nice thing. Not just for embedded systems.

Lua has it under the name "light userdata". You write a C procedure
that creates a void* in any way you please (malloc, mmap, whatever),
lua_pushlightuserdata it and return it to Lua. Make the procedure visible
to Lua via "require".

Lua can't see into the memory; all it can do is assign the pointer to a Lua
name and later pass it as a parameter to other C procedures you have
written that can derefence the pointer and operate on the contents.

If you want object-oriented access to light userdata, you will have
to use debug.setmetatable. Light userdata does not have subtyping,
so there is only one metatable for all.

The manual is quite terse on this topic. You should rather read about it
in "Programming in Lua" <www.lua.org/pil>.

Reply | Threaded
Open this post in threaded view
|

AW: Execution of pre compiled lua code from memory mapped files

Hans Riekehof
Hi,

this is not exactly what i mean. Its not about interacting with C/C++ because this is already available through the API.
If I have compiled C/C++ code I think its already the case, that the ram consumption is not that high if you register this in a lua state.

My idea is to have this for pre compiled lua code because I want to write a lot of code in Lua and not in C/C++.

Best,
Hans



-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Dirk Laurie
Gesendet: Donnerstag, 28. Mai 2015 10:57
An: Lua mailing list
Betreff: Re: Execution of pre compiled lua code from memory mapped files

2015-05-28 10:28 GMT+02:00 Hans Riekehof <[hidden email]>:

> But I still think that including memory mapping directly to the Lua language
> would be a nice thing. Not just for embedded systems.

Lua has it under the name "light userdata". You write a C procedure
that creates a void* in any way you please (malloc, mmap, whatever),
lua_pushlightuserdata it and return it to Lua. Make the procedure visible
to Lua via "require".

Lua can't see into the memory; all it can do is assign the pointer to a Lua
name and later pass it as a parameter to other C procedures you have
written that can derefence the pointer and operate on the contents.

If you want object-oriented access to light userdata, you will have
to use debug.setmetatable. Light userdata does not have subtyping,
so there is only one metatable for all.

The manual is quite terse on this topic. You should rather read about it
in "Programming in Lua" <www.lua.org/pil>.

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Dirk Laurie-2
2015-05-28 11:21 GMT+02:00 Hans Riekehof <[hidden email]>:

> My idea is to have this for pre compiled lua code because I want to
> write a lot of code in Lua and not in C/C++.

Lua compiles to bytecode for a virtual machine. That VM is different
for every major Lua release: 5.1, 5.2, 5.3 etc, but the same within minor
releases 5.2.1, 5.2.2 etc. However, it is not supposed to portable across
different architectures.

I'm not entirely sure I understand what you want. Can you show us
what the Lua code would look like in a postulated Lua dialect that has
this feature?

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

BogdanM
In reply to this post by Hans Riekehof
Hi,

On Thu, May 28, 2015 at 11:28 AM, Hans Riekehof <[hidden email]> wrote:

Hi Bogdan,

 

you are right i will also open an issue for this in the elua github repo. But I still think that including memory mapping directly to the Lua language would be a nice thing. Not just for embedded systems.


I think I understand where you're coming from, but simply doing mmap() on precompiled bytecode will not help. Lua doesn't simply use the precompiled bytecode representation as-is, it will still allocate RAM to hold various variables and some internal data while reading the bytecode file. In other words, the compiled bytecode file is not a "frozen" memory image, it is still interpreted.

Best,
Bogdan

 

Best,

Hans


Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

KHMan
In reply to this post by Dirk Laurie-2
On 5/28/2015 5:42 PM, Dirk Laurie wrote:

> 2015-05-28 11:21 GMT+02:00 Hans Riekehof:
>
>> My idea is to have this for pre compiled lua code because I want to
>> write a lot of code in Lua and not in C/C++.
>
> Lua compiles to bytecode for a virtual machine. That VM is different
> for every major Lua release: 5.1, 5.2, 5.3 etc, but the same within minor
> releases 5.2.1, 5.2.2 etc. However, it is not supposed to portable across
> different architectures.
>
> I'm not entirely sure I understand what you want. Can you show us
> what the Lua code would look like in a postulated Lua dialect that has
> this feature?

Dirk, IMHO you should mostly leave this to Bogdan and eLua users,
they have been tackling low-RAM issues for a long time and can
obviously engage in productive dialogue.

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia


Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Dirk Laurie-2
2015-05-28 12:32 GMT+02:00 KHMan <[hidden email]>:

> Dirk, IMHO you should mostly leave this to Bogdan and eLua users, they have
> been tackling low-RAM issues for a long time and can obviously engage in
> productive dialogue.

Hans said:

> This feature would also be useful in Linux and Windows.

That seemed to indicate that eLua expertise was not a prerequisite.

But you are right in advising me to shut up. Trying to be helpful to people
who describe themselves as "quite new to lua" is in general a silly idea.

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

BogdanM


On Thu, May 28, 2015 at 2:00 PM, Dirk Laurie <[hidden email]> wrote:
2015-05-28 12:32 GMT+02:00 KHMan <[hidden email]>:

> Dirk, IMHO you should mostly leave this to Bogdan and eLua users, they have
> been tackling low-RAM issues for a long time and can obviously engage in
> productive dialogue.

Hans said:

> This feature would also be useful in Linux and Windows.

That seemed to indicate that eLua expertise was not a prerequisite.

But you are right in advising me to shut up. Trying to be helpful to people
who describe themselves as "quite new to lua" is in general a silly idea.

 
By all means, please don't keep quiet. I'm pretty much always looking for new ideas of how to make Lua more RAM-friendly (without criplling it in the process :) )

Thanks,
Bogdan 

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

KHMan
In reply to this post by Dirk Laurie-2
On 5/28/2015 7:00 PM, Dirk Laurie wrote:
> 2015-05-28 12:32 GMT+02:00 KHMan:
>
>> Dirk, IMHO you should mostly leave this to Bogdan and eLua users, they have
>> been tackling low-RAM issues for a long time and can obviously engage in
>> productive dialogue.
>
> Hans said:
>
>> This feature would also be useful in Linux and Windows.

Sure, this is one sentence.
The primary issue is still solidly in embedded dev, RAM size.

Regarding mmap() in mainstream CPUs, in most applications, data or
data manipulation will be massively larger than bytecode size. I
doubt mmap() will result in any kind of performance gain, since
bytecode reads still contends with data and objects in the
D-caches of the CPU -- none of that I/O behaviour changed. I don't
see any big win...

> That seemed to indicate that eLua expertise was not a prerequisite.
[snip snip snip snip snip]

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia


Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Jay Carlson
In reply to this post by Hans Riekehof
>
> On 2015-05-27, at 5:54 AM, Hans Riekehof <[hidden email]> wrote:
>
> If you call mod = require 'mod' the loader can recognize if mod is in a pre-compiled file which is memory mapable. In Linux this would turn out in a call to mmap() in an embedded system you can
> return a pointer with the starting address of the file in memory.
> Than you can start using mod as you want but it is not copied into the lua state.

On Linux and Windows, mmap can deliver copy-on-write pages. If you’re working on something like an ESP8266, there is no MMU to let you remap pages when they are written to, so some logic is harder.

It may be easier to design something for compile-time. Create some .o files, and try to make them as "const" as possible. On Linux and Windows, you will get copy-on-write for free. On embedded chips, you will have at least figured out which parts of your design need to be allocated in RAM; the linker will take care of placing them in the right spot and initializing them.

eLua's rommable tables can work as backing tables for this purpose: only changed entries are stored in a real table.

Jay
Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Marc Balmer
In reply to this post by Hans Riekehof


Am 27.05.15 um 11:54 schrieb Hans Riekehof:

> Hi everyone,
>
> i am quite new to lua and i want to execute pre compiled lua code directly from a memory mapped file.  The background is this:
>
> I am working in the embedded systems area and as you know memory consumption especially RAM usage is critical. In the system i am currently working on with a small team, there is the nice situation that we have a quite large flash. The uC in this system is capable of executing code directly from flash. From this flash the uC boots a small operating system which includes basic POSIX functionality like file operations etc.
> On top of that we also included Lua as a script language based on a implementation from the eLua-project. In the eLua Version, which is Lua 5.1, there is a patch included which should save RAM usage.
> I know now that the patch is not working as I expected because the pre compiled modules are still copied into the "package.loaded" table and therefore consume a lot of RAM.
>
> The best thing would be, if we could execute pre compiled lua code from a memory mapped file. Than we could have a lot of pre compiled lua modules in the flash and work with them without wasting RAM. This feature would also be useful in Linux and Windows. In Linux you could use mmap() in windows there is an equivalent I don't know right now.
> I would like to make a feature request on this for some next lua Version :) I think a generic implementation of this would be great in a lot of use cases. It could also be a feature you can activate or deactivate during the compilation of lua. I imagine something like this:
>
> If you call mod = require 'mod' the loader can recognize if mod is in a pre-compiled file which is memory mapable. In Linux this would turn out in a call to mmap() in an embedded system you can
> return a pointer with the starting address of the file in memory.
> Than you can start using mod as you want but it is not copied into the lua state.
>
> If there is another way or if you have other ideas how to save RAM in a lua sate please tell me.

That is exactly what we do:  We mmap a file.  Just we added another
little complexion:  Our bytecode is encrypted on disc, so we mmap the
encrypted bytecode, decrypt it, and run it.

works like a charm.


Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Tim Hill
In reply to this post by Dirk Laurie-2

> On May 28, 2015, at 2:42 AM, Dirk Laurie <[hidden email]> wrote:
>
> 2015-05-28 11:21 GMT+02:00 Hans Riekehof <[hidden email]>:
>
>> My idea is to have this for pre compiled lua code because I want to
>> write a lot of code in Lua and not in C/C++.
>
> Lua compiles to bytecode for a virtual machine. That VM is different
> for every major Lua release: 5.1, 5.2, 5.3 etc, but the same within minor
> releases 5.2.1, 5.2.2 etc. However, it is not supposed to portable across
> different architectures.
>
> I'm not entirely sure I understand what you want. Can you show us
> what the Lua code would look like in a postulated Lua dialect that has
> this feature?
>

The poster wants to execute directly from a bytecode file via memory-mapping to save on RAM footprint by avoiding the need to copy/load the bytecode into memory before execution. The issue of portability isn't a problem here since (presumably) the bytecode and Lua interpreter are all embedded in the flash/ROM image at the same time.

Given that Lua is designed for embedded systems it’s a bit surprising that this isn’t part of the core API.

—Tim


Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Ashwin Hirschi

> The poster wants to execute directly from a bytecode file via  
> memory-mapping to save on RAM footprint by avoiding the need to  
> copy/load the bytecode into memory before execution. The issue of  
> portability isn't a problem here since (presumably) the bytecode and Lua  
> interpreter are all embedded in the flash/ROM image at the same time.
>
> Given that Lua is designed for embedded systems it’s a bit surprising  
> that this isn’t part of the core API.

Ummm, of course there's embedding and then there's embedding.

Lua was designed as an embedded *language*, in order to extend & enhance  
host applications [read: other software].

Running Lua on embedded *systems* only became somewhat of a sport later on  
in the life of Lua.

Ashwin.

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Michael Welsh Duggan
In reply to this post by Tim Hill
Tim Hill <[hidden email]> writes:

>> On May 28, 2015, at 2:42 AM, Dirk Laurie <[hidden email]> wrote:
>>
>> 2015-05-28 11:21 GMT+02:00 Hans Riekehof <[hidden email]>:
>>
>>> My idea is to have this for pre compiled lua code because I want to
>>> write a lot of code in Lua and not in C/C++.
>>
>> Lua compiles to bytecode for a virtual machine. That VM is different
>> for every major Lua release: 5.1, 5.2, 5.3 etc, but the same within minor
>> releases 5.2.1, 5.2.2 etc. However, it is not supposed to portable across
>> different architectures.
>>
>> I'm not entirely sure I understand what you want. Can you show us
>> what the Lua code would look like in a postulated Lua dialect that has
>> this feature?
>
> The poster wants to execute directly from a bytecode file via
> memory-mapping to save on RAM footprint by avoiding the need to
> copy/load the bytecode into memory before execution. The issue of
> portability isn't a problem here since (presumably) the bytecode and
> Lua interpreter are all embedded in the flash/ROM image at the same
> time.
>
> Given that Lua is designed for embedded systems it’s a bit surprising
> that this isn’t part of the core API.

I'm confused.  Although not part of the core library, luaL_loadbufferx()
exists, and is implemented in a very small amount of code:

  typedef struct LoadS {
    const char *s;
    size_t size;
  } LoadS;
 
  static const char *getS (lua_State *L, void *ud, size_t *size) {
    LoadS *ls = (LoadS *)ud;
    (void)L;  /* not used */
    if (ls->size == 0) return NULL;
    *size = ls->size;
    ls->size = 0;
    return ls->s;
  }
 
  LUALIB_API int luaL_loadbufferx (lua_State *L, const char *buff, size_t size,
                                   const char *name, const char *mode) {
    LoadS ls;
    ls.s = buff;
    ls.size = size;
    return lua_load(L, getS, &ls, name, mode);
  }

I use luaL_loadbufferx() to compile compiled lua into my binaries (along
with a small script that converts the binary compiled lua file into a C
array).  I feel that I must be missing something about the original
question.

--
Michael Welsh Duggan
([hidden email])

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Luiz Henrique de Figueiredo
> I'm confused.  Although not part of the core library, luaL_loadbufferx()
> exists, and is implemented in a very small amount of code:

The point is that when Lua loads precompiled Lua code, it *copies* the
data to its internal data structures.

The question is about how to avoid this copy, especially in RAM
starved systems, when precompiled code is already stored in ROM.

The answer is that this is not very easy to do because the internal data
structures are not prepared to handle some data in Lua core land and
some data in user land, although recently it does support strings in
user land (at a cost, since it avoids interning them).

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Michael Welsh Duggan
Luiz Henrique de Figueiredo <[hidden email]> writes:

>> I'm confused.  Although not part of the core library, luaL_loadbufferx()
>> exists, and is implemented in a very small amount of code:
>
> The point is that when Lua loads precompiled Lua code, it *copies* the
> data to its internal data structures.
>
> The question is about how to avoid this copy, especially in RAM
> starved systems, when precompiled code is already stored in ROM.
>
> The answer is that this is not very easy to do because the internal data
> structures are not prepared to handle some data in Lua core land and
> some data in user land, although recently it does support strings in
> user land (at a cost, since it avoids interning them).

Thanks.  I am happy that I now understand the issue.  (And happier that
it doesn't obviate the reason I use this to put compiled lua code in my
binaries.)

--
Michael Welsh Duggan
([hidden email])

Reply | Threaded
Open this post in threaded view
|

Re: Execution of pre compiled lua code from memory mapped files

Jay Carlson
In reply to this post by Luiz Henrique de Figueiredo
On 2015-05-29, at 10:24 AM, Luiz Henrique de Figueiredo <[hidden email]> wrote:
>
>> I'm confused.  Although not part of the core library, luaL_loadbufferx()
>> exists, and is implemented in a very small amount of code:
>
> The point is that when Lua loads precompiled Lua code, it *copies* the
> data to its internal data structures.
>
> The question is about how to avoid this copy, especially in RAM
> starved systems, when precompiled code is already stored in ROM.

For people just joining in, check out http://www.eluaproject.net/doc/v0.9/en_arch_ltr.html for a discussion of what eLua does already.

I ran into the problem in my Lua-FLTK days (4.0, 2001?), where my tolua binding created one table per class for its methods. Not very memory-friendly, and the data structures were not sharable across Lua invocations the way C shared libraries can be.

Jay