late binding functions to save ram on embedded systems

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

late binding functions to save ram on embedded systems

me the user
I was looking for ways to free up ram on embedded systems using lua and ran across e-lua
specifically the LuaTinyRam patches I looked through the source and replicated it on my side
to see what kind of ram savings it brought around 10-15k on this specific application pretty good!

Anyways, having hand applied the patches I realized how much of a maintenance nightmare
these changes would add, So I came up with something I think might work well and would like
to get some opinions.

My Idea is to use a form of late binding basically instead of luaL_register adding tables instead
it would store the address of the table and add an __index and __pairs method to the meta table
this is lua 5.1 so I also had to change the pairs function a bit


[lauxlib.c]
------------------------------------------------------------------------------------
static int lookup_latebind_func(lua_State *L)
{
  const luaL_Reg *l;
  /* name is top of stack */
  const char * name = lua_tostring (L, -1);

  luaL_argcheck(L, lua_istable(L, -2), -2, "table expected");
  lua_getmetatable (L, -2);
 
  /* lookup our virtual function(s) */
  for(int i = lua_objlen(L, -1); i > 0; i--) {
    lua_rawgeti (L, -1, i);
    l = (const luaL_Reg *) lua_touserdata (L, -1);
    lua_pop(L, 1);

    if(!l)
      break;

    for (; l->name; l++) {
      if(!name || strcmp(name, l->name) == 0) {
        /* if we use a function it gets added to the base table*/
        lua_pushcclosure(L, l->func, 0);

        if(name) {
          lua_pushvalue(L, -1); /* dupe closure */
          lua_setfield (L, -5, l->name);
          /* returns the closure */
          return 1;
        }
        else
          lua_setfield (L, -4, l->name);
      }
    }
  }
  lua_pop(L, 2); /* base table is top of stack */
  return 0;
}


static int latebind_func_pairs(lua_State *L)
{
  const luaL_Reg *l;

  /* base table is top of stack */
  luaL_argcheck(L, lua_istable(L, -1), -1, "table expected");
  lua_getmetatable (L, -1);

  lua_getglobal(L, "pairs");  /* function to be called / returned */

  /* clone the base table */
  lua_newtable(L);
  lua_pushnil(L);
  while(lua_next(L, -5) != 0) {
    lua_pushvalue(L, -2); /* dupe key */
    lua_insert(L, -2);
    lua_settable(L, -4);
  }

  /* add our dynamic functions */
  for(int i = lua_objlen(L, -3); i > 0; i--) {
    lua_rawgeti (L, -3, i);
    l = (const luaL_Reg *) lua_touserdata (L, -1);
    lua_pop(L, 1);

    if(!l)
      break;

    for (; l->name; l++)
    {  
      lua_pushcclosure(L, l->func, 0);
      lua_setfield (L, -2, l->name);
    }
  }

  lua_call(L, 1, 3);

  return 3;
}


LUALIB_API void (luaL_register_late) (lua_State *L, const char *libname,
                                const luaL_Reg *l) {
  if(!libname)
  {
    /* if there is no libname register normally */
    luaI_openlib(L, libname, l, 0);
    return;
  }
   
  static const struct luaL_reg virt_lib [] =
  {
    {"__latebind", lookup_latebind_func},
    {NULL, NULL}
  };

  luaI_openlib(L, libname, virt_lib, 0);

  if(!lua_getmetatable(L, -1))
    lua_createtable(L, 5, 2); //-2

  lua_pushcfunction(L, latebind_func_pairs);
  lua_setfield(L, -2, "__pairs");

  lua_pushcfunction(L, lookup_latebind_func);
  lua_setfield(L, -2, "__index");

  lua_pushinteger(L, lua_objlen(L, -1) + 1);
  lua_pushlightuserdata (L, (void *) l);
  lua_settable(L, -3);
 
  lua_setmetatable(L, -2);
}

[lbaselib.c]
-------------------------------------------------------------------------------------
static int luaB_pairs (lua_State *L) {
  if (!luaL_getmetafield(L, 1, "__pairs")) {  /* no metamethod? */
    luaL_checktype(L, 1, LUA_TTABLE);
    lua_pushvalue(L, lua_upvalueindex(1));  /* return generator, */
    lua_pushvalue(L, 1);  /* state, */
    lua_pushnil(L);  /* and initial value */
  }
  else {
    lua_pushvalue(L, 1);  /* argument 'self' to metamethod */
    lua_call(L, 1, 3);  /* get 3 values from metamethod */
  }
 
  return 3;
}


By adding the __pairs metamethod it allows us to still lookup functions within the tables
when functions are called within a latebound table the __index metamethod adds them
to the table and returns the closure this makes it faster after the first call to a function

The __latebind key allows a way to see which tables are latebound and if desired
bind all the functions by passing nil as the name __latebind(nil, t)

Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

Philipp Janda
Hi!

Am 12.09.2018 um 06:06 schröbte me the user:
> I was looking for ways to free up ram on embedded systems using lua and ran across e-lua
> specifically the LuaTinyRam patches I looked through the source and replicated it on my side
> to see what kind of ram savings it brought around 10-15k on this specific application pretty good!
>
> Anyways, having hand applied the patches I realized how much of a maintenance nightmare
> these changes would add, So I came up with something I think might work well and would like
> to get some opinions.

I have written a (limited) port[1] of eLua's rotables using the Lua C
API. Maybe you are interested. You'd still have to patch the Lua
standard modules to actually use those rotables, though.

Philipp

   [1]:  https://github.com/siffiejoe/lua-rotable



Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

BogdanM
In reply to this post by me the user
Hi,
On Wed, Sep 12, 2018 at 7:06 AM me the user <[hidden email]> wrote:

I was looking for ways to free up ram on embedded systems using lua and ran across e-lua
specifically the LuaTinyRam patches I looked through the source and replicated it on my side
to see what kind of ram savings it brought around 10-15k on this specific application pretty good!

Anyways, having hand applied the patches I realized how much of a maintenance nightmare
these changes would add, So I came up with something I think might work well and would like
to get some opinions.

My Idea is to use a form of late binding basically instead of luaL_register adding tables instead
it would store the address of the table and add an __index and __pairs method to the meta table
this is lua 5.1 so I also had to change the pairs function a bit

That looks like a nice idea, one that is much easier to implement and maintain. I am the author of the LTR patch and I fully agree that maintaining it between various versions of Lua is a nightmare, which is why I never tried to apply it to Lua 5.2 and above. Did you run some benchmarks to check how much memory your solution saves?

Thanks,
Bogdan 
Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

Dirk Laurie-2
In reply to this post by me the user
... me the user <[hidden email]> ...

GMail rightly sent this to Spam, and the only reason I see it now is
because a list member I normally respect has chosen to reply.

This amount of anonymity is going too far. We've had several list
posters who use pseudonyms with unreplyable e-mail addresses, and I am
fine with that — one gets to know the list personality like a
character in a book — but when messages from different people would
all look as if sent by the same Midwich Cuckoo, I opt out.

Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

me the user
In reply to this post by me the user

>That looks like a nice idea, one that is much easier to implement and maintain. I am the author of the LTR patch and I fully agree that maintaining it between various >versions of Lua is a nightmare, which is why I never tried to apply it to Lua 5.2 and above. Did you run some benchmarks to check how much memory your solution >saves?

Bogdan, I was able to add the baselib as well as a few other tables that couldn't be readonly
using luas garbage collection memory counting
I got 19KB ram saved at load
versus the 15KB I got with your LTR patch


> I have written a (limited)
> port[1] of eLua's rotables using the Lua C
> API. Maybe you are interested. You'd still
> have to patch the Lua
> standard modules to
> actually use those rotables, though.
 
 > Philipp
 
 >   [1]: 
 > https://github.com/siffiejoe/lua-rotable
 
Phillipp, Thanks I'll have a look into your module unfortunately no stdlib
so bsearch will have to go right off the bat but it looks like we had similar ideas
 
 

Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

Hugo Musso Gualandi
In reply to this post by me the user

On Sep 12, 2018 4:11 AM, Dirk Laurie <[hidden email]> wrote:
>
> GMail sent this to Spam, ...

I think this might be simply because they used an yahoo email address.

Apparently, yahoo servers don't send the proper email headers for mailing list posts so they get marked as spam. This has been a recurring problem here on lua-l.
Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

Jonathan Goble
On Wed, Sep 12, 2018, 10:19 AM Hugo Musso Gualandi <[hidden email]> wrote:
I think this might be simply because they used an yahoo email address.

This is correct. 

Apparently, yahoo servers don't send the proper email headers for mailing list posts so they get marked as spam. This has been a recurring problem here on lua-l.

This is not correct. 

As I think I've said a few times, the issue is that several years ago, Yahoo published a DMARC reject policy, which requires all compliant mail servers to reject (or mark as spam) any email that contains a Yahoo address in the From line but which did not originate from Yahoo's own servers. This intentional action by Yahoo broke all mailing lists because mailing lists receive the incoming copy from the Yahoo user and then resend it from their own servers, causing DMARC verification to fail.

Yahoo's recommended solution IIRC is for lists to change the From header to the list's own address, which violates the email standard that the From line should never contain any name or address other than that of the person that authored the message. I think there are a few partial workarounds that the mailing list administrators can apply, but none are ideal. The least of the evils IMO is to simply ban Yahoo addresses from 5
posting to the list.

You can get more information by Googling "yahoo dmarc reject policy mailing list". 
Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

me the user-2
In reply to this post by me the user
Apparently <me.theuser> yahoo email won't work with this mailing list so I have created this gmail one instead Ive also cleaned up the code a bit to save some bytes and make it a bit easier to follow

I found out it works perfectly well as a lua_register replacement but a bit more testing needs done before I'd declare it a success

lauxlib.c
----------------------------------------------------------------------------------------------------------
static int latebind_func_index(lua_State *L)
{
  const luaL_Reg *llib;
  /* name @ top of stack */
  const char *name = lua_tostring (L, -1);

  luaL_argcheck(L, lua_istable(L, -2) && lua_getmetatable(L, -2), -2,
                "latebind table expected");

  /* lookup late bound func(s), lua_objlen allows check of multiple luaL_Reg */
  for(int i = lua_objlen(L, -1); i > 0; i--) {
    lua_rawgeti (L, -1, i);
    llib = (const luaL_Reg *) lua_touserdata (L, -1);
    lua_pop(L, 1);

    if(!llib)
      continue;

    for (; llib->name; llib++) {
      if(!name || strcmp(name, llib->name) == 0) {
        /* if function is used it is added to the base table */
        lua_pushcclosure(L, llib->func, 0);

        if(!name) /* nil name binds all functions in table immediately */
          lua_setfield (L, -4, llib->name);
        else {
          lua_pushvalue(L, -1); /* dupe closure */
          lua_setfield (L, -5, llib->name);
          /* returns the closure */
          return 1;
        }
      }
    }
  }
  lua_pop(L, 2); /* base table is top of stack */
  return 0;
}

static int latebind_func_pairs(lua_State *L)
{
  luaL_argcheck(L, lua_istable(L, 1), 1, "table expected");
  lua_getglobal(L, "pairs");  /* function to be called / returned */

  lua_createtable(L, 0, 15);
  lua_getmetatable (L, 1);
  lua_setmetatable (L, 3);

  lua_pushnil(L); /*nil name retrieves all late bound functions */
  latebind_func_index(L);

  lua_pushnil(L); /* remove metatable */
  lua_setmetatable (L, 3);

  /* clone base table */
  lua_pushnil(L); /* first key */
  while(lua_next(L, 1) != 0) { 
    lua_pushvalue(L, -2); /* dupe key */
    lua_insert(L, -2); /* Stk = kkv */
    lua_settable(L, 3);
  }

  lua_call(L, 1, 3); /* pairs(t_cloned) */

  return 3;
}

LUALIB_API void (luaL_register) (lua_State *L, const char *libname,
                                const luaL_Reg *l) {
  if(!libname)
  {
    /* if there is no libname register normally */
    luaI_openlib(L, libname, l, 0);
    return;
  }

  /* store this table instead of of the passed luaL_Reg table */
  static const struct luaL_reg late_lib [] =
  {
    {"__latebind", latebind_func_index},
    {NULL, NULL}
  };

  static const struct luaL_reg late_meta [] =
  {
    {"__index", latebind_func_index},
    {"__pairs", latebind_func_pairs},
    {NULL, NULL}
  };

  luaI_openlib(L, libname, late_lib, 0);

  if(!lua_getmetatable(L, -1))
    lua_createtable(L, 5, 2);

  luaI_openlib(L, NULL, late_meta, 0);

  /* address of luaL_Reg is stored in numbered [1] key indices of metatable */

  lua_pushlightuserdata (L, (void *) l);
  lua_rawseti (L, -2, lua_objlen(L, -2) + 1);

  lua_setmetatable(L, -2);
}


lbaselib.c (lua 5.1)
----------------------------------------------------------------------------------------------------------
static int luaB_pairs (lua_State *L) {
  if (!luaL_getmetafield(L, 1, "__pairs")) {  /* no metamethod? */
    luaL_checktype(L, 1, LUA_TTABLE);
    lua_pushvalue(L, lua_upvalueindex(1));  /* return generator, */
    lua_pushvalue(L, 1);  /* state, */
    lua_pushnil(L);  /* and initial value */
  }
  else {
    lua_pushvalue(L, 1);  /* argument 'self' to metamethod */
    lua_call(L, 1, 3);  /* get 3 values from metamethod */
  }
  
  return 3;
}
Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

BogdanM

On Thu, Sep 13, 2018 at 8:44 AM Walter anderson <[hidden email]> wrote:
I found out it works perfectly well as a lua_register replacement but a bit more testing needs done before I'd declare it a success

I find it really useful to run the Lua test suite (https://www.lua.org/tests/) after every change that I do to the Lua source code. It caught loads of errors in my changes, especially in the LTR patch. It might help you too.

Best,
Bogdan
Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

me the user-2
It'd be nice if it weren't so dependent on floats being scattered all through the code our implementation is pretty custom anyways
I spent an hour trying to get around all the floats and exponents but the underlying parts worked well

On Thu, Sep 13, 2018 at 3:38 AM, Bogdan Marinescu <[hidden email]> wrote:

On Thu, Sep 13, 2018 at 8:44 AM Walter anderson <[hidden email]> wrote:
I found out it works perfectly well as a lua_register replacement but a bit more testing needs done before I'd declare it a success

I find it really useful to run the Lua test suite (https://www.lua.org/tests/) after every change that I do to the Lua source code. It caught loads of errors in my changes, especially in the LTR patch. It might help you too.

Best,
Bogdan

Reply | Threaded
Open this post in threaded view
|

Re: late binding functions to save ram on embedded systems

me the user-2
One issue I noted in further testing is when you try and set a new metatable to a latebound table you wipe out the mechanism for binding functions
I used the __metatable method to block this but I really don't want to make that permenant
so I added removal of the metatable when you bind all functions

I also added the __call metamethod to make it a bit more intuitive to bind a function

for instance.. the math table
f_add = math("add") -- binds add function

local t = {dummy = "dummy"}
setmetatable(math, t) -- ERROR 'cannot change a protected metatable'

math(nil) -- binds all math functions
local oldt = getmetatable(math) -- oldt == nil
setmetatable(math, t) -- success

here is the patch I submitted to our repository http://gerrit.rockbox.org/r/#/c/1922/

On Fri, Sep 14, 2018 at 9:05 AM, me the user <[hidden email]> wrote:
It'd be nice if it weren't so dependent on floats being scattered all through the code our implementation is pretty custom anyways
I spent an hour trying to get around all the floats and exponents but the underlying parts worked well

On Thu, Sep 13, 2018 at 3:38 AM, Bogdan Marinescu <[hidden email]> wrote:

On Thu, Sep 13, 2018 at 8:44 AM Walter anderson <[hidden email]> wrote:
I found out it works perfectly well as a lua_register replacement but a bit more testing needs done before I'd declare it a success

I find it really useful to run the Lua test suite (https://www.lua.org/tests/) after every change that I do to the Lua source code. It caught loads of errors in my changes, especially in the LTR patch. It might help you too.

Best,
Bogdan