Sharing userdata among stats.

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Sharing userdata among stats.

Laurent FAILLIE
Hi,

I'm building a multi-threaded application and all threads carry their own stat.
Now, I have to create an object to share data among those stats, named SelFIFO : this object is NEVER subject to be collected, mutex are handled at C side and slaves only get a C pointer to userdata.

My concern is "how can I insert this pointer to slave stats ?".

So, in the main thread, this object is created like that :

struct SelFIFO *q = (struct SelFIFO *)lua_newuserdata(L, sizeof(struct SelFIFO));
assert(q);
luaL_getmetatable(L, "SelFIFO");
lua_setmetatable(L, -2);
...

but, in the slave thread, I got only 'q' pointer.
How can I push it in slave's stat ?
How can I associate a "SelFIFO" to it ?

Thanks

Laurent

Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Sean Conner
It was thus said that the Great Laurent FAILLIE once stated:
> Hi,
> I'm building a multi-threaded application and all threads carry their own
> stat.Now, I have to create an object to share data among those stats,
> named SelFIFO : this object is NEVER subject to be collected, mutex are
> handled at C side and slaves only get a C pointer to userdata.

  By "stat" do you mean "Lua state"?

> My concern is "how can I insert this pointer to slave stats ?".
> So, in the main thread, this object is created like that :
> struct SelFIFO *q = (struct SelFIFO *)lua_newuserdata(L, sizeof(struct SelFIFO));
> assert(q);
> luaL_getmetatable(L, "SelFIFO");
> lua_setmetatable(L, -2);
> ...
> but, in the slave thread, I got only 'q' pointer.How can I push it in slave's stat ?How can I associate a "SelFIFO" to it ?
> Thanks
> Laurent
> ps: full source code is https://github.com/destroyedlolo/Selene/blob/master/src/SelFIFO.c

  Some random comments:  you don't need checkSelFIFO()---luaL_checkudata()
will do what you want.  Second, DO NOT USE assert() TO CHECK THE RETURN CODE
FOR FUNCTIONS!  In particular, this is wrong:

        assert(q->name = strdup(n));

  Unless you are coding in C++ (and you are not), you do not need to cast
the return code from lua_newuserdata() or any other function that returns a
void *.  In fact, you should NOT do that as it can hide bugs.

  The easiest way to get what you want is to change sff_create():

        static int sff_create(lua_State *L)
        {
          const char      *n = luaL_checkstring(L,1);
          char            *copy;
          struct SelFIFO **q = lua_newuserdata(L,sizeof(struct SelFIFO **));
       
          /*-------------------------------------
          ; copy the string from Lua and bail if we can't copy it
          ;------------------------------------------------------*/
       
          copy = strdup(n);
       
          if (copy == NULL)
          {
            lua_pushnil(L);
            return 1;
          }
       
          /*------------------------------------------------
          ; allocate memory for the SelFIFO.  Bail if we can't create it.
          ;-------------------------------------------------------------*/
       
          *q = calloc(1,sizeof(struct SelFIFO));
       
          if (*q == NULL)
          {
            lua_pushnil(L);
            return 1;
          }
       
          pthread_mutex_init(&q->mutex,NULL);
       
          q->h      = hash(n);
          q->name   = copy;
          q->first  = NULL;
          q->last   = NULL;
          q->next   = firstFifo;
          firstFifo = q;

          return 1;
        }

  Then sff_find() becomes:

        static int sff_find(lua_State *L)
        {
          const char *n = luaL_checkstring(L,1);
          int         h = hash(n);
         
          for (struct SelFIFO *p = firstFifo ; p ; p = p->next)
          {
            if ((h == p->h) && (strcmp(n,p->name) == 0))
            {
              struct SelFIFO **q = lua_newuserdata(L,sizeof(struct SelFIFO **));
              *q = p;
              return 1;
            }
          }
       
          return 0;
        }
       
  By using a pointer-to-a-pointer, it's easier to push the FIFO into other
states.

  -spc (Hope this helps some)
 

Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
In reply to this post by Laurent FAILLIE
Thanks Sean,

I've converted to pointer to pointer and it's working fine.

> Some random comments:  you don't need checkSelFIFO()---luaL_checkudata() will do what you want.

It's only to embbed type checking with luaL_argcheck().
As per my reading, luaL_checkudata() only returns a NULL in case of check failure, isn't it ?
(it would be more efficient to be an inline function ... and it will be).

> Second, DO NOT USE assert() TO CHECK THE RETURN CODE
> FOR FUNCTIONS!

It is only by laziness :) I have done that for very small allocations: if they fail, in any case, the application is close to crash facing a severe starvation of resources.

Thanks again.

Laurent

Le Jeudi 22 juin 2017 1h33, Sean Conner <[hidden email]> a écrit :


It was thus said that the Great Laurent FAILLIE once stated:
> Hi,
> I'm building a multi-threaded application and all threads carry their own
> stat.Now, I have to create an object to share data among those stats,
> named SelFIFO : this object is NEVER subject to be collected, mutex are
> handled at C side and slaves only get a C pointer to userdata.

  By "stat" do you mean "Lua state"?

> My concern is "how can I insert this pointer to slave stats ?".
> So, in the main thread, this object is created like that :
> struct SelFIFO *q = (struct SelFIFO *)lua_newuserdata(L, sizeof(struct SelFIFO));
> assert(q);
> luaL_getmetatable(L, "SelFIFO");
> lua_setmetatable(L, -2);
> ...
> but, in the slave thread, I got only 'q' pointer.How can I push it in slave's stat ?How can I associate a "SelFIFO" to it ?
> Thanks
> Laurent
> ps: full source code is https://github.com/destroyedlolo/Selene/blob/master/src/SelFIFO.c

  Some random comments:  you don't need checkSelFIFO()---luaL_checkudata()
will do what you want.  Second, DO NOT USE assert() TO CHECK THE RETURN CODE
FOR FUNCTIONS!  In particular, this is wrong:

    assert(q->name = strdup(n));

  Unless you are coding in C++ (and you are not), you do not need to cast
the return code from lua_newuserdata() or any other function that returns a
void *.  In fact, you should NOT do that as it can hide bugs.

  The easiest way to get what you want is to change sff_create():

    static int sff_create(lua_State *L)
    {
      const char      *n = luaL_checkstring(L,1);
      char            *copy;
      struct SelFIFO **q = lua_newuserdata(L,sizeof(struct SelFIFO **));
   
      /*-------------------------------------
      ; copy the string from Lua and bail if we can't copy it
      ;------------------------------------------------------*/
   
      copy = strdup(n);
   
      if (copy == NULL)
      {
        lua_pushnil(L);
        return 1;
      }
   
      /*------------------------------------------------
      ; allocate memory for the SelFIFO.  Bail if we can't create it.
      ;-------------------------------------------------------------*/
   
      *q = calloc(1,sizeof(struct SelFIFO));
   
      if (*q == NULL)
      {
        lua_pushnil(L);
        return 1;
      }
   
      pthread_mutex_init(&q->mutex,NULL);
   
      q->h      = hash(n);
      q->name  = copy;
      q->first  = NULL;
      q->last  = NULL;
      q->next  = firstFifo;
      firstFifo = q;

      return 1;
    }

  Then sff_find() becomes:

    static int sff_find(lua_State *L)
    {
      const char *n = luaL_checkstring(L,1);
      int        h = hash(n);
     
      for (struct SelFIFO *p = firstFifo ; p ; p = p->next)
      {
        if ((h == p->h) && (strcmp(n,p->name) == 0))
        {
          struct SelFIFO **q = lua_newuserdata(L,sizeof(struct SelFIFO **));

          *q = p;

          return 1;
        }
      }
   
      return 0;
    }
   
  By using a pointer-to-a-pointer, it's easier to push the FIFO into other
states.

  -spc (Hope this helps some)




Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Sean Conner
It was thus said that the Great Laurent FAILLIE once stated:
> Thanks Sean,
> I've converted to pointer to pointer and it's working fine.
>
> > Some random comments:  you don't need checkSelFIFO()---luaL_checkudata()
> > will do what you want.
>
> It's only to embbed type checking with luaL_argcheck().As per my reading,
> luaL_checkudata() only returns a NULL in case of check failure, isn't it ?
> (it would be more efficient to be an inline function ... and it will be).

  It takes a close reading of the manual, but luaL_checkudata() has an
indicator (described in section 4.8 of the Lua 5.3 manual) of:

        [-0,+0,v]

which indicates it can intentionally throw an error, and yes, if the given
item at the index isn't a userdata with the proper type (checked by the
metatable of the given name), it throws an error.  If it returns, it will
never return a NULL value.

> > Second, DO NOT USE assert() TO CHECK THE RETURN CODE> FOR FUNCTIONS!
>
> It is only by laziness :) I have done that for very small allocations: if
> they fail, in any case, the application is close to crash facing a severe
> starvation of resources.

  Best to either return a Lua nil, or throw a Lua error:

        if (ptr == NULL)
          return luaL_error(L,"OUT OF MEMORY!");

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
ok, for checkSelFIFO(), I keep it in my todo list.

About assert() : you're right. I'll convert it to an Lua error raising.

Anyway, as SelFIFO is working well, I'm now able to move ahead with my smarthome dashboard. Thanks again.




Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
Well well well,

I'm still facing a nasty problem that drive me nut :(

In the slave, I'm able to retrieve my object and when I display its metatable, it's  displaying my methods :
local fifo = SelFIFO.Find("toto")
local mt = getmetatable(fifo)
for k,v in pairs(mt) do print(k,v) end

result :
dump    function: 0x14ddf50
list    function: 0x14ddf70
Pop    function: 0x187d360
__index    table: 0x1762aa8
Push    function: 0x1762ad0

but when I'm calling them
fifo:dump()

it's saying

/home/laurent/Toile/Inputs/20_MQTTLog.lua:25: attempt to call method 'dump' (a nil value)

I can't understand how the metatable can be filled whereas calling member raise an error :(

Any tip welcome :)

Laurent



Le Jeudi 22 juin 2017 23h29, Laurent FAILLIE <[hidden email]> a écrit :


ok, for checkSelFIFO(), I keep it in my todo list.

About assert() : you're right. I'll convert it to an Lua error raising.

Anyway, as SelFIFO is working well, I'm now able to move ahead with my smarthome dashboard. Thanks again.







Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Sean Conner
It was thus said that the Great Laurent FAILLIE once stated:

> Well well well,
> I'm still facing a nasty problem that drive me nut :(
> In the slave, I'm able to retrieve my object and when I display its metatable, it's  displaying my methods :local fifo = SelFIFO.Find("toto")local mt = getmetatable(fifo)
> for k,v in pairs(mt) do print(k,v) end
> result :dump    function: 0x14ddf50
> list    function: 0x14ddf70
> Pop    function: 0x187d360
> __index    table: 0x1762aa8
> Push    function: 0x1762ad0
>
> but when I'm calling themfifo:dump()
> it's saying
>
> /home/laurent/Toile/Inputs/20_MQTTLog.lua:25: attempt to call method 'dump' (a nil value)
> I can't understand how the metatable can be filled whereas calling member raise an error :(
> Any tip welcome :)

  Does mt == mt.__index?

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
I tried

        local q3 = SelFIFO.Find("myFifo")
        local mt = getmetatable(q3)
        mt.__index = mt
        q3:dump()

but the result is the same :
    Selenites/FIFO.sel:27: attempt to call method 'dump' (a nil value)

Anyway, it's strange it's working in my main thread and not the slave one : state initialization code is strictly the same.

Best regards,

Laurent


Le Mercredi 28 juin 2017 0h02, Sean Conner <[hidden email]> a écrit :


It was thus said that the Great Laurent FAILLIE once stated:

> Well well well,
> I'm still facing a nasty problem that drive me nut :(
> In the slave, I'm able to retrieve my object and when I display its metatable, it's  displaying my methods :local fifo = SelFIFO.Find("toto")local mt = getmetatable(fifo)
> for k,v in pairs(mt) do print(k,v) end
> result :dump    function: 0x14ddf50
> list    function: 0x14ddf70
> Pop    function: 0x187d360
> __index    table: 0x1762aa8
> Push    function: 0x1762ad0
>
> but when I'm calling themfifo:dump()
> it's saying
>
> /home/laurent/Toile/Inputs/20_MQTTLog.lua:25: attempt to call method 'dump' (a nil value)
> I can't understand how the metatable can be filled whereas calling member raise an error :(
> Any tip welcome :)

  Does mt == mt.__index?

  -spc




Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
Hum, another thing strange.

I tested the following code

    local q3 = SelFIFO.Find("myFifo")
    local mt = getmetatable(q3)
    print('--- mt ----')
    for k,v in pairs(mt) do print(k,v) end

    print('--- index ---')
    print(mt.__index)
    for k,v in pairs(mt.__index) do print(k,v) end

But it's output is :

    --- mt ----
    dump    function: 0x13e7270
    list    function: 0x13e7290
    Pop    function: 0x13e4188
    __index    table: 0x13e7058
    Push    function: 0x13e4148
    --- index ---
    nil
*E* (launch) Selenites/FIFO.sel:31: bad argument #1 to 'pairs' (table expected, got nil)

How it's possible to got something during table scanning but being unable to retrieve it ?

The C code behind is the following (lua 5.1) :

static const struct luaL_reg SelFFLib [] = {
    {"Create", sff_create},
    {"Find", sff_find},
    {NULL, NULL}
};

static const struct luaL_reg SelFFM [] = {
    {"Push", sff_push},
    {"Pop", sff_pop},
/*    {"HowMany", sff_HowMany}, */
    {"dump", sff_dump},
    {"list", sff_list},
    {NULL, NULL}
};


void init_SelFIFO( lua_State *L ){    /* CAUTION : called also when thread are created */
    luaL_newmetatable(L, "SelFIFO");
    lua_pushstring(L, "__index");
    lua_pushvalue(L, -2);
    lua_settable(L, -3);    /* metatable.__index = metatable */
    luaL_register(L, NULL, SelFFM);
    luaL_register(L,"SelFIFO", SelFFLib);
}



And this function is called by :

    lua_State *tstate = luaL_newstate(); /* Initialise new state for the thread */
    assert(tstate);
    luaL_openlibs( tstate );
    init_shared_Lua( tstate );
    init_SelFIFO( tstate );



Thanks.

Laurent


Le Mercredi 28 juin 2017 17h04, Laurent FAILLIE <[hidden email]> a écrit :


I tried

        local q3 = SelFIFO.Find("myFifo")
        local mt = getmetatable(q3)
        mt.__index = mt
        q3:dump()

but the result is the same :
    Selenites/FIFO.sel:27: attempt to call method 'dump' (a nil value)

Anyway, it's strange it's working in my main thread and not the slave one : state initialization code is strictly the same.

Best regards,

Laurent

Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Sean Conner
It was thus said that the Great Laurent FAILLIE once stated:
>
> And this function is called by :
>     lua_State *tstate = luaL_newstate(); /* Initialise new state for the  
> thread */
>     assert(tstate);
>     luaL_openlibs( tstate );
>     init_shared_Lua( tstate );
>     init_SelFIFO( tstate );
> code : https://github.com/destroyedlolo/Selene/blob/master/src/SelEvent.c

  Wrong file---the code is in selene.c (and similar in SelMQTT.c).  But the
one thing that *really* stands out is this:                                

        lua_xmove( L, tstate, 1 );        

  You might want to double check the documentation for lua_xmove() because
it's not doing what you think it's doing.  I'm not sure that's what's
causing the issue you are seeing, but it's probably causing *some* issue.

  -spc                                                                    


Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
Hi Sean,


On 29/06/2017 23:51, Sean Conner wrote:
It was thus said that the Great Laurent FAILLIE once stated:

Wrong file---the code is in selene.c (and similar in SelMQTT.c).  But the
one thing that *really* stands out is this:

    lua_xmove( L, tstate, 1 );

   You might want to double check the documentation for lua_xmove() because
it's not doing what you think it's doing.  I'm not sure that's what's
causing the issue you are seeing, but it's probably causing *some* issue.
It's here to push the function from the main state to the slave one as per lua_pcall()'s documentation.
It is here to push the Lua function to "launch" into the slave's state, from lua_*call() functions documentation.
In Selene.c, the function is Detach() first and only argument.
In SelMQTT.c, the function is retrieved from LUA_REGISTRY.

I don't think there is any trouble here as my home's dashboard is running for months on 24/7 basis and this part of code is called on several MQTT message arrivals which happening several time per hours (most of message handling is done using another mechanism as dealing with graphics so not suitable to be done asynchronously).
The same code is used on the automation tools as well which drives shutters and other things.

Both of them have months of uptime without trouble so I guess they are stable.

But back to my issue : for me, it's a bit illogical to get a metatable but not being able to call it's "member". Perhaps I missed something when initialing the slave's state ... but can't find what. And all other stuffs are working find fine in slave ...
I think I would have to jump into Lua's own code to understand what's happening ...

Thanks

Laurent
Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Sean Conner
It was thus said that the Great Laurent Faillie once stated:

> Hi Sean,
>
>
> On 29/06/2017 23:51, Sean Conner wrote:
> >It was thus said that the Great Laurent FAILLIE once stated:
> >
> >Wrong file---the code is in selene.c (and similar in SelMQTT.c). But the
> >one thing that *really* stands out is this:
> >
> >    lua_xmove( L, tstate, 1 );
> >
> >   You might want to double check the documentation for lua_xmove()
> >because
> >it's not doing what you think it's doing.  I'm not sure that's what's
> >causing the issue you are seeing, but it's probably causing *some* issue.
> >It's here to push the function from the main state to the slave one as
> >per lua_pcall()'s documentation.
> It is here to push the Lua function to "launch" into the slave's state,
> from lua_*call() functions documentation.
> In Selene.c, the function is Detach() first and only argument.
> In SelMQTT.c, the function is retrieved from LUA_REGISTRY.

  I don't know how it works then.  Because here's the code in question (from
selene.c):

        lua_State *tstate = luaL_newstate();
        assert(tstate);
        luaL_openlibs( tstate );
        init_shared_Lua( tstate );
        init_SelFIFO( tstate );
        lua_xmove( L, tstate, 1 );

And now the descrption from the Lua 5.1 documentation:

        lua_xmove [-?, +?, -]

        void lua_xmove (lua_State *from, lua_State *to, int n);

        Exchange values between different threads of the *same* global
        state.

        This function pops n values from the stack from, and pushes them
        onto the stack to.

You aren't moving data from the same global state, so that it's working is
... puzzling.  When I tried the following (using Lua 5.1):

        #include <stdio.h>

        #include <lua.h>
        #include <lualib.h>
        #include <lauxlib.h>
 
        int main(void)
        {
          lua_State *L1 = luaL_newstate();
          lua_State *L2 = luaL_newstate();
         
          luaL_openlibs(L1);
          luaL_openlibs(L2);
       
          luaL_loadstring(L1,"function foo() io.stdout:write('hello world\\n') end");
          lua_call(L1,0,0);
         
          lua_getglobal(L1,"foo");
          lua_call(L1,0,0);
         
          lua_getglobal(L1,"foo");
          lua_xmove(L1,L2,1);
          printf(">>> %s\n",luaL_typename(L2,-1));
          lua_call(L2,0,0);
         
          lua_close(L1);
          lua_close(L2);
         
          return 0;
        }

I get:

[spc]lucy:/tmp>./a.out
hello world
>>> function
PANIC: unprotected error in call to Lua API ([string "function foo()
io.stdout:write('hello world..."]:1: attempt to index field 'stdout' (a userdata value))
[spc]lucy:/tmp>

  Which, now that I think on it, is kind of like your problem, isn't it?

> I don't think there is any trouble here as my home's dashboard is
> running for months on 24/7 basis and this part of code is called on
> several MQTT message arrivals which happening several time per hours
> (most of message handling is done using another mechanism as dealing
> with graphics so not suitable to be done asynchronously).
> The same code is used on the automation tools as well which drives
> shutters and other things.
>
> Both of them have months of uptime without trouble so I guess they are
> stable.
>
> But back to my issue : for me, it's a bit illogical to get a metatable
> but not being able to call it's "member". Perhaps I missed something
> when initialing the slave's state ... but can't find what. And all other
> stuffs are working find fine in slave ...
> I think I would have to jump into Lua's own code to understand what's
> happening ...

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Laurent FAILLIE
Arg, in this case, I'm facing a STRONG problem as I'm using this mechanism in other stuffs (like Marcel which is the daemon providing most of MQTT data from my 1-wire network).
I don't have the choice, I need multi-threading as MQTT data have to arrive asynchronously (there is also a synchronous mode, but it not suitable as my tool is doing heavy graphical tasks, and it's cleaners in architectural point of view).

Obviously, I can delegate what I want to do to C side but in this case, I will loose the flexibility the have Lua code in early message arrival ... and all in all, I'm interested to know how I can exchange stuffs b/w totally different states.
Reply | Threaded
Open this post in threaded view
|

Re: Sharing userdata among stats.

Sean Conner
It was thus said that the Great Laurent FAILLIE once stated:

> Arg, in this case, I'm facing a STRONG problem as I'm using this mechanism
> in other stuffs (like Marcel which is the daemon providing most of MQTT
> data from my 1-wire network).I don't have the choice, I need
> multi-threading as MQTT data have to arrive asynchronously (there is also
> a synchronous mode, but it not suitable as my tool is doing heavy
> graphical tasks, and it's cleaners in architectural point of view).
>
> Obviously, I can delegate what I want to do to C side but in this case, I
> will loose the flexibility the have Lua code in early message arrival ...
> and all in all, I'm interested to know how I can exchange stuffs b/w
> totally different states.

  You are now talking about serialization.  Upshot:  for nil, booleans,
numbers, strings and tables (consisting of the previous types and containing
no cycles) there are any number of solutions available [1].  But for tables
containing cycles, userdata, lightuserdata, functions and threads ... not so
much.  I know of one module that can easily handle table cycles [2] but not
functions, threads, or userdata.

  For userdata, it might be possible to serialize, but 1) it is really
dependent upon the userdata in question and 2) will need code for both
encoding and decoding to work.  Lightuserdata ... more problematic as Lua
really treats the pointer as a glorified integer.

  Functions are an interesting problem.  Yes, you can get the bytecode
(string.dump()) but there are issues:  1) the default serialization of
functions will only work on the same architecture and 2) does not contain
the current values of any upvalues.  1) is not an issue with your usecase,
but 2) is.  Again, that can be worked around by not only encoding the
function (via string.dump()) but also including any upvalues, which can get
quite interesting (oh, an upvalue might be math.floor() or _G itself!).

  A few years ago I was able to do this (using an initial version of my CBOR
module [2]) and here's what I ended up having to do (NOTE: CBOR allows the
semtantic tagging of values, which makes this much easier):

        1) A function ended up being a CBOR array (if I recall) tagged as a
           Lua function (the array is tagged).  The first element was either
           a string, meaning the actual name of a known function, or a CBOR
           array, comprising the actual bytecode (which should technically
           be architecturally neutral but still limited to the same version
           of Lua, like Lua 5.1 to 5.1, or Lua 5.3 to 5.3).

        2) The upvalues in another array (and if one of the upvalues was a
           function, well, here we go again ... ),

        3) followed by the constants array, then any strings, then debugging
           information---basically, I wrote my own version of string.dump()
           on steroids [3].

I never finished it, not really needing it and having grown weary of the
whole endevour.  For me, in this instance, knowing what needs to be done was
good enough (wow!  harder than I thought).

  -spc

[1] http://lua-users.org/wiki/TableSerialization

[2] https://github.com/spc476/CBOR
        (disclaimer:  I wrote this)

[3] So, if one of the upvalues was actually _G, then I would tag the
        string "_G" with *LuaKnownValue* so I knew to use _G on the
        receiving end.  All known globals like math and io.stdin were
        handled this way.


Reply | Threaded
Open this post in threaded view
|

Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Sean Conner
It was thus said that the Great Sean Conner once stated:
>
>   Functions are an interesting problem.  Yes, you can get the bytecode
> (string.dump()) but there are issues:  1) the default serialization of
> functions will only work on the same architecture and 2) does not contain
> the current values of any upvalues.  1) is not an issue with your usecase,
> but 2) is.  Again, that can be worked around by not only encoding the
> function (via string.dump()) but also including any upvalues, which can get
> quite interesting (oh, an upvalue might be math.floor() or _G itself!).

  So I played around with this today, and I created a github repository for
those interested:

        https://github.com/spc476/LuaFunctionSerialize

  Right now, the code only supports Lua 5.1, and is not meant for any real
production (it's just a "proof-of-concept" right now).  But it works.  At
least, for the one complicated recursive function I've tested so far.  

        1) It uses string.dump() to get the bytecode
        2) It serializes any upvalues [1]
        3) It only serializes the environment if it's the global environment
        4) It can handle any normal Lua global value (like io.stdin or getmetatable)
        5) It will probably fail horribly if given a Lua function written in
           C (I did not check for this condition)

  So there you have it.  Something to play around with.

  -spc

[1] bools, numbers, tables and functions.  Tables can have cycles.

Reply | Threaded
Open this post in threaded view
|

Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Benoit Germain-2

2017-07-01 4:48 GMT+02:00 Sean Conner <[hidden email]>:
It was thus said that the Great Sean Conner once stated:
>
>   Functions are an interesting problem. 

Hi,

Have you had a look at Lanes? It has a pretty comprehensive state<->state data moving implementation, including functions of course, supporting Lua 5.1->5.3 and LuaJIT as well.


Cheers,

--
Benoit.
Reply | Threaded
Open this post in threaded view
|

Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Sven Olsen
In reply to this post by Sean Conner
Functions are an interesting problem.  Yes, you can get the bytecode (string.dump()) but there are issues:  

This is a bit OT, but I was also working on a multithreading problem yesterday, and ended up rolling a rather dirty C-hack to solve the following problem:

Given a Lua 5.2 function with upvalues that are only strings, numbers, booleans, nils, or _ENV, create a new closure that contains at most one upvalue (equal to the parent function's _ENV).  Any non-_ENV upvalues should be converted to constants, using the value currently stored in the upvalue.

My excuse for writing this (rather odd) function was that string.dump is a great tool for moving simple functions from one thread to another; but, 'prepping' a function for transfer via string.dump can be harder than it needs to be, if said function happens to include a lot of upvalues that are simple types.  In the case of such upvalues, just converting the upvalue ref to a constant ref is usually the behavior I want, and writing a function that hacks into the guts of Lua and does exactly that is relatively straightforward (in C, anyhow).

If anyone wants a copy of the code, let me know.  

Similarly, of course, if anyone wants to tell me what a fool I am for solving this particular problem in this particular way, constructive criticism is always welcome :)

-Sven


On Fri, Jun 30, 2017 at 7:48 PM, Sean Conner <[hidden email]> wrote:
It was thus said that the Great Sean Conner once stated:
>
>   Functions are an interesting problem.  Yes, you can get the bytecode
> (string.dump()) but there are issues:  1) the default serialization of
> functions will only work on the same architecture and 2) does not contain
> the current values of any upvalues.  1) is not an issue with your usecase,
> but 2) is.  Again, that can be worked around by not only encoding the
> function (via string.dump()) but also including any upvalues, which can get
> quite interesting (oh, an upvalue might be math.floor() or _G itself!).

  So I played around with this today, and I created a github repository for
those interested:

        https://github.com/spc476/LuaFunctionSerialize

  Right now, the code only supports Lua 5.1, and is not meant for any real
production (it's just a "proof-of-concept" right now).  But it works.  At
least, for the one complicated recursive function I've tested so far.

        1) It uses string.dump() to get the bytecode
        2) It serializes any upvalues [1]
        3) It only serializes the environment if it's the global environment
        4) It can handle any normal Lua global value (like io.stdin or getmetatable)
        5) It will probably fail horribly if given a Lua function written in
           C (I did not check for this condition)

  So there you have it.  Something to play around with.

  -spc

[1]     bools, numbers, tables and functions.  Tables can have cycles.