Need some help to use TFUNCTION

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Need some help to use TFUNCTION

Laurent FAILLIE
Hello,

I'm working on a C programs using Lua as embedded language.

At Lua side, I'm defining a table on witch a filed need to be a function as :

_, err = Broker:subscribe {
        { topic = "/tata", function = myfunc },
        { topic = "tutu", function = func2 }
}

And I know at C side I have to look up for TFUNCTION for "function" field.
My reading function is creating a structure containing
- a string, the topic,
- and a "reference" or something to keep track of the function

now my question :
1/ How can I store TFUNCTION at C side ?
2/ How my C code can "call" this Lua function ?

If it's not so clear, please have a look on my code here : https://github.com/destroyedlolo/Selene/blob/master/src/MQTT.c
especially on line 79.

Thanks

Laurent

---
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.
Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Brigham Toskin
You use the API function lua_call and its cousins to call a function you have on the stack. As far as saving it in C land, I don't think you can export it across the API barrier; someone please correct me if I'm wrong. If you do need a persistent handle to it for future use, what you can do is put it in the registry (basically a globally-available table for use by clients of the Lua API).

On Tue, Jun 2, 2015 at 4:26 PM, Laurent FAILLIE <[hidden email]> wrote:
Hello,

I'm working on a C programs using Lua as embedded language.

At Lua side, I'm defining a table on witch a filed need to be a function as :

_, err = Broker:subscribe {
        { topic = "/tata", function = myfunc },
        { topic = "tutu", function = func2 }
}

And I know at C side I have to look up for TFUNCTION for "function" field.
My reading function is creating a structure containing
- a string, the topic,
- and a "reference" or something to keep track of the function

now my question :
1/ How can I store TFUNCTION at C side ?
2/ How my C code can "call" this Lua function ?

If it's not so clear, please have a look on my code here : https://github.com/destroyedlolo/Selene/blob/master/src/MQTT.c
especially on line 79.

Thanks

Laurent

---
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



--
Brigham Toskin
Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Sean Conner
In reply to this post by Laurent FAILLIE
It was thus said that the Great Laurent FAILLIE once stated:

> Hello,
> I'm working on a C programs using Lua as embedded language.
> At Lua side, I'm defining a table on witch a filed need to be a function as :
> _, err = Broker:subscribe {
>         { topic = "/tata", function = myfunc },
>         { topic = "tutu", function = func2 }}
> And I know at C side I have to look up for TFUNCTION for "function" field.My reading function is creating a structure containing
> - a string, the topic,
> - and a "reference" or something to keep track of the function
>
> now my question :1/ How can I store TFUNCTION at C side ?2/ How my C code can "call" this Lua function ?
> If it's not so clear, please have a look on my code here : https://github.com/destroyedlolo/Selene/blob/master/src/MQTT.c
> especially on line 79.
> Thanks
> Laurent

  There are a few ways to do this.  The easiest way is to store it in a
global variable in the Lua state:

        static int foo(lua_State *L)
        {
          luaL_checktype(L,1,LUA_TFUNCTION);
          lua_pushvalue(L,1); /* make sure it's at the top of the stack */
          lua_setglobal(L,"myreffunction");
          return 0;
        }
       
  And to call it:
 
  static int bar(lua_State *L)
  {
   lua_getglobal(L,"myreffunction");
   lua_call(L,0,0); /* no parameters */
   return 0;
  }
 
  Another way is to store a reference in the Lua Registry:
 
  int myfuncref;
 
  static int foo(lua_State *L)
  {
   
   luaL_checktype(L,1,LUA_TFUNCTION);
   myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
   lua_pushinteger(L,myfuncref);
   lua_pushvalue(L,1);
   lua_settable(L,LUA_REGISTRYINDEX);
   return 0;
  }
 
  And to call that:
 
  static int bar(lua_State *L)
  {
   lua_pushinteger(L,myfuncref);
   lua_gettable(L,LUA_REGISTRYINDEX);
   lua_call(L,0,0);
   return 0;
  }
 
  Personally, I would store a reference in the Lua Registry as it won't
pollute the global space.

  -spc (This should be enough to get you started)


Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Nagaev Boris
In reply to this post by Brigham Toskin
>> now my question :
>> 1/ How can I store TFUNCTION at C side ?
>> 2/ How my C code can "call" this Lua function ?

There are several ways to do it:
 * registry (see Sean Conner's answer)
 * global variable (see Sean Conner's answer)
 * probably you use the function only when dealing with some other
function or a userdata object. In the former case you can put the
function to an upvalue of another function (see lua_pushcclosure). In
the latter case you can store the function in a field of the userdata
object's environment/user value (see lua_setfenv for Lua 5.1,
lua_setuservalue for Lua 5.2).

I'd prefer third method, because it doesn't involve changing global
state (global variables or registry).


--


Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Roberto Ierusalimschy
In reply to this post by Sean Conner
>   Another way is to store a reference in the Lua Registry:
>  
>   int myfuncref;
>  
>   static int foo(lua_State *L)
>   {
>    
>    luaL_checktype(L,1,LUA_TFUNCTION);
>    myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
>    lua_pushinteger(L,myfuncref);
>    lua_pushvalue(L,1);
>    lua_settable(L,LUA_REGISTRYINDEX);
>    return 0;
>   }

luaL_ref does all the work (see the manual). This function should be
like this:

  static int foo(lua_State *L) {
     luaL_checktype(L,1,LUA_TFUNCTION);
     myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
     return 0;
    }


>   And to call that:
>  
>   static int bar(lua_State *L)
>   {
>    lua_pushinteger(L,myfuncref);
>    lua_gettable(L,LUA_REGISTRYINDEX);
>    lua_call(L,0,0);
>    return 0;
>   }

(It is simpler to use 'lua_rawgeti'...)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Laurent FAILLIE
Ok, thanks to all for your replies.

I'll make some tests with the provided information and will post the result (if it can help someone else).

Thanks

Laurent----
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



Le Mercredi 3 juin 2015 14h19, Roberto Ierusalimschy <[hidden email]> a écrit :


>  Another way is to store a reference in the Lua Registry:


>      int myfuncref;
>     
>      static int foo(lua_State *L)
>      {
>       
>        luaL_checktype(L,1,LUA_TFUNCTION);
>        myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
>        lua_pushinteger(L,myfuncref);
>        lua_pushvalue(L,1);
>        lua_settable(L,LUA_REGISTRYINDEX);
>        return 0;
>      }

luaL_ref does all the work (see the manual). This function should be
like this:

      static int foo(lua_State *L) {
          luaL_checktype(L,1,LUA_TFUNCTION);
          myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
          return 0;
        }


>  And to call that:

>      static int bar(lua_State *L)
>      {
>        lua_pushinteger(L,myfuncref);
>        lua_gettable(L,LUA_REGISTRYINDEX);
>        lua_call(L,0,0);
>        return 0;
>      }

(It is simpler to use 'lua_rawgeti'...)

-- Roberto



Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Laurent FAILLIE
Hello,

I have another question related to Lua's stat which is expected to be passed as 1st arg for Lua's function : is it thread safe ?

My application handles MQTT messages in asynchronous mode meaning messages handling (and consequently the Lua callbacks I want to call for every arriving messages) is running in a separate thread ... but it is safe to use the same lua_stat ? what will happen if a message arrive whereas the main thread is already running a Lua function ?

Thanks

Laurent
 
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



Le Mercredi 3 juin 2015 15h03, Laurent FAILLIE <[hidden email]> a écrit :


Ok, thanks to all for your replies.

I'll make some tests with the provided information and will post the result (if it can help someone else).

Thanks

Laurent----
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



Le Mercredi 3 juin 2015 14h19, Roberto Ierusalimschy <[hidden email]> a écrit :


>  Another way is to store a reference in the Lua Registry:


>      int myfuncref;
>     
>      static int foo(lua_State *L)
>      {
>       
>        luaL_checktype(L,1,LUA_TFUNCTION);
>        myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
>        lua_pushinteger(L,myfuncref);
>        lua_pushvalue(L,1);
>        lua_settable(L,LUA_REGISTRYINDEX);
>        return 0;
>      }

luaL_ref does all the work (see the manual). This function should be
like this:

      static int foo(lua_State *L) {
          luaL_checktype(L,1,LUA_TFUNCTION);
          myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
          return 0;
        }


>  And to call that:

>      static int bar(lua_State *L)
>      {
>        lua_pushinteger(L,myfuncref);
>        lua_gettable(L,LUA_REGISTRYINDEX);
>        lua_call(L,0,0);
>        return 0;
>      }

(It is simpler to use 'lua_rawgeti'...)

-- Roberto





Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Brigham Toskin
If you mean the lua_State* that represents your running program, no it is not thread safe by default. There are probably a few kinds of things that are safe to do in combinations from two or more threads, but that is probably an implementation detail rather than a language feature.

For instance, I could imagine you safely reading a global from two threads if it doesn't do Bad Things™ to the stack (it might..?), but there's nothing to stop you from reading and writing the same global variable from two threads at the same time, which would be bad.

On Sat, Jun 6, 2015 at 9:50 AM, Laurent FAILLIE <[hidden email]> wrote:
Hello,

I have another question related to Lua's stat which is expected to be passed as 1st arg for Lua's function : is it thread safe ?

My application handles MQTT messages in asynchronous mode meaning messages handling (and consequently the Lua callbacks I want to call for every arriving messages) is running in a separate thread ... but it is safe to use the same lua_stat ? what will happen if a message arrive whereas the main thread is already running a Lua function ?

Thanks

Laurent
 
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



Le Mercredi 3 juin 2015 15h03, Laurent FAILLIE <[hidden email]> a écrit :


Ok, thanks to all for your replies.

I'll make some tests with the provided information and will post the result (if it can help someone else).

Thanks

Laurent----
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/               /(
/ Dico       / / Pleins d'autres fautes sur
/________/ /
(#######( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



Le Mercredi 3 juin 2015 14h19, Roberto Ierusalimschy <[hidden email]> a écrit :


>  Another way is to store a reference in the Lua Registry:


>      int myfuncref;
>     
>      static int foo(lua_State *L)
>      {
>       
>        luaL_checktype(L,1,LUA_TFUNCTION);
>        myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
>        lua_pushinteger(L,myfuncref);
>        lua_pushvalue(L,1);
>        lua_settable(L,LUA_REGISTRYINDEX);
>        return 0;
>      }

luaL_ref does all the work (see the manual). This function should be
like this:

      static int foo(lua_State *L) {
          luaL_checktype(L,1,LUA_TFUNCTION);
          myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
          return 0;
        }


>  And to call that:

>      static int bar(lua_State *L)
>      {
>        lua_pushinteger(L,myfuncref);
>        lua_gettable(L,LUA_REGISTRYINDEX);
>        lua_call(L,0,0);
>        return 0;
>      }

(It is simpler to use 'lua_rawgeti'...)

-- Roberto








--
Brigham Toskin
Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Nagaev Boris
In reply to this post by Laurent FAILLIE
On Sat, Jun 6, 2015 at 4:50 PM, Laurent FAILLIE <[hidden email]> wrote:
> Hello,
>
> I have another question related to Lua's stat which is expected to be passed
> as 1st arg for Lua's function : is it thread safe ?

No.

I use definitions of thread-safety and reentrancy from Qt documentation.

By default [*], Lua functions are reentrant (i.e., C functions could
be called safely with different Lua states by multiple threads at the
same moment), but not thread-safe (i.e., a Lua state must not be used
by two threads at the same moment).

You can use N Lua states in N threads (1 thread = 1 Lua state), but
you could not use 1 Lua state in N threads.

One Lua state can be used by multiple threads, but not at the same
moment. In this case a Lua state is "passed" from thread to thread.
Some applications use multiple OS threads, multiple Lua states, each
with multiple Lua threads. For example, nginx does so. It can spawn
multiple workers (= OS threads). Each worker starts a Lua state. Lua
states are shared by OS threads, but no Lua state is operated by two
OS threads at the same moment. When HTTP request is accepted, it is
passed to one of workers, starting a Lua thread. If then request
handler makes something asynchronous (i.e., database query), this Lua
thread is yield'ed. When that asynchronous task is finished, then the
Lua thread resumes the request handler. It can happen in any worker
though (but in same Lua state, of course). Selection of a worker
depends on availability of workers and on configuration. For instance,
it makes sense to wait for the worker which worked with this Lua
thread before to reuse processor cache.

Thus it is thread-safe to reuse Lua state (e.g., resume Lua thread) in
other OS thread.


> My application handles MQTT messages in asynchronous mode meaning messages
> handling (and consequently the Lua callbacks I want to call for every
> arriving messages) is running in a separate thread ... but it is safe to use
> the same lua_stat ? what will happen if a message arrive whereas the main
> thread is already running a Lua function ?

I hope I answered this part of your question as well.

By the way, consider using Lua threads instead of callbacks, as nginx
does. Lua code using nginx functions looks like synchronous code.

Compare:

  users = database.query "select * from users;"

with

  database.query("select * from users;", function(users)
    -- do smth with users
  end)

[*] You can achieve thread-safety by defining functions lua_lock and
lua_unlock, which are called by all Lua API functions.

>
> Thanks
>
> Laurent
>
> The misspelling master is on the Web.
>   _________ 100 % Dictionnary Free !
> /               /(
> / Dico       / / Pleins d'autres fautes sur
> /________/ /
> (#######( / http://destroyedlolo.info
> Quoi, des fautes d'orthographe! Pas possible ;-D.
>
>
>
> Le Mercredi 3 juin 2015 15h03, Laurent FAILLIE <[hidden email]> a écrit
> :
>
>
> Ok, thanks to all for your replies.
>
> I'll make some tests with the provided information and will post the result
> (if it can help someone else).
>
> Thanks
>
> Laurent----
> The misspelling master is on the Web.
>   _________ 100 % Dictionnary Free !
> /               /(
> / Dico       / / Pleins d'autres fautes sur
> /________/ /
> (#######( / http://destroyedlolo.info
> Quoi, des fautes d'orthographe! Pas possible ;-D.
>
>
>
> Le Mercredi 3 juin 2015 14h19, Roberto Ierusalimschy
> <[hidden email]> a écrit :
>
>
>>  Another way is to store a reference in the Lua Registry:
>>
>>      int myfuncref;
>>
>>      static int foo(lua_State *L)
>>      {
>>
>>        luaL_checktype(L,1,LUA_TFUNCTION);
>>        myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
>>        lua_pushinteger(L,myfuncref);
>>        lua_pushvalue(L,1);
>>        lua_settable(L,LUA_REGISTRYINDEX);
>>        return 0;
>>      }
>
> luaL_ref does all the work (see the manual). This function should be
> like this:
>
>       static int foo(lua_State *L) {
>           luaL_checktype(L,1,LUA_TFUNCTION);
>           myfuncref = luaL_ref(L,LUA_REGISTRYINDEX);
>           return 0;
>         }
>
>
>>  And to call that:
>>
>>      static int bar(lua_State *L)
>>      {
>>        lua_pushinteger(L,myfuncref);
>>        lua_gettable(L,LUA_REGISTRYINDEX);
>>        lua_call(L,0,0);
>>        return 0;
>>      }
>
> (It is simpler to use 'lua_rawgeti'...)
>
> -- Roberto
>
>
>
>
>



--


Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Laurent FAILLIE
Le Samedi 6 juin 2015 22h22, Nagaev Boris <[hidden email]> a écrit :



Thanks Boris for your reply. Searching for additional information on ideas you provided, I found this interesting page : http://lua-users.org/wiki/ThreadsTutorial

First of all, but if I rewrite entirely the MQTT stack, I have to keep my current implementation where Paho library spawn a new C thread to handle incoming messages. The question now is how to get the new Lua_stat ?

- I rejected the use of lua_open() as I need access to global variables and I don't want to reopen each and every libraries.

- so let's go with lua_newthread() : as per my understanding (let me know if I missed anything), it will not spawn really a new thread (in system point of view) but "only" create a kind of forked Lua stat allowing my thread to live its local life but still being able to interact with global environment.

But, as per this wiki, I'll have to create my own locking function, right ?

- I noticed also lua_newstate() but I wasn't able to determine if global variables are still accessible or not.


So, all in all, my new code may looks like something like :

struct _context {
    lua_State *L;
    ... other context stuffs, bla bla ...
};

int msgarrived(void *actx, char *topic, int tlen, MQTTClient_message *msg){
    struct _context *ctx = (struct _context *)actx;    // to avoid zillion of casting

    ...

    lua_pushnil(ctx->L);    // push nil in the alternate Lua stack

    ...
}

void myinit( lua_State *L){
    struct _context ctx;     // in fact, it will not be a stack object, but it's only an example

    ctx.L = lua_newthread( L);

    MQTTClient_setCallbacks( client, &ctx, connlost, msgarrived, NULL);
}

Best regards,

Laurent

Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Nagaev Boris
On Sat, Jun 6, 2015 at 10:55 PM, Laurent FAILLIE <[hidden email]> wrote:

> Le Samedi 6 juin 2015 22h22, Nagaev Boris <[hidden email]> a écrit :
>
>
>
> Thanks Boris for your reply. Searching for additional information on ideas
> you provided, I found this interesting page :
> http://lua-users.org/wiki/ThreadsTutorial
>
> First of all, but if I rewrite entirely the MQTT stack, I have to keep my
> current implementation where Paho library spawn a new C thread to handle
> incoming messages. The question now is how to get the new Lua_stat ?

Why to spawn new OS thread per incoming message? The most appealing
advantage of asynchronous networking is that you don't have to spawn
new OS thread per each incoming message. Start from this point: single
OS thread, single Lua state, multiple Lua threads. Maybe you don't
need OS threads at all.

If you should use multiple OS threads, then you can move to multiple
OS threads, multiple Lua states model. Single Lua state allows to use
global variables. For multiple Lua states you have to implement global
variables replacement yourself. See nginx's ngx.shared.DICT.

See also luajit.io [1], it looks like nginx rewritten in LuaJIT.

[1] https://github.com/kingluo/luajit.io


> - I rejected the use of lua_open() as I need access to global variables and
> I don't want to reopen each and every libraries.

To open or not to open libraries is a separate issue.

> - so let's go with lua_newthread() : as per my understanding (let me know if
> I missed anything), it will not spawn really a new thread (in system point
> of view) but "only" create a kind of forked Lua stat allowing my thread to
> live its local life but still being able to interact with global
> environment.
>
> But, as per this wiki, I'll have to create my own locking function, right ?

lua_newthread creates Lua thread (aka coroutine, green thread). Lua
thread is a part of Lua state so it can access global variables.
Remember that Lua state itself is not thread-safe, that is why
multiple Lua threads started from same Lua state must not be touched
by multiple OS threads at the same moment.

Actually, you have to implement lua_lock and lua_unlock if you really
want this, but it is bad idea, because such locking would nullify
performance advantage of multiple threads - most time is spent inside
Lua API calls which are mutually exclusive locked. In practice this
scheme can work even slower than one thread program.


> - I noticed also lua_newstate() but I wasn't able to determine if global
> variables are still accessible or not.

Global variables are not accessible.

luaL_newstate / lua_newstate create independent Lua state. Global
variables are not inherited. Nothing is inherited. They are completely
independent!


>
>
> So, all in all, my new code may looks like something like :
>
> struct _context {
>     lua_State *L;
>     ... other context stuffs, bla bla ...
> };
>
> int msgarrived(void *actx, char *topic, int tlen, MQTTClient_message *msg){
>     struct _context *ctx = (struct _context *)actx;    // to avoid zillion
> of casting
>
>     ...
>
>     lua_pushnil(ctx->L);    // push nil in the alternate Lua stack
>
>     ...
> }
>
> void myinit( lua_State *L){
>     struct _context ctx;     // in fact, it will not be a stack object, but
> it's only an example
>
>     ctx.L = lua_newthread( L);
>
>     MQTTClient_setCallbacks( client, &ctx, connlost, msgarrived, NULL);
> }
>
> Best regards,
>
> Laurent
>



--


Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Brigham Toskin
In reply to this post by Nagaev Boris
On Sat, Jun 6, 2015 at 1:21 PM, Nagaev Boris <[hidden email]> wrote:
You can use N Lua states in N threads (1 thread = 1 Lua state), but
you could not use 1 Lua state in N threads.

Do you know if this applies to the "daughter states" created by coroutine threads? Is something like this e.g. what Lanes does?
 
--
Brigham Toskin
Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Brigham Toskin
... assuming you didn't modify any globals.

On Sat, Jun 6, 2015 at 5:25 PM, Brigham Toskin <[hidden email]> wrote:
On Sat, Jun 6, 2015 at 1:21 PM, Nagaev Boris <[hidden email]> wrote:
You can use N Lua states in N threads (1 thread = 1 Lua state), but
you could not use 1 Lua state in N threads.

Do you know if this applies to the "daughter states" created by coroutine threads? Is something like this e.g. what Lanes does?
 
--
Brigham Toskin



--
Brigham Toskin
Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Coda Highland
In reply to this post by Brigham Toskin
On Sat, Jun 6, 2015 at 5:25 PM, Brigham Toskin <[hidden email]> wrote:

> On Sat, Jun 6, 2015 at 1:21 PM, Nagaev Boris <[hidden email]> wrote:
>>
>> You can use N Lua states in N threads (1 thread = 1 Lua state), but
>> you could not use 1 Lua state in N threads.
>
>
> Do you know if this applies to the "daughter states" created by coroutine
> threads? Is something like this e.g. what Lanes does?
>
> --
> Brigham Toskin

Coroutines aren't a separate global state, so those aren't threadsafe.

Lanes fires up completely separate global Lua states and has its own
communication system between them.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

RE: Need some help to use TFUNCTION

Thijs Schreijer
In reply to this post by Laurent FAILLIE

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Laurent FAILLIE
> Sent: zaterdag 6 juni 2015 18:51
> To: Lua mailing list
> Subject: Re: Need some help to use TFUNCTION
>
> Hello,
>
>
> I have another question related to Lua's stat which is expected to be passed
> as 1st arg for Lua's function : is it thread safe ?
>
> My application handles MQTT messages in asynchronous mode meaning messages
> handling (and consequently the Lua callbacks I want to call for every
> arriving messages) is running in a separate thread ... but it is safe to use
> the same lua_stat ? what will happen if a message arrive whereas the main
> thread is already running a Lua function ?
>
> Thanks
>
> Laurent
>
>

The naming of some elements in the C api is a bit misleading. Personally I wish that all wording with "thread" would be changed to "coroutine", and the lua_State would be renamed to "executionstack" or something more appropriate.

Have a look here [1] for some "definitions" I came up with when I started with multi-threading and wrote my DarkSideSync library. To deal with multithreading there are a number of libraries that handle things for Lua
- Lanes [2] (create new states and threads from Lua)
- Rings [3] (lighter than Lanes, but also lesser capabilities)
- llthreads [4]
- darksidesync [5] (a helper library to synchronise background tasks/os-threads back to a single Lua state)
- ZMQ [6] (or equivalents) to communicate between multiple states (to work around lacking shared global data)

I'm not familiar with Paho, but how is is used? Is the main loop on the C side and are messages passed to Lua to be handled with scripting, or is the main loop on the Lua side? Does Paho create the new os threads? Do you have control over that behaviour? Does it also generate new Lua states?


Thijs

[1] http://www.thijsschreijer.nl/blog/?p=693
[2] http://cmr.github.io/lanes/ 
[3] http://keplerproject.github.io/rings/ 
[4] https://github.com/Neopallium/lua-llthreads 
[5] https://github.com/Tieske/DarkSideSync 
[6] http://zeromq.org/bindings:lua 


Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Laurent FAILLIE
In reply to this post by Nagaev Boris



Le Dimanche 7 juin 2015 1h38, Nagaev Boris <[hidden email]> a écrit :


On Sat, Jun 6, 2015 at 10:55 PM, Laurent FAILLIE <[hidden email]> wrote:

> Why to spawn new OS thread per incoming message?

Sorry, I wasn't clean : not per incoming message.
The main program and the incoming loop are running in separate thread meaning msg handling is done in background whatever the main program is doing : the old good preemptive mode. But messages callbacks are running within this second thread.

> See also luajit.io [1], it looks like nginx rewritten in LuaJIT.

I will, thanks for the link.

> lua_newthread creates Lua thread (aka coroutine, green thread).

Cooperative multitasking and not preemptive multi-tasking, right ?

> Lua thread is a part of Lua state so it can access global variables.
> Remember that Lua state itself is not thread-safe, that is why
> multiple Lua threads started from same Lua state must not be touched
> by multiple OS threads at the same moment.

So it's not what I'm looking for.

> luaL_newstate / lua_newstate create independent Lua state. Global
> variables are not inherited. Nothing is inherited. They are completely
> independent!

So my "this morning" idea : I'll use lua_newstate() so my Lua part will be thread safe (not the same Lua_state at all). This will prevent as well callback to call global stuffs. And I'll create some C helpers to do IPC.

Thanks for your patience and clear explanation

Laurent
 
Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Laurent FAILLIE
In reply to this post by Thijs Schreijer
Le Dimanche 7 juin 2015 11h15, Thijs Schreijer <[hidden email]> a écrit :



> The naming of some elements in the C api is a bit misleading. Personally I wish that all wording with "thread"
> would be changed to "coroutine", and the lua_State would be renamed to "executionstack" or something
> more appropriate.

Yes, I totally second that, it's a bit confusing, especially because Lua's thread aren't preemptive.

Thanks for the links, I'll check them if I can't succeed with my current thinking;

> I'm not familiar with Paho, but how is is used? Is the main loop on the C side and are messages passed to Lua
> to be handled with scripting, or is the main loop on the Lua side? Does Paho create the new os threads?
> Do you have control over that behaviour? Does it also generate new Lua states?

Paho ( https://eclipse.org/paho/ ) is a messaging library implementing MQTT protocol, mostly associated with IoT stuffs.
It supports several implementations : Java, C, C++ and even a native Lua library which is not finished up to now.
I'm using the C implementation, which has nothing to do with Lua at all.

My project is the following :

- I have several daemons running in my basement server to do "smart housing", and monitoring several figures of my home : electricity consumption and production, temperature in rooms, pool, health of my Internet connection, and so on and so forth ... https://github.com/destroyedlolo/TeleInfod and https://github.com/destroyedlolo/Marcel are some example of such daemon.
- Daemons and my supervision tool are exchanging data using MQTT messaging : the big advantage is it runs in pub/sub model so I can add / remove / modify any part of my network without touching the remaining.

Now, I'm working now on the "home dashboard" part which will plug-in on this network and will display some valuables informations on a screen on my living room.

- The main work is done in C and takes in charge all the dirty / low level work. This technical layer is https://github.com/destroyedlolo/Selene
- Selene embeds Lua as end user scripting language, which will be used to build the final dashboard (has to be done). And to respond to your question, yes, messages are handled in C then passed to Lua part for end user oriented processing.

The main advantage is a clear separation b/w the technical layers and the end user interface itself.

- Laurent


Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Laurent FAILLIE
So here I am : the full source code can be found https://github.com/destroyedlolo/Selene/blob/master/src/MQTT.c

1/ Environment creation :

struct enhanced_client {
    MQTTClient client;
    lua_State *L;
    struct _topic *subscriptions;
};

...

        /* Creating Lua data */
    eclient = (struct enhanced_client *)lua_newuserdata(L, sizeof(struct enhanced_client));
    luaL_getmetatable(L, "SelMQTT");
    lua_setmetatable(L, -2);
    eclient->subscriptions = NULL;
    eclient->L = luaL_newstate();
    luaL_openlibs(eclient->L);
...

2/ registering the callback

...
        lua_pushstring(L, "func");    /* Argument cames from a table */
        lua_gettable(L, -2);
        if( lua_type(L, -1) != LUA_TFUNCTION ){
            lua_pop(L, 1);    /* Pop the result */
            lua_pushnil(L);
            lua_pushstring(L, "Subscribe() : topics needs associated function");
            return 2;
        }
        lua_xmove( L, eclient->L, 1 );    /* Move the function to the callback's stack */
        func = luaL_ref(eclient->L,LUA_REGISTRYINDEX);    /* Reference the function in callbacks' context */
...

3/ calling the callback function

 int msgarrived(void *actx, char *topic, int tlen, MQTTClient_message *msg){
/* handle message arrival and call associated function.
 * NOTE : up to now, only textual topics & messages are
 * correctly handled (lengths are simply ignored)
 */
    struct enhanced_client *ctx = actx;    /* To avoid numerous cast */
    struct _topic *tp;

    for(tp = ctx->subscriptions; tp; tp = tp->next){    /* Looks for the corresponding function */
        if(!strcmp(tp->topic, topic)){    /* AF : wildcard to be done */
            lua_rawgeti( ctx->L, LUA_REGISTRYINDEX, tp->func);    /* retrieves the function */
            lua_pushstring( ctx->L, topic);
            lua_pushstring( ctx->L, msg->payload);
            lua_pcall( ctx->L, 2, 0, 0);    /* Call Lua callback function */
        }
    }

    MQTTClient_freeMessage(&msg);
    MQTTClient_free(topic);
    return 1;
}


print '\nTesting broker connection'
print '--------------------------'

-- Callbacks
function handle_tata( topic, msg )
    print("Lua received t:'" .. topic .."' m:'".. msg .. "'\n");
end


function handle_toto( topic, msg )
    print("Lua received t:'" .. topic .."' m:'".. msg .. "'\n");
end

-- Connection, subscription and finally waiting for messages
Brk, err = SelMQTT.connect( "tcp://localhost:1883", { reliable=false  } )
if not Brk then
    print( err )
    return
end

_, err = Brk:subscribe( { { topic = "/tata/#", func=handle_tata, qos=SelMQTT.QoSConst("QoS0") }, { topic = "/toto",func=handle_toto } } )
if err then
    print( err )
    return
end

print "Hit 'enter' key to exit"
io.stdin:read'*l'    -- wait for enter

And it seems it's working pretty well.
If I enter in another shell :
---
torchwood ~ # mosquitto_pub -h torchwood.local -t '/toto' -m 'Coucou'
---

I got 
---
torchwood Selene # Selenites/MQTT.sel
Testing broker connection
--------------------------
Hit 'enter' key to exit
Lua received t:'/toto' m:'Coucou'
---

Can someone having a look and tell me if I made a mistake ?

Now, I'll have to work data shared b/w threads :)

Thanks for all.

Laurent

---
The misspelling master is on the Web.
  _________ 100 % Dictionnary Free !
/                 /(
/ Dico        / / Pleins d'autres fautes sur
/________ / /
(########( / http://destroyedlolo.info
Quoi, des fautes d'orthographe! Pas possible ;-D.



Le Dimanche 7 juin 2015 12h01, Laurent FAILLIE <[hidden email]> a écrit :


Le Dimanche 7 juin 2015 11h15, Thijs Schreijer <[hidden email]> a écrit :



> The naming of some elements in the C api is a bit misleading. Personally I wish that all wording with "thread"
> would be changed to "coroutine", and the lua_State would be renamed to "executionstack" or something
> more appropriate.

Yes, I totally second that, it's a bit confusing, especially because Lua's thread aren't preemptive.

Thanks for the links, I'll check them if I can't succeed with my current thinking;

> I'm not familiar with Paho, but how is is used? Is the main loop on the C side and are messages passed to Lua
> to be handled with scripting, or is the main loop on the Lua side? Does Paho create the new os threads?
> Do you have control over that behaviour? Does it also generate new Lua states?

Paho ( https://eclipse.org/paho/ ) is a messaging library implementing MQTT protocol, mostly associated with IoT stuffs.
It supports several implementations : Java, C, C++ and even a native Lua library which is not finished up to now.
I'm using the C implementation, which has nothing to do with Lua at all.

My project is the following :

- I have several daemons running in my basement server to do "smart housing", and monitoring several figures of my home : electricity consumption and production, temperature in rooms, pool, health of my Internet connection, and so on and so forth ... https://github.com/destroyedlolo/TeleInfod and https://github.com/destroyedlolo/Marcel are some example of such daemon.
- Daemons and my supervision tool are exchanging data using MQTT messaging : the big advantage is it runs in pub/sub model so I can add / remove / modify any part of my network without touching the remaining.

Now, I'm working now on the "home dashboard" part which will plug-in on this network and will display some valuables informations on a screen on my living room.

- The main work is done in C and takes in charge all the dirty / low level work. This technical layer is https://github.com/destroyedlolo/Selene
- Selene embeds Lua as end user scripting language, which will be used to build the final dashboard (has to be done). And to respond to your question, yes, messages are handled in C then passed to Lua part for end user oriented processing.

The main advantage is a clear separation b/w the technical layers and the end user interface itself.

- Laurent




Reply | Threaded
Open this post in threaded view
|

Re: Need some help to use TFUNCTION

Nagaev Boris
In reply to this post by Brigham Toskin
On Sun, Jun 7, 2015 at 12:25 AM, Brigham Toskin <[hidden email]> wrote:

> On Sat, Jun 6, 2015 at 1:21 PM, Nagaev Boris <[hidden email]> wrote:
>>
>> You can use N Lua states in N threads (1 thread = 1 Lua state), but
>> you could not use 1 Lua state in N threads.
>
>
> Do you know if this applies to the "daughter states" created by coroutine
> threads? Is something like this e.g. what Lanes does?
>
> --
> Brigham Toskin

"Daughter state" is a part of original Lua state, so all limitations
apply: you can't use them in multiple OS threads at the same moment.


--


Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

RE: Need some help to use TFUNCTION

Thijs Schreijer
In reply to this post by Laurent FAILLIE


> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Laurent FAILLIE
> Sent: zondag 7 juni 2015 14:55
> To: Lua mailing list
> Subject: Re: Need some help to use TFUNCTION
>
> So here I am : the full source code can be found
> https://github.com/destroyedlolo/Selene/blob/master/src/MQTT.c
>
>
> 1/ Environment creation :
>
>
> struct enhanced_client {
>     MQTTClient client;
>     lua_State *L;
>     struct _topic *subscriptions;
> };
>
> ...
>
>
>         /* Creating Lua data */
>     eclient = (struct enhanced_client *)lua_newuserdata(L, sizeof(struct
> enhanced_client));
>     luaL_getmetatable(L, "SelMQTT");
>     lua_setmetatable(L, -2);
>     eclient->subscriptions = NULL;
>     eclient->L = luaL_newstate();
>     luaL_openlibs(eclient->L);
> ...
>
>
> 2/ registering the callback
>
> ...
>
>         lua_pushstring(L, "func");    /* Argument cames from a table */
>         lua_gettable(L, -2);
>         if( lua_type(L, -1) != LUA_TFUNCTION ){
>             lua_pop(L, 1);    /* Pop the result */
>             lua_pushnil(L);
>             lua_pushstring(L, "Subscribe() : topics needs associated
> function");
>             return 2;
>         }
>         lua_xmove( L, eclient->L, 1 );    /* Move the function to the
> callback's stack */
>         func = luaL_ref(eclient->L,LUA_REGISTRYINDEX);    /* Reference the
> function in callbacks' context */
> ...
>
>
> 3/ calling the callback function
>
>
>  int msgarrived(void *actx, char *topic, int tlen, MQTTClient_message *msg){
> /* handle message arrival and call associated function.
>  * NOTE : up to now, only textual topics & messages are
>  * correctly handled (lengths are simply ignored)
>  */
>     struct enhanced_client *ctx = actx;    /* To avoid numerous cast */
>     struct _topic *tp;
>
>     for(tp = ctx->subscriptions; tp; tp = tp->next){    /* Looks for the
> corresponding function */
>         if(!strcmp(tp->topic, topic)){    /* AF : wildcard to be done */
>             lua_rawgeti( ctx->L, LUA_REGISTRYINDEX, tp->func);    /*
> retrieves the function */
>             lua_pushstring( ctx->L, topic);
>             lua_pushstring( ctx->L, msg->payload);
>             lua_pcall( ctx->L, 2, 0, 0);    /* Call Lua callback function */
>         }
>     }
>
>     MQTTClient_freeMessage(&msg);
>     MQTTClient_free(topic);
>     return 1;
> }
>
> 4/ Finally, a Lua example :
> https://github.com/destroyedlolo/Selene/blob/master/Selenites/MQTT.sel
>
> print '\nTesting broker connection'
> print '--------------------------'
>
> -- Callbacks
> function handle_tata( topic, msg )
>     print("Lua received t:'" .. topic .."' m:'".. msg .. "'\n");
> end
>
>
> function handle_toto( topic, msg )
>     print("Lua received t:'" .. topic .."' m:'".. msg .. "'\n");
> end
>
> -- Connection, subscription and finally waiting for messages
> Brk, err = SelMQTT.connect( "tcp://localhost:1883", { reliable=false  } )
> if not Brk then
>     print( err )
>     return
> end
>
> _, err = Brk:subscribe( { { topic = "/tata/#", func=handle_tata,
> qos=SelMQTT.QoSConst("QoS0") }, { topic = "/toto",func=handle_toto } } )
> if err then
>     print( err )
>     return
> end
>
> print "Hit 'enter' key to exit"
> io.stdin:read'*l'    -- wait for enter
>
> And it seems it's working pretty well.
> If I enter in another shell :
> ---
> torchwood ~ # mosquitto_pub -h torchwood.local -t '/toto' -m 'Coucou'
> ---
>
> I got
> ---
> torchwood Selene # Selenites/MQTT.sel
> Testing broker connection
> --------------------------
> Hit 'enter' key to exit
> Lua received t:'/toto' m:'Coucou'
> ---
>
> Can someone having a look and tell me if I made a mistake ?

After a quick glance; it's just single state code. If you mix (preemptive os) threads in this state, it will fail sooner or later.

From what I understand; you have a C program, which embeds Lua, and the Paho library. The Paho library creates a separate background OS thread for the async handling of incoming data. So you have two OS threads that might access the Lua states simultaneously, which requires synchronisation.

>
> Now, I'll have to work data shared b/w threads :)

Yes you have to, see remark above...

If you create multiple States, and sync between them using mechanism X, you might as well directly use mechanism X from the Paho callbacks and use a single Lua state. No?

Thijs

>
> Thanks for all.
>
> Laurent
>
> ---
> The misspelling master is on the Web.
>   _________ 100 % Dictionnary Free !
> /                 /(
> / Dico        / / Pleins d'autres fautes sur
> /________ / /
> (########( / http://destroyedlolo.info
> Quoi, des fautes d'orthographe! Pas possible ;-D.
>
>
> Le Dimanche 7 juin 2015 12h01, Laurent FAILLIE <[hidden email]> a écrit
> :
>
> Le Dimanche 7 juin 2015 11h15, Thijs Schreijer <[hidden email]> a
> écrit :
>
>
> > The naming of some elements in the C api is a bit misleading. Personally I
> wish that all wording with "thread"
> > would be changed to "coroutine", and the lua_State would be renamed to
> "executionstack" or something
> > more appropriate.
>
> Yes, I totally second that, it's a bit confusing, especially because Lua's
> thread aren't preemptive.
>
> Thanks for the links, I'll check them if I can't succeed with my current
> thinking;
>
> > I'm not familiar with Paho, but how is is used? Is the main loop on the C
> side and are messages passed to Lua
> > to be handled with scripting, or is the main loop on the Lua side? Does
> Paho create the new os threads?
> > Do you have control over that behaviour? Does it also generate new Lua
> states?
>
> Paho ( https://eclipse.org/paho/ ) is a messaging library implementing MQTT
> protocol, mostly associated with IoT stuffs.
> It supports several implementations : Java, C, C++ and even a native Lua
> library which is not finished up to now.
> I'm using the C implementation, which has nothing to do with Lua at all.
>
> My project is the following :
>
> - I have several daemons running in my basement server to do "smart
> housing", and monitoring several figures of my home : electricity
> consumption and production, temperature in rooms, pool, health of my
> Internet connection, and so on and so forth ...
> https://github.com/destroyedlolo/TeleInfod and
> https://github.com/destroyedlolo/Marcel are some example of such daemon.
> - Daemons and my supervision tool are exchanging data using MQTT messaging :
> the big advantage is it runs in pub/sub model so I can add / remove / modify
> any part of my network without touching the remaining.
>
> Now, I'm working now on the "home dashboard" part which will plug-in on this
> network and will display some valuables informations on a screen on my
> living room.
>
> - The main work is done in C and takes in charge all the dirty / low level
> work. This technical layer is https://github.com/destroyedlolo/Selene
> - Selene embeds Lua as end user scripting language, which will be used to
> build the final dashboard (has to be done). And to respond to your question,
> yes, messages are handled in C then passed to Lua part for end user oriented
> processing.
>
> The main advantage is a clear separation b/w the technical layers and the
> end user interface itself.
>
> - Laurent
>
>

12