Lua callback in another thread

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua callback in another thread

Laurent FAILLIE
Hello,

I have a C multi-threaded program using Lua as user end scripting language ( https://github.com/destroyedlolo/Selene , I already presented some months back :) ).

My main thread is used also to declare some Lua callbacks needs to be called when an MQTT data arrived. So I keep a reference to this function using classical luaL_ref().

Now, my problem : an MQTT data arrives and is handled by another thread, on which I need to created a dedicated Lua_State for this newly created thread. But how can I retrieve then launch this function in the new lua_state.
Because, for the moment, I'm retrieving the function from the main state, using xmove() it to the new one and this call it ... obviously, it's crashing as the main state may be busy by other tasks.
In addition, I don't want to block the main thread using mutex as I don't want to have to rely on main thead activities.

So any idea welcome :)

Thanks

Laurent


Reply | Threaded
Open this post in threaded view
|

RE: Lua callback in another thread

Thijs Schreijer

I had the same issue long ago with the PUPnP library, hence I wrote “darksidesync” [1], which signals the main thread using a udp packet (or just by polling). Not the most elegant solution, but works and is very portable (though I think the Windows build is broken right now)

 

Thijs

 

[1] http://tieske.github.io/DarkSideSync/topics/readme.md.html

 

Van: [hidden email] [mailto:[hidden email]] Namens Laurent FAILLIE
Verzonden: Wednesday, 20 April, 2016 11:33
Aan: Lua Mailing List
Onderwerp: Lua callback in another thread

 

Hello,

 

I have a C multi-threaded program using Lua as user end scripting language ( https://github.com/destroyedlolo/Selene , I already presented some months back :) ).

 

My main thread is used also to declare some Lua callbacks needs to be called when an MQTT data arrived. So I keep a reference to this function using classical luaL_ref().

 

Now, my problem : an MQTT data arrives and is handled by another thread, on which I need to created a dedicated Lua_State for this newly created thread. But how can I retrieve then launch this function in the new lua_state.

Because, for the moment, I'm retrieving the function from the main state, using xmove() it to the new one and this call it ... obviously, it's crashing as the main state may be busy by other tasks.

In addition, I don't want to block the main thread using mutex as I don't want to have to rely on main thead activities.

 

So any idea welcome :)

 

Thanks

 

Laurent

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Daurnimator
On 22 April 2016 at 15:08, Thijs Schreijer <[hidden email]> wrote:

> I had the same issue long ago with the PUPnP library, hence I wrote
> “darksidesync” [1], which signals the main thread using a udp packet (or
> just by polling). Not the most elegant solution, but works and is very
> portable (though I think the Windows build is broken right now)
>
>
>
> Thijs
>
>
>
> [1] http://tieske.github.io/DarkSideSync/topics/readme.md.html
>
>
>
> Van: [hidden email] [mailto:[hidden email]] Namens
> Laurent FAILLIE
> Verzonden: Wednesday, 20 April, 2016 11:33
> Aan: Lua Mailing List
> Onderwerp: Lua callback in another thread
>
>
>
> Hello,
>
>
>
> I have a C multi-threaded program using Lua as user end scripting language (
> https://github.com/destroyedlolo/Selene , I already presented some months
> back :) ).
>
>
>
> My main thread is used also to declare some Lua callbacks needs to be called
> when an MQTT data arrived. So I keep a reference to this function using
> classical luaL_ref().
>
>
>
> Now, my problem : an MQTT data arrives and is handled by another thread, on
> which I need to created a dedicated Lua_State for this newly created thread.
> But how can I retrieve then launch this function in the new lua_state.
>
> Because, for the moment, I'm retrieving the function from the main state,
> using xmove() it to the new one and this call it ... obviously, it's
> crashing as the main state may be busy by other tasks.
>
> In addition, I don't want to block the main thread using mutex as I don't
> want to have to rely on main thead activities.
>
>
>
> So any idea welcome :)

Infact, sending a packet to a socket *is* actually the best option for
this sort of thing.
On linux, you have a special fast kernel provided file descriptor type
'eventfd' just for this purpose.
In general, you can fall back to using a pipe() or socketpair() and
writing a single byte so that the other end polls readable.

Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Victor Bombi
In reply to this post by Laurent FAILLIE

why not use lanes?: It is already done and controls thread creation and thread intercomunication.
----- Original Message -----
Sent: Wednesday, April 20, 2016 11:33 AM
Subject: Lua callback in another thread

Hello,

I have a C multi-threaded program using Lua as user end scripting language ( https://github.com/destroyedlolo/Selene , I already presented some months back :) ).

My main thread is used also to declare some Lua callbacks needs to be called when an MQTT data arrived. So I keep a reference to this function using classical luaL_ref().

Now, my problem : an MQTT data arrives and is handled by another thread, on which I need to created a dedicated Lua_State for this newly created thread. But how can I retrieve then launch this function in the new lua_state.
Because, for the moment, I'm retrieving the function from the main state, using xmove() it to the new one and this call it ... obviously, it's crashing as the main state may be busy by other tasks.
In addition, I don't want to block the main thread using mutex as I don't want to have to rely on main thead activities.

So any idea welcome :)

Thanks

Laurent


Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Viacheslav Usov
In reply to this post by Laurent FAILLIE
On Wed, Apr 20, 2016 at 11:33 AM, Laurent FAILLIE <[hidden email]> wrote:
Hello,

I have a C multi-threaded program using Lua as user end scripting language ( https://github.com/destroyedlolo/Selene , I already presented some months back :) ).

My main thread is used also to declare some Lua callbacks needs to be called when an MQTT data arrived. So I keep a reference to this function using classical luaL_ref().

Now, my problem : an MQTT data arrives and is handled by another thread, on which I need to created a dedicated Lua_State for this newly created thread. But how can I retrieve then launch this function in the new lua_state.
Because, for the moment, I'm retrieving the function from the main state, using xmove() it to the new one and this call it ... obviously, it's crashing as the main state may be busy by other tasks.
In addition, I don't want to block the main thread using mutex as I don't want to have to rely on main thead activities.

So any idea welcome :)


Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Laurent FAILLIE
Hi Viacheslav,

I read this page but unfortunately, it only explains how to lock and mutex b/w thread, not how to share a function reference w/o locking.

Best regards,

Laurent


Le Vendredi 22 avril 2016 10h42, Viacheslav Usov <[hidden email]> a écrit :


On Wed, Apr 20, 2016 at 11:33 AM, Laurent FAILLIE <[hidden email]> wrote:
Hello,

I have a C multi-threaded program using Lua as user end scripting language ( https://github.com/destroyedlolo/Selene , I already presented some months back :) ).

My main thread is used also to declare some Lua callbacks needs to be called when an MQTT data arrived. So I keep a reference to this function using classical luaL_ref().

Now, my problem : an MQTT data arrives and is handled by another thread, on which I need to created a dedicated Lua_State for this newly created thread. But how can I retrieve then launch this function in the new lua_state.
Because, for the moment, I'm retrieving the function from the main state, using xmove() it to the new one and this call it ... obviously, it's crashing as the main state may be busy by other tasks.
In addition, I don't want to block the main thread using mutex as I don't want to have to rely on main thead activities.

So any idea welcome :)


Cheers,
V.



Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Laurent FAILLIE
In reply to this post by Victor Bombi
Hi Victor,

Le 22/04/2016 09:57, Victor Bombi a écrit :
why not use lanes?: It is already done and controls thread creation and thread intercomunication.


I had a look on Lanes.
But it seems to me it isn't able to handle thread that are created externally.
I mean, I can't let Lanes creates threads : they are created by Paho MQTT library when messages arrives.
Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Viacheslav Usov
In reply to this post by Laurent FAILLIE
On Fri, Apr 22, 2016 at 12:31 PM, Laurent FAILLIE <[hidden email]> wrote:

I read this page but unfortunately, it only explains how to lock and mutex b/w thread, not how to share a function reference w/o locking.

The page explains how you could use one Lua environment from multiple threads. It does need a mutex to protect its internal structures, but that does not make you "rely on main thread activities". With this approach, both of your threads can call lua_pcall with the same Lua environment concurrently, meaning that callbacks can run concurrently with the main thread.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Laurent FAILLIE
Le 22/04/2016 14:08, Viacheslav Usov a écrit :
On Fri, Apr 22, 2016 at 12:31 PM, Laurent FAILLIE <[hidden email]> wrote:

I read this page but unfortunately, it only explains how to lock and mutex b/w thread, not how to share a function reference w/o locking.

The page explains how you could use one Lua environment from multiple threads. It does need a mutex to protect its internal structures, but that does not make you "rely on main thread activities". With this approach, both of your threads can call lua_pcall with the same Lua environment concurrently, meaning that callbacks can run concurrently with the main thread.
Not concurrently as this page explains : "The method above for implementing threading in Lua using a global mutex is inefficient on multiprocessor systems. As one thread holds the global mutex, other threads are waiting for it. Thus only one Lua thread may be running at a time, regardless of the number of processors in the system."
So, if I have a long running thread, it will block others to run as well.

The only way is to have a dedicated state per thread ... but my problem is to pass function reference to them.
Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Viacheslav Usov
On Fri, Apr 22, 2016 at 4:50 PM, Laurent Faillie <[hidden email]> wrote:

> So, if I have a long running thread, it will block others to run as well.

Only if the thread runs pure Lua without C/Lua transitions (that includes calling Lua's standard library functions) and never does string/table manipulations. I doubt your "main thread" is like that.

But, of course, you know better.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Victor Bombi
In reply to this post by Laurent FAILLIE

I used it for luaRtAudio.
Created a lane with a function that was going to be called by a callback thread from RtAudio.
<A href="mhtml:{B286F767-57C0-4A44-9BB9-C21E2D97B311}mid://00000486/!x-usc:https://github.com/sonoro1234/luaRtAudio">https://github.com/sonoro1234/luaRtAudio
This lane had to be blocked to avoid concurrent use as can be seen in RtAudio.lua
----- Original Message -----
Sent: Friday, April 22, 2016 1:13 PM
Subject: Re: Lua callback in another thread

Hi Victor,

Le 22/04/2016 09:57, Victor Bombi a écrit :
why not use lanes?: It is already done and controls thread creation and thread intercomunication.


I had a look on Lanes.
But it seems to me it isn't able to handle thread that are created externally.
I mean, I can't let Lanes creates threads : they are created by Paho MQTT library when messages arrives.
Reply | Threaded
Open this post in threaded view
|

RE: Lua callback in another thread

Thijs Schreijer
In reply to this post by Daurnimator


> -----Oorspronkelijk bericht-----
> Van: [hidden email] [mailto:[hidden email]]
> Namens Daurnimator
> Verzonden: Friday, 22 April, 2016 8:52
> Aan: Lua mailing list
> Onderwerp: Re: Lua callback in another thread
>
> On 22 April 2016 at 15:08, Thijs Schreijer <[hidden email]>
> wrote:
> > I had the same issue long ago with the PUPnP library, hence I wrote
> > “darksidesync” [1], which signals the main thread using a udp packet
> > (or just by polling). Not the most elegant solution, but works and is
> > very portable (though I think the Windows build is broken right now)
> >
> >
> >
> > Thijs
> >
> >
> >
> > [1] http://tieske.github.io/DarkSideSync/topics/readme.md.html
> >
> >
> >
> > Van: [hidden email] [mailto:[hidden email]]
> > Namens Laurent FAILLIE
> > Verzonden: Wednesday, 20 April, 2016 11:33
> > Aan: Lua Mailing List
> > Onderwerp: Lua callback in another thread
> >
> >
> >
> > Hello,
> >
> >
> >
> > I have a C multi-threaded program using Lua as user end scripting
> > language ( https://github.com/destroyedlolo/Selene , I already
> > presented some months back :) ).
> >
> >
> >
> > My main thread is used also to declare some Lua callbacks needs to be
> > called when an MQTT data arrived. So I keep a reference to this
> > function using classical luaL_ref().
> >
> >
> >
> > Now, my problem : an MQTT data arrives and is handled by another
> > thread, on which I need to created a dedicated Lua_State for this newly
> created thread.
> > But how can I retrieve then launch this function in the new lua_state.
> >
> > Because, for the moment, I'm retrieving the function from the main
> > state, using xmove() it to the new one and this call it ... obviously,
> > it's crashing as the main state may be busy by other tasks.
> >
> > In addition, I don't want to block the main thread using mutex as I
> > don't want to have to rely on main thead activities.
> >
> >
> >
> > So any idea welcome :)
>
> Infact, sending a packet to a socket *is* actually the best option for
> this sort of thing.
> On linux, you have a special fast kernel provided file descriptor type
> 'eventfd' just for this purpose.
> In general, you can fall back to using a pipe() or socketpair() and
> writing a single byte so that the other end polls readable.

The way I generally use DarkSideSync, is with Copas as a scheduler, with an event driven paradigm. All code is scheduled as tasks, combined with timers and socket events.
The incoming mqtt stuff would just be another socket event in your case.

Whether or not you can implement it in your application I don't know, but my first guess would be to decouple the main application from the Lua state as well. Or stated otherwise, run a Lua state to do your Lua stuff and feed it with events from both the mqtt stack as well as the main application, both through DSS.

It works best with an event driven paradigm.

Btw; Another example, the GPIO interface for the raspberry pi [1] also uses DSS for background event triggering from foreign threads.

[1] GPIO Lua; https://github.com/Tieske/rpi-gpio/blob/master/lua/README.md 
[2] example code; https://github.com/Tieske/rpi-gpio/blob/master/lua/scripts/test6_callbacks.lua


Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Laurent FAILLIE
In reply to this post by Daurnimator
Hello,

Le 22/04/2016 08:52, Daurnimator a écrit :
> Infact, sending a packet to a socket *is* actually the best option for
> this sort of thing. On linux, you have a special fast kernel provided
> file descriptor type 'eventfd' just for this purpose. In general, you
> can fall back to using a pipe() or socketpair() and writing a single
> byte so that the other end polls readable.

My issue is not to signalize anything b/w thread, it's already done
using eventfb and socket signaling :)
My issue is to pass a function reference defined in the main thread to
the newly created one w/o blocking the main one.

Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Laurent FAILLIE
In reply to this post by Thijs Schreijer

Hello all,

So the solution I found : a created a dedicated State containing only
functions' references and I put a mutex to protect it : the lock is used
only when I lookup for a function.

So consequentely :
- my main thread is totally decoupled from other ones (so never locked
by any other activities)
- childs ones are blocked only when at least 2 of them are looking up
for a function ... so almost never.

Bye

Reply | Threaded
Open this post in threaded view
|

Re: Lua callback in another thread

Oleg Popov
In reply to this post by Laurent FAILLIE
On Sat, Apr 23, 2016 at 05:18:59PM +0200, Laurent Faillie wrote:

> Hello,
>
> Le 22/04/2016 08:52, Daurnimator a écrit :
> > Infact, sending a packet to a socket *is* actually the best option for
> > this sort of thing. On linux, you have a special fast kernel provided
> > file descriptor type 'eventfd' just for this purpose. In general, you
> > can fall back to using a pipe() or socketpair() and writing a single
> > byte so that the other end polls readable.
>
> My issue is not to signalize anything b/w thread, it's already done
> using eventfb and socket signaling :)
> My issue is to pass a function reference defined in the main thread to
> the newly created one w/o blocking the main one.

And... what are you going to do with that reference? Because references
are just numbers, passing them through sockets is trivial, but they will
be meaningless to another lua_State. If you are going to just pass the
reference eventually back to the main thread, you may handle it as
opaque number. But if you are going to call that function from the
separate thread, you are in trouble. If both threads use the same state,
then you MUST block. And if the state is different, then the only way to
pass objects between states is to copy them, like LuaLanes does. The
copying is difficult, slow and unreliable, and the copy of the function
won't be able to modify the original state (all its upvalues will be
copies of the originals, and _ENV will be that of the new lua_State).