Wrapping multithreaded c lib

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Wrapping multithreaded c lib

Gregg Reynolds-2
Hi folks,

Thinking about writing a Lua binding for https://www.iotivity.org over the holidays. What could be more fun?

But I'm a little worried about threading. Using the lib involves callbacks, each of which is on a separate thread. I.e. you pass a cb to the send fn, and the engine, when it receives a response, passes control to the call back, with the incoming msg as an arg, but on a new thread.

My understanding is that I would just need to create a new Lua state for each thread, to pass to the call back.

Seems reasonable to me, but it also looks like one of those things that turn out to be much more complicated than they seem. For example, if the same callback can occur multiple times, can I save one Lua state per callback and reuse it? And what if I want to share some data across threads?

Any examples of this sorta thing out there? Any guidance appreciated.

Gregg


Reply | Threaded
Open this post in threaded view
|

Re: Wrapping multithreaded c lib

Laurent FAILLIE
Hello Gregg,

I did a low level framework in order to build dashboards using MQTT / DirectFB and obviously Lua for user scripts : it uses intensively multi-threading with different State as well as interthreading data exchange and notification as you described :

https://github.com/destroyedlolo/Selene

It is working pretty well and driving my smart home automation for months without a glitch.

Notwithstanding, having this approach is creating lot of side effect in your code : as example, strings are stored in a lookup table that seems not shared among threads.
So simple comparison like :

if truc == "machin" ...

may fails ... only of that.

The more solid solution seems to me to patch Lua to allows multithreading (you have to define in lua's own code some function implementing locking and such ... sorry I don't have the link).
I didn't choose this solution as I wanted to stick with standard Lua distribution as it is found in most Linux distribution (so without this patch).

Best regards,

Laurent



Le vendredi 22 décembre 2017 à 01:24:02 UTC+1, Gregg Reynolds <[hidden email]> a écrit :


Hi folks,

Thinking about writing a Lua binding for https://www.iotivity.org over the holidays. What could be more fun?

But I'm a little worried about threading. Using the lib involves callbacks, each of which is on a separate thread. I.e. you pass a cb to the send fn, and the engine, when it receives a response, passes control to the call back, with the incoming msg as an arg, but on a new thread.

My understanding is that I would just need to create a new Lua state for each thread, to pass to the call back.

Seems reasonable to me, but it also looks like one of those things that turn out to be much more complicated than they seem. For example, if the same callback can occur multiple times, can I save one Lua state per callback and reuse it? And what if I want to share some data across threads?

Any examples of this sorta thing out there? Any guidance appreciated.

Gregg


Reply | Threaded
Open this post in threaded view
|

Re: Wrapping multithreaded c lib

Miha Cooper
Hello Gregg,

You can check out our multithreading solution:
https://github.com/effil/effil

We have made exactly what you need: threading based on independent Lua states, data exchange and sharing between states.

It's written in C++, so you will be able to use it with IoTivity's C++ API. Check it out, maybe there is something useful for your project.


On 22 December 2017 at 12:05, Laurent FAILLIE <[hidden email]> wrote:
Hello Gregg,

I did a low level framework in order to build dashboards using MQTT / DirectFB and obviously Lua for user scripts : it uses intensively multi-threading with different State as well as interthreading data exchange and notification as you described :


It is working pretty well and driving my smart home automation for months without a glitch.

Notwithstanding, having this approach is creating lot of side effect in your code : as example, strings are stored in a lookup table that seems not shared among threads.
So simple comparison like :

if truc == "machin" ...

may fails ... only of that.

The more solid solution seems to me to patch Lua to allows multithreading (you have to define in lua's own code some function implementing locking and such ... sorry I don't have the link).
I didn't choose this solution as I wanted to stick with standard Lua distribution as it is found in most Linux distribution (so without this patch).

Best regards,

Laurent



Le vendredi 22 décembre 2017 à 01:24:02 UTC+1, Gregg Reynolds <[hidden email]> a écrit :


Hi folks,

Thinking about writing a Lua binding for https://www.iotivity.org over the holidays. What could be more fun?

But I'm a little worried about threading. Using the lib involves callbacks, each of which is on a separate thread. I.e. you pass a cb to the send fn, and the engine, when it receives a response, passes control to the call back, with the incoming msg as an arg, but on a new thread.

My understanding is that I would just need to create a new Lua state for each thread, to pass to the call back.

Seems reasonable to me, but it also looks like one of those things that turn out to be much more complicated than they seem. For example, if the same callback can occur multiple times, can I save one Lua state per callback and reuse it? And what if I want to share some data across threads?

Any examples of this sorta thing out there? Any guidance appreciated.

Gregg





--
With best regards,
Kupriyanov Mikhail.
Reply | Threaded
Open this post in threaded view
|

Re: Wrapping multithreaded c lib

Viacheslav Usov
In reply to this post by Gregg Reynolds-2
On Fri, Dec 22, 2017 at 1:23 AM, Gregg Reynolds <[hidden email]> wrote:

>  But I'm a little worried about threading. Using the lib involves callbacks, each of which is on a separate thread. I.e. you pass a cb to the send fn, and the engine, when it receives a response, passes control to the call back, with the incoming msg as an arg, but on a new thread.

I am not familiar with that framework, so I can only say a few general things.

> My understanding is that I would just need to create a new Lua state for each thread, to pass to the call back.

If you are completely sure that a given callback can only be called from a particular thread, or at least at can only be called once before it returns control, then this approach would work (0).

If, however, a given callback can be invoked by multiple threads without serialisation, then you will have to either (A) serialise access to the Lua stack or (B) use a multithread-safe Lua state. In case (A), care must be taken to serialise ALL kinds of access to the state, in case (B) you need to build your Lua library in a special way, which may or may not be possible depending on your needs to use third-party Lua libraries.

>  Seems reasonable to me, but it also looks like one of those things that turn out to be much more complicated than they seem. For example, if the same callback can occur multiple times, can I save one Lua state per callback and reuse it?

It is not clear what "multiple times" and "save one Lua state" mean here.

> And what if I want to share some data across threads?

In case (0) or case (A), you would need something else to share data. The framework itself might provide means to that effect, check it out.

In case (B), all callbacks can use the same Lua environment concurrently, so sharing is natural.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Wrapping multithreaded c lib

Thijs Schreijer
In reply to this post by Gregg Reynolds-2
Hi Greg,

I specifically wrote “darksidesync” for this purpose. At the time to create a binding to LibUPnP. It does the exact same thing, it starts background threads for listening and all callbacks are on those background threads.

The main premise is that the main-application loop is on the Lua side, while the C lib runs the background threads and generates the call-backs

It’s been a while since I visited that code though.


Thijs

On 22 Dec 2017, at 01:23, Gregg Reynolds <[hidden email]> wrote:

Hi folks,

Thinking about writing a Lua binding for https://www.iotivity.org over the holidays. What could be more fun?

But I'm a little worried about threading. Using the lib involves callbacks, each of which is on a separate thread. I.e. you pass a cb to the send fn, and the engine, when it receives a response, passes control to the call back, with the incoming msg as an arg, but on a new thread.

My understanding is that I would just need to create a new Lua state for each thread, to pass to the call back.

Seems reasonable to me, but it also looks like one of those things that turn out to be much more complicated than they seem. For example, if the same callback can occur multiple times, can I save one Lua state per callback and reuse it? And what if I want to share some data across threads?

Any examples of this sorta thing out there? Any guidance appreciated.

Gregg