Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Laurent FAILLIE

Hello,

 

I’m thinking to migrate my very own home automation tools from 5.1 to 5.3 …

One of the biggest problem I’m facing with Lua is multi-threading : all these tools are events driven, especially,  each and every MQTT message arrival triggers a new thread running some Lua code, and this is driven from C side.

 On 5.3 documentation, I can read

Each Lua state has one or more threads, which correspond to independent, cooperative lines of execution. The type lua_State (despite its name) refers to a thread. (Indirectly, through the thread, it also refers to the Lua state associated to the thread.)” but cooperative looks to me it still not real preemptive multi-threading but still co-routines like, isn’t it ?

 In my current application (using Lua 5.1), I’m creating new State for each thread but it create some issues :

*         Global variables can’t be used … but frankly speaking, it’s avoiding to deal with concurrent access and I created C side shared object for this which doing the job pretty well.

*         More problematic : strings comparison doesn’t work at all

If my_var == “toto”

                Will always fails. I got from this list it’s because there is a lookup table to speed up and this table is stick to the main State.

 What would be the best solution to avoid this 2nd nasty issue ? Is implementing lua_(un)lock()still the solution in 5.3 ? Does it mean I will have only a single Lua_State shared with all threads ?

Any better solution

 

Thanks

 

Laurent



Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Thomas Jericke
On 27.03.2018 17:07, Laurent FAILLIE wrote:

Hello,

 

I’m thinking to migrate my very own home automation tools from 5.1 to 5.3 …

One of the biggest problem I’m facing with Lua is multi-threading : all these tools are events driven, especially,  each and every MQTT message arrival triggers a new thread running some Lua code, and this is driven from C side.

 On 5.3 documentation, I can read

Each Lua state has one or more threads, which correspond to independent, cooperative lines of execution. The type lua_State (despite its name) refers to a thread. (Indirectly, through the thread, it also refers to the Lua state associated to the thread.)” but cooperative looks to me it still not real preemptive multi-threading but still co-routines like, isn’t it ?

 In my current application (using Lua 5.1), I’m creating new State for each thread but it create some issues :

*         Global variables can’t be used … but frankly speaking, it’s avoiding to deal with concurrent access and I created C side shared object for this which doing the job pretty well.

*         More problematic : strings comparison doesn’t work at all

If my_var == “toto”

                Will always fails. I got from this list it’s because there is a lookup table to speed up and this table is stick to the main State.

 What would be the best solution to avoid this 2nd nasty issue ? Is implementing lua_(un)lock()still the solution in 5.3 ? Does it mean I will have only a single Lua_State shared with all threads ?

Any better solution

 

Thanks

 

Laurent



It is possible to add preemptive scheduling to Lua >5.2 using hooks and yields. But I would ask myself first, if you really need preemptive scheduling and in which situations.
So my advice is, to look into a cooperative solution for your problem first.
--
Thomas
Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Javier Guerra Giraldez
On 28 March 2018 at 06:47, Thomas Jericke <[hidden email]> wrote:
> *         More problematic : strings comparison doesn’t work at all
>
> If my_var == “toto”
>
>                 Will always fails. I got from this list it’s because there
> is a lookup table to speed up and this table is stick to the main State.

this should work.  my hunch is that that "my_var" isn't correctly
interned.  if it's a real Lua string, then it will be comparable.  If
it's a userdata or some reference to an "external" string, then it's
your responsibility to add the metamethods to do the comparison.


--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Laurent FAILLIE
In reply to this post by Thomas Jericke
Thanks Thomas for your reply.

> Le mercredi 28 mars 2018 à 07:47:33 UTC+2, Thomas Jericke <[hidden email]> a écrit :

> It is possible to add preemptive scheduling to Lua >5.2 using hooks and yields.

Can you point me out some docs, examples, tutorials about that ?

> But I would ask myself first, if you really need preemptive scheduling and in which situations.
> So my advice is, to look into a cooperative solution for your problem first.

My goal is to provide to end user the possibility to create Blockly like scripts which will be converted to Lua for execution (as Domoticz is doing).
The problems with cooperative approach is the scripts have to contains some explicite yield() which, de facto, impact the design of scripts themself, something I would like to avoid.

Using real preemptive scheduling approach is an easiest concept :
* each task is focusing only on its jobs
* their scripts doesn't have to take care of what happening outside their own duty (shared objects access arbitration is already done at C side).

The other added value is a badly written script can't block everything (bad remembering of m$-win 3.xx era).

Bye

Laurent


Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Thomas Jericke
On 28.03.2018 11:38, Laurent FAILLIE wrote:
Thanks Thomas for your reply.

> Le mercredi 28 mars 2018 à 07:47:33 UTC+2, Thomas Jericke [hidden email] a écrit :

> It is possible to add preemptive scheduling to Lua >5.2 using hooks and yields.

Can you point me out some docs, examples, tutorials about that ?
The best description of my own solution is in the slides of my speak:
https://www.lua.org/wshop13/Jericke.pdf

Page 18 and following.

> But I would ask myself first, if you really need preemptive scheduling and in which situations.
> So my advice is, to look into a cooperative solution for your problem first.

My goal is to provide to end user the possibility to create Blockly like scripts which will be converted to Lua for execution (as Domoticz is doing).
The problems with cooperative approach is the scripts have to contains some explicite yield() which, de facto, impact the design of scripts themself, something I would like to avoid.
I thought that as well when I started. Currently I came to the conclusion that yields are seldom explicit in my code. Most of the time they are part of an API call.
That means, my API is built in a way, that every blocking API call yields and only stops the Lua thread and not the Lua state/interpreter.

Using real preemptive scheduling approach is an easiest concept :
* each task is focusing only on its jobs
* their scripts doesn't have to take care of what happening outside their own duty (shared objects access arbitration is already done at C side).

The other added value is a badly written script can't block everything (bad remembering of m$-win 3.xx era).

I know that problem. I am using a count hook to check if a script goes rogue. This doesn't work in all cases, but I can live with some responsibility for the scripter.
--
Thomas
Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Robert Virding-2
Another solution to the scheduling problem is to use luerl, https://github.com/rvirding/luerl , which runs a Lua inside the Erlang system. It is in fact implemented in Erlang. This gives you complete access to all of Erlang's concurrency and built-in pre-emptive scheduling. However, it also means that the Lua is not directly callable from C. Also it is only at 5.2 and not 5.3.

Robert


On 28 March 2018 at 13:22, Thomas Jericke <[hidden email]> wrote:
On 28.03.2018 11:38, Laurent FAILLIE wrote:
Thanks Thomas for your reply.

> Le mercredi 28 mars 2018 à 07:47:33 UTC+2, Thomas Jericke [hidden email] a écrit :

> It is possible to add preemptive scheduling to Lua >5.2 using hooks and yields.

Can you point me out some docs, examples, tutorials about that ?
The best description of my own solution is in the slides of my speak:
https://www.lua.org/wshop13/Jericke.pdf

Page 18 and following.

> But I would ask myself first, if you really need preemptive scheduling and in which situations.
> So my advice is, to look into a cooperative solution for your problem first.

My goal is to provide to end user the possibility to create Blockly like scripts which will be converted to Lua for execution (as Domoticz is doing).
The problems with cooperative approach is the scripts have to contains some explicite yield() which, de facto, impact the design of scripts themself, something I would like to avoid.
I thought that as well when I started. Currently I came to the conclusion that yields are seldom explicit in my code. Most of the time they are part of an API call.
That means, my API is built in a way, that every blocking API call yields and only stops the Lua thread and not the Lua state/interpreter.

Using real preemptive scheduling approach is an easiest concept :
* each task is focusing only on its jobs
* their scripts doesn't have to take care of what happening outside their own duty (shared objects access arbitration is already done at C side).

The other added value is a badly written script can't block everything (bad remembering of m$-win 3.xx era).

I know that problem. I am using a count hook to check if a script goes rogue. This doesn't work in all cases, but I can live with some responsibility for the scripter.
--
Thomas

Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Laurent FAILLIE
In reply to this post by Thomas Jericke



Le mercredi 28 mars 2018 à 13:23:19 UTC+2, Thomas Jericke <[hidden email]> a écrit :


> The best description of my own solution is in the slides of my speak:
> https://www.lua.org/wshop13/Jericke.pdf

> Page 18 and following.

If I correctly understood, it's still not OS native threading isn't it ? But co-routines with some code to secure and makes their management easier, isn't it ?

But I'm thinking of another solution :

* A main Lua State that will only parse Lua source code and potentially run some initialization code before thread are launched.

* Then I create sub State using lua_newthread() and the corresponding State will be attached to a new OS thread.

As a consequence, all thread/State will share the same global objects (that will solve my issue about strings comparison I have will lua_newstate() ) but can run TOTALLY concurrently.

What to you think about this approach ? Even if slave States are not expecting to modify any global objects, do you think it is worth to implement lua_lock and lua_unlock ?

Thanks

Laurent

Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

云风 Cloud Wu
In reply to this post by Robert Virding-2
Robert Virding <[hidden email]>于2018年3月28日周三 下午8:49写道:
Another solution to the scheduling problem is to use luerl, https://github.com/rvirding/luerl , which runs a Lua inside the Erlang system. It is in fact implemented in Erlang. This gives you complete access to all of Erlang's concurrency and built-in pre-emptive scheduling. However, it also means that the Lua is not directly callable from C. Also it is only at 5.2 and not 5.3.


I tried to embed lua into Erlang in my project ( https://github.com/cloudwu/skynet )  at first, and then I found implement a lightweight framework in C to replace Erlang/OTP is a better choice. So I rewrote it. And now it's widely used in China , dozens of online game servers base on this project .

Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Thomas Jericke
In reply to this post by Laurent FAILLIE
On 29.03.2018 11:28, Laurent FAILLIE wrote:



Le mercredi 28 mars 2018 à 13:23:19 UTC+2, Thomas Jericke [hidden email] a écrit :


> The best description of my own solution is in the slides of my speak:
> https://www.lua.org/wshop13/Jericke.pdf

> Page 18 and following.

If I correctly understood, it's still not OS native threading isn't it ? But co-routines with some code to secure and makes their management easier, isn't it ?
Yes and no. I don't use OS threading to schedule the Lua threads. All Lua threads run in one C thread.
On the other hand, I would not call them co-routines anymore as the scheduler I implemented is preemptive.

But I'm thinking of another solution :

* A main Lua State that will only parse Lua source code and potentially run some initialization code before thread are launched.

* Then I create sub State using lua_newthread() and the corresponding State will be attached to a new OS thread.

As a consequence, all thread/State will share the same global objects (that will solve my issue about strings comparison I have will lua_newstate() ) but can run TOTALLY concurrently.

What to you think about this approach ? Even if slave States are not expecting to modify any global objects, do you think it is worth to implement lua_lock and lua_unlock ?

Thanks

Laurent

I can't give you a final answer as I never tried to use multiple C threads on the same global Lua state.
As far as Lua 5.1 and Lua 5.2 goes I came to the conclution, that Lua runs in the locked state for most of the time and therfore you wont be able to run the co-routines totally concurrently.

I would actually be interested in that approach, if someone has some experience with it.
--
Thomas
Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Marc Balmer


Am 29.03.2018 um 12:37 schrieb Thomas Jericke <[hidden email]>:

On 29.03.2018 11:28, Laurent FAILLIE wrote:



Le mercredi 28 mars 2018 à 13:23:19 UTC+2, Thomas Jericke [hidden email] a écrit :


> The best description of my own solution is in the slides of my speak:
> https://www.lua.org/wshop13/Jericke.pdf

> Page 18 and following.

If I correctly understood, it's still not OS native threading isn't it ? But co-routines with some code to secure and makes their management easier, isn't it ?
Yes and no. I don't use OS threading to schedule the Lua threads. All Lua threads run in one C thread.
On the other hand, I would not call them co-routines anymore as the scheduler I implemented is preemptive.

But I'm thinking of another solution :

* A main Lua State that will only parse Lua source code and potentially run some initialization code before thread are launched.

* Then I create sub State using lua_newthread() and the corresponding State will be attached to a new OS thread.

As a consequence, all thread/State will share the same global objects (that will solve my issue about strings comparison I have will lua_newstate() ) but can run TOTALLY concurrently.

What to you think about this approach ? Even if slave States are not expecting to modify any global objects, do you think it is worth to implement lua_lock and lua_unlock ?

Thanks

Laurent

I can't give you a final answer as I never tried to use multiple C threads on the same global Lua state.
As far as Lua 5.1 and Lua 5.2 goes I came to the conclution, that Lua runs in the locked state for most of the time and therfore you wont be able to run the co-routines totally concurrently.

I would actually be interested in that approach, if someone has some experience with it.
--
Thomas

I successfully implemented and used this approach:

- A Lua state is created in a POSIX thread, one state per thread (I called these "nodes")
- Nodes use ZeroMQ message queues to communicate

-> The Lua states run concurrently and have a means to communicate.

Worked like charm.

A Lua binding for ZeroMQ can be found here: https://github.com/arcapos/luazmq

If some is interested in the node creation/thread stuff, mail me offlist. 

- mb



Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Thomas Jericke
On 29.03.2018 12:44, Marc Balmer wrote:


Am 29.03.2018 um 12:37 schrieb Thomas Jericke <[hidden email]>:

On 29.03.2018 11:28, Laurent FAILLIE wrote:



Le mercredi 28 mars 2018 à 13:23:19 UTC+2, Thomas Jericke [hidden email] a écrit :


> The best description of my own solution is in the slides of my speak:
> https://www.lua.org/wshop13/Jericke.pdf

> Page 18 and following.

If I correctly understood, it's still not OS native threading isn't it ? But co-routines with some code to secure and makes their management easier, isn't it ?
Yes and no. I don't use OS threading to schedule the Lua threads. All Lua threads run in one C thread.
On the other hand, I would not call them co-routines anymore as the scheduler I implemented is preemptive.

But I'm thinking of another solution :

* A main Lua State that will only parse Lua source code and potentially run some initialization code before thread are launched.

* Then I create sub State using lua_newthread() and the corresponding State will be attached to a new OS thread.

As a consequence, all thread/State will share the same global objects (that will solve my issue about strings comparison I have will lua_newstate() ) but can run TOTALLY concurrently.

What to you think about this approach ? Even if slave States are not expecting to modify any global objects, do you think it is worth to implement lua_lock and lua_unlock ?

Thanks

Laurent

I can't give you a final answer as I never tried to use multiple C threads on the same global Lua state.
As far as Lua 5.1 and Lua 5.2 goes I came to the conclution, that Lua runs in the locked state for most of the time and therfore you wont be able to run the co-routines totally concurrently.

I would actually be interested in that approach, if someone has some experience with it.
--
Thomas

I successfully implemented and used this approach:

- A Lua state is created in a POSIX thread, one state per thread (I called these "nodes")
- Nodes use ZeroMQ message queues to communicate

-> The Lua states run concurrently and have a means to communicate.

Worked like charm.

A Lua binding for ZeroMQ can be found here: https://github.com/arcapos/luazmq

If some is interested in the node creation/thread stuff, mail me offlist. 

- mb



Do you have a global state (created with lua_newstate) or a Lua thread (created with lua_newthread) per POSIX thread?

--

Thomas

Reply | Threaded
Open this post in threaded view
|

Re: Moving from Lua 5.1 to 5.3 … (yet another multithreading discuss)

Marc Balmer


Am 29.03.2018 um 12:53 schrieb Thomas Jericke <[hidden email]>:
[ ...]

Do you have a global state (created with lua_newstate) or a Lua thread (created with lua_newthread) per POSIX thread?


A global state (lua_newstate()).