Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Laurent FAILLIE

Hi Sean,

Thanks for your explanation and your code :)
But the think I'm missing is ... why I need to serialize function ?
I mean I know that function are variables like others and I'm propagating them form one thread to another :
1/ or by pushing the function itself as in Detach() : xmove to the main state to the slave one ...
In this case, code launched in the slave thread CAN access to global stuffs (with all the risks it's implying).
2/ or by retrieving first the function content from Lua's registry (https://www.lua.org/pil/27.3.1.html) and then pushing it using again xmove.
In this case, slave code CAN'T access to globals.

The only thing that is not working in both cases is metatable as we are discussing.

It seems from our discuss I'm cheating a bit here but I didn't get the limitation "can move from the same GLOBAL STATE". Simply ... why ? The only danger I can imagine is usage counters are squeezed and objects are subject to be collected. It is definitively not the case here as I'm dealing with constant code : it is stored "somewhere" in Lua's memory and stay until process death as not attached to any dynamic object.


Le Samedi 1 juillet 2017 4h49, Sean Conner <[hidden email]> a écrit :

>  So I played around with this today, and I created a github repository for
> those interested:
>    https://github.com/spc476/LuaFunctionSerialize

I had a look and will deeply study it but ... the processing is made at Lua's side (which mean to me it will impact my application user scripts).

Anyway, I found a workaround by exposing method FIFO:Push() as a function, FIFO.PushFIFO() and it's working.

I'm currently running intensive testing to check there is no side effects or instability.

Rgds,

Laurent
Reply | Threaded
Open this post in threaded view
|

Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Laurent FAILLIE
It's seems string comparisons is not working ether :(

    if topic == "messages" then
        print 'found it"
    end

will not work if this code is launched at MQTT message arrival (so in my famous slave thread).
I read from here https://stackoverflow.com/questions/27633331/can-i-check-strings-equality-in-lua/27634079 that strings are "interned" resulting to a pointer comparison ... not working when using different states, voiding array lookup as well :(

Uncomenting lines 74 and 75, you will see something like :
    >>>    Marcel.prod/Log/Information    nil
    .    "Marcel.prod/Log/Information"    0
    .    "messages2"    1
So t2u table contains an entry for key 'Marcel.prod/Log/Information' but trying to do t2u[topic] ... fails :(

I did good an nice thing with my application (I will publish a video of my dashboard soon :) ), but I think I reach the limit of Lua in a multithreaded environment :(




 
The misspelling master is on the Web.   _________ 100 % Dictionnary Free ! /                    /( / Dico          / / Pleins d'autres fautes sur /________ / / (########( / http://destroyedlolo.info Quoi, des fautes d'orthographe! Pas possible ;-D.


Le Samedi 1 juillet 2017 14h24, Laurent FAILLIE <[hidden email]> a écrit :



Hi Sean,

Thanks for your explanation and your code :)
But the think I'm missing is ... why I need to serialize function ?
I mean I know that function are variables like others and I'm propagating them form one thread to another :
1/ or by pushing the function itself as in Detach() : xmove to the main state to the slave one ...
In this case, code launched in the slave thread CAN access to global stuffs (with all the risks it's implying).
2/ or by retrieving first the function content from Lua's registry (https://www.lua.org/pil/27.3.1.html) and then pushing it using again xmove.
In this case, slave code CAN'T access to globals.

The only thing that is not working in both cases is metatable as we are discussing.

It seems from our discuss I'm cheating a bit here but I didn't get the limitation "can move from the same GLOBAL STATE". Simply ... why ? The only danger I can imagine is usage counters are squeezed and objects are subject to be collected. It is definitively not the case here as I'm dealing with constant code : it is stored "somewhere" in Lua's memory and stay until process death as not attached to any dynamic object.


Le Samedi 1 juillet 2017 4h49, Sean Conner <[hidden email]> a écrit :

>  So I played around with this today, and I created a github repository for
> those interested:
>    https://github.com/spc476/LuaFunctionSerialize

I had a look and will deeply study it but ... the processing is made at Lua's side (which mean to me it will impact my application user scripts).

Anyway, I found a workaround by exposing method FIFO:Push() as a function, FIFO.PushFIFO() and it's working.

I'm currently running intensive testing to check there is no side effects or instability.

Rgds,

Laurent


Reply | Threaded
Open this post in threaded view
|

Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Sean Conner
It was thus said that the Great Laurent FAILLIE once stated:
>
> I did good an nice thing with my application (I will publish a video of my
> dashboard soon :) ), but I think I reach the limit of Lua in a
> multithreaded environment :(

  Have you thought of recompiling liblua with an implementation for
lua_lock() and lua_unlock()?  That way, you can have a single Lua state that
can be shared among threads.  And Lua is small enough that you can embed the
source within the project itself.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Laurent FAILLIE
When starting my projects (as I said, Lua is used in several of my tools dealing with MQTT and smart housing), I made the balance b/w pros and cons : I decided to stick as possible with my distribution standard packages in order to make ownership easy to other people.
I'm using Gentoo and none of packaged version propose lua_lock() mechanism (even if I know it's quite easy to implement) and 5.1 was, and still, the proposed stable version.

Till now, even if sometime it's harder than expected, I've been able to keep dirty work in the framework itself (named "Toile") keeping "user side code" clean. In addition, I achieved almost all my objectives (lightweight, fast, open, clean, expandable, ...).
I will take such radical change only it I'm stuck and it would require strong modification and potentially would creates other issues : especially performances ones (because if Lua engine is requiring locking too often, it will void real time aspect, which is a critical point when speaking about IoT and automation).

More generally, there are ofter discuss in this list about Lua future : usual ":? is missing" is a gadget, but a clean and mastered threading capability is really a must nowday :)



Le Samedi 1 juillet 2017 23h48, Sean Conner <[hidden email]> a écrit :


It was thus said that the Great Laurent FAILLIE once stated:
>
> I did good an nice thing with my application (I will publish a video of my
> dashboard soon :) ), but I think I reach the limit of Lua in a
> multithreaded environment :(

  Have you thought of recompiling liblua with an implementation for
lua_lock() and lua_unlock()?  That way, you can have a single Lua state that
can be shared among threads.  And Lua is small enough that you can embed the
source within the project itself.


  -spc