Passing function b/w mutithreaded states

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing function b/w mutithreaded states

Laurent FAILLIE
Hi all,

While migrating my application from Lua 5.1 to Lua 5.3, I'm trying correct multi-threading issues I'm encounterning. I already created some discusses about that on this mailing list ... but, clearly, I'm missing something :)

I'm using dedicated States using luaL_newstate() for each and every thread created, and my issue is pass a function to execute from my master threads to slave ones. I'm using 2 mechanisms (in both cases, L is the State from the master thread, that parsed the source code ) :

1/ The function to call is passed as a function resulting following code :

static void *launchfunc(void *arg){
    if(lua_pcall( (lua_State *)arg, 0, 1, 0))
        fprintf(stderr, "*E* (launch) %s\n", lua_tostring((lua_State *)arg, -1));
    
    lua_close((lua_State *)arg);
    return NULL;
}

int SelDetach( lua_State *L ){
    if(lua_type(L, 1) != LUA_TFUNCTION ){
        lua_pushnil(L);
        lua_pushstring(L, "Function needed as 1st argument of Selene.Detach()");
        return 2;
    }

    pthread_t tid;    /* No need to be kept */
    lua_State *tstate = luaL_newstate(); /* Initialise new state for the thread */
    assert(tstate);
    luaL_openlibs( tstate );
    lua_xmove( L, tstate, 1 );

    if(pthread_create( &tid, &thread_attr, launchfunc,  tstate) < 0){
        fprintf(stderr, "*E* Can't create a new thread : %s\n", strerror(errno));
        lua_pushnil(L);
        lua_pushstring(L, strerror(errno));
        return 2;
    }

    return 0;
}

In this case :
* my slave function CAN access to global variables (which has to be avoided as dangerous : there is no management of concurrent acces).
* I have no issue when comparing strings.

2/ The function to call is passed only as a reference resulting following code :

        ...
        lua_pushstring(L, "func");    /* retreaving from arguments tables
        lua_gettable(L, -2);
        if( lua_type(L, -1) != LUA_TFUNCTION )
            lua_pop(L, 1);    /* Pop the result */
        else {
            lua_xmove( L, eclient->L, 1 );    /* Move the function to the callback's stack */
            func = luaL_ref(eclient->L,LUA_REGISTRYINDEX);    /* Reference the function in callbacks' context */
        }
        ...

        ...
        lua_State *tstate = luaL_newstate();    /* State dedicated to this thread */
        assert(tstate);
        luaL_openlibs( tstate );
                
        pthread_mutex_lock( &ctx->access_ctrl );    /* Exclusive access to the broker's stat needed */
        lua_rawgeti( ctx->L, LUA_REGISTRYINDEX, tp->func);    /* retrieves the function */
        lua_xmove( ctx->L, tstate, 1 );
        pthread_mutex_unlock( &ctx->access_ctrl );
        lua_pushstring( tstate, topic);
        lua_pushstring( tstate, cpayload);
        if(lua_pcall( tstate, 2, 1, 0)){    /* Call Lua callback function */
            fprintf(stderr, "*E* (msg arrival) %s\n", lua_tostring(tstate, -1));
            lua_pop(tstate, 2); /* pop error message and NIL from the stack */
        } else if(tp->trigger != LUA_REFNIL){
            if(lua_toboolean(tstate, -1))
                pushtask( tp->trigger, tp->trigger_once );
            lua_pop(tstate, 1);    /* remove the return code */
        }
        lua_close( tstate ); /* Remove this thread own state */
        ...

In this case, global variables are unknown BUT I'm in trouble when I'm comparing a variable with a string like
    if myvar == "toto" then ...
even with myvar has the good value, the comparison is failling.

So, what should I do to correctly call a function in another thread ?

Thanks

Laurent

Reply | Threaded
Open this post in threaded view
|

Re: Passing function b/w mutithreaded states

Andrew Gierth
>>>>> "Laurent" == Laurent FAILLIE <[hidden email]> writes:

 Laurent> So, what should I do to correctly call a function in another
 Laurent> thread ?

Dump the function to a bytecode string and load the string in the other
state.

You can't safely do it any other way, because each "main" state has its
own independent garbage collection, and can't know whether any of its
objects are referenced by other main states. Even if you worked around
that by holding references to everything, you'd fall foul of memory
synchronization issues, string interning issues, and all sorts of other
breakage (some of which you mentioned in your message).

If you turn on API checks, you'll notice that lua_xmove will abort if
you try and move values between independent states; this operation is
forbidden. The only way to communicate values between independent states
is to serialize them in some form.

(cqueues has an example, you might look at that)

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Passing function b/w mutithreaded states

Laurent FAILLIE
Thanks Andrew for these valuable information.
I got from cqueues source code (in fact, Lua's lstrlib.c), I have to use lua_dump() and a wrapper to store the result.
Then, I'll have to find how to load it back in a new state.

Thanks again.


Le lundi 9 avril 2018 à 21:36:15 UTC+2, Andrew Gierth <[hidden email]> a écrit :


>>>>> "Laurent" == Laurent FAILLIE <[hidden email]> writes:

Laurent> So, what should I do to correctly call a function in another
Laurent> thread ?

Dump the function to a bytecode string and load the string in the other

state.


You can't safely do it any other way, because each "main" state has its
own independent garbage collection, and can't know whether any of its
objects are referenced by other main states. Even if you worked around
that by holding references to everything, you'd fall foul of memory
synchronization issues, string interning issues, and all sorts of other
breakage (some of which you mentioned in your message).

If you turn on API checks, you'll notice that lua_xmove will abort if
you try and move values between independent states; this operation is
forbidden. The only way to communicate values between independent states
is to serialize them in some form.

(cqueues has an example, you might look at that)

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Passing function b/w mutithreaded states

Francisco Olarte
In reply to this post by Andrew Gierth
Andrew:

On Mon, Apr 9, 2018 at 9:36 PM, Andrew Gierth
<[hidden email]> wrote:
> ... The only way to communicate values between independent states
> is to serialize them in some form.

Not really. We do it by just recursively copying, similar to a
serialize+deserialize loop, nearly the same problems, but you avoid
all the problems of inventing a format and managing the serialized
format ( and encounter nearly the same problems, loop detection,
functions, userdata.. )

Francisco Olarte.