Is it possible to yield/resume the main thread?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Is it possible to yield/resume the main thread?

Stefan-2
Hello,

is it possible to yield and resume the main thread, even if the
yielding function runs in another coroutine?

(related: http://lua-users.org/lists/lua-l/2019-05/msg00081.html )

My Lua script runs in an event-driven environment and I need to
halt it to until an external resource is ready. This works really
nice as long as the state is started with lua_resume and the
C callbacks yield back to the "ground" layer of C.

But when the script calls these functions in a coroutine, they just
yield the script's coroutine and don't stop the execution of Lua.

I tried to yield the main thread directly, but resuming it continues
execution at the wrong point:

print 'A'
cb() --> lua_yield(main thread) / Should pause the script
print 'B'
coroutine.resume(coroutine.create(function()
   print '   X'
   cb() --> lua_yield(main thread) / Should pause the script
   print '   Y'
end))
print 'C'

Prints (Y is missing):

[Resume state]
A
[Yielded]
[Resume state]
B
   X
[Yielded]
[Resume state]
C
[Finished]

Now my questions are:
A) Can a CFunction call lua_yield with anything other than the
   lua_State passed as argument?
B) If not, can the entire script be yielded/resumed directly in
   standard Lua?

I'm sorry for the naive questions, but I couldn't find a direct
statement in the manual.

Here is the C test program:

// gcc -Wall -Ilua-5.4.0/src main.c lua-5.4.0/src/liblua.a -ldl -lm
#include <stdio.h>
#include <lua.h>
#include <lualib.h>
#include <lauxlib.h>

lua_State* L;

int luacb(lua_State* T)
{
    return lua_yield(L, 0);
}

int main()
{
    L = luaL_newstate();
    luaL_openlibs(L);
    lua_register(L, "cb", luacb);
    const char script[] =
        "print 'A'\n"
        "cb()\n"
        "print 'B'\n"
        "coroutine.resume(coroutine.create(function()\n"
        "   print '   X'\n"
        "   cb()\n"
        "   print '   Y'\n"
        "end))\n"
        "print 'C'\n";
    if (luaL_loadbufferx (L, script, sizeof script - 1, "script", "t")
!= LUA_OK) {
        printf("[Error] %s\n", lua_tostring(L, -1));
        goto close_state;
    }
    for (;;) {
        printf("[Resume state]\n");
        int nresults = 0;
        int lr = lua_resume(L, NULL, 0, &nresults);
        switch (lr) {
            case LUA_OK:
                lua_pop(L, nresults);
                printf("[Finished]\n");
                goto close_state;
            case LUA_YIELD:
                lua_pop(L, nresults);
                printf("[Yielded]\n");
                continue;
            default: {
                printf("[Error] %s\n", lua_tostring(L, -1));
                goto close_state;
            }
        }
    }
close_state:
    lua_close(L);
}
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Francisco Olarte
On Fri, Jul 3, 2020 at 2:50 PM Stefan <[hidden email]> wrote:
>
> Hello,
>
> is it possible to yield and resume the main thread, even if the
> yielding function runs in another coroutine?

Although you maay be able to yield from the main thread I do not think
this is totally supported given:

"coroutine.isyieldable ()
Returns true when the running coroutine can yield.
A running coroutine is yieldable if ***it is not the main thread****
and it is not inside a non-yieldable C function. "

Probably "lua_isyieldable [-0, +0, –]
int lua_isyieldable (lua_State *L);
Returns 1 if the given coroutine can yield, and 0 otherwise. "

could help you assert this.

Anyway, you do not have to use the main coroutine to run your thread,
why don't you just use an alternate one and lua_yield on whatever
lua_state is passed into your C callback?

If I'm not too confused just try:

> int main()
> {
>     L = luaL_newstate();
>     luaL_openlibs(L);
>     lua_register(L, "cb", luacb);

MAINL = luaL_newstate(), openlibs and register in mainL, then do...

    L = lua_newthread(MAINL);

All the rest mainly unchanged.
>     const char script[] =
...
   }
> close_state:
>     lua_close(L);

And  change this to lua_close(MAINL) here.

i.e., the same thing you could do in pure lua. ( you create the
coroutine with the function you want to run and then resume it, you
cannot resume main thread or create the main thread with an arbitrary
function ). Bear in mind the C code in main is running in the main
thread, so it is resuming itself, and chaos ensues sooner or later.

I do a variation of this with some event handler code, for IVRs. When
I get a call I start a coroutine with the IVR from main and yield from
it when it needs to do blocking stuff. And for some administrative
tasks which I need to call and may need to do blocking things I create
a one-off coroutine to do the work which can yield if needed, although
it normally does not do it. Some registry storing with lightuserdata
to C events ( the blocking things are typically handled with a C
function which takes a pointer which it passes to a C callback when
done, which you normally use for some struct used on the callback. I
use this pointer as key to save the coroutine in the registry so the C
callbacks now which coroutine to resume ) and you can juggle a lot of
events seamlessly.

For some other stuff I've found easier to just handling all the
yielding and resuming in lua, the main coroutine just runs a
scheduler, an when it has no slave coroutine to run I just return
giving the C calling code an indication of what to do/queue.

I've been using coroutine packages since CP/M and MSDOS, even written
some for those, and I've normally found "do not run app code in the
main coroutine" to be the easier thing. In fact package I wrote having
the resume/yield style did not have a main coroutine.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Stefan-2

Am 03.07.2020 um 18:49 schrieb Francisco Olarte:
> Although you maay be able to yield from the main thread I do not think
> this is totally supported given:
>
> "coroutine.isyieldable ()
> Returns true when the running coroutine can yield.
> A running coroutine is yieldable if ***it is not the main thread****
> and it is not inside a non-yieldable C function. "

Hmm.. this doesn't seem to be correct. Here is a test program[1] that
loads und runs the following buffer in the main thread:

print(coroutine.isyieldable(),coroutine.isyieldable(coroutine.running()))

With Lua 5.3.5:
lua_call   --> false false
lua_callk  --> false false
lua_pcall  --> false false
lua_pcallk --> false false
lua_resume --> true true

With Lua 5.4.0:
lua_call   --> false false
lua_callk  --> true true
lua_pcall  --> false false
lua_pcallk --> true true
lua_resume --> true true

The same program, with another script:

if coroutine.isyieldable() then coroutine.yield('hi') end

With Lua 5.3.5:
lua_call   --> (nothing)
lua_callk  --> (nothing)
lua_pcall  --> LUA_OK
lua_pcallk --> LUA_OK
lua_resume --> LUA_YIELD

With Lua 5.4.0:
lua_call   --> (nothing)
lua_callk  --> PANIC: unprotected error in call to Lua API (hi)
lua_pcall  --> LUA_OK
lua_pcallk --> PANIC: unprotected error in call to Lua API (hi)
lua_resume --> LUA_YIELD

Just yielding:

coroutine.yield('hi')

With Lua 5.3.5:
lua_call   --> PANIC: unprotected error in call to Lua API (attempt to
yield from outside a coroutine)
lua_callk  --> PANIC: unprotected error in call to Lua API (attempt to
yield from outside a coroutine)
lua_pcall  --> Error: attempt to yield from outside a coroutine
lua_pcallk --> Error: attempt to yield from outside a coroutine
lua_resume --> LUA_YIELD

With Lua 5.4.0:
lua_call   --> PANIC: unprotected error in call to Lua API (attempt to
yield from outside a coroutine)
lua_callk  --> PANIC: unprotected error in call to Lua API (hi)
lua_pcall  --> Error: attempt to yield from outside a coroutine
lua_pcallk --> PANIC: unprotected error in call to Lua API (hi)
lua_resume --> LUA_YIELD


What is going on?!


[1] Test program, comments toggled

gcc (Debian 9.3.0-14) 9.3.0

gcc -Wall -Wextra -Wno-unused -Ilua/lua-5.4.0/src main2.c
lua/lua-5.4.0/src/liblua.a -ldl -lm && ./a.out

gcc -Wall -Wextra -Wno-unused -Ilua/lua-5.3.5/src main2.c
lua/lua-5.3.5/src/liblua.a -ldl -lm && ./a.out

#include <stdio.h>
#include <lua.h>
#include <lualib.h>
#include <lauxlib.h>

static int k(lua_State *L, int status, lua_KContext ctx) { (void)L;
(void)status; (void)ctx; return 0; }

int main()
{
    lua_State* L = luaL_newstate();
    luaL_openlibs(L);
    const char script[] = "coroutine.yield('hi')";
    if (luaL_loadbufferx(L, script, sizeof script - 1, "script", "t") !=
LUA_OK) {
        printf("Error: %s\n", lua_tostring(L, -1));
        return 2;
    }
    int nres;
    lua_call(L, 0, 0);
    // lua_callk(L, 0, 0, (lua_KContext)0, k);
    // switch (lua_pcall(L, 0, 0, 0))
    // switch (lua_pcallk(L, 0, 0, 0, (lua_KContext)0, k))
    // switch (lua_resume(L, NULL, 0, &nres)) /* Lua 5.4.0 */
    // switch (lua_resume(L, NULL, 0))  /* Lua 5.3.5 */
    // {
        // case LUA_OK:
            // printf("LUA_OK\n");
            // return 0;
        // case LUA_YIELD:
            // printf("LUA_YIELD\n");
            // return 1;
        // default: {
            // printf("Error: %s\n", lua_tostring(L, -1));
            // return 2;
        // }
    // }
}
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Stefan-2
In reply to this post by Francisco Olarte
Am 03.07.2020 um 18:49 schrieb Francisco Olarte:

> Anyway, you do not have to use the main coroutine to run your thread,
> why don't you just use an alternate one and lua_yield on whatever
> lua_state is passed into your C callback?
>
> If I'm not too confused just try:
>
>> int main()
>> {
>>     L = luaL_newstate();
>>     luaL_openlibs(L);
>>     lua_register(L, "cb", luacb);
>
> MAINL = luaL_newstate(), openlibs and register in mainL, then do...
>
>     L = lua_newthread(MAINL);
>
> All the rest mainly unchanged.
>>     const char script[] =
> ...
>    }
>> close_state:
>>     lua_close(L);
>
> And  change this to lua_close(MAINL) here.

I tried this approach as well and it gives the same result - it
skips the 'Y'.

The 5.1 manual is more direct about what yield does:
"When a C function calls lua_yield in that way, the running coroutine
suspends its execution, and the call to lua_resume that started this
coroutine returns."

But is this still true? Or can the CFunction yield a different thread
as well?

After yielding the 3 different threads in the callback (the one passed
as parameter, the main thread, the extra thread created as you
suggested) on 5.1.5, 5.3.5, 5.4.0 and LuaJIT2.1 it looks like they
all do different things, including crashing, printing a random double,
or ignoring the second yield.

To me it seems that it is not safe to yield directly into the
"ground layer" C.

> i.e., the same thing you could do in pure lua. ( you create the
> coroutine with the function you want to run and then resume it, you
> cannot resume main thread or create the main thread with an arbitrary
> function ). Bear in mind the C code in main is running in the main
> thread, so it is resuming itself, and chaos ensues sooner or later.
>
> I do a variation of this with some event handler code, for IVRs. When
> I get a call I start a coroutine with the IVR from main and yield from
> it when it needs to do blocking stuff. And for some administrative
> tasks which I need to call and may need to do blocking things I create
> a one-off coroutine to do the work which can yield if needed, although
> it normally does not do it. Some registry storing with lightuserdata
> to C events ( the blocking things are typically handled with a C
> function which takes a pointer which it passes to a C callback when
> done, which you normally use for some struct used on the callback. I
> use this pointer as key to save the coroutine in the registry so the C
> callbacks now which coroutine to resume ) and you can juggle a lot of
> events seamlessly.
>
> For some other stuff I've found easier to just handling all the
> yielding and resuming in lua, the main coroutine just runs a
> scheduler, an when it has no slave coroutine to run I just return
> giving the C calling code an indication of what to do/queue.
>
> I've been using coroutine packages since CP/M and MSDOS, even written
> some for those, and I've normally found "do not run app code in the
> main coroutine" to be the easier thing. In fact package I wrote having
> the resume/yield style did not have a main coroutine.
>
> Francisco Olarte.
>

Thanks for the interesting story! Are there multiple threads or
processors involved or is it fully cooperative?
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Francisco Olarte
In reply to this post by Stefan-2
Stefan:

On Fri, Jul 3, 2020 at 9:29 PM Stefan <[hidden email]> wrote:

> Am 03.07.2020 um 18:49 schrieb Francisco Olarte:
> > Although you maay be able to yield from the main thread I do not think
> > this is totally supported given:
> > "coroutine.isyieldable ()
> > Returns true when the running coroutine can yield.
> > A running coroutine is yieldable if ***it is not the main thread****
> > and it is not inside a non-yieldable C function. "
> Hmm.. this doesn't seem to be correct. Here is a test program[1] that
> loads und runs the following buffer in the main thread:
> print(coroutine.isyieldable(),coroutine.isyieldable(coroutine.running()))

... exhaustive example....

Stefan , although it may work some times I do not think, given how
easy it is to avoid it, resuming the main thread is a good idea. A BAD
program does not need to malfunction in every situation, just in one.

In asymmetric coroutines y normally assume a coroutine can be resumed
when it is suspended, and it is suspended when it is just created but
not started or has called yield, neither of which is true in main, I
see it as started from creation and not being able to yield because it
has no one to yield to.

Given lua uses the same struct in the C API it may well be posible to
call the functions and obtain some aparently correct bu tsubtly
corrupted results. You can use them, but IMO you are inflicting
problems in yourself.

Your exhaustive exploration of ( what I personally consider bizarre )
ways of (ab)using the API is good, but I am not personally going to
analyze it too much. Just seeing the difference in behaviours between
versions makes me worry you are hitting some kind of what in C is
called "undefined behaviour" at its finest.

To me it seems whenever you call lua_resume the API assumes you are
doing it on a resumable coroutine, and acts accordingly without
checking too much, so sometimes it does the right thing, sometimes it
does not. Paraphrasing AST, it is a servant, not a nanny. And nearly
something for yield.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Francisco Olarte
In reply to this post by Stefan-2
Stefan:


On Sat, Jul 4, 2020 at 1:14 AM Stefan <[hidden email]> wrote:

>
> Am 03.07.2020 um 18:49 schrieb Francisco Olarte:
> > Anyway, you do not have to use the main coroutine to run your thread,
> > why don't you just use an alternate one and lua_yield on whatever
> > lua_state is passed into your C callback?
> >
> > If I'm not too confused just try:
> >
> >> int main()
> >> {
> >>     L = luaL_newstate();
> >>     luaL_openlibs(L);
> >>     lua_register(L, "cb", luacb);
> >
> > MAINL = luaL_newstate(), openlibs and register in mainL, then do...
> >
> >     L = lua_newthread(MAINL);
> >
> > All the rest mainly unchanged.
> >>     const char script[] =
> > ...
> >    }
> >> close_state:
> >>     lua_close(L);
> >
> > And  change this to lua_close(MAINL) here.
>
> I tried this approach as well and it gives the same result - it
> skips the 'Y'.

You are totally right, I forgot you where using:
>>>>
lua_State* L;
int luacb(lua_State* T)
{
    return lua_yield(L, 0);
}
<<<<

Which, IMO, is a recipe for disaster, I should have noted it and added
"change lua_yield to use the state passed to it, lua_yield(T,0)".

The fact that coroutine.yield does not take a co parameter like resume
or status, clearly hints to the state used in lua_yield not being used
to "direct" the yield, but just for all the other stuff the api does.
I did not check that too much because it never occurred to me someone
would try to yield from a "hidden" ( global, not passed in ) pointer
in a callback.

> The 5.1 manual is more direct about what yield does:
> "When a C function calls lua_yield in that way, the running coroutine
> suspends its execution, and the call to lua_resume that started this
> coroutine returns."
>
> But is this still true? Or can the CFunction yield a different thread
> as well?

You might be able to do it, but IMO you are in for a lot of pain.

Lua yield is a primitive for asymmetric coroutines, you do not "yield
to a coroutine", in Lua , you just yield to your parent. If you want
symmetric ones you can build a kernel for that, in a hundred lines
supporting it, it is a classical programming exercise, basically you
use the main thread as a supervisor, the only one calling resume, and
make it be the only one calling resume and bridging amongst the other
coroutines, and optimize from there.

> After yielding the 3 different threads in the callback (the one passed
> as parameter, the main thread, the extra thread created as you
> suggested) on 5.1.5, 5.3.5, 5.4.0 and LuaJIT2.1 it looks like they
> all do different things, including crashing, printing a random double,
> or ignoring the second yield.

I think this is because you are constantly invoking undefined
behaviour, but I'm not that versed.

> To me it seems that it is not safe to yield directly into the
> "ground layer" C.

That I take as granted, especifically using an external state. Even in
lua it is not that safe to blindly yield. Normally when I'm doing
coroutines, typically to linearize flows on event driven systems, I
run everything inside a coroutine, typically defined in the lua side,
I just use the main thread to dispatch and I normally never
resume/yield in C ( even handlers go to main, main identifies handler,
resumes it, handlers does it work. Async callbacks in C just fire some
stuff and return an identifying token to lua, which the lua wrapper
typically stores somewhere and return, its caller eventually yields in
lua, main thread returns and a later event reconstructs the token and
calls the main thread which uses the token to resume the appropiate
coroutine to handle it ).


> > I've been using coroutine packages since CP/M and MSDOS, even written
> > some for those, and I've normally found "do not run app code in the
> > main coroutine" to be the easier thing. In fact package I wrote having
> > the resume/yield style did not have a main coroutine.
> Thanks for the interesting story! Are there multiple threads or
> processors involved or is it fully cooperative?

It's CP/M (8080) / MSDOS (8086), no multicore there. Anyway, when I've
done this can of things in some of this things my coroutines where in
a scheduler which worked more or less like a lua state, you could
invoke it from different threads but only one at a time. I normally
did not run coroutines for work splitting, but for synchronization of
logic.

In recent ones I tend to not write any of them anymore, there are too
many good coroutine packages to bother. What I have this days is
typically a thread drinking from a message queue running a lua state
with coroutines, a thread pool doing http, voip and other stuff
getting the work to do from another queue and posting results to lua.
Lua does high level control logic, it's fast enough for all I do once
I offload the waiting or slow stuff ( sometimes I offload preparsing
of complex documents), and I use it specifically because coroutines
are easy and just work.

Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Stefan-2
In reply to this post by Francisco Olarte
04.07.2020 um 11:31 schrieb Francisco Olarte:

> Stefan , although it may work some times I do not think, given how
> easy it is to avoid it, resuming the main thread is a good idea. A BAD
> program does not need to malfunction in every situation, just in one.
>
> In asymmetric coroutines y normally assume a coroutine can be resumed
> when it is suspended, and it is suspended when it is just created but
> not started or has called yield, neither of which is true in main, I
> see it as started from creation and not being able to yield because it
> has no one to yield to.
>
> Given lua uses the same struct in the C API it may well be posible to
> call the functions and obtain some aparently correct bu tsubtly
> corrupted results. You can use them, but IMO you are inflicting
> problems in yourself.
>
> Your exhaustive exploration of ( what I personally consider bizarre )
> ways of (ab)using the API is good, but I am not personally going to
> analyze it too much. Just seeing the difference in behaviours between
> versions makes me worry you are hitting some kind of what in C is
> called "undefined behaviour" at its finest.
>
> To me it seems whenever you call lua_resume the API assumes you are
> doing it on a resumable coroutine, and acts accordingly without
> checking too much, so sometimes it does the right thing, sometimes it
> does not. Paraphrasing AST, it is a servant, not a nanny. And nearly
> something for yield.
>
> Francisco Olarte.
>

I'm sorry if this came across negative. The goal of this post is *NOT*
to exploit the C API, it is to find out how to use it safely and avoid
the "C API undefined behaviour". When this tiny script:

if coroutine.isyieldable() then
  coroutine.yield()
end

crashes, clearly something went wrong on the C side.
So far my conclusions are:

* Never run the main thread with anything other than lua_call or
  lua_pcall
* Never lua_yield anything other than the currently running coroutine

I.e., the answer to "Is it possible to yield/resume the main thread?" is
no.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Francisco Olarte
Stefan:

On Sat, Jul 4, 2020 at 2:55 PM Stefan <[hidden email]> wrote:
...
> I'm sorry if this came across negative. The goal of this post is *NOT*
> to exploit the C API, it is to find out how to use it safely and avoid
> the "C API undefined behaviour".

Understood.

> When this tiny script:
> if coroutine.isyieldable() then
>   coroutine.yield()
> end
> crashes, clearly something went wrong on the C side.

Well, that does not crash the default C host, lua. IMO what is
happening in the C samples is C code managed to put the lua_state in a
state where it can be crashed by running the tiny script.


> So far my conclusions are:
> * Never run the main thread with anything other than lua_call or
>   lua_pcall

IMO this is one thing which you have got wrong. I like to consider it
as if the main thread is running as soon as you do lua_open. It is
just not running lua code. lua_*call just make it run it. If you view
it like this it is less confusing. When I start the REPL an sitting at
the prompt the main thread is running, and waiting in C code. When I
execute io.stdin:read() it is running too, and waiting in C code
invoked from lua code. But it is running.

AAMOF, it you look at coroutine.runnig sources in lcorolib.c you'll
find "  if (L == co) lua_pushliteral(L, "running");".

> * Never lua_yield anything other than the currently running coroutine

I think lua_yield() is not going to be a problem if you pass the
proper argument, the state pointer you normally pass around all your C
code. A lua_state carries a coroutine context, but the main one
carries some other things. The fact that lua_yieldk contains the line
"      luaG_runerror(L, "attempt to yield from outside a coroutine");"
makes me think you would just get an error if you do this and the
state you passed happens to be the main thread.

I suspect what is happening is calling lua_resume on the main thread,
is incorrect but unchecked. This leads to corruption of things which
then lets you call lua_yield avoiding the checks.

In fact I see in your example totally normal results in all cases
except the lua_resume case, where I consider everything after running
lua_resume is meaningless, as that is incorrect, as the main thread
cannot be suspended ( every attempt to yield fails ) and you can only
resume a suspended coroutine ( but lua_resume does not seem to check
that ).

And I am not too sure of the utility of the parameter version of
coroutine.isyieldable(). I mean, only a running coroutine can yield.
You can test a suspended coroutine to see if it could yield as soon as
it starts running, but it can put itself on an unyieldable state ( by
entering a non-yieldable C function) as soon as you resume it, but I
suppose for some cases using external communication mediums it can
work ( maybe pushing a yielding callback or a non yielding one on the
coroutines stack because you know it is going to call that, things
like this ).

> I.e., the answer to "Is it possible to yield/resume the main thread?" is no.

Correct. I thought you were trying to asses the problems of calling
lua_yield in a C callback being unsafe (crashing), which is not, from
what I see it is as safe as calling coroutine.yield in lua as long as
you do not mess with states, the worst that can happen is you get an
error somewhere.


Francisco Olarte.
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Stefan-2
Am 04.07.2020 um 18:21 schrieb Francisco Olarte:

>> So far my conclusions are:
>> * Never run the main thread with anything other than lua_call or
>>   lua_pcall
>
> IMO this is one thing which you have got wrong. I like to consider it
> as if the main thread is running as soon as you do lua_open. It is
> just not running lua code. lua_*call just make it run it. If you view
> it like this it is less confusing. When I start the REPL an sitting at
> the prompt the main thread is running, and waiting in C code. When I
> execute io.stdin:read() it is running too, and waiting in C code
> invoked from lua code. But it is running.

Ah yes - now I see. luaB_costatus always gives "running" for the main
thread, "dead" for a thread with no bytecode (even a fresh one) and
"suspended" for one with a function loaded.

>
> AAMOF, it you look at coroutine.runnig sources in lcorolib.c you'll
> find "  if (L == co) lua_pushliteral(L, "running");".
>
>> * Never lua_yield anything other than the currently running coroutine
>
> I think lua_yield() is not going to be a problem if you pass the
> proper argument, the state pointer you normally pass around all your C
> code. A lua_state carries a coroutine context, but the main one
> carries some other things. The fact that lua_yieldk contains the line
> "      luaG_runerror(L, "attempt to yield from outside a coroutine");"
> makes me think you would just get an error if you do this and the
> state you passed happens to be the main thread.

Since the argument to a C function is the currently running thread,
this is the only valid way, I think (probably what you mean, too):

int c_func(lua_State* L)
or
int c_func_k(lua_State *L, int status, lua_KContext ctx)
{
    ...
    return lua_yield(L, ...);
    or
    return lua_yieldk(L, ...);
}

>
> I suspect what is happening is calling lua_resume on the main thread,
> is incorrect but unchecked. This leads to corruption of things which
> then lets you call lua_yield avoiding the checks.
>
> In fact I see in your example totally normal results in all cases
> except the lua_resume case, where I consider everything after running
> lua_resume is meaningless, as that is incorrect, as the main thread
> cannot be suspended ( every attempt to yield fails ) and you can only
> resume a suspended coroutine ( but lua_resume does not seem to check
> that ).

I've actually used the lua_resume(main_L) / lua_yield(L) quite a bit
and it generally works (without other coroutines). I never
suspected that the main thread cannot be used as a coroutine.
LUA_USE_APICHECK also doesn't detect this.

>
> And I am not too sure of the utility of the parameter version of
> coroutine.isyieldable(). I mean, only a running coroutine can yield.
> You can test a suspended coroutine to see if it could yield as soon as
> it starts running, but it can put itself on an unyieldable state ( by
> entering a non-yieldable C function) as soon as you resume it, but I
> suppose for some cases using external communication mediums it can
> work ( maybe pushing a yielding callback or a non yielding one on the
> coroutines stack because you know it is going to call that, things
> like this ).
>
>> I.e., the answer to "Is it possible to yield/resume the main thread?" is no.
>
> Correct. I thought you were trying to asses the problems of calling
> lua_yield in a C callback being unsafe (crashing), which is not, from
> what I see it is as safe as calling coroutine.yield in lua as long as
> you do not mess with states, the worst that can happen is you get an
> error somewhere.
>
>
> Francisco Olarte.
>

lua_isyieldable seems to be wrong:

lua_State* MAINL = luaL_newstate();
printf("luaL_newstate() = %p\n", (void*)MAINL);
printf("lua_isyieldable(MAINL) = %d\n", lua_isyieldable(MAINL));
printf("lua_status(MAINL) = %d\n", lua_status(MAINL));
printf("luaL_openlibs(MAINL)\n");
luaL_openlibs(MAINL);
printf("lua_isyieldable(MAINL) = %d\n", lua_isyieldable(MAINL));
printf("lua_status(MAINL) = %d\n", lua_status(MAINL));
lua_State* NTHRL = lua_newthread(MAINL);
printf("lua_newthread(MAINL) = %p\n", (void*)NTHRL);
printf("lua_isyieldable(NTHRL) = %d\n", lua_isyieldable(NTHRL));
printf("lua_status(NTHRL) = %d\n", lua_status(NTHRL));

This prints with Lua 5.4:
luaL_newstate() = 0x55d6635ff2a8
lua_isyieldable(MAINL) = 1
lua_status(MAINL) = 0
luaL_openlibs(MAINL)
lua_isyieldable(MAINL) = 1
lua_status(MAINL) = 0
lua_newthread(MAINL) = 0x55d663605d78
lua_isyieldable(NTHRL) = 1
lua_status(NTHRL) = 0

but with Lua 5.3:
luaL_newstate() = 0x5644debdf2a8
lua_isyieldable(MAINL) = 0
lua_status(MAINL) = 0
luaL_openlibs(MAINL)
lua_isyieldable(MAINL) = 0
lua_status(MAINL) = 0
lua_newthread(MAINL) = 0x5644debe6688
lua_isyieldable(NTHRL) = 0
lua_status(NTHRL) = 0

Maybe it is a side-effect of this change:

Lua 5.4.0  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> coroutine.isyieldable(coroutine.create(function() end))
true

Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> coroutine.isyieldable(coroutine.create(function() end))
false
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to yield/resume the main thread?

Francisco Olarte
Stefan:

On Sun, Jul 5, 2020 at 2:28 AM Stefan <[hidden email]> wrote:
....
> Since the argument to a C function is the currently running thread,
> this is the only valid way, I think (probably what you mean, too):

Exactly, avoid stashing a lua_state* somewher and using it, work only
with the one passed in and,
in very specialized setups, whith aone obtained from lua_newthread/getthread.

....
> I've actually used the lua_resume(main_L) / lua_yield(L) quite a bit
> and it generally works (without other coroutines). I never
> suspected that the main thread cannot be used as a coroutine.
> LUA_USE_APICHECK also doesn't detect this.

This happens a lot, some APIs do not check for speed/whatever reason,
so when you do bad things it sometimes works as you expect, classic
undefined behaviour.

....
> lua_isyieldable seems to be wrong:
...
> This prints with Lua 5.4:
...
> lua_isyieldable(MAINL) = 1
...
> lua_isyieldable(NTHRL) = 1
...
> but with Lua 5.3:
...
> lua_isyieldable(MAINL) = 0
...
> lua_isyieldable(NTHRL) = 0

Yep, it seems it has changed somehow.

I think lua_isyieldable may be meant to be used only with the


> Maybe it is a side-effect of this change:
> Lua 5.4.0  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> > coroutine.isyieldable(coroutine.create(function() end))
> true
> Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> > coroutine.isyieldable(coroutine.create(function() end))
> false

I fear you are using the docs for 5.4 and comparing apples to orange,
isyieldable does not take parameters in 5.3, so it is ignoring the
coroutine you pass and acting on the current thread:

5.3
static int luaB_yieldable (lua_State *L) {
  lua_pushboolean(L, lua_isyieldable(L));
  return 1;
}

5.4
static int luaB_yieldable (lua_State *L) {
  lua_State *co = lua_isnone(L, 1) ? L : getco(L);
  lua_pushboolean(L, lua_isyieldable(co));
  return 1;
}



And I feel isyieldable is not supposed to be used to try to make
"directed yields", but just a check to lookup whether you are going to
hit the "This function can raise an error if it is called from a
thread with a pending C call with no continuation function, or it is
called from a thread that is not running inside a resume (e.g., the
main thread). " condition ( first branch mainly ) when you have a C
function which has alternative approaches ( like an http handler,
doing a request asynchronously when it can and taking a heavier
synchonous approach when not ).

Francisco Olarte.