Forcing coroutine.resume to call the yielding function again

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Forcing coroutine.resume to call the yielding function again

Andreas Falkenhahn-2
I have a C function which looks like this:

        static int testfunc(lua_State *L)
        {
                static int flag = 0;
       
                if(!flag) {
                        flag = 1;
                        printf("First call\n");
                        return lua_yield(L, 0);
                } else {
                        printf("Second call\n");
                }

                return 0;
        }

It is called from a coroutine like this:

        co = coroutine.create(function()
                print("foo")
                testfunc()
                print("bar")
        end)

        coroutine.resume(co)
        coroutine.resume(co)

Normally, the second call to coroutine.resume() will continue executing the code at

        print("bar")

because testfunc() yields. However, instead of doing that, I'd like resume() to resume code execution at testfunc() instead of print("bar").

So I've hacked lua_yield() to return -2 instead of -1 and then in lvm.c/OP_CALL I'm checking for -2 and, if it's found, decrement the PC stored in "savedpc" to make lua_resume() run OP_CALL again. The code in lvm.c looks like this:

        ...
          } else if (firstResult > L->top) {  /* yield? */
                lua_assert(L->ci->state == (CI_C | CI_YIELD));
                (L->ci - 1)->u.l.savedpc = (firstResult == L->top + 2) ? pc - 1 : pc;
                (L->ci - 1)->state = CI_SAVEDPC;
                return NULL;
          }
        ...

However, it doesn't work as expected because on resume, OP_CALL doesn't seem to be able to find "testfunc", presumably because the result of the OP_GETGLOBAL preceding OP_CALL isn't in the VM registers any more.

When I decrement the PC by 2 instead of 1, everything works fine because OP_GETGLOBAL is called then to resolve the "testfunc" reference so that OP_CALL can find it. But this is confusing me a little because I thought that yielding/resuming would preserve the complete VM states so that it should be possible to force the VM to just re-execute the OP_CALL that was responsible for yielding but apparently that's not the case.

So is there any other way to force the VM to re-run the function that yielded when resuming code execution on a coroutine?

Note that I'm still on Lua 5.0.

--
Best regards,
 Andreas Falkenhahn                          mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Philippe Verdy
What you want is simply impossible to do in C: your testfunct uses a "return lua_yield(...)" (basically a trailing call that a C compiler may eventually optimize into a jump in some platform-specific conditions, but not always like in Lua). That lua_yield() calls is still in C, and the C stack pointer will return to the point where this call was done, i.e. still inside the "testunct" function where it will return (an optimizing compiler may eventually have resolved it as a jump (after popping the current stack frame of "testfunc" and repolacing it with the stack frame for "lua_yeild" just before dong the jump. The return address of the "testunction()" call is still in the C stack and not modified, but it points to the instruction AFTER the call to testfunc; there's no way to restart the function from the beginning, simply because there's no warranty that the stackframe inside the function "testfunc" is still there and valid (it may have already been destroyed and replaced by the stackframe for lua_yield(...).

Yielding coroutines is not supposed to perform such loop implicitly (and tricking the Lua VM will not help you) : you have to write the loop explicitly yourself in C, around the call to lua_yield() inside the "testfunct" function, in order to allow it to loop, and use a continuation condition for your loop, that may depend on the value returned by lua_yield(), i.e. the value passed by the coroutine.resume(value), or on the value of an external variable...

Coroutines are just pausing and always resuming **after** at the point where they yielded, but **never before that point**.


Le dim. 18 août 2019 à 18:13, Andreas Falkenhahn <[hidden email]> a écrit :
I have a C function which looks like this:

        static int testfunc(lua_State *L)
        {
                static int flag = 0;

                if(!flag) {
                        flag = 1;
                        printf("First call\n");
                        return lua_yield(L, 0);
                } else {
                        printf("Second call\n");
                }

                return 0;
        }

It is called from a coroutine like this:

        co = coroutine.create(function()
                print("foo")
                testfunc()
                print("bar")
        end)

        coroutine.resume(co)
        coroutine.resume(co)

Normally, the second call to coroutine.resume() will continue executing the code at

        print("bar")

because testfunc() yields. However, instead of doing that, I'd like resume() to resume code execution at testfunc() instead of print("bar").

So I've hacked lua_yield() to return -2 instead of -1 and then in lvm.c/OP_CALL I'm checking for -2 and, if it's found, decrement the PC stored in "savedpc" to make lua_resume() run OP_CALL again. The code in lvm.c looks like this:

        ...
          } else if (firstResult > L->top) {  /* yield? */
                lua_assert(L->ci->state == (CI_C | CI_YIELD));
                (L->ci - 1)->u.l.savedpc = (firstResult == L->top + 2) ? pc - 1 : pc;
                (L->ci - 1)->state = CI_SAVEDPC;
                return NULL;
          }
        ...

However, it doesn't work as expected because on resume, OP_CALL doesn't seem to be able to find "testfunc", presumably because the result of the OP_GETGLOBAL preceding OP_CALL isn't in the VM registers any more.

When I decrement the PC by 2 instead of 1, everything works fine because OP_GETGLOBAL is called then to resolve the "testfunc" reference so that OP_CALL can find it. But this is confusing me a little because I thought that yielding/resuming would preserve the complete VM states so that it should be possible to force the VM to just re-execute the OP_CALL that was responsible for yielding but apparently that's not the case.

So is there any other way to force the VM to re-run the function that yielded when resuming code execution on a coroutine?

Note that I'm still on Lua 5.0.

--
Best regards,
 Andreas Falkenhahn                          mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Andreas Falkenhahn-2
I think it is possible, you just need the right trick ;) I've now come up with something that seems to do the job: The key to success was realizing that it is perfectly legal for Lua hooks to yield. Since Lua hooks can be called before every instruction I can just use this insight to yield before OP_CALL instead of during it.

In other words, I just needed to add some code right to the spot where Lua will execute LUA_MASKCOUNT hooks: The code to be added checks if the next instruction is OP_CALL and it also checks if the C function to be executed is one of those few special C functions that should be able to yield and be called again on resume. If it is one of those functions, I'm preparing the stack and call the respective function manually and then I'm calling lua_yield() to yield control back to the callee. Lua now thinks a hook has yielded which is perfectly legal. Et voila, once resume() is called it will now start resuming execution right at the OP_CALL instruction that I've intercepted via a pseudo-hook. All VM states preserved. Seems to work just fine.

I'll have to do some more tests but AFAICS it does exactly what I need and it also seems safe to do so because it makes use of the Lua feature that hooks are allowed to yield so I can just yield before OP_CALL, manually run the C function and then have resume() continue at OP_CALL precisely.

On 19.08.2019 at 01:23 Philippe Verdy wrote:

> What you want is simply impossible to do in C: your testfunct uses
> a "return lua_yield(...)" (basically a trailing call that a C
> compiler may eventually optimize into a jump in some
> platform-specific conditions, but not always like in Lua). That
> lua_yield() calls is still in C, and the C stack pointer will return
> to the point where this call was done, i.e. still inside the
> "testunct" function where it will return (an optimizing compiler may
> eventually have resolved it as a jump (after popping the current
> stack frame of "testfunc" and repolacing it with the stack frame for
> "lua_yeild" just before dong the jump. The return address of the
> "testunction()" call is still in the C stack and not modified, but
> it points to the instruction AFTER the call to testfunc; there's no
> way to restart the function from the beginning, simply because
> there's no warranty that the stackframe inside the function
> "testfunc" is still there and valid (it may have already been
> destroyed and replaced by the stackframe for lua_yield(...).


> Yielding coroutines is not supposed to perform such loop implicitly
> (and tricking the Lua VM will not help you) : you have to write the
> loop explicitly yourself in C, around the call to lua_yield() inside
> the "testfunct" function, in order to allow it to loop, and use a
> continuation condition for your loop, that may depend on the value
> returned by lua_yield(), i.e. the value passed by the
> coroutine.resume(value), or on the value of an external variable...


> Coroutines are just pausing and always resuming **after** at the
> point where they yielded, but **never before that point**.


> Le dim. 18 août 2019 à 18:13, Andreas Falkenhahn
> <[hidden email]> a écrit :

> I have a C function which looks like this:
>  
>          static int testfunc(lua_State *L)
>          {
>                  static int flag = 0;
>  
>                  if(!flag) {
>                          flag = 1;
>                          printf("First call\n");
>                          return lua_yield(L, 0);
>                  } else {
>                          printf("Second call\n");
>                  }
>  
>                  return 0;
>          }
>  
>  It is called from a coroutine like this:
>  
>          co = coroutine.create(function()
>                  print("foo")
>                  testfunc()
>                  print("bar")
>          end)
>  
>          coroutine.resume(co)
>          coroutine.resume(co)
>  
>  Normally, the second call to coroutine.resume() will continue executing the code at
>  
>          print("bar")
>  
>  because testfunc() yields. However, instead of doing that, I'd
> like resume() to resume code execution at testfunc() instead of print("bar").
>  
>  So I've hacked lua_yield() to return -2 instead of -1 and then in
> lvm.c/OP_CALL I'm checking for -2 and, if it's found, decrement the
> PC stored in "savedpc" to make lua_resume() run OP_CALL again. The code in lvm.c looks like this:
>  
>          ...
>            } else if (firstResult > L->top) {  /* yield? */
>                  lua_assert(L->ci->state == (CI_C | CI_YIELD));
>                  (L->ci - 1)->u.l.savedpc = (firstResult == L->top + 2) ? pc - 1 : pc;
>                  (L->ci - 1)->state = CI_SAVEDPC;
>                  return NULL;
>            }
>          ...
>  
>  However, it doesn't work as expected because on resume, OP_CALL
> doesn't seem to be able to find "testfunc", presumably because the
> result of the OP_GETGLOBAL preceding OP_CALL isn't in the VM registers any more.
>  
>  When I decrement the PC by 2 instead of 1, everything works fine
> because OP_GETGLOBAL is called then to resolve the "testfunc"
> reference so that OP_CALL can find it. But this is confusing me a
> little because I thought that yielding/resuming would preserve the
> complete VM states so that it should be possible to force the VM to
> just re-execute the OP_CALL that was responsible for yielding but apparently that's not the case.
>  
>  So is there any other way to force the VM to re-run the function
> that yielded when resuming code execution on a coroutine?
>  
>  Note that I'm still on Lua 5.0.
>  
>  --
>  Best regards,
>   Andreas Falkenhahn                          mailto:[hidden email]
>  
>  
>  


--
Best regards,
 Andreas Falkenhahn                            mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Philippe Verdy
But the "trick" has no warranty to work in C: once you've past over the point of execution where you called a function (here lua_yield()) there's no warranty at all that you can return back to the point where you started the function calling it, because the environment is now different: the state of variables/registers/parameters is lost, and there's still NOTHING left explicitly in your code that allows you to come back and resurrect the initial state at start of the function in which you called lua_yield  (at end of your example function, and possibly even after its termination when you call lua_yield as a trailing return).
Really, think about recreating your C function using an EXPLICIT loop and use the value that MUST be returned from lua_yield() and should be the value passed to coroutine.resume(value) in Lua. Returning at any other point of execution in C will just give unpredicable result and forcibly ignores the value given to coroutine.resume(value).

In summary you C function should do something like:
 my_cfunction() {
    do {
     //...
    } while (lua_yield());
  }
where it explicitly uses the value passed in Lua (here as a boolean, tested by the "while" loop, and that can be passed in Lua as "coroutine.resume(true)" to restart the loop in the function, or "coroutine.resume(false)" to terminate it). You can pass any value in "coroutine.resume(value)" to control how your C function should behave.
But remember that coroutine.resume(...) is not supposed to restart any function but only continue its execution, exactly in its current context and flow, just after the point where lua_yield() returns.

Modifying the VM to do something else will break many uses inclujding in existing libraries where lua_yield() must return immediately after its call and use its value.
Remember that you can use lua_yield inside a sequence of instructions that is not supposed to be interrupted in the middle like in:
 my_cfunction() {
    do_something();
    lua_yield();
    do_something_else();
  }
If you change that behavior in the VM to restart the C function, "do_something_else()" will never be called and you'll loop indefinitely on executing "do_something()".

So your approach is definitely broken. It's not the way to go. Use an explicit loop in C, and you solve the problem CLEANLY without changing anything in the Lua VM.


Le lun. 19 août 2019 à 16:08, Andreas Falkenhahn <[hidden email]> a écrit :
I think it is possible, you just need the right trick ;) I've now come up with something that seems to do the job: The key to success was realizing that it is perfectly legal for Lua hooks to yield. Since Lua hooks can be called before every instruction I can just use this insight to yield before OP_CALL instead of during it.

In other words, I just needed to add some code right to the spot where Lua will execute LUA_MASKCOUNT hooks: The code to be added checks if the next instruction is OP_CALL and it also checks if the C function to be executed is one of those few special C functions that should be able to yield and be called again on resume. If it is one of those functions, I'm preparing the stack and call the respective function manually and then I'm calling lua_yield() to yield control back to the callee. Lua now thinks a hook has yielded which is perfectly legal. Et voila, once resume() is called it will now start resuming execution right at the OP_CALL instruction that I've intercepted via a pseudo-hook. All VM states preserved. Seems to work just fine.

I'll have to do some more tests but AFAICS it does exactly what I need and it also seems safe to do so because it makes use of the Lua feature that hooks are allowed to yield so I can just yield before OP_CALL, manually run the C function and then have resume() continue at OP_CALL precisely.

On 19.08.2019 at 01:23 Philippe Verdy wrote:

> What you want is simply impossible to do in C: your testfunct uses
> a "return lua_yield(...)" (basically a trailing call that a C
> compiler may eventually optimize into a jump in some
> platform-specific conditions, but not always like in Lua). That
> lua_yield() calls is still in C, and the C stack pointer will return
> to the point where this call was done, i.e. still inside the
> "testunct" function where it will return (an optimizing compiler may
> eventually have resolved it as a jump (after popping the current
> stack frame of "testfunc" and repolacing it with the stack frame for
> "lua_yeild" just before dong the jump. The return address of the
> "testunction()" call is still in the C stack and not modified, but
> it points to the instruction AFTER the call to testfunc; there's no
> way to restart the function from the beginning, simply because
> there's no warranty that the stackframe inside the function
> "testfunc" is still there and valid (it may have already been
> destroyed and replaced by the stackframe for lua_yield(...).


> Yielding coroutines is not supposed to perform such loop implicitly
> (and tricking the Lua VM will not help you) : you have to write the
> loop explicitly yourself in C, around the call to lua_yield() inside
> the "testfunct" function, in order to allow it to loop, and use a
> continuation condition for your loop, that may depend on the value
> returned by lua_yield(), i.e. the value passed by the
> coroutine.resume(value), or on the value of an external variable...


> Coroutines are just pausing and always resuming **after** at the
> point where they yielded, but **never before that point**.


> Le dim. 18 août 2019 à 18:13, Andreas Falkenhahn
> <[hidden email]> a écrit :

> I have a C function which looks like this:

>          static int testfunc(lua_State *L)
>          {
>                  static int flag = 0;

>                  if(!flag) {
>                          flag = 1;
>                          printf("First call\n");
>                          return lua_yield(L, 0);
>                  } else {
>                          printf("Second call\n");
>                  }

>                  return 0;
>          }

>  It is called from a coroutine like this:

>          co = coroutine.create(function()
>                  print("foo")
>                  testfunc()
>                  print("bar")
>          end)

>          coroutine.resume(co)
>          coroutine.resume(co)

>  Normally, the second call to coroutine.resume() will continue executing the code at

>          print("bar")

>  because testfunc() yields. However, instead of doing that, I'd
> like resume() to resume code execution at testfunc() instead of print("bar").

>  So I've hacked lua_yield() to return -2 instead of -1 and then in
> lvm.c/OP_CALL I'm checking for -2 and, if it's found, decrement the
> PC stored in "savedpc" to make lua_resume() run OP_CALL again. The code in lvm.c looks like this:

>          ...
>            } else if (firstResult > L->top) {  /* yield? */
>                  lua_assert(L->ci->state == (CI_C | CI_YIELD));
>                  (L->ci - 1)->u.l.savedpc = (firstResult == L->top + 2) ? pc - 1 : pc;
>                  (L->ci - 1)->state = CI_SAVEDPC;
>                  return NULL;
>            }
>          ...

>  However, it doesn't work as expected because on resume, OP_CALL
> doesn't seem to be able to find "testfunc", presumably because the
> result of the OP_GETGLOBAL preceding OP_CALL isn't in the VM registers any more.

>  When I decrement the PC by 2 instead of 1, everything works fine
> because OP_GETGLOBAL is called then to resolve the "testfunc"
> reference so that OP_CALL can find it. But this is confusing me a
> little because I thought that yielding/resuming would preserve the
> complete VM states so that it should be possible to force the VM to
> just re-execute the OP_CALL that was responsible for yielding but apparently that's not the case.

>  So is there any other way to force the VM to re-run the function
> that yielded when resuming code execution on a coroutine?

>  Note that I'm still on Lua 5.0.

>  --
>  Best regards,
>   Andreas Falkenhahn                          mailto:[hidden email]





--
Best regards,
 Andreas Falkenhahn                            mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Philippe Verdy
In reply to this post by Andreas Falkenhahn-2
Another example in C:
{
  dosomething();

  MUTEX *m=getmutex(...);
  while (! lockmutex()) lua_yield();
  do_something_with_the_mutex_owned();
  releasemutex(m);

  dosomethingelse();
}

or:
void sleepfor(int milliseconds)
{
   int until=now()+milliseconds;
   while (now() < until) lua_yield();
   return;
}

lua_yield() is used as a *cooperative* (non-multithreading) mean of allowing concurrent routines to run; it is not preemptive (no need of interrupting the thread at arbitrary points of their execution) and allows the coroutine to eventually (not always) adapt what it will do after lua_yield() returns; for that, the calling function can use the value returned from lua_yield() or use any state in the program (or in external variables of its choice) to decide if, how and where it will continue its execution.

The calling function can as well ignore that value and any other external variable and continue its execution as it wishes (as if it was a debugging breakpoint only): outside the value returned from lua_yield(), the current state of execution and all local variables (including the current value of parameters which may have changed in the function) will keep their semantic of value: lua_yield() is NOT a conditional control of execution in the calling function, it's just a normal C function call that behaves like pause(), except that you don't know if or when lua_yield() will ever return (but if it does, it will return just like a normal C function). That's the whole spirit of "coroutines": **cooperative** multitasking.

There are many ways your C function can handle its execution after lua_yield() returns, and all these ways are possible (and already exist in the standard Lua library, notably for performing "blocking" I/O (including networking: sending or receiving packets; or for handling an UI and even rendering animations/video/audio, or to give control to the user that may want to use the main UI thread to react and instruct your long-running coroutine to be interrupted and stop its execution, but it's still up yo the coroutine that will have to terminate itself and decide to return and up yo it to check the external state set by the UI to detect that the user wanted to interrupt its execution).


Le lun. 19 août 2019 à 16:08, Andreas Falkenhahn <[hidden email]> a écrit :
I think it is possible, you just need the right trick ;) I've now come up with something that seems to do the job: The key to success was realizing that it is perfectly legal for Lua hooks to yield. Since Lua hooks can be called before every instruction I can just use this insight to yield before OP_CALL instead of during it.

In other words, I just needed to add some code right to the spot where Lua will execute LUA_MASKCOUNT hooks: The code to be added checks if the next instruction is OP_CALL and it also checks if the C function to be executed is one of those few special C functions that should be able to yield and be called again on resume. If it is one of those functions, I'm preparing the stack and call the respective function manually and then I'm calling lua_yield() to yield control back to the callee. Lua now thinks a hook has yielded which is perfectly legal. Et voila, once resume() is called it will now start resuming execution right at the OP_CALL instruction that I've intercepted via a pseudo-hook. All VM states preserved. Seems to work just fine.

I'll have to do some more tests but AFAICS it does exactly what I need and it also seems safe to do so because it makes use of the Lua feature that hooks are allowed to yield so I can just yield before OP_CALL, manually run the C function and then have resume() continue at OP_CALL precisely.

On 19.08.2019 at 01:23 Philippe Verdy wrote:

> What you want is simply impossible to do in C: your testfunct uses
> a "return lua_yield(...)" (basically a trailing call that a C
> compiler may eventually optimize into a jump in some
> platform-specific conditions, but not always like in Lua). That
> lua_yield() calls is still in C, and the C stack pointer will return
> to the point where this call was done, i.e. still inside the
> "testunct" function where it will return (an optimizing compiler may
> eventually have resolved it as a jump (after popping the current
> stack frame of "testfunc" and repolacing it with the stack frame for
> "lua_yeild" just before dong the jump. The return address of the
> "testunction()" call is still in the C stack and not modified, but
> it points to the instruction AFTER the call to testfunc; there's no
> way to restart the function from the beginning, simply because
> there's no warranty that the stackframe inside the function
> "testfunc" is still there and valid (it may have already been
> destroyed and replaced by the stackframe for lua_yield(...).


> Yielding coroutines is not supposed to perform such loop implicitly
> (and tricking the Lua VM will not help you) : you have to write the
> loop explicitly yourself in C, around the call to lua_yield() inside
> the "testfunct" function, in order to allow it to loop, and use a
> continuation condition for your loop, that may depend on the value
> returned by lua_yield(), i.e. the value passed by the
> coroutine.resume(value), or on the value of an external variable...


> Coroutines are just pausing and always resuming **after** at the
> point where they yielded, but **never before that point**.


> Le dim. 18 août 2019 à 18:13, Andreas Falkenhahn
> <[hidden email]> a écrit :

> I have a C function which looks like this:

>          static int testfunc(lua_State *L)
>          {
>                  static int flag = 0;

>                  if(!flag) {
>                          flag = 1;
>                          printf("First call\n");
>                          return lua_yield(L, 0);
>                  } else {
>                          printf("Second call\n");
>                  }

>                  return 0;
>          }

>  It is called from a coroutine like this:

>          co = coroutine.create(function()
>                  print("foo")
>                  testfunc()
>                  print("bar")
>          end)

>          coroutine.resume(co)
>          coroutine.resume(co)

>  Normally, the second call to coroutine.resume() will continue executing the code at

>          print("bar")

>  because testfunc() yields. However, instead of doing that, I'd
> like resume() to resume code execution at testfunc() instead of print("bar").

>  So I've hacked lua_yield() to return -2 instead of -1 and then in
> lvm.c/OP_CALL I'm checking for -2 and, if it's found, decrement the
> PC stored in "savedpc" to make lua_resume() run OP_CALL again. The code in lvm.c looks like this:

>          ...
>            } else if (firstResult > L->top) {  /* yield? */
>                  lua_assert(L->ci->state == (CI_C | CI_YIELD));
>                  (L->ci - 1)->u.l.savedpc = (firstResult == L->top + 2) ? pc - 1 : pc;
>                  (L->ci - 1)->state = CI_SAVEDPC;
>                  return NULL;
>            }
>          ...

>  However, it doesn't work as expected because on resume, OP_CALL
> doesn't seem to be able to find "testfunc", presumably because the
> result of the OP_GETGLOBAL preceding OP_CALL isn't in the VM registers any more.

>  When I decrement the PC by 2 instead of 1, everything works fine
> because OP_GETGLOBAL is called then to resolve the "testfunc"
> reference so that OP_CALL can find it. But this is confusing me a
> little because I thought that yielding/resuming would preserve the
> complete VM states so that it should be possible to force the VM to
> just re-execute the OP_CALL that was responsible for yielding but apparently that's not the case.

>  So is there any other way to force the VM to re-run the function
> that yielded when resuming code execution on a coroutine?

>  Note that I'm still on Lua 5.0.

>  --
>  Best regards,
>   Andreas Falkenhahn                          mailto:[hidden email]





--
Best regards,
 Andreas Falkenhahn                            mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Tim Hill
In reply to this post by Andreas Falkenhahn-2


> On Aug 19, 2019, at 7:07 AM, Andreas Falkenhahn <[hidden email]> wrote:
>
> I think it is possible, you just need the right trick ;) I've now come up with something that seems to do the job: The key to success was realizing that it is perfectly legal for Lua hooks to yield. Since Lua hooks can be called before every instruction I can just use this insight to yield before OP_CALL instead of during it.
>

No, Philippe is quite correct, there is no “trick” you can pull to work around this. To implement coroutines, Lua uses “long jumps” in C, and these implicitly demolish the C call stack. This means that lua_yield() NEVER RETURNS and any function that calls lua_yield() is abandoned (and any nested outer C functions as well). Whatever you do can not work around the fact that to yield a coroutine MUST demolish the C stack but the very way in which Lua intermixes the Lua and C stacks.

—Tim



Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Andreas Falkenhahn-2
There's probably a misunderstanding here: Of course I'm not intending to have the C function continue where it left off. This is obviously impossible. I just want the C function to be called again. The C function code will always be run from its entry point, not from somewhere in between.

On 20.08.2019 at 08:58 Tim Hill wrote:

>> On Aug 19, 2019, at 7:07 AM, Andreas Falkenhahn <[hidden email]> wrote:

>> I think it is possible, you just need the right trick ;) I've now come up with something that seems to do the job: The key to success was realizing that it is perfectly legal for Lua hooks to yield. Since Lua hooks can be called before every instruction I can just use this insight to yield before OP_CALL instead of during it.


> No, Philippe is quite correct, there is no “trick” you can pull to
> work around this. To implement coroutines, Lua uses “long jumps” in
> C, and these implicitly demolish the C call stack. This means that
> lua_yield() NEVER RETURNS and any function that calls lua_yield() is
> abandoned (and any nested outer C functions as well). Whatever you
> do can not work around the fact that to yield a coroutine MUST
> demolish the C stack but the very way in which Lua intermixes the Lua and C stacks.

> —Tim






--
Best regards,
 Andreas Falkenhahn                            mailto:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Philippe Verdy
In reply to this post by Tim Hill
Actually the "long jump" does NOT destroy the call stack: it goes to some piece of code that will save the call stack state, before foing to another suitable and runnnable coroutine for which it will restore its call stack.

The long jump in C by itself does not destroy anything, except in its default implementation (in the main application thread) that does this extreme cleanup (and it is used to allow terminating a program properly after catching all pending exceptions; eventually the long jump handler to which you go can avoid this cleanup.
Long jumps in C are used for implementing exception handlers, they are also used by C++'s own exception handler (which of course must not destroy the call stack).

As you can see in the example shown in https://www.tutorialspoint.com/setjump-and-longjump-in-c,
the local variables are preserved, meaning that the call stack is also preserved, and in that example (which creates a loop) the C function doing that will return normally to the caller after performing the loops, which are counted normally.

All I said is that long jumps are just like breakpoints in a debugger: it preserves the current execution state, which can be saved before switching to another context by terminating the jump to the location immediately after where "setjmp()" was called in that other context, where the state (including its call stack) will be restored.

The setjmp() and longjmp() functions in C do not perform any context switching or something that would allow the call stack to be preserved: it's up to the C code to do that class stack saving/restoring. That's why setjmp() and longjmp() are rarely used directly in C applications, but used within a library that will handle properly the context switching (and this is the case for "Structured Error Handling" (SEH) in C, or standard C++ exception handlers, or for the Lua_C library that implements the Lua's coroutines, or for other similar mechanisms used in OSes (like the cooperative "multitasking" in old 16-bit versions of Windows, whose kernel still was not preemptive).

Preemptive multitasking (i.e. true multithreading) however can be implemented in C (or other languages), by adding a hardware timer interrupt handler that will encapsulate the setjmp/longjmp: this allows creating a true multithreading kernel in DOS for example.

[I did that in 1982-1984 in Pascal, with basic timer interrupt handlers (but without even needing to use setjmp/longjmp but handling the context switching, including switching stacks and saving/restoring CPU/FPU registers, with a small x86 assembly code). It was used for a networking server application and at that time there was still no free Linux, and 16-bit Windows did not even exist to implement its cooperative multitasking also based on coroutines. My code created a really multithreaded program, but there was initially no virtual memory management so no real multiprocessing in the first version and it was still constrained by the 640KB limit in which the OS was also living, plus 384KB for devices; later I added the use of DOS's VMM/EMM for switching also memory contexts and allow true multitasking, extending the total usable memory usable to several megabytes where each process could have 196KB of process-private memory in the paged device area plus some other pages in the low virtual 640KB space, when DOS only allowed 640KB of virtual memory in a single process; it worked even without hardware CPU support for the virtual memory, but used the EMM hardware in the external north-bridge chipset controlling memory paging, using the "EMM286" DOS driver, and later the EMM386 DOS driver using the first hardware integrated CPU support; later came virtual DOS and the first x86 free Unix-based kernels including Linux; it took many years before Windows switched to Win32 with a true preemptive multiprocessing, not just preemptive multithreading, and a decent virtual memory manager; during these years, Unix and NT/Win32 were completely proprietary and costly). Finally this server was extended to use 32-bit registers and 48-bit hardware memory addresses, and all the OS was running in a separate process, allowing each process to use up to 1MB, and even more when using explicit paging, for file-mappings or I/O buffers, networking packets, audio/video frames, or memory-resident databases and filesystem caches.]

"setjmp/longjmp" are core elements needed in C which is very complex to write and test (it's implemented in assembly and highly platform/CPU specific), but beside this pair that can be isolated in a small piece of code, you can then implement very powerful things and reimplement any VM or complete OS with them, using portable code.


Le mar. 20 août 2019 à 09:55, Tim Hill <[hidden email]> a écrit :


> On Aug 19, 2019, at 7:07 AM, Andreas Falkenhahn <[hidden email]> wrote:
>
> I think it is possible, you just need the right trick ;) I've now come up with something that seems to do the job: The key to success was realizing that it is perfectly legal for Lua hooks to yield. Since Lua hooks can be called before every instruction I can just use this insight to yield before OP_CALL instead of during it.
>

No, Philippe is quite correct, there is no “trick” you can pull to work around this. To implement coroutines, Lua uses “long jumps” in C, and these implicitly demolish the C call stack. This means that lua_yield() NEVER RETURNS and any function that calls lua_yield() is abandoned (and any nested outer C functions as well). Whatever you do can not work around the fact that to yield a coroutine MUST demolish the C stack but the very way in which Lua intermixes the Lua and C stacks.

—Tim



Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Gé Weijers
In reply to this post by Andreas Falkenhahn-2


On Tue, Aug 20, 2019 at 4:24 AM Andreas Falkenhahn <[hidden email]> wrote:
There's probably a misunderstanding here: Of course I'm not intending to have the C function continue where it left off. This is obviously impossible. I just want the C function to be called again. The C function code will always be run from its entry point, not from somewhere in between.


You'd be better off switching to Lua 5.3, where you can use lua_yieldk to specify a C function that's called on resume. This function will find the Lua stack as you left it, with the results yielded replaced by the arguments to resume.

I have not used lua_yieldk much, but I do use lua_callk an awful lot so Lua code can call C code that then calls Lua code, which can yield if it wants to.


--

Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Tim Hill
In reply to this post by Philippe Verdy


On Aug 20, 2019, at 9:23 AM, Philippe Verdy <[hidden email]> wrote:

Actually the "long jump" does NOT destroy the call stack: it goes to some piece of code that will save the call stack state, before foing to another suitable and runnnable coroutine for which it will restore its call stack.


Apologies, I should have been more precise. What I meant by “destroy” was that a C long jump discards the C stack frames for any C functions called after the context used for the long jump. I was not referring to the Lua stack, which of course is managed outside of the C setjmp/longjmp mechanism.

—Tim


Reply | Threaded
Open this post in threaded view
|

Re: Forcing coroutine.resume to call the yielding function again

Philippe Verdy
In reply to this post by Gé Weijers
No, there's a solution, but it's do do in the C function itself, not in the Lua VM.
I already gave an example, all that is needed for that use is to write your C function as a loop around lua_yield(), allowing the code to bo restarted at the next loop...
But note that the C function will still not be "recalled", only a loop will occur, meaning that local variables and local function parameters are kept with their current value and not with the values they had on the first entry in the C function: if you need the function parameter values to be preserved, you need to write your C function using additional local variables that can be modified, and initialized from the function parameters.
Then use the return value of lua_yield() (or of other external variables accessible from the local varaibles or parameters or from gobal varaibles) to decide if you need to loop again.

Basically lua_yield() behaves like setjmp(): it also returns the value given in coroutine.resume(value), like setjmp() returns the value given in longjmp(); but the difference is that setjmp() will return one or more times from a single call; the first time it will return 0 after saving the current stack frame pointer, the next times it will return the value given in the second parameter to longjmp(), which "unwinds" the stack by restoring the stack frame pointer.

Note also that both lua_yield() and setjmp() are creating a "register barrier": this means that any variables that were cached in CPU registers before the call are no longer valid, they are considered "scratch registers", so the C compiler will need to reload these registers from local variables or parameters from the stack, or from external variables: setjmp() does NOT save these scratch registers, it just saves the stack frame pointer.

Implementing setjmp() is quite simple, what can be complex to implement in "longjmp()" because it may need to prepare the CPU in a state that allows unwinding the stack by overwriting the current frame pointer. But longjmp() does not free up any other resources: by unwinding the stack abruptly, it may leave leaked resources like dynamically allocated memory, open file handles (that won't be closed at all), locked mutexes and so on. If some of these leaked resources can cause the program state to become completely unworkable and invalid, special OS-specific behavior will be made (eventually there will also need to patch setjmp() so that it saves a bit more data to allow "safe" unwinding so that the program will not simply crash: this behavior is completely dependent of the OS-specific ABI and sometimes the C compiler ABI itself (and notably the way it performs optimizations, or allows debugging).

That's why "lua_yield" does not attempt to "emulate" what setjmp()/longjmp() do, it just use them directly as implemented by the C compiler and platform (doing anything else would make the Lua VM highly non-portable, and the same is true if you attempt top "patch" incorrectly what lua_yield() does by "playing" with the stack yourself: it will never be safe. So your code may work on one machine, with a specific OS, a specific compiler, a specific compilation mode, and for some specific CPU models or in specific contexts of use possibly depending on other workloads or depending if the OS is a native OS or a virtualized OS, but will crash on another machine or will have unexpected results and unpredictable behavior; it's exactly the same kind of problems you would encounter by trying to use dynamically allocated memory after if has been explicitly freed, even if it's not erased/overwritten immediately when freeing it). The C compiler never warranties you can safely return to a point of execution and state before the state that was explicitly preserved (here by setjmp() or by lua_yield(), which is not the point where your C function was first called) unless you've explicitly prepared your C code to allow such "restart" (i.e. by using an explicit loop in C).




Le mar. 20 août 2019 à 20:11, Gé Weijers <[hidden email]> a écrit :


On Tue, Aug 20, 2019 at 4:24 AM Andreas Falkenhahn <[hidden email]> wrote:
There's probably a misunderstanding here: Of course I'm not intending to have the C function continue where it left off. This is obviously impossible. I just want the C function to be called again. The C function code will always be run from its entry point, not from somewhere in between.


You'd be better off switching to Lua 5.3, where you can use lua_yieldk to specify a C function that's called on resume. This function will find the Lua stack as you left it, with the results yielded replaced by the arguments to resume.

I have not used lua_yieldk much, but I do use lua_callk an awful lot so Lua code can call C code that then calls Lua code, which can yield if it wants to.


--