Deep resume and yield for nested coroutine

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Deep resume and yield for nested coroutine

pocomane

I am playing with the lua vm in the browser [1]. Both browser and lua have a collaborative multitasking design, so I would like to use coroutine for the interaction.

The whole lua script is wrapped in a coroutine, and the browser resume it until it goes to the 'dead' state. Ideally I can wrap browser actions in some lua function that yields. On the javascript side there will be something that parses the yielded values.

This does not works when the "Browser interaction api" is called from a sub-coroutine created in the lua code. Or, at least, I need to propagate the "Yielding" to the parent coroutine, and the "Resuming" to the child one [2].

Obviously, to make this 'Deep yielding/resuming' as much transparent as possible, I have to replace coroutine.resume and friends. So I ask:

Am I missing something? Is there some simpler solution?

It seems to me a quite natural scenario for a coroutine, also in a non-browser environment...

pocomane

[2] I.e. I need something like the follolwing prototype code :

```
-- Nested Coroutine

local THREAD = 1
local SUCCESS = 2
local TARGET = 3
local PARENT = 4

local nccurrent = nil

local function nccreate(f)
  local nc = {0,0,0,0}
  nc[THREAD] = coroutine.create(f)
  return nc
end

local function ncyield(nc, ...)
  nc[TARGET] = nc[THREAD]
  return coroutine.yield(nc, ...)
end

local function ncresume(nc, ...)
  nc[PARENT] = nccurrent
  nccurrent = nc
  return (function(a, b, ...)
      nc[SUCCESS] = a
      if coroutine.status(nc[THREAD]) ~= 'suspended' then
        return b, ...
      end
      if nc == b then
          b = nil
      end
      nc[TARGET] = b
      if nc[PARENT] == nil then
        return b, ...
      else
        return coroutine.yield(b, ...)
      end
  end)(coroutine.resume(nc[THREAD], ...))
end
``

Usage example:

```
local top, middle, bottom
local out = ''

out = out .. 'a'

top = nccreate(function()
  out = out .. 'b'
  
  middle = nccreate(function()
    out = out .. 'c'
    
    bottom = nccreate(function()
        out = out .. 'd'
        
        ncyield(middle)
        
        out = out .. '-UNREACHED-'
    end)
    ncresume(bottom)
    
    out = out .. '-UNREACHED-'
  end)
  ncresume(middle)
  
  out = out .. 'f'
end)
ncresume(top)

out = out .. 'e'
ncresume(top)

assert(out=='abcdef')
```

A production code should handle several cases (e.g. errors) and I am a bit unsure about its performance.

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

pocomane


Il gio 9 mag 2019, 11:33 pocomane <[hidden email]> ha scritto:

I am playing with the lua vm in the browser [1]. Both browser and lua have a collaborative multitasking design, so I would like to use coroutine for the interaction.

The whole lua script is wrapped in a coroutine, and the browser resume it until it goes to the 'dead' state. Ideally I can wrap browser actions in some lua function that yields. On the javascript side there will be something that parses the yielded values.

This does not works when the "Browser interaction api" is called from a sub-coroutine created in the lua code. Or, at least, I need to propagate the "Yielding" to the parent coroutine, and the "Resuming" to the child one [2].

...

It seems to me a quite natural scenario for a coroutine, also in a non-browser environment...

pocomane

[2] I.e. I need something like the follolwing prototype code :


My previous example was wrong, so I attach a new simpler one. I hope that it is more readable. The question is still:

Is there some other solution that I can not think of?

I just want to fully stop the lua VM, to process some data outside lua, and then to restart the VM exactly where it did stop. To do this transparently with the proposed code, a lot of auxiliary code is required, e.g. to decide if perform a `nestresume` or a normal one, to handle errors, to wrap the standard coroutine functions, and so on.

```

local function nestresume(nc, ...)

      local res = table.pack(coroutine.resume(nc, ...))

      -- Skip the yield/resume propagation as needed
      if not coroutine.isyieldable() or coroutine.status(nc) ~= 'suspended' then
        return table.unpack(res)
      end

      -- Propagate the yielding upstream
      local res = table.pack(coroutine.yield(table.unpack(res)))

      -- Propagate the resuming downstream
      return nestresume(nc, table.unpack(res)) 
end

-- Usage example:

local outer, inner
local out = ''

outer = coroutine.create(function()
  
  inner = coroutine.create(function()
    out = out .. 'c'
    coroutine.yield()
    out = out .. 'e' -- with std resume, this whould have been skipped
  end)

  out = out .. 'b'
  nestresume(inner)
  out = out .. 'f'
end)

out = out .. 'a'
nestresume(outer)
out = out .. 'd' -- with std resume, this would have been run after 'f'
nestresume(outer) -- with std resume, this was not required

print(out)
assert(out=='abcdef')
```

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

Sean Conner
It was thus said that the Great pocomane once stated:
>
> I just want to fully stop the lua VM, to process some data outside lua, and
> then to restart the VM exactly where it did stop. To do this transparently
> with the proposed code, a lot of auxiliary code is required, e.g. to decide
> if perform a `nestresume` or a normal one, to handle errors, to wrap the
> standard coroutine functions, and so on.

  Perhaps I don't understand what's going on, but once Lua calls into C, the
Lua VM isn't running.  Are you perhaps looking for some form of preemptive
task switching for Lua coroutines?

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

Francisco Olarte
In reply to this post by pocomane
I skipped the first message but:

On Tue, May 14, 2019 at 5:22 PM pocomane <[hidden email]> wrote:
> Il gio 9 mag 2019, 11:33 pocomane <[hidden email]> ha scritto:
>> The whole lua script is wrapped in a coroutine, and the browser resume it until it goes to the 'dead' state. Ideally I can wrap browser actions in some lua function that yields. On the javascript side there will be something that parses the yielded values.
>> This does not works when the "Browser interaction api" is called from a sub-coroutine created in the lua code. Or, at least, I need to propagate the "Yielding" to the parent coroutine, and the "Resuming" to the child one [2].

Why do you need this? I did a similar thing, more or less:
- when you start the lua stuff, you create a corourine and start it.
- When lua code wants to call API it calls a routine published in the
interpreter which yields a special token ( a lightuserdata, or the
funcion per se ) plus the function it wants to call and the
parameters, control returns to C.
- C records the coroutine which yielded somewhere ( registry? ), and
calls the API.
- Api comes back, recover the suspended coroutine from where you
stored it and return it.

If API has created a coroutine, the main one is already "dormant", and
the child, which will be woken up on API return, is the responsible of
returning to it.

IIRC I also had to send API messages ( and call lua from message
handlers ), and what I did was to pass a "usertoken" in the message,
which was a pointer to a heap allocated pointer to the (main) lua
state and use it as a lightuserdata key in the registry to the
coroutine, so the api code, when invoked on message return, could
recover the state pointer, recover the coroutine from the registry,
free the usertoken and resume the correct coroutine.

I do not have the code handy, and it was tailored for the specific
host, but I do not remember having problems.

> I just want to fully stop the lua VM, to process some data outside lua, and then to restart the VM exactly where it did stop. To do this transparently with the proposed code, a lot of auxiliary code is required, e.g. to decide if perform a `nestresume` or a normal one, to handle errors, to wrap the standard coroutine functions, and so on.

I do not think you need to resume the main coroutine, just resume the
one calling the API.

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

Philippe Verdy
Isn't it possible to just compile Lua with a C compiler targetting 

WebAssembly

?
It is more or less an hypervisor implemented in Javascript used to boot a Linux-like kernel, and already working across 4 major browsers (Firefox, Chrome/Chromium, Safari and IE) on Windows, MacOSX and Linux (with the builtin Javascript engines of these browsers) ?
Did someone try it ? Could Lua be successfully compiled with its toolchain (which includes the C compiler and most shell and dev tools from Linux) ?
I cannot be so bad in perfomance, given the demos including games made with Unity. The mini Linux kernel run by Webasembly boots really fast and applications made with it are also very small to download. Behind the scene, there's the LLVM engine, but we've seen Mozilla and Google working to boost WebAssembly. And soon Microsoft will abandon Edge to rebuild it from the Webkit/Chromium base, so it will hopefully support WebAssembly)


Le mar. 14 mai 2019 à 17:47, Francisco Olarte <[hidden email]> a écrit :
I skipped the first message but:

On Tue, May 14, 2019 at 5:22 PM pocomane <[hidden email]> wrote:
> Il gio 9 mag 2019, 11:33 pocomane <[hidden email]> ha scritto:
>> The whole lua script is wrapped in a coroutine, and the browser resume it until it goes to the 'dead' state. Ideally I can wrap browser actions in some lua function that yields. On the javascript side there will be something that parses the yielded values.
>> This does not works when the "Browser interaction api" is called from a sub-coroutine created in the lua code. Or, at least, I need to propagate the "Yielding" to the parent coroutine, and the "Resuming" to the child one [2].

Why do you need this? I did a similar thing, more or less:
- when you start the lua stuff, you create a corourine and start it.
- When lua code wants to call API it calls a routine published in the
interpreter which yields a special token ( a lightuserdata, or the
funcion per se ) plus the function it wants to call and the
parameters, control returns to C.
- C records the coroutine which yielded somewhere ( registry? ), and
calls the API.
- Api comes back, recover the suspended coroutine from where you
stored it and return it.

If API has created a coroutine, the main one is already "dormant", and
the child, which will be woken up on API return, is the responsible of
returning to it.

IIRC I also had to send API messages ( and call lua from message
handlers ), and what I did was to pass a "usertoken" in the message,
which was a pointer to a heap allocated pointer to the (main) lua
state and use it as a lightuserdata key in the registry to the
coroutine, so the api code, when invoked on message return, could
recover the state pointer, recover the coroutine from the registry,
free the usertoken and resume the correct coroutine.

I do not have the code handy, and it was tailored for the specific
host, but I do not remember having problems.

> I just want to fully stop the lua VM, to process some data outside lua, and then to restart the VM exactly where it did stop. To do this transparently with the proposed code, a lot of auxiliary code is required, e.g. to decide if perform a `nestresume` or a normal one, to handle errors, to wrap the standard coroutine functions, and so on.

I do not think you need to resume the main coroutine, just resume the
one calling the API.

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

William Ahern
On Tue, May 14, 2019 at 07:09:46PM +0200, Philippe Verdy wrote:
> Isn't it possible to just compile Lua with a C compiler targetting
> WebAssembly <https://webassembly.org/>?

FWIW, WebAssembly is hostile to coroutines and stackful coroutines in
particular. It does not make a good target for Lua, Go, or similar languages
that reify the logical execution stack. They require an entire extra layer
of indirection which will always make them second-class citizens in WASM.

That said, you can of course compile the PUC Lua VM to WASM today, and
people have. I don't see much value in it, but nothing is stopping you. Give
it a try and many things will become clear in ways that would be difficult
to communicate.

> It is more or less an hypervisor implemented in Javascript used to boot a
> Linux-like kernel, and already working across 4 major browsers (Firefox,
> Chrome/Chromium, Safari and IE) on Windows, MacOSX and Linux (with the
> builtin Javascript engines of these browsers) ?
> Did someone try it ? Could Lua be successfully compiled with its toolchain
> (which includes the C compiler and most shell and dev tools from Linux) ?
> I cannot be so bad in perfomance, given the demos including games made with
> Unity. The mini Linux kernel run by Webasembly boots really fast and
> applications made with it are also very small to download. Behind the
> scene, there's the LLVM engine, but we've seen Mozilla and Google working
> to boost WebAssembly. And soon Microsoft will abandon Edge to rebuild it
> from the Webkit/Chromium base, so it will hopefully support WebAssembly)
>
>
> Le mar. 14 mai 2019 à 17:47, Francisco Olarte <[hidden email]> a
> écrit :
>
> > I skipped the first message but:
> >
> > On Tue, May 14, 2019 at 5:22 PM pocomane <[hidden email]> wrote:
> > > Il gio 9 mag 2019, 11:33 pocomane <[hidden email]> ha scritto:
> > >> The whole lua script is wrapped in a coroutine, and the browser resume
> > it until it goes to the 'dead' state. Ideally I can wrap browser actions in
> > some lua function that yields. On the javascript side there will be
> > something that parses the yielded values.
> > >> This does not works when the "Browser interaction api" is called from a
> > sub-coroutine created in the lua code. Or, at least, I need to propagate
> > the "Yielding" to the parent coroutine, and the "Resuming" to the child one
> > [2].
> >
> > Why do you need this? I did a similar thing, more or less:
> > - when you start the lua stuff, you create a corourine and start it.
> > - When lua code wants to call API it calls a routine published in the
> > interpreter which yields a special token ( a lightuserdata, or the
> > funcion per se ) plus the function it wants to call and the
> > parameters, control returns to C.
> > - C records the coroutine which yielded somewhere ( registry? ), and
> > calls the API.
> > - Api comes back, recover the suspended coroutine from where you
> > stored it and return it.
> >
> > If API has created a coroutine, the main one is already "dormant", and
> > the child, which will be woken up on API return, is the responsible of
> > returning to it.
> >
> > IIRC I also had to send API messages ( and call lua from message
> > handlers ), and what I did was to pass a "usertoken" in the message,
> > which was a pointer to a heap allocated pointer to the (main) lua
> > state and use it as a lightuserdata key in the registry to the
> > coroutine, so the api code, when invoked on message return, could
> > recover the state pointer, recover the coroutine from the registry,
> > free the usertoken and resume the correct coroutine.
> >
> > I do not have the code handy, and it was tailored for the specific
> > host, but I do not remember having problems.
> >
> > > I just want to fully stop the lua VM, to process some data outside lua,
> > and then to restart the VM exactly where it did stop. To do this
> > transparently with the proposed code, a lot of auxiliary code is required,
> > e.g. to decide if perform a `nestresume` or a normal one, to handle errors,
> > to wrap the standard coroutine functions, and so on.
> >
> > I do not think you need to resume the main coroutine, just resume the
> > one calling the API.
> >
> > Francisco Olarte.
> >
> >

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

Sean Conner
It was thus said that the Great William Ahern once stated:

> On Tue, May 14, 2019 at 07:09:46PM +0200, Philippe Verdy wrote:
> > Isn't it possible to just compile Lua with a C compiler targetting
> > WebAssembly <https://webassembly.org/>?
>
> FWIW, WebAssembly is hostile to coroutines and stackful coroutines in
> particular. It does not make a good target for Lua, Go, or similar languages
> that reify the logical execution stack. They require an entire extra layer
> of indirection which will always make them second-class citizens in WASM.
>
> That said, you can of course compile the PUC Lua VM to WASM today, and
> people have. I don't see much value in it, but nothing is stopping you. Give
> it a try and many things will become clear in ways that would be difficult
> to communicate.

  And let us not forget that C should not be used.  As someone on this list
recently said:

> In summary, C is a malware to eradicate, unless it is implemented by
> replacing all undefined behaviors from the so called "standard" by defined
> behavior.
>
> For now the C standard is not made for any one. Its goals are stupid. The
> whole team of this standard body (most of them from ANSI) should be fired
> (or another standard body should take ove the goal of redefining it).
> Everyone in the computing industry will be interested and now every one in
> the world uses computers and are interested: this goal is no longer
> technical, it has become a fundamental human right to absolutely preserve,
> but voluntarily ignored by the existing standard body which behaves like
> devil.

  So we should all forget about C entirely.

  -spc (Unfortunately, that means PUC Lua goes away as well ... )
 

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

William Ahern
In reply to this post by William Ahern
On Tue, May 14, 2019 at 12:12:03PM -0700, William Ahern wrote:
> On Tue, May 14, 2019 at 07:09:46PM +0200, Philippe Verdy wrote:
> > Isn't it possible to just compile Lua with a C compiler targetting
> > WebAssembly <https://webassembly.org/>?
>
> FWIW, WebAssembly is hostile to coroutines and stackful coroutines in
> particular. It does not make a good target for Lua, Go, or similar languages
> that reify the logical execution stack. They require an entire extra layer
> of indirection which will always make them second-class citizens in WASM.

For reference:

 http://troubles.md/posts/why-do-we-need-the-relooper-algorithm-again/
 https://github.com/WebAssembly/design/issues/796#issuecomment-401310366


Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

pocomane
On Tue, May 14, 2019 at 7:10 PM Philippe Verdy <[hidden email]> wrote:
>
> Isn't it possible to just compile Lua with a C compiler targetting  WebAssembly?
...
> Did someone try it ? Could Lua be successfully compiled with its toolchain (which includes the C compiler and most shell and dev tools from Linux) ?

I do not understand if you are asking for *Other* ones doing that...
however, yes, I did in [1]. There is also an online version if you
want to make some experimentation (serach for "playground" in [1]).

> I cannot be so bad in perfomance, given the demos including games made with Unity. The mini Linux kernel run by Webasembly boots really fast and applications made with it are also very small to download. Behind the scene, there's the LLVM engine, but we've seen Mozilla and Google working to boost WebAssembly. And soon Microsoft will abandon Edge to rebuild it from the Webkit/Chromium base, so it will hopefully support WebAssembly)
>

Whithout an heavy use of coroutine/error (i.e. longjmp), I cannot see
any performance loss wrt PUC-Rio lua compiled to binary. By the way,
it was not a very deep benchmark.

On Tue, May 14, 2019 at 5:41 PM Sean Conner <[hidden email]> wrote:
>   Perhaps I don't understand what's going on, but once Lua calls into C, the
> Lua VM isn't running.  Are you perhaps looking for some form of preemptive
> task switching for Lua coroutines?

> Le mar. 14 mai 2019 à 17:47, Francisco Olarte <[hidden email]> a écrit :
>> Why do you need this? I did a similar thing, more or less:

I did this mostly to avoid messing with emscripten interface. I liked
to keep that code small, and write as much as I can in the sole lua
and javascript. However, I agreed with you: a proper C api published
to the lua interpreter is better than my proposed solution. This is
why I was asking for alternative in my fist message.

On Tue, May 14, 2019 at 9:12 PM William Ahern
<[hidden email]> wrote:
> FWIW, WebAssembly is hostile to coroutines and stackful coroutines in
> particular. It does not make a good target for Lua, Go, or similar languages
> that reify the logical execution stack. They require an entire extra layer
> of indirection which will always make them second-class citizens in WASM.

True, but this is just a limitation of WASM 1.0/MVP. For now, there is
no way to implement longjmp fully in WASM, so the toolchain must
switch back and forth between WASM and javascript to handle it.
However there is already some work for a better implementation that we
could have in the next WASM revision [2].

pocomane

[1] https://github.com/pocomane/walua
[2] https://github.com/WebAssembly/exception-handling/blob/master/proposals/Exceptions.md

Reply | Threaded
Open this post in threaded view
|

Re: Deep resume and yield for nested coroutine

Francisco Olarte
pocomane:

On Wed, May 15, 2019 at 7:31 AM pocomane <[hidden email]> wrote:
... Snip, snip...,
> > Le mar. 14 mai 2019 à 17:47, Francisco Olarte <[hidden email]> a écrit :
> >> Why do you need this? I did a similar thing, more or less:
>
> I did this mostly to avoid messing with emscripten interface. I liked
> to keep that code small, and write as much as I can in the sole lua
> and javascript. However, I agreed with you: a proper C api published
> to the lua interpreter is better than my proposed solution. This is
> why I was asking for alternative in my fist message.

Ok, I see it. No C code wanted, gluing is more difficult. I normally
have some c/c++ gluing every system around, so I do not have this kind
of problems.

Francisco Olarte.