lua_yieldk/lua_resume not removing yielded values

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

lua_yieldk/lua_resume not removing yielded values

Jerome Vuarand
Hi list,

I'm trying to port to C a framework of mine that is basically a
coroutine scheduler hidden under a pseudo-blocking I/O API (not mature
enough to be released, but ask for a URL if interested). I have
several pure-Lua implementations (with various backend dependencies),
and I'm trying to re-implement it in C. That was the perfect occasion
to dive into Lua 5.2's new lua_yieldk feature. But I'm faced with an
oddity, which I believe is either a bug or a documentation problem.

When a first call to lua_resume returns, only the yielded values are
on the stack. Any other values in the thread stack (in the
lua_CFunction that called lua_yieldk) are not accessible (but will be
in the continuation). So far so good. However if I let those return
values there, push a couple other ones, and call lua_resume with nargs
equal to 1 or 2, the continuation receives 1) the non returned value
from the initial coroutine resume at the bottom, 2) the nargs values
passed to the second resume at the top, but also 3) any other yielded
values, and any other pushed values before the second call to
lua_resume. In other words it seems the nargs of a second lua_resume
seems to be ignored, and all values on the thread stack at the moment
of the resume are passed to the continuation function.

Here is a small program showing the problem:

#include <lua.h>
#include <lauxlib.h>
#include <stdlib.h>

static int func_k1(lua_State* L)
{
    int i;
    printf("continuation got %d values\n", lua_gettop(L));
    for (i=1; i<=lua_gettop(L); ++i)
    {
        printf("\t%s %s\n", luaL_typename(L, i), lua_type(L,
i)==LUA_TSTRING ? lua_tostring(L, i) : "");
    }
    return 0;
}

static int func(lua_State* L)
{
    int i;
    lua_pushstring(L, "data 1");
    lua_pushstring(L, "data 2");
    lua_pushstring(L, "yielded value 1");
    lua_pushstring(L, "yielded value 2");
    printf("coroutine yields %d values out of %d:\n", 2, lua_gettop(L));
    for (i=1; i<=lua_gettop(L); ++i)
    {
        printf("\t%s %s\n", luaL_typename(L, i), lua_type(L,
i)==LUA_TSTRING ? lua_tostring(L, i) : "");
    }
    return lua_yieldk(L, 2, 0, func_k1);
}

int main()
{
    lua_State* L,* T;
    int result, i;
    L = luaL_newstate();
    T = lua_newthread(L);
    lua_pushcfunction(T, func);
    lua_pushstring(T, "starting value 1");
    lua_pushstring(T, "starting value 2");
    result = lua_resume(T, L, 2);
    if (result != 0 && result != LUA_YIELD)
        printf("error in first resume: %s\n", lua_tostring(T, -1));
    else
    {
        printf("main thread got %d values:\n", lua_gettop(T));
        for (i=1; i<=lua_gettop(T); ++i)
        {
            printf("\t%s %s\n", luaL_typename(T, i), lua_type(T,
i)==LUA_TSTRING ? lua_tostring(T, i) : "");
        }
    }
    lua_pushstring(T, "resumed value 1");
    lua_pushstring(T, "resumed value 2");
    printf("resuming with 1 argument out of %d:\n", lua_gettop(T));
    for (i=1; i<=lua_gettop(T); ++i)
    {
        printf("\t%s %s\n", luaL_typename(T, i), lua_type(T,
i)==LUA_TSTRING ? lua_tostring(T, i) : "");
    }
    result = lua_resume(T, L, 1);
    if (result != 0 && result != LUA_YIELD)
        printf("error in second resume: %s\n", lua_tostring(T, -1));
    lua_close(L);
    return 0;
}

And here is the output I get:

coroutine yields 2 values out of 6:
        string starting value 1
        string starting value 2
        string data 1
        string data 2
        string yielded value 1
        string yielded value 2
main thread got 2 values:
        string yielded value 1
        string yielded value 2
resuming with 1 argument out of 4:
        string yielded value 1
        string yielded value 2
        string resumed value 1
        string resumed value 2
continuation got 8 values
        string starting value 1
        string starting value 2
        string data 1
        string data 2
        string yielded value 1
        string yielded value 2
        string resumed value 1
        string resumed value 2

Instead I expected the last section to read:
continuation got 5 values
        string starting value 1
        string starting value 2
        string data 1
        string data 2
        string resumed value 2

Did I misinterpret something, or is there a problem somewhere (with
lua_resume, lua_yieldk, or the doc of one of these)?

Doub.

Reply | Threaded
Open this post in threaded view
|

Re: lua_yieldk/lua_resume not removing yielded values

Roberto Ierusalimschy
> [...] That was the perfect occasion
> to dive into Lua 5.2's new lua_yieldk feature. But I'm faced with an
> oddity, which I believe is either a bug or a documentation problem.
>
> When a first call to lua_resume returns, only the yielded values are
> on the stack. Any other values in the thread stack (in the
> lua_CFunction that called lua_yieldk) are not accessible (but will be
> in the continuation). So far so good. However if I let those return
> values there, push a couple other ones, and call lua_resume with nargs
> equal to 1 or 2, the continuation receives 1) the non returned value
> from the initial coroutine resume at the bottom, 2) the nargs values
> passed to the second resume at the top, but also 3) any other yielded
> values, and any other pushed values before the second call to
> lua_resume. In other words it seems the nargs of a second lua_resume
> seems to be ignored, and all values on the thread stack at the moment
> of the resume are passed to the continuation function.

This is a documentation problem. When a coroutine resumes after the
first time, the stack cannot contain anything besides the arguments to
resume. 'nargs' is mostly ignored (except for error handling and API
checks).

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: lua_yieldk/lua_resume not removing yielded values

Roberto Ierusalimschy
> > [...] In other words it seems the nargs of a second lua_resume
> > seems to be ignored, and all values on the thread stack at the moment
> > of the resume are passed to the continuation function.
>
> This is a documentation problem. When a coroutine resumes after the
> first time, the stack cannot contain anything besides the arguments to
> resume. 'nargs' is mostly ignored (except for error handling and API
> checks).

The documentation is not that wrong, after all. It says:

  To resume a coroutine, you put on its stack only the values to be
  passed as results from yield, and then call lua_resume.

Note the "only".  It is not very clear about removing the old results,
but it is very clear about not pushing extra elements.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: lua_yieldk/lua_resume not removing yielded values

Jerome Vuarand
In reply to this post by Roberto Ierusalimschy
2013/2/15 Roberto Ierusalimschy <[hidden email]>:
> This is a documentation problem. When a coroutine resumes after the
> first time, the stack cannot contain anything besides the arguments to
> resume. 'nargs' is mostly ignored (except for error handling and API
> checks).

The API could be simplified by removing that nargs argument, to avoid
confusion. For startup it would be lua_gettop(thread)-1 (one being for
the function), and for resuming it would be lua_gettop(thread).

But that's a big API change, so in the meantime I'll call lua_resume
with nargs based on lua_gettop, and I'll make sure I clean up the
stack after lua_resume returned and I'm done with the result values.

Thanks for the answer.

Reply | Threaded
Open this post in threaded view
|

Re: lua_yieldk/lua_resume not removing yielded values

Patrick Donnelly
In reply to this post by Roberto Ierusalimschy
On Fri, Feb 15, 2013 at 7:17 AM, Roberto Ierusalimschy
<[hidden email]> wrote:
> This is a documentation problem. When a coroutine resumes after the
> first time, the stack cannot contain anything besides the arguments to
> resume. 'nargs' is mostly ignored (except for error handling and API
> checks).
>
> -- Roberto
>



--
- Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: lua_yieldk/lua_resume not removing yielded values

Patrick Donnelly
In reply to this post by Roberto Ierusalimschy
[Apologies I accidentally hit send on the previous reply.]

On Fri, Feb 15, 2013 at 7:34 AM, Roberto Ierusalimschy
<[hidden email]> wrote:

>> > [...] In other words it seems the nargs of a second lua_resume
>> > seems to be ignored, and all values on the thread stack at the moment
>> > of the resume are passed to the continuation function.
>>
>> This is a documentation problem. When a coroutine resumes after the
>> first time, the stack cannot contain anything besides the arguments to
>> resume. 'nargs' is mostly ignored (except for error handling and API
>> checks).
>
> The documentation is not that wrong, after all. It says:
>
>   To resume a coroutine, you put on its stack only the values to be
>   passed as results from yield, and then call lua_resume.
>
> Note the "only".  It is not very clear about removing the old results,
> but it is very clear about not pushing extra elements.

It would seem to me that this is inconsistent with the rest of the
API. After all, you don't have to clear the stack of other items
before calling lua_call. It would seem removing extraneous items on
the stack on resume is the sensible and expected behavior.

The documentation could be clearer regardless.

--
- Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: lua_yieldk/lua_resume not removing yielded values

Jerome Vuarand
2013/2/15 Patrick Donnelly <[hidden email]>:
> It would seem to me that this is inconsistent with the rest of the
> API. After all, you don't have to clear the stack of other items
> before calling lua_call. It would seem removing extraneous items on
> the stack on resume is the sensible and expected behavior.

If you want to be consistent with lua_call, you wouldn't remove the
values but let them there, still accessible after lua_resume returns,
but not accessible to the coroutine itself (in the continuation
function). Or maybe that's what you meant (ie. removing them from the
grasp of the continuation).