A question about "yieldables"

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

A question about "yieldables"

Soni "They/Them" L.
Lua 5.2 added the ability to yield thru C functions, but most functions aren't yieldable (e.g. string.gsub, load, table.sort). Does the manual require that those functions be non-yieldable? or could a conforming implementation have them be yieldable? (Also why are they not yieldable in the reference implementation?)
-- 
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.
Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

Roberto Ierusalimschy
> Lua 5.2 added the ability to yield thru C functions, but most
> functions aren't yieldable (e.g. string.gsub, load, table.sort).
> Does the manual require that those functions be non-yieldable? or
> could a conforming implementation have them be yieldable?

No. Yes.


> (Also why are they not yieldable in the reference implementation?)

It does not seem to be a good cost-benefit approach. Making them
yieldable will probably make them more complex and slower in the "normal"
case. ('load' has an extra difficulty that it would require changes in
the C API; table.sort has an extra difficulty that it is recursive;
string.gsub may be not that problematic, after all.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

Sean Conner
In reply to this post by Soni "They/Them" L.
It was thus said that the Great Soni L. once stated:
> Lua 5.2 added the ability to yield thru C functions, but most functions
> aren't yieldable (e.g. string.gsub, load, table.sort). Does the manual
> require that those functions be non-yieldable? or could a conforming
> implementation have them be yieldable? (Also why are they not yieldable
> in the reference implementation?)

  You could, in theory, rewrite the problematic functions in Lua itself,
calling coroutine.yield() as required.

  -spc
 

Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

Paul K-2
In reply to this post by Roberto Ierusalimschy
>> could a conforming implementation have them be yieldable?
>
Yes.

>> (Also why are they not yieldable in the reference implementation?)
> It does not seem to be a good cost-benefit approach. Making them yieldable will probably make them more complex and slower in the "normal" case.

I'm curious as to what changes at the C level are required to make them yieldable. For example, is it possible to make Lpeg handling of callback functions (Cmt and /func) yieldable?

Paul.

Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

Tim Hill

> On Feb 23, 2015, at 1:44 PM, Paul K <[hidden email]> wrote:
>
> >> could a conforming implementation have them be yieldable?
> > Yes.
>
> >> (Also why are they not yieldable in the reference implementation?)
> > It does not seem to be a good cost-benefit approach. Making them yieldable will probably make them more complex and slower in the "normal" case.
>
> I'm curious as to what changes at the C level are required to make them yieldable. For example, is it possible to make Lpeg handling of callback functions (Cmt and /func) yieldable?
>
> Paul.
>

The problem is that yieldable functions lose their stack on a yield (because of long jumps), and have to keep all function state in an auxiliary data structure that is then passed by Lua to a “continuation” function. Converting simple functions to this model is a little messy but not too hard (you move all local state and variables into the structure), but it gets much messier with recursive functions, since you have to simulate the recursion stack within the continuation state. it can be done, but it’s not nice.

—Tim




Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

Thomas Jericke
In reply to this post by Soni "They/Them" L.
On 02/23/2015 01:14 AM, Soni L. wrote:
Lua 5.2 added the ability to yield thru C functions, but most functions aren't yieldable (e.g. string.gsub, load, table.sort). Does the manual require that those functions be non-yieldable? or could a conforming implementation have them be yieldable? (Also why are they not yieldable in the reference implementation?)
-- 
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.
I made several functions yieldable for my system. The most prominent one was 'require'. That makes it possible for me to debug in required scripts.

It might be interesting to have a bundel of patches on the Lua Wike. The best organization would probably be one patch per function.
--
Thomas
Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

Valerio
In reply to this post by Soni "They/Them" L.
This library provides a yieldable gsplit:


there might be other functions available as well.

best
valerio

On Mon, Feb 23, 2015 at 1:14 AM, Soni L. <[hidden email]> wrote:
Lua 5.2 added the ability to yield thru C functions, but most functions aren't yieldable (e.g. string.gsub, load, table.sort). Does the manual require that those functions be non-yieldable? or could a conforming implementation have them be yieldable? (Also why are they not yieldable in the reference implementation?)
-- 
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.

Reply | Threaded
Open this post in threaded view
|

Re: A question about "yieldables"

William Ahern
In reply to this post by Soni "They/Them" L.
On Sun, Feb 22, 2015 at 09:14:09PM -0300, Soni L. wrote:
> Lua 5.2 added the ability to yield thru C functions, but most functions
> aren't yieldable (e.g. string.gsub, load, table.sort). Does the manual
> require that those functions be non-yieldable? or could a conforming
> implementation have them be yieldable? (Also why are they not yieldable
> in the reference implementation?)
>

tostring isn't yieldable, either, which is problematic when trying to
implement lazy evaluation. Imagine passing an object around which is tied to
an HTTP POST request. Also imagine you're using a non-blocking I/O type
framework. You don't have to bother reading the POST body until somebody
calls tostring on the object, which they may never get around to because of
filters, access restrictions, etc. Lazy evaluation let's you do this without
requiring application code to understand the underlying API or semantics;
all they need to do is use tostring.

I added a yieldable tostring as part of an auxlib module
(cqueues.auxlib.tostring). I submitted a patch based on my code but never
heard back.

In Lua 5.3 tostring is implemented in a single line by calling
luaL_tolstring. The yielable version is about 15 lines (and thus 15x
longer!). Some of that is reimplementing some boilerplate code behind the
luaL_tolstring call. A better patch might refactor some of the Lua
internals, such as maybe adding a luaL_callmetak function.