To-be-closed variables and coroutines

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

To-be-closed variables and coroutines

Egor Skriptunoff-2
Hi!

To-be-closed variables were introduced as a more strict (and preferred) mechanism than GC finalizers.
But in suspended and normal coroutines the __close metamethods does not run on program exit!

If a coroutine is created with coroutine.create,
we need to not forget to invoke coroutine.close somewhere before the program exits.
And if a coroutine is created with coroutine.wrap,
we don't even have the ability to invoke coroutine.close!
As a result, we have to duplicate all __close functionality in finalizers.
Why such inconvenience?

To-be-closed variables should be reliable and self-sufficient.
The suggestion:
When VM is closing, coroutine.close() should be automatically invoked for every non-dead coroutine in the VM.
This way we could forget about __gc and work only with __close.

Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Andrew Gierth
>>>>> "Egor" == Egor Skriptunoff <[hidden email]> writes:

 Egor> As a result, we have to duplicate all __close functionality in
 Egor> finalizers. Why such inconvenience?

Other than the auto_close module I mentioned recently (which exists
solely as a proxy for __close methods), every real-life __close
metamethod I have written so far has been ... just a reference to the
finalizer.

--
Andrew.
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Egor Skriptunoff-2
In reply to this post by Egor Skriptunoff-2

The suggestion:
When VM is closing, coroutine.close() should be automatically invoked for every non-dead coroutine in the VM.

Dead coroutines (after an error is raised) can have non-closed to-be-closed variables too.
So, the suggestion is: All coroutines in the VM should be implicitly closed on VM exit.
Of course it would be better to implicitly close dead coroutines at the moment they become dead.
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Dibyendu Majumdar
On Thu, 30 Jul 2020 at 18:01, Egor Skriptunoff
<[hidden email]> wrote:
>
>
>> The suggestion:
>> When VM is closing, coroutine.close() should be automatically invoked for every non-dead coroutine in the VM.
>
>
> Dead coroutines (after an error is raised) can have non-closed to-be-closed variables too.
> So, the suggestion is: All coroutines in the VM should be implicitly closed on VM exit.
> Of course it would be better to implicitly close dead coroutines at the moment they become dead.

Repeating myself - the 'defer' statement leads to no such issue or
expectation - as it is explicitly related to a scope exit and does not
promise to close anything!

In my opinion it is perfectly correct that the close method is not
invoked unless scope is exited. You just need to ensure that the close
logic is also invoked by the GC, which will presumably cleanup
everything eventually.
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Egor Skriptunoff-2
On Thu, Jul 30, 2020 at 9:08 PM Dibyendu Majumdar wrote:
You just need to ensure that the close
logic is also invoked by the GC

If I understand your words correctly, you suggest the following:
if a coroutine is exiting due to error,
all the "defer" statements will be executed by GC when collecting the coroutine?


Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Dibyendu Majumdar
On Thu, 30 Jul 2020 at 22:27, Egor Skriptunoff
<[hidden email]> wrote:

>
> On Thu, Jul 30, 2020 at 9:08 PM Dibyendu Majumdar wrote:
>>
>> You just need to ensure that the close
>> logic is also invoked by the GC
>
>
> If I understand your words correctly, you suggest the following:
> if a coroutine is exiting due to error,
> all the "defer" statements will be executed by GC when collecting the coroutine?
>
>

Not exactly ... I am saying that the close() method should invoke the
finalizer on the object - and of course the finalizer needs to be
idempotent so that it can be safely invoked multiple times.
Eventually every object will be garbage and the object finalizer will
clean it up even if the close() never got called.

Regards
Dibyendu
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

pocomane
In reply to this post by Dibyendu Majumdar
On Thu, 30 Jul 2020 at 18:01, Egor Skriptunoff wrote:
> Dead coroutines (after an error is raised) can have non-closed to-be-closed variables too.
> So, the suggestion is: All coroutines in the VM should be implicitly closed on VM exit.
> Of course it would be better to implicitly close dead coroutines at the moment they become dead.

coroutine.wrap closes them automatically [4]

On Thu, Jul 30, 2020 at 8:08 PM Dibyendu Majumdar wrote:
> Repeating myself - the 'defer' statement leads to no such issue or
> expectation - as it is explicitly related to a scope exit and does not
> promise to close anything!

I tried to run some tests with and without the defer patch [1][2]. I
created a coroutine with coroutine.create and I run it with
coroutine.resume. In the coroutine a defer/<close> is "Registered" and
an error is raised.

It seems to me that both the codes just do not call the finalization
code [3]. Calling coroutine.close (or using coroutine.wrap) just makes
the finalizer be called in both the cases (with or without the patch).

What am I missing ?

Just to be clear, I am not fully sure what the "Right" behaviour
should be. I was just puzzled by Dibyendu's statement that the defer
patch behaves differently in this case. I also wrote a different defer
patch [5] and it behaves the same.

--

[1] The code for the defer is:
```
local thread = coroutine.create(function()
  defer
    print"coroutine defer"
  end
  error"an error in the goroutine"
  print"coroutine ends"
end)
local ok= coroutine.resume(thread)
if not ok then
  -- This should be the right thing to do (and indeed coroutine.wrap does it),
  -- but it is skipped for the sake of the test
  -- -- coroutine.close(thread)
end
print"program exit"
```

[2] the code for the <close> variable is:
```
local thread = coroutine.create(function()
  local please <close> = setmetatable({}, {__close = function()
    print"coroutine toclose"
  end})
  error"an error in the goroutine"
  print"coroutine ends"
end)
local ok= coroutine.resume(thread)
if not ok then
  -- This should be the right thing to do (and indeed coroutine.wrap does it),
  -- but it is skipped for the sake of the test
  -- -- coroutine.close(thread)
end
print"program exit"
```

[3] In both the cases I got
```
program exit
```
i.e.  print"coroutine defer/toclose" is not called

[4] The code
```
local thread = coroutine.wrap(function()
  local please <close> = setmetatable({}, {__close = function()
    print"coroutine toclose"
  end})
  error"an error in the goroutine"
  print"coroutine ends"
end)
thread()
print"program exit"
```
prints "coroutine toclose" before quitting with an error.

[5] A defer statement is part of the do-attibute patch. It does not
introduce any new keyword, since it uses a syntax like do <defer> end.
It works by creating an "Hidden" to-be-closed variable.
https://github.com/pocomane/lua-do-attribute
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Egor Skriptunoff-2
On Fri, Jul 31, 2020 at 1:13 PM Mimmo Mane wrote:
> So, the suggestion is: All coroutines in the VM should be implicitly closed on VM exit.

coroutine.wrap closes them automatically [4]

[4] The code
```
local thread = coroutine.wrap(function()
  local please <close> = setmetatable({}, {__close = function()
    print"coroutine toclose"
  end})
  error"an error in the goroutine"
  print"coroutine ends"
end)
thread()
print"program exit"
```
prints "coroutine toclose" before quitting with an error.

Yes, if an error is raised, the coroutine is correctly closed.
But not every code raises an error  :-)


local thread = coroutine.wrap(function()
   local please <close> = setmetatable({}, {__close = function()
      print"coroutine toclose"
   end})
   print"coroutine yields"
   coroutine.yield()
   print"coroutine ends"
end)
thread()
print"program exit"

I don't see "coroutine toclose" printed.
The Lua manual says we should invoke coroutine.close().
But we can't  :-)
 
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

pocomane
On Fri, Jul 31, 2020 at 2:20 PM Egor Skriptunoff
<[hidden email]> wrote:

> local thread = coroutine.wrap(function()
>    local please <close> = setmetatable({}, {__close = function()
>       print"coroutine toclose"
>    end})
>    print"coroutine yields"
>    coroutine.yield()
>    print"coroutine ends"
> end)
> thread()
> print"program exit"
>
> I don't see "coroutine toclose" printed.
> The Lua manual says we should invoke coroutine.close().
> But we can't  :-)

Yes, I think you are right: coroutine.wrap is not usable for now.
However, I would avoid the "Do something at exit" policy. I would
prefer to change the coroutine.wrap implementation to something
__gc/__close -aware [1].

For create/resume instead, I am not fully sure that any automatism is
a good idea. Maybe a lower level design is more flexible: if you
really need, you can use them to build wrappers with automatic
__gc/__close calls [1].

--

[1] Something like:

local function cowrap(f)
  local thread = coroutine.create(f)
  local closed = false
  local function doclose()
    if not closed then
      coroutine.close(thread)
      closed = true
    end
  end
  return setmetatable({},{
    __call = function(_,...)
      return coroutine.resume(thread,...)
    end,
    __close=doclose,
    __gc=doclose,
  })
end
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Dibyendu Majumdar
In reply to this post by pocomane
On Fri, 31 Jul 2020 at 11:13, Mimmo Mane <[hidden email]> wrote:

>
> Just to be clear, I am not fully sure what the "Right" behaviour
> should be. I was just puzzled by Dibyendu's statement that the defer
> patch behaves differently in this case. I also wrote a different defer
> patch [5] and it behaves the same.
>

Hi, I did not mean to say that the 'defer'  patch solves it.
All I was saying is that the 'defer' statement makes less promises as
it makes it more explicit that some code (it can anything) will be
executed when the scope is exited.
But basically that is all that to-be-closed variables do - a promise
to run some code if the scope is exited. They do not promise to close
anything. They are not finalizers.
And because they are what they are, if a thread does not exit scope
they will never run.

Regards
Dibyendu
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Egor Skriptunoff-2
In reply to this post by pocomane
On Fri, Jul 31, 2020 at 8:19 PM Mimmo Mane wrote:
local function cowrap(f)
  local thread = coroutine.create(f)
  local closed = false
  local function doclose()
    if not closed then
      coroutine.close(thread)
      closed = true
    end
  end
  return setmetatable({},{
    __call = function(_,...)
      return coroutine.resume(thread,...)
    end,
    __close=doclose,
    __gc=doclose,
  })
end

Yes, this would work.
BTW, coroutine.close() is already idempotent, so no need for "if not closed then".

Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Egor Skriptunoff-2
In reply to this post by Dibyendu Majumdar
On Fri, Jul 31, 2020 at 8:54 PM Dibyendu Majumdar wrote:
All I was saying is that the 'defer' statement makes less promises as
it makes it more explicit that some code (it can anything) will be
executed when the scope is exited.
But basically that is all that to-be-closed variables do - a promise
to run some code if the scope is exited. They do not promise to close
anything. They are not finalizers.
And because they are what they are, if a thread does not exit scope
they will never run.

We don't need just a "scope exit event".
The point of why to-be-closed variables exist: we need a "predictable finalizer".
We need to guarantee that a resource will be closed as fast as possible,
whatever happened in the code.

Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Dibyendu Majumdar
On Fri, 31 Jul 2020 at 19:28, Egor Skriptunoff
<[hidden email]> wrote:

>
> On Fri, Jul 31, 2020 at 8:54 PM Dibyendu Majumdar wrote:
>>
>> All I was saying is that the 'defer' statement makes less promises as
>> it makes it more explicit that some code (it can anything) will be
>> executed when the scope is exited.
>> But basically that is all that to-be-closed variables do - a promise
>> to run some code if the scope is exited. They do not promise to close
>> anything. They are not finalizers.
>> And because they are what they are, if a thread does not exit scope
>> they will never run.
>
>
> We don't need just a "scope exit event".
> The point of why to-be-closed variables exist: we need a "predictable finalizer".
> We need to guarantee that a resource will be closed as fast as possible,
> whatever happened in the code.
>

Well, the close() method will be called as soon as you exit the scope!
So it is predictable in that sense.
If the thread is never resumed, it is impossible to invoke the close
method, if you think about it.

Regards
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Egor Skriptunoff-2
On Fri, Jul 31, 2020 at 9:41 PM Dibyendu Majumdar wrote:
If the thread is never resumed, it is impossible to invoke the close
method

Assume we have an hierarchy of nested objects:
- VM
- coroutine
- do-end block
- to-be-closed variable holding allocated resource

The "scope exiting event" means exiting from the immediate parent.
The "coroutine death/collection event" means exiting from the grandparent.
The "VM closing event" means exiting from great grandparent.

If a coroutine is never resumed, we will not receive the innermost event,
but we will receive at least one of the outer events.
All three types of events should be involved in the resource-closing-logic.

Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

Viacheslav Usov
In reply to this post by Egor Skriptunoff-2
On Fri, Jul 31, 2020 at 8:28 PM Egor Skriptunoff
<[hidden email]> wrote:

> We don't need just a "scope exit event".
> The point of why to-be-closed variables exist: we need a "predictable finalizer".
> We need to guarantee that a resource will be closed as fast as possible,
> whatever happened in the code.

Indeed. This seems somehow to have fallen through the cracks.

On November 24th, 2015. Roberto proposed the following syntax:

local <some mark to be invented> name = exp

That was in a thread titled "block-scope finalization" which I started
a few days earlier, with a message that posited that the problem was
"that finalization is non-deterministic, i.e., it is impossible to
predict when it will happen."

In that same message I proposed the following syntax:

block file = io.open('file')

While there are some differences between my original proposal, and
Roberto's original proposal, and ultimately the to-be-closed feature
as we have it now, it seems fairly obvious that this is the same idea:
clean-up when the enclosing block of code is exited.

I reference this particular thread because I think this is when most
of the design decisions were made. Roberto referenced that same
November 24th message as a "technical solution that works" in another
thread in 2018 titled "(not) handling new programming idioms with
grace", whose starter referenced his 2009 thread "state of the Lua
nation on resource cleanup". The "(not) handling..." thread was I
think when the Lua team got serious about implementing the "technical
solution" and that is how we ended up with the current to-be-closed
feature.

It is somewhat ironic that it was Dibyendu Majumdar who did not
believe in 2018 that a "technical solution" existed. Even though he
did not, in my opinion, mention the really problematic aspects that
would make the "technical solution" difficult or impossible, two years
later we still do not have a "technical solution that works period".
Because it is a "technical solution that works sometimes".

What is needed is "that a resource will be closed as fast as possible,
whatever happened in the code", exactly as Egor wrote.

This we do not have.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

DarkWiiPlayer
In reply to this post by Egor Skriptunoff-2
I don't like that suggestion; I'd prefer if coroutines all had `__close`
and possibly also `__gc` metamethod that closes them.

That way one could just define a coroutine as to-be-closed and things
would take care of itself. The `__gc` metamethod would also fix the
`coroutine.wrap` problem, but I don't know if it would be a good idea in
general, as one might not want to close a coroutine for whatever reason.

That being said though, my opinion is that currently `<close>` is
completely worthless because it can't be trusted. The whole point seems
to be to add a predictable and safe way to make sure something
definitely happens, even in case of errors, but that doesn't work
because a coroutine can just fall off the counter and cause memory
leaks, files that stay open, etc.

On 30/07/2020 02:14, Egor Skriptunoff wrote:

> Hi!
>
> To-be-closed variables were introduced as a more strict (and
> preferred) mechanism than GC finalizers.
> But in suspended and normal coroutines the __close metamethods does
> not run on program exit!
>
> If a coroutine is created with coroutine.create,
> we need to not forget to invoke coroutine.close somewhere before the
> program exits.
> And if a coroutine is created with coroutine.wrap,
> we don't even have the ability to invoke coroutine.close!
> As a result, we have to duplicate all __close functionality in finalizers.
> Why such inconvenience?
>
> To-be-closed variables should be *reliable *and*self-sufficient*.
> The suggestion:
> When VM is closing, coroutine.close() should be automatically invoked
> for every non-dead coroutine in the VM.
> This way we could forget about __gc and work only with __close.
>


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: To-be-closed variables and coroutines

John Belmonte
In reply to this post by Egor Skriptunoff-2
On Thu, Jul 30, 2020 at 9:14 AM Egor Skriptunoff <[hidden email]> wrote:
To-be-closed variables were introduced as a more strict (and preferred) mechanism than GC finalizers.
But in suspended and normal coroutines the __close metamethods does not run on program exit!
 ...
To-be-closed variables should be reliable and self-sufficient.
The suggestion:
When VM is closing, coroutine.close() should be automatically invoked for every non-dead coroutine in the VM.

The use case you have in mind may be relatively simple, like freeing some OS resource.  But in the general, tasks of a concurrent program may want to do the following at shutdown:
  1. asynchronous operations under the cooperative scheduler (IO, networking, etc.)
  2. operations that depend on other tasks in the application
I haven't tried to implement such a system in Lua, but do have experience with Python Trio, which is one of frameworks at the forefront of structured concurrency.

Essentially, #1 is going to require cooperation from the task/coroutine scheduler.  It needs to intercept SIGINT / SIGTERM / keyboard interrupt and propagate a cancel signal to all the active tasks, giving their finalizers a chance to run.  In Trio, by default any attempt to block from a task in a cancelled state will raise an exception (which in turn propagates to child and parent tasks).  However, it also has a "shield" feature (usually employed with a timeout), where finalizers can opt in to blocking from a task that is pending cancellation.

Item #2 builds on #1 and is harder to manage.  For example, we have a companion robot-- when the top-level control process receives a SIGTERM, that process is still able to shut down the servos and close the eyelids (requiring networking, animation, and other services provided by other tasks).

The general area is referred to as graceful shutdown-- still a hot topic in the structured concurrency forums.

For Lua, I'm expecting that the to-be-closed variables and coroutine API are sufficient to build upon.

Regards,
--John