Another option for closing variables

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Another option for closing variables

Nevin Flanagan-2
It seems like what is really wanted is a way to clean up a scope when it closes, so it seems like we should just do that. Create a new block frame, such as "defer ... end" or "exiting ... end". Everything that is in scope at the point where the block begins is in scope throughout the block. However, the contents of the block are not executed inline; rather, once execution reaches the point in the code where the block begins, a deferred-execution guarantee is made for the contained code, which will be executed when the scope immediately containing the exiting ... end block is exited (in newest-guarantee-first order).

function foo(bar, ...)
  local baz = bar exiting
    baz.closed = true
  end
  do 
    local quoz = {...} exiting
      setmetatable(quoz, {__mode='kv'})
    end
    quoz[0] = bar.file or error ("'file' field missing from supplied parameter")
    local moz = io.open(quoz[0], 'a+') exiting 
      moz:close()
    end
    if riz(moz, baz) then -- some global function
      return true
    end
  end
  print("warning: operation could not complete")
  return false
end

So let's break this down a little:

After the local baz is assigned from bar, a guarantee is made that baz.closed will be set to true. This will happen even if bar has been changed.

Inside the do block quoz is assigned to a new table created from varargs. A guarantee is then made that before execution exits the do block, this table will receive a metatable that makes its references weak. This will happen BEFORE baz.closed is set.

A new key is added to this table provided that bar.file is not nil or false (it is probably more typical to use assert here, but this still illustrates the principle). If the error is thrown, quoz will get its metatable, then baz.closed will become true, then the error will propagate out of the function. (However, the code just after this, to call moz:close(), will not be called at all, as execution never reached its deferral.)

Provided there is no error, moz is assigned to a newly opened file. A guarantee is then made that this file will be closed before the do block can exit, before the metatable is applied to quoz (this guarantee of reverse execution is something I believe is well-understood to be necessary to keep code maintainable and deterministic). Then, if the riz funciton returns some true value, moz is closed, quoz is set to weak references, baz.closed is set, and execution then leaves foo with the return value true.

If riz returns a false value, execution proceeds out of the do block. moz is closed and quoz is metatabled. The print statement is executed. Finally, returning false triggers setting baz.closed to true before execution returns to the calling function.

The exiting block is not actually part of the assignment syntax. So 

local baz = bar exiting
  baz.closed = true
end

is the same as 

local baz = bar; exiting
  baz.closed = true
end

which is the same as 

local baz = bar
exiting
  baz.closed = true
end

And there is also no reason you can't have an exiting block without any associated variables.
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
пт, 5 июл. 2019 г. в 18:39, Nevin Flanagan <[hidden email]>:
>
> It seems like what is really wanted is a way to clean up a scope when it closes, so it seems like we should just do that. Create a new block frame, such as "defer ... end" or "exiting ... end". Everything that is in scope at the point where the block begins is in scope throughout the block. However, the contents of the block are not executed inline; rather, once execution reaches the point in the code where the block begins, a deferred-execution guarantee is made for the contained code, which will be executed when the scope immediately containing the exiting ... end block is exited (in newest-guarantee-first order).

Why we try to find new way to do this if it could done with existing syntax.
https://raw.githubusercontent.com/kov-serg/lua-aux/master/scope.lua
More over we could control implementation details with several lines of code.

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Coda Highland
On Fri, Jul 5, 2019 at 3:42 PM Sergey Kovalev <[hidden email]> wrote:
пт, 5 июл. 2019 г. в 18:39, Nevin Flanagan <[hidden email]>:
>
> It seems like what is really wanted is a way to clean up a scope when it closes, so it seems like we should just do that. Create a new block frame, such as "defer ... end" or "exiting ... end". Everything that is in scope at the point where the block begins is in scope throughout the block. However, the contents of the block are not executed inline; rather, once execution reaches the point in the code where the block begins, a deferred-execution guarantee is made for the contained code, which will be executed when the scope immediately containing the exiting ... end block is exited (in newest-guarantee-first order).

Why we try to find new way to do this if it could done with existing syntax.
https://raw.githubusercontent.com/kov-serg/lua-aux/master/scope.lua
More over we could control implementation details with several lines of code.


I'm really not a big fan of that. Not only does that have a ton more overhead thanks to needing to construct extra closures, but it messes with control flow -- you can't, for example, optionally return from the code. You have to have the caller inspect the return value and decide whether it needs to return or if it should keep going. At that point, it's barely even syntactic sugar; if you're ALREADY writing a worker function and passing it to another function ("scope" instead of "pcall" but still...) and you have to inspect the return values either way, what have you really gained?

The advantage of having the syntactical support built into the language itself instead of provided via metaprogramming is that it means you can take advantage of all of the other features of the language. I wholly believe that <toclose> serves a purpose that CANNOT be elegantly and efficiently replicated without it. And that's exactly what you should be looking for when adding something to a language.

I mean, yeah, I don't think angle brackets are an ideal syntax for it, and "toclose" is something of an awkward name, but everything else about it I think is sound. The only alternative that's come up in all of this bikeshedding that's equivalently expressive is support for a "finally" block, which has its comparative pros and cons. (Weighing the alternatives, I like finally SLIGHTLY better because it gives better control over error handling, but it has bigger syntactical problems and requires more boilerplate for common use cases.)

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
сб, 6 июл. 2019 г. в 00:24, Coda Highland <[hidden email]>:
> I'm really not a big fan of that. Not only does that have a ton more overhead thanks to needing to construct extra closures, but it messes with control flow -- you can't, for example, optionally return from the code.
You could follow rule: one enter one exit point. So you will no need
in return. Here return used to live then scope, like break in a loop.
Restrictions are always be. There is a lot of cases where you need no
return at all.

> You have to have the caller inspect the return value and decide whether it needs to return or if it should keep going.
There is no need to inspect return values you can pass all you need in upvalues.
function fn(name)
local f,ff,r1,r2

scope(function(auto) f=auto(){ io.open(name) } -- if it unable to open
file it will leave fn with error
  r1=f:read()
end)
-- file1 closed
scope(function(auto) f=auto(){ io.open "file2" }
  r2=f:read()
end)
-- file2 closed

scope(function(auto) f=auto(){ io.open "file3" }
  scope(function(auto) ff=auto(){ io.open "file 4" } -- nested scopes
  end)
end)

print "if we are here it was no errors and we are done"
end

> At that point, it's barely even syntactic sugar; if you're ALREADY writing a worker function and passing it to another function ("scope" instead of "pcall" but still...) and you have to inspect the return values either way, what have you really gained?
If there was a reference counting we could use meta-method
__unreferenced like __gc to achieve behaviour desired. But there
isn't. So it could be done only with functions.
In lack of __unreferenced we could __gc, but with some clauses.

function for_scope(prm,L)
    if L==nil then
        L={list={}}
        function L.auto(close,msg)
            return function(t)
                if type(t)~='table' then error("need table: {
expression }",2) end
                if t[1] then table.insert(L.list,{ arg=t[1], fn=close
or io.close })
                else
                    if msg=='weak' then return table.unpack(t) end
                    error(msg or t[2] or "no resource",2)
                end
                return table.unpack(t)
            end
        end
        function L.defer(fn) L.auto(fn){true} end
        function L.leave()
            for i=#L.list,1,-1 do
                if L.list[i].fn then
                    local fn=L.list[i].fn
                    L.list[i].fn=false
                    fn(L.list[i].arg)
                end
            end
            L.list={}
        end
        return setmetatable(L,{__gc=function(p) p.leave() end})
    else
        L.leave()
    end
end

for scope in for_scope do
    local f=scope.auto(io.close){ io.open("text.txt","w") }
    scope.defer(function() print"defer" end)
    print"body" -- you can use return and break here
end
-- but need call collectgarbage() to force call __gc in case of errors
and returns

> The advantage of having the syntactical support built into the language itself instead of provided via meta-programming is that it means you can take advantage of all of the other features of the language. I wholly believe that <toclose> serves a purpose that CANNOT be elegantly and efficiently replicated without it. And that's exactly what you should be looking for when adding something to a language.

Lua is small language with simple but powerful syntax. Why we want
worthless increase complexity, instead expressiveness. Why
meta-programming should not be used?
There you need extra speed and very fine stack tuning in files open?
Can you give examples?

> I mean, yeah, I don't think angle brackets are an ideal syntax for it, and "toclose" is something of an awkward name, but everything else about it I think is sound. The only alternative that's come up in all of this bikeshedding that's equivalently expressive is support for a "finally" block, which has its comparative pros and cons. (Weighing the alternatives, I like finally SLIGHTLY better because it gives better control over error handling, but it has bigger syntactical problems and requires more boilerplate for common use cases.)

I think angle brackets if awful. Especially for so trivial operation
such as a releasing resources. I think toclose will born more problem
that it solves.
Lua has coroutines. How you will manage finally blocks there. Also
there are a lot of options how to behave in case of errors, returns or
context switching in finally blocks.

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Philippe Verdy
In reply to this post by Nevin Flanagan-2


Le ven. 5 juil. 2019 à 17:39, Nevin Flanagan <[hidden email]> a écrit :
It seems like what is really wanted is a way to clean up a scope when it closes, so it seems like we should just do that. Create a new block frame, such as "defer ... end" or "exiting ... end". Everything that is in scope at the point where the block begins is in scope throughout the block. However, the contents of the block are not executed inline; rather, once execution reaches the point in the code where the block begins, a deferred-execution guarantee is made for the contained code, which will be executed when the scope immediately containing the exiting ... end block is exited (in newest-guarantee-first order).

Isn't it functionally the same as "try..finally" exception handlers ? i.e. this proposal can only work when there are are errors raised and the code cannot continue normally and must pass by exceptionb handlers

The exception handler may retry by using a tralining recursive call, but there's no way to return directly at the point where the exception/error was raised, without reconstructing a safe context (by the trailing recursive call: all variables that were in scope when the exception/error was raised are lost, except those passed in the error object constructor; note that to work correctlt this requires that this error object is created first before entering into the "try..."; that object should then contain all the necessary datastructures that may allow the exception/error handler to get significant data and possibly restart the function recursively by a trailing call in a return statement; an exception handler however has no obligation to perform such recursive call, in which case the code will only continue after the "try...finally", whose internal scope is no longer accessible).

I think that "try...finally" (as defined in an extension for C and then ported to C++ in complement of "try...catch") is the way to go with Lua.



Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Coda Highland
In reply to this post by Sergey Kovalev


On Fri, Jul 5, 2019 at 6:15 PM Sergey Kovalev <[hidden email]> wrote:
сб, 6 июл. 2019 г. в 00:24, Coda Highland <[hidden email]>:
> I'm really not a big fan of that. Not only does that have a ton more overhead thanks to needing to construct extra closures, but it messes with control flow -- you can't, for example, optionally return from the code.
You could follow rule: one enter one exit point. So you will no need
in return. Here return used to live then scope, like break in a loop.
Restrictions are always be. There is a lot of cases where you need no
return at all.

No, thank you. I really don't have any interest in going back to programming in Pascal. One enter is fine, but one exit is unnecessarily restrictive -- even loops have break statements, and that's multiple exits. 
 
> At that point, it's barely even syntactic sugar; if you're ALREADY writing a worker function and passing it to another function ("scope" instead of "pcall" but still...) and you have to inspect the return values either way, what have you really gained?
If there was a reference counting we could use meta-method
__unreferenced like __gc to achieve behaviour desired. But there
isn't. So it could be done only with functions.

You mean... the very functionality you're trying to suggest this workaround function to deal with not having? Gee, I wonder why people have been saying it's a good idea.

 
> The advantage of having the syntactical support built into the language itself instead of provided via meta-programming is that it means you can take advantage of all of the other features of the language. I wholly believe that <toclose> serves a purpose that CANNOT be elegantly and efficiently replicated without it. And that's exactly what you should be looking for when adding something to a language.

Lua is small language with simple but powerful syntax. Why we want
worthless increase complexity, instead expressiveness. Why
meta-programming should not be used?
There you need extra speed and very fine stack tuning in files open?
Can you give examples?

It's nice to have the ability to do metaprogramming, of course. I'm glad that Lua is a flexible enough language to make it possible. But when it takes a huge glob of metaprogramming to make a syntactically-awkward way of doing something, and the language creators are offering a way to provide the very tool you said would make it work better, then... why resist that feature? it's an opt-in feature, too, and Lua's optimizations still work if you don't use it. pcall() is capable of doing this all by itself without anything fancier, but it's awkward to program with and pure-Lua metaprogramming can't solve the problems it has.
 
Lua has coroutines. How you will manage finally blocks there. Also
there are a lot of options how to behave in case of errors, returns or
context switching in finally blocks.

Coroutines have the same problem with <toclose>. Coroutines have the same problem with your scope() function, too. You have to choose the right tool for the job, and if you have a resource that really has to be cleaned up in a coroutine that might never run to completion, then that's what __gc is for. <toclose> and finally and scopes, regardless of how you implement them, are for fine-grained control when you need to make sure that a function doesn't exit without running a certain piece of code. 
 
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Coda Highland
In reply to this post by Philippe Verdy


On Fri, Jul 5, 2019 at 7:00 PM Philippe Verdy <[hidden email]> wrote:


Le ven. 5 juil. 2019 à 17:39, Nevin Flanagan <[hidden email]> a écrit :
It seems like what is really wanted is a way to clean up a scope when it closes, so it seems like we should just do that. Create a new block frame, such as "defer ... end" or "exiting ... end". Everything that is in scope at the point where the block begins is in scope throughout the block. However, the contents of the block are not executed inline; rather, once execution reaches the point in the code where the block begins, a deferred-execution guarantee is made for the contained code, which will be executed when the scope immediately containing the exiting ... end block is exited (in newest-guarantee-first order).

Isn't it functionally the same as "try..finally" exception handlers ? i.e. this proposal can only work when there are are errors raised and the code cannot continue normally and must pass by exceptionb handlers

try...finally isn't about exception handling. it interacts with exception handling, to be sure, but the entire point is that it's code that runs whether or not there's an error, or even if there's a return statement.
 
The exception handler may retry by using a tralining recursive call, but there's no way to return directly at the point where the exception/error was raised, without reconstructing a safe context (by the trailing recursive call: all variables that were in scope when the exception/error was raised are lost, except those passed in the error object constructor; note that to work correctlt this requires that this error object is created first before entering into the "try..."; that object should then contain all the necessary datastructures that may allow the exception/error handler to get significant data and possibly restart the function recursively by a trailing call in a return statement; an exception handler however has no obligation to perform such recursive call, in which case the code will only continue after the "try...finally", whose internal scope is no longer accessible).

I think that "try...finally" (as defined in an extension for C and then ported to C++ in complement of "try...catch") is the way to go with Lua.

While I do like try...finally in terms of expressiveness and robustness (it's way more well-defined than <toclose> in the face of multiple errors) it's also a lot more effort on the programmer's part and doesn't solve the "forgot to release it" issue -- it just makes it easier to make sure it happens. I think <toclose> is a better fit for existing Lua programming paradigms.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Philippe Verdy


Le sam. 6 juil. 2019 à 06:48, Coda Highland <[hidden email]> a écrit :


On Fri, Jul 5, 2019 at 7:00 PM Philippe Verdy <[hidden email]> wrote:


Le ven. 5 juil. 2019 à 17:39, Nevin Flanagan <[hidden email]> a écrit :
It seems like what is really wanted is a way to clean up a scope when it closes, so it seems like we should just do that. Create a new block frame, such as "defer ... end" or "exiting ... end". Everything that is in scope at the point where the block begins is in scope throughout the block. However, the contents of the block are not executed inline; rather, once execution reaches the point in the code where the block begins, a deferred-execution guarantee is made for the contained code, which will be executed when the scope immediately containing the exiting ... end block is exited (in newest-guarantee-first order).

Isn't it functionally the same as "try..finally" exception handlers ? i.e. this proposal can only work when there are are errors raised and the code cannot continue normally and must pass by exceptionb handlers

try...finally isn't about exception handling. it interacts with exception handling, to be sure, but the entire point is that it's code that runs whether or not there's an error, or even if there's a return statement.

You did not need to add this precision because I added it myself in several parts, notably when I refered to the fact that they were ported from C to C++ as a complement of exception handling. But in the Windows API the concept of "try...finally" is called "Structured Exception Handling (SEH)" and not "Structured Error Handling". That's because exception handling in C++ is implemented with SEH, i.e. "try...catch" internally uses "try..finally", in an "hidden way", but the "try..finally" you use in C++ is to add code to the "finally" block (the rest of this block is generated by the initial exception handler, and even if you don't write "try..finally" in C++, there's a "try..finally" block intrinsicly generated for your "try..catch" block, as if you had explicitly added an empty "finally" block.


 
The exception handler may retry by using a tralining recursive call, but there's no way to return directly at the point where the exception/error was raised, without reconstructing a safe context (by the trailing recursive call: all variables that were in scope when the exception/error was raised are lost, except those passed in the error object constructor; note that to work correctlt this requires that this error object is created first before entering into the "try..."; that object should then contain all the necessary datastructures that may allow the exception/error handler to get significant data and possibly restart the function recursively by a trailing call in a return statement; an exception handler however has no obligation to perform such recursive call, in which case the code will only continue after the "try...finally", whose internal scope is no longer accessible).

I think that "try...finally" (as defined in an extension for C and then ported to C++ in complement of "try...catch") is the way to go with Lua.

While I do like try...finally in terms of expressiveness and robustness (it's way more well-defined than <toclose> in the face of multiple errors) it's also a lot more effort on the programmer's part and doesn't solve the "forgot to release it" issue -- it just makes it easier to make sure it happens. I think <toclose> is a better fit for existing Lua programming paradigms.

try..finally are perfect to handle the "toclose" concept: instead of declaring toclose variables, you just declare local variables within the "try" block. When the block will exit for any reason (explicit return of function, exception/error, or because you reach its "ending" instruction (not marked by an "end" keyword but by the continuation of the code that reached the position just after the last token before the "finally" keyword) all local variables declared in the "try" sub-block get out of scope, are definitely no logner reachable, can have their values dereferenced (and then garbage collected).

Yes this requires discipline: to be able to close a file that has been opened in the "try" block, its file variable must be declared BEFORE entering the "try..finally...end" block (you can also declare your error tracking variables outside of it, it's not necesssary to create/instanciate a new error object within the protected block, as it's enough to pass the values or references of the values you want to give to the error handler to one of these outer variables.

Then the finally block has all it needs to decide (looking at these outer variables that indicate a status or the presence of an error) if:
- it will not handle that error/exception, in which case it can rethrow it (in Lua it can use the "error" Lua keyword for that)
- it will handle the error but will return from the function (not allowing to continue after the end of the "finally...end" subblock) with specific values
- it will return with a recursive trailing call to the function to retry, after having performed recovery/cleanup and possibly modified the values to submit as parameters to the recursive call
- it will modifiy some other status allowing an external code (enclosing the "try...finally...end" block) to adopt another behavior (for exemple set a condition that will break an enclosing loop), and then continue just after the "end"
- it will do nothing and will then let the code continue after the "end".

Typically the "finally...end" would behave in Lua like an anonymous function, called using a trailing recursive call from within the "try...finally" subblock, each time there's an error or exception occuring in it (the "error" can come from the code inside the "try...finally" subblock or from anyone of the functions it called: because this will be like if the code performed a return with a trailing function call, it will behave almost like a jump: no stack used at all (for the case of "error", it firsts calls the generic error handler that will lookup the call stack to unwind it up to the last exception handler, and then will unwind the stack and restore the context that was current at the start of the "try...finally...end" block just before performing the equivalent of a return with a trailing call to the error handler: this will also be a simple jump, so there should be no problem in terms of memory allocation, notably on the stack, because Lua ensures a minimum stack size left at top to be able to perform a call to its internal error/exception handler, that does not need any parameter from the code).

Note that this also requires discipline from the programmer: they must avoid using "error(complex expression)" to build a new object with complex function calls or evaluations that could themselves generate errors/exceptions; using the "error" keyword in Lua should be done with minimal parameters, already constructed before, or using simple values like strings, numbers or a reference to a preexisting value).

And with try..finally..end, it's then trivial to write or port the classic "try..catch" exception handlers of C++.

In summary, no need of "toclose" (whose semantic, scope and lifetime is ill-defined). No need of any specific modification to the garbage collector.

But if a programmer is lazy and declares a new varaible inside the "try...finally" 1st subblock, the consequences will be also the same as if it had forgotten to use the "toclose" feature. This is also much easier to maintain! No need to use any specific item in the metatable (in fact we don't even need any metatable, and this works as well on any variables, not jsut varaibles containing reference-type values for tables or for userdata, thread objects...)




Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Andrew Starks-2
In reply to this post by Coda Highland


> On Jul 5, 2019, at 16:24, Coda Highland <[hidden email]> wrote:
>
> I mean, yeah, I don't think angle brackets are an ideal syntax for it, and "toclose" is something of an awkward name, but everything else about it I think is sound.

Not sure where to stick these comments, so instead of starting another thread...

<toclose> solves a problem I’ve actually had: getting rid of a big resource in a deterministic way, regardless of error, not waiting for __gc. Since this solves that problem in a way I can understand, thumbs up!

Making a wild guess: in 10k lines of code, I may use something like this in 5 places, almost always in library code, not application scripts. I don’t mind it being verbose or even a little awkward. Some may argue that the syntax should stand out and be overly clear (no multiple assignments that might make order of release slightly ambiguous or at least something I may have to look up). When handling resources and memory, I like to go slow and careful because I make lots of mistakes.

As for the word, “close” makes sense enough to me because that’s what I care about (and release is a longer word to type). I don’t care that the tiny little variable is garbage collected. I care that the big giant resource that the variable is holding on to is closed/released as soon as possible.

This looks like a nice enough mechanism to do that.

I’ve read most, but not all, of the debates around this. I have not gotten the sense that the controversy is because this doesn’t work for the intended purpose; just that it’s awkward or that people wished it worked differently / more flexibly.

Does it solve a real problem that is worth solving? Does Lua 5.4’s <toclose> feature work for that intended purpose?  Does it break anything? Given its use cases, is it clear enough and does the syntax fit for the purpose?

-Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Coda Highland


On Sat, Jul 6, 2019 at 10:58 AM Andrew Starks <[hidden email]> wrote:


> On Jul 5, 2019, at 16:24, Coda Highland <[hidden email]> wrote:
>
> I mean, yeah, I don't think angle brackets are an ideal syntax for it, and "toclose" is something of an awkward name, but everything else about it I think is sound.

Not sure where to stick these comments, so instead of starting another thread...

<toclose> solves a problem I’ve actually had: getting rid of a big resource in a deterministic way, regardless of error, not waiting for __gc. Since this solves that problem in a way I can understand, thumbs up!

Making a wild guess: in 10k lines of code, I may use something like this in 5 places, almost always in library code, not application scripts. I don’t mind it being verbose or even a little awkward. Some may argue that the syntax should stand out and be overly clear (no multiple assignments that might make order of release slightly ambiguous or at least something I may have to look up). When handling resources and memory, I like to go slow and careful because I make lots of mistakes.

As for the word, “close” makes sense enough to me because that’s what I care about (and release is a longer word to type). I don’t care that the tiny little variable is garbage collected. I care that the big giant resource that the variable is holding on to is closed/released as soon as possible.

This looks like a nice enough mechanism to do that.

I’ve read most, but not all, of the debates around this. I have not gotten the sense that the controversy is because this doesn’t work for the intended purpose; just that it’s awkward or that people wished it worked differently / more flexibly.

Does it solve a real problem that is worth solving? Does Lua 5.4’s <toclose> feature work for that intended purpose?  Does it break anything? Given its use cases, is it clear enough and does the syntax fit for the purpose?

-Andrew

Even if I think the cosmetics aren't perfect I wouldn't complain about it staying as-is. I think what's currently there is good enough.

I haven't been following everything. Some of the rabbit holes have been too deep for me, so I've been tuning out threads that start going into too much detail about unnecessary alternatives.

I would probably end up using it more than you, at least in terms of uses per kloc. A lot of it is going to have to do with use cases -- I would turn to <toclose> in MOST places I would have otherwise used an IIFE. And it would really cut down on boilerplate for things I do a lot.

So in my opinion, my answers to your question are: 

Yes, it solves a real problem worth solving. Because there are techniques you could have already used it mostly just makes it easier and more efficient, so it's not a critical feature, but the cost of providing it appears to be small enough to make it worthwhile.

Yes, it works for that purpose as-is. 

No, I don't see a way it could break any existing scenarios; there's one edge case that it introduces that could possibly break (resolution of multiple errors in the callback) but C++ programmers are already used to the notion that you're not supposed to do that. 

Yes, given its use cases it's clear and the syntax fits the purpose, but the name "close" is insufficiently broad because most of MY uses for it don't have anything to do with closing anything.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
In reply to this post by Coda Highland
сб, 6 июл. 2019 г. в 07:44, Coda Highland <[hidden email]>:

>
>
>
> On Fri, Jul 5, 2019 at 6:15 PM Sergey Kovalev <[hidden email]> wrote:
>>
>> сб, 6 июл. 2019 г. в 00:24, Coda Highland <[hidden email]>:
>> You could follow rule: one enter one exit point. So you will no need
>> in return. Here return used to live then scope, like break in a loop.
>
> No, thank you. I really don't have any interest in going back to programming in Pascal. One enter is fine, but one exit is unnecessarily restrictive -- even loops have break statements, and that's multiple exits.
>
It's not a problem. It could be rewritten to use returns:

function for_scope(body)
    local list,res={}
    local function auto(close,msg)
        return function(t)
            if type(t)~='table' then error("need table: { expression }",2) end
            if t[1] then table.insert(list,{ arg=t[1], fn=close or io.close })
            else
                if msg=='weak' then return table.unpack(t) end
                error(msg or t[2] or "no resource",2)
            end
            return table.unpack(t)
        end
    end
    local function defer(fn) auto(fn){true} end
    local ok,err=pcall(function() res={body(auto,defer)} end)
    for i=#list,1,-1 do list[i].fn(list[i].arg) end
    if not ok then
        if type(err)~='string' then error(err,2)
        else error("scope error\nlua: "..err,2) end
    end
    if #res>0 then return res end
end

function test()
  for t in for_scope,function(auto,defer) -- scope.begin
    local f,err=auto(io.close,"weak") { io.open "test2.txt" }
    print(f,err)
    defer(function() print "defer" end)
    return auto(){ io.open "readme.txt" }:read()
  end do return table.unpack(t) end -- scope.end
end
print(test())

-- output:
-- nil    test2.txt: No such file or directory
-- defer
-- readme

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Gé Weijers


On Mon, Jul 8, 2019 at 7:14 AM Sergey Kovalev <[hidden email]> wrote:
function for_scope(body)
    local list,res={}
    local function auto(close,msg)
        return function(t)
            if type(t)~='table' then error("need table: { expression }",2) end
            if t[1] then table.insert(list,{ arg=t[1], fn=close or io.close })
            else
                if msg=='weak' then return table.unpack(t) end
                error(msg or t[2] or "no resource",2)
            end
            return table.unpack(t)
        end
    end
    local function defer(fn) auto(fn){true} end
    local ok,err=pcall(function() res={body(auto,defer)} end)
    for i=#list,1,-1 do list[i].fn(list[i].arg) end
    if not ok then
        if type(err)~='string' then error(err,2)
        else error("scope error\nlua: "..err,2) end
    end
    if #res>0 then return res end
end

function test()
  for t in for_scope,function(auto,defer) -- scope.begin
    local f,err=auto(io.close,"weak") { io.open "test2.txt" }
    print(f,err)
    defer(function() print "defer" end)
    return auto(){ io.open "readme.txt" }:read()
  end do return table.unpack(t) end -- scope.end
end
print(test())



Yes, you can do without "local <toclose>", but I would not call your example readable code. The point of adding features like this to a language is to make programs simpler and easier to understand by increasing the expressive power of the language.

This is a whole lot more readable:

local <toclose> f = assert(io.open("test.txt", "r"))
-- do something with f
return value

I think the "<toclose>" facility is a fair tradeoff between added language complexity and expressive power. It's also likely to be much faster because there's no need to create closures, no need to pack or unpack parameter lists, and no need to use 'pcall' or forward the error.


-- 
--

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
пн, 8 июл. 2019 г. в 20:33, Gé Weijers <[hidden email]>:
>
> Yes, you can do without "local <toclose>", but I would not call your example readable code. The point of adding features like this to a language is to make programs simpler and easier to understand by increasing the expressive power of the language.
>
> This is a whole lot more readable:
>
> local <toclose> f = assert(io.open("test.txt", "r"))
> -- do something with f
> return value

I think this is readable enough:
local f=auto(io.close){ io.open("test.txt","r") }
-- do something with f
return value

And do it same things.
Boiler plate code "scope.begin" and "scope.end" be hidden for example
by transpiler or preprocessor.

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
вт, 9 июл. 2019 г. в 01:23, Sergey Kovalev <[hidden email]>:
>
> пн, 8 июл. 2019 г. в 20:33, Gé Weijers <[hidden email]>:
> >
> > Yes, you can do without "local <toclose>", but I would not call your example readable code. The point of adding features like this to a language is to make programs simpler and easier to understand by increasing the expressive power of the language.
> >
> > This is a whole lot more readable:
> >
> > local <toclose> f = assert(io.open("test.txt", "r"))

I think this is readable enough:
local f=auto(io.close){ io.open("test.txt","r") }
And it does do same things.

or even if you have to open group of files:
local files={}
for name in names do
  table.insert(files, auto(io.close){ io.open(name) } )
end

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Coda Highland
In reply to this post by Sergey Kovalev


On Mon, Jul 8, 2019 at 5:23 PM Sergey Kovalev <[hidden email]> wrote:
пн, 8 июл. 2019 г. в 20:33, Gé Weijers <[hidden email]>:
>
> Yes, you can do without "local <toclose>", but I would not call your example readable code. The point of adding features like this to a language is to make programs simpler and easier to understand by increasing the expressive power of the language.
>
> This is a whole lot more readable:
>
> local <toclose> f = assert(io.open("test.txt", "r"))
> -- do something with f
> return value

I think this is readable enough:
local f=auto(io.close){ io.open("test.txt","r") }
-- do something with f
return value

And do it same things.
Boiler plate code "scope.begin" and "scope.end" be hidden for example
by transpiler or preprocessor.


Well YEAH, of course you can do it if you add a preprocessing phase. That's cheating. :P It's not really Lua anymore if you do that.

/s/ Adam 
Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Dirk Laurie-2
In reply to this post by Gé Weijers
Op Ma. 8 Jul. 2019 om 19:33 het Gé Weijers <[hidden email]> geskryf:

> Yes, you can do without "local <toclose>", but I would not call your example readable code. The point of adding features like this to a language is to make programs simpler and easier to understand by increasing the expressive power of the language.
>
> This is a whole lot more readable:
>
> local <toclose> f = assert(io.open("test.txt", "r"))
> -- do something with f
> return value
>
> I think the "<toclose>" facility is a fair tradeoff between added language complexity and expressive power. It's also likely to be much faster because there's no need to create closures, no need to pack or unpack parameter lists, and no need to use 'pcall' or forward the error.

The VM instructions in lopcodes.c are

opmode(0, 0, 0, 0, iABC)              /* OP_TBC */
opmode(0, 0, 0, 0, iABC)              /* OP_CLOSE */

So 'local <toclose> x = initializer' generates the same code as 'local
x = initlalizer', plus ' TBC lx' where lx is the number of the local
variable x, plus 'CLOSE lx' at the end of the scope.

Given that there is to be a name for the attribute that x should have,
only two bytes '<>' of syntax is used up, and those bytes provide the
feature of associating other attributes with a local variable.

Personally (as I have argued in another post) I think the name
'close', with metamethod '__toclose', too specific, since it is merely
a method called when a local name with which that variable is
associated goes out of scope. There is no need at all for the value
itself to end its existence: it could have other names. 'close' is
appropriate for a file, but the value really could be anything, e.g. a
custom message handler.

-- Dirk

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
вт, 9 июл. 2019 г. в 19:44, Dirk Laurie <[hidden email]>:
>
> Op Ma. 8 Jul. 2019 om 19:33 het Gé Weijers <[hidden email]> geskryf:
>

> > I think the "<toclose>" facility is a fair tradeoff between added language complexity and expressive power. It's also likely to be much faster because there's no need to create closures, no need to pack or unpack parameter lists, and no need to use 'pcall' or forward the error.
>
> The VM instructions in lopcodes.c are
>
> opmode(0, 0, 0, 0, iABC)              /* OP_TBC */
> opmode(0, 0, 0, 0, iABC)              /* OP_CLOSE */
>
> So 'local <toclose> x = initializer' generates the same code as 'local
> x = initlalizer', plus ' TBC lx' where lx is the number of the local
> variable x, plus 'CLOSE lx' at the end of the scope.
>
> Given that there is to be a name for the attribute that x should have,
> only two bytes '<>' of syntax is used up, and those bytes provide the
> feature of associating other attributes with a local variable.
>
> Personally (as I have argued in another post) I think the name
> 'close', with metamethod '__toclose', too specific, since it is merely
> a method called when a local name with which that variable is
> associated goes out of scope. There is no need at all for the value
> itself to end its existence: it could have other names. 'close' is
> appropriate for a file, but the value really could be anything, e.g. a
> custom message handler.
>
Syntax for toclose is very inconsistent. For example defer
implementation in lua 5.4:

defer=setmetatable({},{__sub=function(t,f)
    return setmetatable({},{__close=f})
end})

usage:
do
  local<toclose>_=defer-function() print "defer" end
  print "body"
end

output:
body
defer

1. I have to declare name.
2. Syntax is inconsistent. It looks like squashed alien:
"local ‘<’ name ‘>’ Name ‘=’ exp"
only one name and must be one expression.
local <toclose> name=123 -- will cause error but it after end of block
and if no error in block
local <toclose> name -- error need expression
local <toclose> name=nil -- fine. will work without error
The way it could be used is to raw. And should be found more suitable syntax.

ps: local <const> a={} a.x=1 print(a.x)
What is this constants for? Area where they could be helpful is too limited.
Isolate block or function from up-values will be much useful. This
could solve problem of isolating code from current context.

Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Lorenzo Donati-3
In reply to this post by Andrew Starks-2
On 06/07/2019 17:58, Andrew Starks wrote:

>
>
>> On Jul 5, 2019, at 16:24, Coda Highland <[hidden email]>
>> wrote:
>>
>> I mean, yeah, I don't think angle brackets are an ideal syntax for
>> it, and "toclose" is something of an awkward name, but everything
>> else about it I think is sound.
>
> Not sure where to stick these comments, so instead of starting
> another thread...
>
> <toclose> solves a problem I’ve actually had: getting rid of a big
> resource in a deterministic way, regardless of error, not waiting for
> __gc. Since this solves that problem in a way I can understand,
> thumbs up!
>
> Making a wild guess: in 10k lines of code, I may use something like
> this in 5 places, almost always in library code, not application
> scripts. I don’t mind it being verbose or even a little awkward. Some
> may argue that the syntax should stand out and be overly clear (no
> multiple assignments that might make order of release slightly
> ambiguous or at least something I may have to look up). When handling
> resources and memory, I like to go slow and careful because I make
> lots of mistakes.
>
> As for the word, “close” makes sense enough to me because that’s what
> I care about (and release is a longer word to type). I don’t care
> that the tiny little variable is garbage collected. I care that the
> big giant resource that the variable is holding on to is
> closed/released as soon as possible.
>
> This looks like a nice enough mechanism to do that.
>
> I’ve read most, but not all, of the debates around this. I have not
> gotten the sense that the controversy is because this doesn’t work
> for the intended purpose; just that it’s awkward or that people
> wished it worked differently / more flexibly.
>
> Does it solve a real problem that is worth solving? Does Lua 5.4’s
> <toclose> feature work for that intended purpose?  Does it break
> anything? Given its use cases, is it clear enough and does the syntax
> fit for the purpose?
>
> -Andrew
>

(Also related to Coda Highland's reply to your post)

BTW (@Coda): yes I, got almost lost in the rabbit holes, too! So I may
be repeating things here (sorry then).

@Andrew:

You make a nice analysis of the situation, and you come to the
conclusion that a little syntactic/semantic awkwardness is acceptable,
because the problem this new mechanism solve "elegantly" (essentially
RAII) is very important and critical in resource management.

I agree to a point: we would really benefit a clean mechanism for RAII.

The problem I see with the awkwardness is that it may bite us heavily in
the future (5.1 "module" thing, anyone?).
I would really prefer to wait a little more for such a feature until Lua
Team has come up with a smoother mechanism.

The ugliness of the "toclose" name is just one thing. The not-so-nice
interaction with multiple assignment and other idioms is yet another
one. Who knows which kind of antipattern could come up after we use it
for a while.

And, BTW, readability in code is not to be underestimated. Especially in
resource management. If the code is managed by multiple persons,
especially over a long span of time, nasty error could be introduced
because of miscommunication (high level code is more a mean of
communication between people - even a programmer and himself - rather
than a way to give a computer orders).

I almost feel that /there has to be a better way/ and the current way
doesn't feel quite right.

For example, the fact that a /variable/ is marked, but /an object/ is
"closed" (using its metamethod) is something smelly. The whole point of
RAII is to make very explicit the connection between a resource and the
code that handle its release and the time this code is run.
The current Lua 5.4 mechanism is not so explicit, IMO.

For that matters, I'm beginning to feel a sort of "defer..end" block
could be a better, more general, more explicit, solution.

I love Lua ability to keep the code clean (and clear) in common use
cases (although it has its dark corners, e.g. the general for loop
syntax - maybe it's me, but I keep forgetting what are those pesky three
parameters after "in" keyword <ugh>). I'd like to stay that way.

Cheers!

-- Lorenzo












Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Hakki Dogusan-3
Hi,

On 12.07.2019 12:36, Lorenzo Donati wrote:

> On 06/07/2019 17:58, Andrew Starks wrote:
>[snip]
>
> I almost feel that /there has to be a better way/ and the current way
> doesn't feel quite right.
>
> For example, the fact that a /variable/ is marked, but /an object/ is
> "closed" (using its metamethod) is something smelly. The whole point of
> RAII is to make very explicit the connection between a resource and the
> code that handle its release and the time this code is run.
> The current Lua 5.4 mechanism is not so explicit, IMO.
>

Seconded.

This "variable-value" thing already existed, but it becomes confusing.
IMHO solution should not touch "value/class" metatable.

Maybe something like following is better?
local a = ..
a.__on_scope_exit = function .. end

(Although I'm still thinking that similar to setmetatable, there maybe
setattribute function for variables)


> For that matters, I'm beginning to feel a sort of "defer..end" block
> could be a better, more general, more explicit, solution.
>
> I love Lua ability to keep the code clean (and clear) in common use
> cases (although it has its dark corners, e.g. the general for loop
> syntax - maybe it's me, but I keep forgetting what are those pesky three
> parameters after "in" keyword <ugh>). I'd like to stay that way.
>
> Cheers!
>
> -- Lorenzo
>
>
>
>
>
>
>
>
>
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Another option for closing variables

Sergey Kovalev
In reply to this post by Coda Highland
I found the way to do it in lua without preprocessing. But only in 5.4.
There is 4-th parameter in for loop introduced in lua 5.4 so
now it look like:

require "scope54" --
https://raw.githubusercontent.com/kov-serg/lua-aux/master/scope54.lua

for auto,defer in scope() do
  defer(function() print"defer" end)
  local f=auto(io.close){ io.open "scope54.lua"}
  print(f:read())
end

I think this is enough readable and looks nice.

вт, 9 июл. 2019 г. в 02:54, Coda Highland <[hidden email]>:
>
> Well YEAH, of course you can do it if you add a preprocessing phase. That's cheating. :P It's not really Lua anymore if you do that.
>
> /s/ Adam

12