Why do we need “toclose” (or those angle brackets) at all?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Why do we need “toclose” (or those angle brackets) at all?

Paul Ducklin
Here’s something I don’t understand.

Why do we need the ugly, weird-looking and controversial <toclose> notation at all? (Also, “toclose” isn’t quite the right word anyway, but that is another matter.)

We don’t annotate variable declarations with <tofinalize> when they have a finalizer - we just set a __gc metamethod and that tells Lua it’s a <tofinalize> variable automatically.

A <toclose> variable *must have* a __close metamethod or else an error is raised when it goes out of scope. So why can’t the presence of a __close metamethod be the signal to Lua that it is handling a to-be-closed variable?

The <toclose> flag for a variable should simply be set automatically at runtime when the setmetatable() function is called for the variable.



PS. The __close metamethod is not documented in the Metatables and Metamethods section (2.4) of the manual. It should be. (The __gc metamethod is mentioned there but as a bit of an afterthought. Both of those special metamethods ought to be included in the formal list of events that activate metamethods, for completeness and clarity.)
Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

David Heiko Kolf-2
Paul Ducklin wrote:
> A <toclose> variable *must have* a __close metamethod or else an error is raised when it goes out of scope. So why can’t the presence of a __close metamethod be the signal to Lua that it is handling a to-be-closed variable?
>
> The <toclose> flag for a variable should simply be set automatically at runtime when the setmetatable() function is called for the variable.

Lua cannot easily tell the intended scope of a variable without some
extra informations. Consider this case:

  function foo (fname)
    local f, err = io.open(fname, "w")
    if not f then
      print(("Failed to open %s: %s"):format(fname, err))
    else
      bar(f)
    end
  end

Now what does "bar(f)" do to the file? It could just write some stuff in
the file and it should be closed once f in "foo" goes out of scope:

  function bar (f)
    f:write("some stuff")
  end

However, it could also save the file handle somewhere else where it is
expected to be still open at a later point:

  function bar (f)
    activefile = f
  end

How would Lua know there is still a reference to the file? At least
without running the GC every time a variable with a __close metamethod
goes out of scope.

Best regards

David

Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Egor Skriptunoff-2
In reply to this post by Paul Ducklin
On Fri, Jun 14, 2019 at 2:27 PM Paul Ducklin wrote:

The <toclose> flag for a variable should simply be set automatically at runtime when the setmetatable() function is called for the variable.



To-be-closed variables should be visible at compile-time because they affect VM instructions in the compiled bytecode.
For example, tailcall is not allowed in the scope of to-be-closed variable.


Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Sergey Zakharchenko
In reply to this post by Paul Ducklin
Paul,

> Why do we need the ugly, weird-looking and controversial <toclose> notation at all?

Indeed, so many people don't like it, but fail to notice one thing.
You don't have to use it!

local function scope(object)
    return next,{[object]=1},nil,object
end

for file in scope(io.open("file")) do
    print("I can use file ",file)
end

print"File has been closed!"

People trying to get others to use variants of 'with' don't notice
it's there already.

Please. Things are going fine.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Paul Ducklin
In reply to this post by Egor Skriptunoff-2
Then perhaps the issue would better be solved with new keywords rather than trying to add modifiers to “local”. For clarity, I would much prefer to write:

do
  local a,b,c
  autoclose e,f,g
  const t = 2*math.pi
  ...
end

to declare what are essentially three very different sorts of local object, rather than denote (using that rather clumsy angle-bracket syntax) that they are all local variables with slightly different flavours.

As many have said, the idea of a “constant variable” is kind of weird. And if a to-be-closed variable is such a different sort of beast to a regular local variable that it needs different compile time opcodes, then it doesn’t seem weird to have a unique keyword to declare it. In fact, it feels weird to use the same keyword “local” to declare them all, especially the idea of a “local constant variable”. 

(The idea of “constant variables” sounds very strange indeed - as strange as saying that local variables that are initialised to a fixed value, as in the code “local flag = true”, should be called “variable constants” :-).


On 14 Jun 2019, at 13:02, Egor Skriptunoff <[hidden email]> wrote:

On Fri, Jun 14, 2019 at 2:27 PM Paul Ducklin wrote:

The <toclose> flag for a variable should simply be set automatically at runtime when the setmetatable() function is called for the variable.



To-be-closed variables should be visible at compile-time because they affect VM instructions in the compiled bytecode.
For example, tailcall is not allowed in the scope of to-be-closed variable.


Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Paul Ducklin
In reply to this post by Sergey Zakharchenko
In your example, does your scope() function absolutely guarantee that the file you opened in the call to scope() will synchronously and reliably be closed after the “end” of the for loop but before the final call to print()?

Doesn’t the closing of the file in your code depend in some way on an asynchronous part of the Lua core, namely the garbage collector?



> On 14 Jun 2019, at 13:16, Sergey Zakharchenko <[hidden email]> wrote:
>
> Paul,
>
>> Why do we need the ugly, weird-looking and controversial <toclose> notation at all?
>
> Indeed, so many people don't like it, but fail to notice one thing.
> You don't have to use it!
>
> local function scope(object)
>    return next,{[object]=1},nil,object
> end
>
> for file in scope(io.open("file")) do
>    print("I can use file ",file)
> end
>
> print"File has been closed!"
>
> People trying to get others to use variants of 'with' don't notice
> it's there already.
>
> Please. Things are going fine.
>
> Best regards,
>
> --
> DoubleF
>
Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Sergey Zakharchenko

Paul,

> In your example, does your scope() function absolutely guarantee that the file you opened in the call to scope() will synchronously and reliably be closed after the “end” of the for loop but before the final call to print()?

The 5.4 'for' loop does (we return the file as a closin value for it). The function itself can be implemented in different ways, that's just one of them. As long as it doesn't itself cause some error (out of memory constructing the table?), the guarantee should hold.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Paul Ducklin
Ah!

I missed that rather important detail about an iterator function now returning *four* values, not three.

FWIW I think that is a significant change from 5.3 to 5.4 and really ought to be added to the “main changes” section in the 5.4.0 manual. 

The manual mentions that the semantics of the numeric “for” has changed so I really think it is worth mentioning this change in the generic “for” as well. Maybe say words something like this:

* generic for loop iteration now allows for automatic variable cleanup when the loop ends




On 14 Jun 2019, at 13:57, Sergey Zakharchenko <[hidden email]> wrote:

Paul,

> In your example, does your scope() function absolutely guarantee that the file you opened in the call to scope() will synchronously and reliably be closed after the “end” of the for loop but before the final call to print()?

The 5.4 'for' loop does (we return the file as a closin value for it). The function itself can be implemented in different ways, that's just one of them. As long as it doesn't itself cause some error (out of memory constructing the table?), the guarantee should hold.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Ryan Ford-2
In reply to this post by Paul Ducklin
On Fri, 14 Jun 2019 11:27:27 +0000
Paul Ducklin <[hidden email]> wrote:

> Here’s something I don’t understand.
>
> Why do we need the ugly, weird-looking and controversial <toclose> notation at all? (Also, “toclose” isn’t quite the right word anyway, but that is another matter.)

I could be wrong, but I believe it's related to the new generational gc. If it doesn't get collected early, it could be a while before it does get collected, so now you'll have a way to get it taken care of when the variable goes out of scope and would then no longer be needed.

That's my limited understanding


Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

Rodrigo Azevedo
In reply to this post by Paul Ducklin
Em sex, 14 de jun de 2019 às 09:52, Paul Ducklin
<[hidden email]> escreveu:

>
> Then perhaps the issue would better be solved with new keywords rather than trying to add modifiers to “local”. For clarity, I would much prefer to write:
>
> do
>   local a,b,c
>   autoclose e,f,g
>   const t = 2*math.pi
>   ...
> end
>
> to declare what are essentially three very different sorts of local object, rather than denote (using that rather clumsy angle-bracket syntax) that they are all local variables with slightly different flavours.

They are not slightly different flavors, but three different,
additive, concepts indeed.

local Name = value --> lexically scoped Name
local & constant  Name = value -->  local AND constante value for Name
local & caught Name = value  --> local AND to be __caught when out of scope

You can even understand like "local&constant"  or "local&caught" as
qualifiers for Names that can be highlighted (as 'local' is in fact)
by your favorite editor.

> As many have said, the idea of a “constant variable” is kind of weird. And if a to-be-closed variable is such a different sort of beast to a regular local variable that it needs different compile time opcodes, then it doesn’t seem weird to have a unique keyword to declare it. In fact, it feels weird to use the same keyword “local” to declare them all, especially the idea of a “local constant variable”.

<toclose> is ugly, moreover, 'toclose' is not a good word. In fact, we
can make many things with "Name"  when it is out of scope not directly
related with 'close' or 'finalize' anything, which will make 'close'
and 'finalize' - like words very confusing when programmers try to use
__caught to solve other problems.

"constant variable" is a misinterpretation: the 'Name' is 'local',
namely, lexically scoped, AND maybe 'constant' or 'caught'. The
'variable' is only a (common) subtype that is the default for
convenience :)

--
Rodrigo Azevedo Moreira da Silva

Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

David Heiko Kolf-2
In reply to this post by Sergey Zakharchenko
Sergey Zakharchenko wrote:

> Paul,
>
>> Why do we need the ugly, weird-looking and controversial <toclose> notation at all?
>
> Indeed, so many people don't like it, but fail to notice one thing.
> You don't have to use it!
>
> local function scope(object)
>     return next,{[object]=1},nil,object
> end
>
> for file in scope(io.open("file")) do
>     print("I can use file ",file)
> end

Oh, I missed those changes to the for loop as well. This example however
would ignore any errors, io.open would return nil as the first return
value and the loop never runs.

I tried extending your example, but my result doesn't look pretty in my
opinion:

  local function scoperesult (t, c)
      if not c then
          return true, table.unpack(t, 1, t.n)
      else
          return nil
      end
  end

  function scope (...)
      local t = { ... }
      t.n = select("#", ...)
      return scoperesult, t, nil, (...)
  end

  for _, f, err in scope(io.open("test.dat") do
      print(f, err)
  end

It works, but it requires a new temporary table and an otherwise unused
loop control variable.

I still hope for some syntax like
  local <toclose> f, err = io.open("test.dat")

Or whatever is used instead of "<toclose>", I don't care that much about
it, it doesn't look particularly ugly to me.

Best regards,

David

Reply | Threaded
Open this post in threaded view
|

Re: Why do we need “toclose” (or those angle brackets) at all?

René Rebe
In reply to this post by Paul Ducklin
Hi,

On 14 Jun 2019, at 13:27, Paul Ducklin <[hidden email]> wrote:

Here’s something I don’t understand.

Why do we need the ugly, weird-looking and controversial <toclose> notation at all? (Also, “toclose” isn’t quite the right word anyway, but that is another matter.)

Yeah, well, I also wanted to nearly ask myself as well, but then did not want to start a flamewar.

From my point of view I would prefer “local” without the controversial <> notation as well:

local const foo = 123

I understand that change would render existing code incompatible which already used “const” for their own variables (well, seriously?, … their bad ;-)
I also understand the controversial and wired-looking <> notation is more flexible for the future. But do we really need that (keeping Lua small and such)?
Well, one could probably roll their own syntactic sugar to translate “ const “ to “ <const> “ though.

Btw. I have not looked at the implementation details, but it would be cool if a global:

local const debug = false

could be used by the compiler to optimize all the debug conditionals away in the bytecode, … at least that would be one useful use case for myself.

Thanks,
René

We don’t annotate variable declarations with <tofinalize> when they have a finalizer - we just set a __gc metamethod and that tells Lua it’s a <tofinalize> variable automatically.

A <toclose> variable *must have* a __close metamethod or else an error is raised when it goes out of scope. So why can’t the presence of a __close metamethod be the signal to Lua that it is handling a to-be-closed variable?

The <toclose> flag for a variable should simply be set automatically at runtime when the setmetatable() function is called for the variable.



PS. The __close metamethod is not documented in the Metatables and Metamethods section (2.4) of the manual. It should be. (The __gc metamethod is mentioned there but as a bit of an afterthought. Both of those special metamethods ought to be included in the formal list of events that activate metamethods, for completeness and clarity.)

-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin, https://exactcode.com