[NoW] Once more about to-be-closed variables

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[NoW] Once more about to-be-closed variables

Egor Skriptunoff-2

1) About limitations.
Why to-be_closed variables must be initialized at the very beginning?
Why they must be const?
Why an error must be raised if a value doesn't have the metamethod?
Are all these limitations necessary?
The concept of "to-be-closed" variables would be more clean and easy-to-understand without all these restrictions.
I mean, all the specific actions should be taken only when variable is going out of scope and only if it has the metamethod.
Besides that, such variable could be used as usual local variable.


2) About naming.
"close" says about the value (the value will be closed) and says nothing about the variable
"scope" says about the variable (the variable goes out of scope) and says nothing about the value
We need a word emphasizing the semantical link between "value" and "variable"
There was a suggestion "bind" (the value is bound to the variable)
I suggest another one: "evict" (the value is evicted from the local variable when the variable goes out of scope)
local <evictable> x = setmetatable(obj, {__evict = ...})
The name doesn't insist on what user should use this feature for (closing/finalizing/releasing/destructing/unlocking).
IMO, the word "evict" explains the purpose of such variables better than the word "bind".
The phrase "evictable variables" is good enough too.
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Sean Conner

  Um ... it's Tuesday.  Shouldn't this come out tomorrow?

It was thus said that the Great Egor Skriptunoff once stated:
> 1) *About limitations.*
> Why to-be_closed variables must be initialized at the very beginning?

  Because the variable is marked const and can't be updated once marked.

> Why they must be const?

  I think it's to prevent the value from being lost and thus, not closed.
Remember, it's the *variable* that is marked, *NOT* the value.

> Why an error must be raised if a value doesn't have the metamethod?

  I don't like this behavior either.

> Are all these limitations necessary?

  I hope not.  I also hope the single variable limit can be lifted so one
can do (for the poster child example for this concept):

        local <toclose> file,err = io.open(filename)

  It's idiotic that right now, you have to do:

        local file,err = io.open(filename)
        local <toclose> file = file -- [1]

  -spc (assume there's a check of err after the first statement ... )

[1] This brings up a point about luacheck.  Has anyone taken over the
        project?  I would hate to have to stop using luacheck because it
        doesn't support Lua 5.4 syntax.


Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Egor Skriptunoff-2
On Wed, Jul 3, 2019 at 12:37 AM Sean Conner wrote:

  Um ... it's Tuesday.  Shouldn't this come out tomorrow?



It's Wednesday in Moscow.
So, I can start nitpicking.
Apparently you still can not :-)

 
> Why they must be const?

  I think it's to prevent the value from being lost and thus, not closed.


Do we need this "protection"?
Which adversary we should be protected from?

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Patrick Donnelly


On Tue, Jul 2, 2019, 3:02 PM Egor Skriptunoff <[hidden email]> wrote:
 
> Why they must be const?

  I think it's to prevent the value from being lost and thus, not closed.


Do we need this "protection"?
Which adversary we should be protected from?

Yourself, of course!
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Patrick Donnelly
In reply to this post by Egor Skriptunoff-2
On Tue, Jul 2, 2019, 2:09 PM Egor Skriptunoff <[hidden email]> wrote:
Why an error must be raised if a value doesn't have the metamethod?

I do wonder why not allow functions to be assigned and called when the local goes out of scope. It does make some kinds of error handling much simpler. It also is consistent with other metamethods like __index which may be another table or a function.
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Coda Highland
In reply to this post by Egor Skriptunoff-2


On Tue, Jul 2, 2019 at 5:02 PM Egor Skriptunoff <[hidden email]> wrote:
On Wed, Jul 3, 2019 at 12:37 AM Sean Conner wrote:

  Um ... it's Tuesday.  Shouldn't this come out tomorrow?



It's Wednesday in Moscow.
So, I can start nitpicking.
Apparently you still can not :-)

 
> Why they must be const?

  I think it's to prevent the value from being lost and thus, not closed.


Do we need this "protection"?
Which adversary we should be protected from?


Typographical / logical errors. It's defensive programming -- the entire reason for using "const" even though you can theoretically write a correct program without it.

/s/ Adam 
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Sean Conner
In reply to this post by Egor Skriptunoff-2
It was thus said that the Great Egor Skriptunoff once stated:
> On Wed, Jul 3, 2019 at 12:37 AM Sean Conner wrote:
>
> >   Um ... it's Tuesday.  Shouldn't this come out tomorrow?
>
> It's Wednesday in Moscow.
> So, I can start nitpicking.
> Apparently you still can not :-)

  Ah.  Darn timezones.

> > > Why they must be const?
> >
> >   I think it's to prevent the value from being lost and thus, not closed.
>
> Do we need this "protection"?
> Which adversary we should be protected from?

  Typos, mistakes, stuff like that I guess.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Andrew Gierth
In reply to this post by Patrick Donnelly
>>>>> "Patrick" == Patrick Donnelly <[hidden email]> writes:

 >> Why an error must be raised if a value doesn't have the metamethod?

 Patrick> I do wonder why not allow functions to be assigned and called
 Patrick> when the local goes out of scope.

This feature existed when *toclose was first added, and was then
explicitly removed. One problem is that it lets lua code catch errors in
ways that a sandbox can't control.

See https://github.com/lua/lua/commit/4ace93ca6502dd1da38d5c06fa099d229e791ba8

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Egor Skriptunoff-2
In reply to this post by Sean Conner

> > > Why they must be const?
> >
> >   I think it's to prevent the value from being lost and thus, not closed.
>
> Do we need this "protection"?
> Which adversary we should be protected from?

  Typos, mistakes, stuff like that I guess.


We have much bigger problem in Lua: every typo in a local variable's name silently converts it to global.
Compared with this, a "const" looks like a useless toy (if we consider "const" as a protection against typos).
IMO, "to-be-closed variable accidental assignment alert" should be moved from Lua core into external software (such as luacheck)
Besides, it would be useful to have non-constant to-be-closed variables
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Sean Conner
It was thus said that the Great Egor Skriptunoff once stated:
>
> IMO, "to-be-closed variable accidental assignment alert" should be moved
> from Lua core into external software (such as luacheck)
> Besides, it would be useful to have non-constant to-be-closed variables

  So, what exactly should happen in this code?

        local <toclose> f = io.open("a")
        local x = f
        f = io.open("b") -- should 'a' be closed?  or not because of x
        f = io.open("c") -- should 'b' be closed?
       
  -spc (Trying to pin down semantics here ... )


Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Egor Skriptunoff-2
On Wed, Jul 3, 2019 at 9:19 AM Sean Conner <[hidden email]> wrote:
> Besides, it would be useful to have non-constant to-be-closed variables

  So, what exactly should happen in this code?

        local <toclose> f = io.open("a")
        local x = f
        f = io.open("b") -- should 'a' be closed?  or not because of x
        f = io.open("c") -- should 'b' be closed?



In your example only the last value in variable f would be evicted.

Simple example how non-constant to-be-closed variable might be useful:
when you need to reopen a file during file operations (close the file, open it again with another access mode, save new file value into the to-be-closed variable)
It is not known in compile-time would you need to re-open file or not.
This situation requires to-be-closed variable to be writable.

do
   local <toclose> f = io.open("a")
   ...
   if need_to_reopen_this_file then
      f:close()
      f = io.open(...)
   end
   ...
end  -- evict the file value from f

Another possible situation - you are reading multi-volume archive.
Only one volume file is opened at any given time.
To switch a volume, you close the current volume file and open the next volume file.
Single to-be-closed variable is enough to guarantee that the last opened volume file will be closed on eviction.
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Jonathan Goble
In reply to this post by Sean Conner
On Wed, Jul 3, 2019 at 2:19 AM Sean Conner <[hidden email]> wrote:

>
> It was thus said that the Great Egor Skriptunoff once stated:
> >
> > IMO, "to-be-closed variable accidental assignment alert" should be moved
> > from Lua core into external software (such as luacheck)
> > Besides, it would be useful to have non-constant to-be-closed variables
>
>   So, what exactly should happen in this code?
>
>         local <toclose> f = io.open("a")
>         local x = f
>         f = io.open("b") -- should 'a' be closed?  or not because of x
>         f = io.open("c") -- should 'b' be closed?
>
>   -spc (Trying to pin down semantics here ... )

Here's an idea for syntax:

local f = <toclose> io.open("a")

Instead of the variable being marked for closing, the object is
marked. Then there is no confusion over what happens if the object is
assigned to another variable or a different object is assigned to the
same variable, because the variable itself no longer cares whether it
is marked or not.

Of course, you need some definition of an object going out of scope.
Reference counting easily solves that and can be implemented in a way
that only counts references for to-be-closed objects so that the usual
case (not to-be-closed) isn't slowed down. The INCREF and DECREF calls
that are pervasive in Python C extensions aren't an issue in Lua C
extensions, since Lua doesn't allow objects to be held in arbitrary C
variables and so the counting can be handled behind the scenes (add a
reference when adding to the stack, subtract a reference when popping
from the stack). The only issue I can see would be figuring out how to
determine the initial reference count when first marking an object.

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Sean Conner
In reply to this post by Egor Skriptunoff-2
It was thus said that the Great Egor Skriptunoff once stated:

> On Wed, Jul 3, 2019 at 9:19 AM Sean Conner <[hidden email]> wrote:
>
> > > Besides, it would be useful to have non-constant to-be-closed variables
> >
> >   So, what exactly should happen in this code?
> >
> >         local <toclose> f = io.open("a")
> >         local x = f
> >         f = io.open("b") -- should 'a' be closed?  or not because of x
> >         f = io.open("c") -- should 'b' be closed?
> >
> >
> >
> In your example only the last value in variable f would be evicted.

  Is that because of 'x'?  Or because that's the end of the scope?

> Simple example how non-constant to-be-closed variable might be useful:
> when you need to reopen a file during file operations (close the file, open
> it again with another access mode, save new file value into the
> to-be-closed variable)
> It is not known in compile-time would you need to re-open file or not.
> This situation requires to-be-closed variable to be writable.
>
> do
>    local <toclose> f = io.open("a")
>    ...
>    if need_to_reopen_this_file then
>       f:close()
>       f = io.open(...)
>    end
>    ...
> end  -- evict the file value from f
>
> Another possible situation - you are reading multi-volume archive.
> Only one volume file is opened at any given time.
> To switch a volume, you close the current volume file and open the next
> volume file.
> Single to-be-closed variable is enough to guarantee that the last opened
> volume file will be closed on eviction.

  The following worked:

        for i = 1 , 10 do
          local <toclose> f = io.open(string.format("volume.%d",i),"w")
          f:write(tostring(i),"\n")
        end

  Only one file is open at a time.  So there is a way to kind of re-use a
closed variable.

  Now, the following kept the files open unil the end:

        local function process(i)
          if i < 10 do
            local <toclose> f = io.open(string.format("volume.%d",i),"w")
            return process(i+1)
          end
        end

  Just something else to keep in mind.

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Dirk Laurie-2
In reply to this post by Egor Skriptunoff-2
Op Wo. 3 Jul. 2019 om 08:00 het Egor Skriptunoff
<[hidden email]> geskryf:

> We have much bigger problem in Lua: every typo in a local variable's name silently converts it to global.

The fix for that is well known and widely distributed under the name strict.lua.

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Dirk Laurie-2
In reply to this post by Egor Skriptunoff-2
Op Di. 2 Jul. 2019 om 23:10 het Egor Skriptunoff
<[hidden email]> geskryf:

> 2) About naming.
> "close" says about the value (the value will be closed) and says nothing about the variable
> "scope" says about the variable (the variable goes out of scope) and says nothing about the value
> We need a word emphasizing the semantical link between "value" and "variable"

No, we don't need such a word. It suffices to use existing terms properly.

The word "variable" emphasizes the semantic link between "value" and "name."

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Chris Smith
In reply to this post by Sean Conner

> On 3 Jul 2019, at 08:10, Sean Conner <[hidden email]> wrote:
>
> It was thus said that the Great Egor Skriptunoff once stated:
>> On Wed, Jul 3, 2019 at 9:19 AM Sean Conner <[hidden email]> wrote:
>>
>>>> Besides, it would be useful to have non-constant to-be-closed variables
>>>
>>>  So, what exactly should happen in this code?
>>>
>>>        local <toclose> f = io.open("a")
>>>        local x = f
>>>        f = io.open("b") -- should 'a' be closed?  or not because of x
>>>        f = io.open("c") -- should 'b' be closed?
>>>
>>>
>>>
>> In your example only the last value in variable f would be evicted.
>
>  Is that because of 'x'?  Or because that's the end of the scope?

It sounds like it might be better to have __assign and __deassign meta-functions, rather than <toclose>:

do
  f = io.open(‘a’) — __assign called on result of io.open(‘a’)
  local x = f — implicit f.__assign()
  local y = x — implicit x.__assign()

  y = ’something else’ — implicit y.__deassign() followed by (’something else’).__assign()
end — implicit x.__deassign() and y.__deassign() because they are going out of scope.

Regards,
Chris

Chris Smith <[hidden email]>


Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Egor Skriptunoff-2
In reply to this post by Sean Conner


On Wed, Jul 3, 2019 at 10:11 AM Sean Conner wrote:
It was thus said that the Great Egor Skriptunoff once stated:
> Another possible situation - you are reading multi-volume archive.
> Only one volume file is opened at any given time.
> To switch a volume, you close the current volume file and open the next
> volume file.
> Single to-be-closed variable is enough to guarantee that the last opened
> volume file will be closed on eviction.

  The following worked:

        for i = 1 , 10 do
          local <toclose> f = io.open(string.format("volume.%d",i),"w")
          f:write(tostring(i),"\n")
        end

  Only one file is open at a time.  So there is a way to kind of re-use a
closed variable.

  Now, the following kept the files open unil the end:

        local function process(i)
          if i < 10 do
            local <toclose> f = io.open(string.format("volume.%d",i),"w")
            return process(i+1)
          end
        end





This is OK while you can organize your code in the following loop:
   while (file_j_exists) do
      process_next_file(j)
      j = j + 1
   end

But sometimes the logic is complex enough so that you can't simply write the function "process next file".
It might happen that your code has to be a consumer: it needs switching volumes on-the-fly without interrupting main execution stream.
It's a "producer-consumer problem".
 
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Egor Skriptunoff-2
In reply to this post by Dirk Laurie-2
On Wed, Jul 3, 2019 at 10:56 AM Dirk Laurie wrote:
> We have much bigger problem in Lua: every typo in a local variable's name silently converts it to global.

The fix for that is well known and widely distributed under the name strict.lua.



Indeed? 
Try to use strict.lua to find a typo in this code:

   for number = 11, math.huge do
      local sum = 0
      for divisor = 1, number do
         sum = sum + (number % divisor == 0 and divisor or 0)
      end
      if sum/number == 1.8 then
         print("Friend of 10 is "..mumber)   -- the typo is here
         break
      end
   end



There is a problem with strict.lua: you have to write tests to cover all code paths.
Branching in Lua code is possible due to "while/repeat/for", "if/then/else", "goto", and due to short-circuit evaluation of expressions (such as "x = y or z").
Covering all branches by tests is either labour-consuming or impossible (see the example above).

The typo-in-locals problem could be solved only by an external parser/analyzer (luacheck?).
So why not delegating "to-be-closed variables assignment" problem to the same external parser/analyzer?
Particularly taking into consideration that non-constant to-be-closed variables might also be useful in Lua.
 
Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Sergey Kovalev
ср, 3 июл. 2019 г. в 23:16, Egor Skriptunoff <[hidden email]>:
>
> On Wed, Jul 3, 2019 at 10:56 AM Dirk Laurie wrote:
>>
>> > We have much bigger problem in Lua: every typo in a local variable's name silently converts it to global.

> The typo-in-locals problem could be solved only by an external parser/analyzer (luacheck?).

If we would have pure function declaration this problem causes less pain.
Pure function like ordinary function but isolated from global and
up-values. And all variable must be decelerated explicit or
compilation fails.
G="global"
local U="up-value"
pure_function fn(print)
  local L
  L="local"
  print(G) -- compile error undeclared G
  print(U) -- compile error undeclared U
  L2=L -- compile error undeclared L2
end
fn(print)

Reply | Threaded
Open this post in threaded view
|

Re: [NoW] Once more about to-be-closed variables

Egor Skriptunoff-2
On Thu, Jul 4, 2019 at 12:47 AM Sergey Kovalev wrote:

If we would have pure function declaration this problem causes less pain.
Pure function like ordinary function but isolated from global and
up-values.

Pure functions are quite rare.
Upvalues are almost everywhere in Lua :-)

12