Auto-closing with loops

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Auto-closing with loops

Soni "They/Them" L.
(all code here is 100% untested, but see below for english)

---
local with = require "with"

local f = with ^ function(with, do_error)
   for x in with(io.open("/dev/null", "w")) do
     if do_error then error("erroring", 1) end
   end
end
---
local mt = {}
local function tpop(t) return table.remove(t) end
local function with_call(self, ...)
   local function insert_all_nonil(t, v, ...)
     assert(v)
     table.insert(self, v)
     if select('#', ...) > 0 then return insert_all_nonil(t, ...) end
   end
   insert_all_nonil(self, ...)
   return function(n, done)
     if not done then
         return table.unpack(self)
     else
         for v in tpop, self, nil do
           n = n - 1
           if n == 0 then break end
           v:close()
         end
     end
   end, select('#', ...)
end
local with_pow = function(self, f)
   return function(...)
     local w = setmetatable({}, mt)
     local function cleanup()
       for v in tpop, w, nil do
         v:close()
       end
     end
     return xpcall(f, cleanup, w, ...)
   end
end
mt.__call = with_call
mt.__pow = with_pow
return setmetatable({}, mt)
---

(look at the size of this thing!)

The way it works is very simple:

for v1, v2, v3 in with(x1, x2, x3) do ... end

after the with() call, this becomes

for v1, v2, v3 in unpack_or_cleanup, {x1, x2, x3}, nil do ... end

which, if you know how for loops work, basically that nil is the
starting value, so unpack_or_cleanup is called with the table, and the
nil. so it unpacks the table.

that fills in v1, v2, v3. and then the next iteration comes around
and... oh yeah, v1 is the new value, and it's not nil, so we run a
different branch that closes everything, and returns nil, stopping the
iteration.

this is how we close stuff normally.

for errors, we need that "with decorator"[1] to wrap the function in
something that uses xpcall. you need the "with decorator" for every
function you want to use "with" in (or else it may affect unrelated
functions, but I haven't tested that).

oh, and it's nestable. this works: (well, it's untested)

---
local with = require "with"

local f = with ^ function(with, do_error)
   for x in with(io.open("/dev/null", "w")) do
     for y in with(io.open("/dev/null", "w")) do
       error()
     end
   end
end
---

this will call y's close, then x's close. this also works: (also untested)

---
local with = require "with"

local f = with ^ function(with, do_error)
   for x in with({close = function() print(4) end}) do
     for y in with({close = function() print(2) end}) do
       print(1)
     end
     print(3)
   end
   print(5)
end
---

and prints 1, 2, 3, 4, 5 in that order

[1] inspired by https://marc.info/?l=lua-l&m=148745285529221&w=2

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

dyngeccetor8
On 07/19/2018 04:14 AM, Soni "They/Them" L. wrote:

> local with = require "with"
>
> local f = with ^ function(with, do_error)
>   for x in with({close = function() print(4) end}) do
>     for y in with({close = function() print(2) end}) do
>       print(1)
>     end
>     print(3)
>   end
>   print(5)
> end
> ---
>
> and prints 1, 2, 3, 4, 5 in that order

Reading your messages leaves me a feeling that I'm dumb and this is
something outstanding, or that this is some complex way to print
1, 2, 3, 4, 5.

-- Martin

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Soni "They/Them" L.


On 2018-07-19 05:44 PM, dyngeccetor8 wrote:

> On 07/19/2018 04:14 AM, Soni "They/Them" L. wrote:
>> local with = require "with"
>>
>> local f = with ^ function(with, do_error)
>>    for x in with({close = function() print(4) end}) do
>>      for y in with({close = function() print(2) end}) do
>>        print(1)
>>      end
>>      print(3)
>>    end
>>    print(5)
>> end
>> ---
>>
>> and prints 1, 2, 3, 4, 5 in that order
> Reading your messages leaves me a feeling that I'm dumb and this is
> something outstanding, or that this is some complex way to print
> 1, 2, 3, 4, 5.
>
> -- Martin
>

If they were files, instead of tables with a close function, they'd be
actually closed in the correct order.

Oh, I just realized having a "return" or "break" inside the block/for
loop would cause issues... Hmm... Oh well, at least I tried I guess :/

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Javier Guerra Giraldez
In reply to this post by dyngeccetor8
On 19 July 2018 at 21:44, dyngeccetor8 <[hidden email]> wrote:

> On 07/19/2018 04:14 AM, Soni "They/Them" L. wrote:
>> local with = require "with"
>>
>> local f = with ^ function(with, do_error)
>>   for x in with({close = function() print(4) end}) do
>>     for y in with({close = function() print(2) end}) do
>>       print(1)
>>     end
>>     print(3)
>>   end
>>   print(5)
>> end
>> ---
>>
>> and prints 1, 2, 3, 4, 5 in that order
>
> Reading your messages leaves me a feeling that I'm dumb and this is
> something outstanding, or that this is some complex way to print
> 1, 2, 3, 4, 5.

you're not dumb.  it's _very_ rare that code in itself, without any
hint of its purpose, can be an effective way of communication.



--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Soni "They/Them" L.


On 2018-07-20 05:00 AM, Javier Guerra Giraldez wrote:

> On 19 July 2018 at 21:44, dyngeccetor8 <[hidden email]> wrote:
>> On 07/19/2018 04:14 AM, Soni "They/Them" L. wrote:
>>> local with = require "with"
>>>
>>> local f = with ^ function(with, do_error)
>>>    for x in with({close = function() print(4) end}) do
>>>      for y in with({close = function() print(2) end}) do
>>>        print(1)
>>>      end
>>>      print(3)
>>>    end
>>>    print(5)
>>> end
>>> ---
>>>
>>> and prints 1, 2, 3, 4, 5 in that order
>> Reading your messages leaves me a feeling that I'm dumb and this is
>> something outstanding, or that this is some complex way to print
>> 1, 2, 3, 4, 5.
> you're not dumb.  it's _very_ rare that code in itself, without any
> hint of its purpose, can be an effective way of communication.
>
>
>

It's an (attempt at an) "with" block/statement/construct using for
loops. If you'd just read the "for x in with(io.open(filename)) do",
emphasis on the "with"...

(Code is a language like any other, and can be read like any other. I
find this an useful skill to have, as it helps with reverse engineering
and tracking down bugs, and it also helps with reading my and other
ppl's posts.)

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Hisham
In reply to this post by Soni "They/Them" L.
On 19 July 2018 at 21:22, Soni "They/Them" L. <[hidden email]> wrote:
> Oh, I just realized having a "return" or "break" inside the block/for loop
> would cause issues... Hmm... Oh well, at least I tried I guess :/

Yes, that's why people call for an actual feature to support this. It
is not syntactic sugar: implicit immediate release of resources cannot
be implemented in Lua as is.

Note *implicit immediate* together:

You can have implicit release of resources with __gc, but that's not
immediate. (And not being immediate is a problem: I have had
real-world bugs caused by that; files provided by the standard io
library do close() on __gc, but if you rely on that in a loop, your
code works fine for years until you run a loop on a directory with
thousands of files, and then you get a "too many open files" failure.)

You can have immediate release of resources by calling foo:close() by
hand in every possible exit of your function, but that's not implicit.
(And not being implicit is a problem: throw the first stone they who
never forgot to close()/free()/release() a resource is some of their
code paths — that's why languages have garbage collection for memory
management, after all — but memory is not the only kind of resource
managed in a program.)

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Soni "They/Them" L.


On 2018-07-20 07:44 AM, Hisham wrote:

> On 19 July 2018 at 21:22, Soni "They/Them" L. <[hidden email]> wrote:
>> Oh, I just realized having a "return" or "break" inside the block/for loop
>> would cause issues... Hmm... Oh well, at least I tried I guess :/
> Yes, that's why people call for an actual feature to support this. It
> is not syntactic sugar: implicit immediate release of resources cannot
> be implemented in Lua as is.
>
> Note *implicit immediate* together:
>
> You can have implicit release of resources with __gc, but that's not
> immediate. (And not being immediate is a problem: I have had
> real-world bugs caused by that; files provided by the standard io
> library do close() on __gc, but if you rely on that in a loop, your
> code works fine for years until you run a loop on a directory with
> thousands of files, and then you get a "too many open files" failure.)
>
> You can have immediate release of resources by calling foo:close() by
> hand in every possible exit of your function, but that's not implicit.
> (And not being implicit is a problem: throw the first stone they who
> never forgot to close()/free()/release() a resource is some of their
> code paths — that's why languages have garbage collection for memory
> management, after all — but memory is not the only kind of resource
> managed in a program.)
>
> -- Hisham
>
On the other hand, it's a rather complex feature, unlike most of Lua.

And, handling "return" is actually easy now that I think about it (but
still leaves out "break", which, granted, is easy enough to work around)
- just clear out the table and call close on the stuff after returning
from the function.

Anyway, it was just an idea. Sorry it wasn't good.

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Weird Constructor
In reply to this post by Hisham
2018-07-20 12:44 GMT+02:00 Hisham <[hidden email]>:
> You can have immediate release of resources by calling foo:close() by
> hand in every possible exit of your function, but that's not implicit.
> (And not being implicit is a problem: throw the first stone they who
> never forgot to close()/free()/release() a resource is some of their
> code paths — that's why languages have garbage collection for memory
> management, after all — but memory is not the only kind of resource
> managed in a program.)

Memory is also kind of the most unimportant resource to manage.
Important in the sense of "doing something useful", like network
communication, file I/O, calling external APIs for rendering
or generating printed reports.

Thats also why I really like reference counting and/or call stack
based resource management (like C++, Lisp or Scheme supports).

But most of the time I get by with a resource allocating wrapper function
that also catches exceptions, like:

    function with_printer_handle(constructor_args, doItFn)
        local handle = create_printer_obj(constructor_args)

        local s, err = pcall(function() doItfn(handle) end)
        if (s) then
            handle:close()
        else
            handle:close()
            error(err)
        end
    end

I remember that the Parrot VM project a few years back was very convinced
to solve the problem of RAII, timely destruction and GC.
Unfortunately the whole Perl 6 and Parrot VM project(s) fell victim to
the second-system effect.

I would love a Lua with reference counting. I would even be willing to pay
the CPU overhead for that.

Reflecting on the GC concept: Shouldn't we as developers be worried
of introducing such a non-deterministic mechanism into our programs?
We already have to deal with enough non-determinism in our systems:
like kernel controlled preemptive threads and process scheduling
or hard disk and network latencies.
It's obviously a trade off, but I for myself decided that I would rather
like the reference counting trade off. Where I have to take care of
cycles using weak refs, and pay the CPU overhead.
It's not perfect, but I am in control and it solves managing arbitrary resources
in most cases well. And in the other cases, where I generate cycles
(happens quickly if
you use closures) I have a resource leak that is not obvious and hard
to track down.


Greetings

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Kenneth Lorber
In reply to this post by Soni "They/Them" L.
I'm going to stick my neck out and propose that we can attack this from another angle.  (And I will admit from the start that this is not backed up by either code or a deep understanding of the mechanisms I'm proposing to leverage - someone else can make this work or shoot it down.)

We already have a mechanism that catches all exit paths from a function - xpcall().  And it has a way to run code - the message handler.

So what happens if we build on "do..end" to create "via..do..end" (I don't like "with" and this note might be clearer if I don't use "with") that is (mostly?) syntactic sugar that transforms
        via FINALIZER do CHUNK end
into something like
        local chunk = compile CHUNK replacing return... with error( {MAGIC=_ENV,RETURN={...}} )  and error(e,lvl) with error({MAGIC=_ENV,ERROR={e,lvl})
        local finalizer = function(_ENV) FINALIZER end
        local fwrapper = function(e)
                finalizer(e.MAGIC)
                if e.ERROR then error(e.ERROR.e, e.ERROR.lvl) else return e.RETURN end -- unscrambling e.RETURN to .. left as an exercise for the reader
        end
        local status, e = xpcall(CHUNK, fwrapper)
        if status then error(e) else return e end
       
Does this approach offer anything?
Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

pocomane
In reply to this post by Hisham
On Fri, Jul 20, 2018 at 12:44 PM Hisham <[hidden email]> wrote:

>
> On 19 July 2018 at 21:22, Soni "They/Them" L. <[hidden email]> wrote:
> > Oh, I just realized having a "return" or "break" inside the block/for loop
> > would cause issues... Hmm... Oh well, at least I tried I guess :/
>
> Yes, that's why people call for an actual feature to support this. It
> is not syntactic sugar: implicit immediate release of resources cannot
> be implemented in Lua as is.
>
> Note *implicit immediate* together:
>

It is not difficult to implement in lua a couple of functions that let
you to write:

```
local sequence = 'start'

barrier(function() -- Protect code region
  local a = {}

  setfinalizer(a, function(x) -- Call on scope exit
    assert(a == x)
    sequence = sequence .. '; closing a'
  end)

  sequence = sequence .. '; using a'
  error('an error')
end)

assert(sequence == 'start; using a; closing a')
```

It ask you to protect the critical code with 'barrier' and to set the
function to call on exit with 'setfinalizer'. It seems to me enough
implicit and immediate.

Is there some issue I can not see with this semantic?

Here the first, quickest, rawest implementation I can guess to (that
handles nested 'barrier' too):

```
local current_finalizer_list = {}
local function barrier(func)
  local old_finalizer_list = current_finalizer_list
  current_finalizer_list = {}
  pcall(func)
  for i=1,#current_finalizer_list do
    pcall(current_finalizer_list[i])
  end
  current_finalizer_list = old_finalizer_list
end
local function setfinalizer(value, func)
  local result = {}
  current_finalizer_list[1+#current_finalizer_list] = function()
    pcall(func,value)
  end
  return result
end
```

Please note, I am not arguing against any new feature proposed in the last days.

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Hakki Dogusan-3
In reply to this post by Hisham
On 20-07-2018 13:44, Hisham wrote:

> On 19 July 2018 at 21:22, Soni "They/Them" L. <[hidden email]> wrote:
>> Oh, I just realized having a "return" or "break" inside the block/for loop
>> would cause issues... Hmm... Oh well, at least I tried I guess :/
>
> Yes, that's why people call for an actual feature to support this. It
> is not syntactic sugar: implicit immediate release of resources cannot
> be implemented in Lua as is.
>
> Note *implicit immediate* together:
>
> You can have implicit release of resources with __gc, but that's not
> immediate. (And not being immediate is a problem: I have had
> real-world bugs caused by that; files provided by the standard io
> library do close() on __gc, but if you rely on that in a loop, your
> code works fine for years until you run a loop on a directory with
> thousands of files, and then you get a "too many open files" failure.)
>

Could having a "kill" library function solve this "immediate" problem?
ie. kill(o)
- calls __gc if exists
- then deletes o without waiting collecting
- afteruse is user's problem/responsibility

local kill = function(o) -- does not help!
   print("kill:", o.name, o)
   o = nil
   collectgarbage()
end

local mt = {__gc = function(o)
   print("gc:", o.name, o)
   kill(o)
end, }

a0 = {name="a0"}; setmetatable(a0, mt); print(a0.name, a0)
local a1 = {name="a1"}; setmetatable(a1, mt); print(a1.name, a1)
do
   local a2 = {name="a2"}; setmetatable(a2, mt); print(a2.name, a2)
end
kill(a0)
print("no gc", a0.name, a0) -- gc was not called!
local a3 = {name="a3"}; setmetatable(a3, mt); print(a3.name, a3)
a0 = nil; collectgarbage() -- here gc is called!



> You can have immediate release of resources by calling foo:close() by
> hand in every possible exit of your function, but that's not implicit.
> (And not being implicit is a problem: throw the first stone they who
> never forgot to close()/free()/release() a resource is some of their
> code paths — that's why languages have garbage collection for memory
> management, after all — but memory is not the only kind of resource
> managed in a program.)
>
> -- Hisham
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Tom N Harris
In reply to this post by pocomane
On Friday, July 20, 2018 04:55:49 PM pocomane wrote:

> It is not difficult to implement in lua a couple of functions that let
> you to write:
>
> ```
> local sequence = 'start'
>
> barrier(function() -- Protect code region
>   local a = {}
>
>   setfinalizer(a, function(x) -- Call on scope exit
>     assert(a == x)
>     sequence = sequence .. '; closing a'
>   end)
>
>   sequence = sequence .. '; using a'
>   error('an error')
> end)
>
> assert(sequence == 'start; using a; closing a')
> ```

This thread gives me a great sense of déjà vu.

http://lua-users.org/lists/lua-l/2015-11/msg00164.html

--
tom <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Auto-closing with loops

Lorenzo Donati-3
In reply to this post by Soni "They/Them" L.
On 20/07/2018 12:23, Soni "They/Them" L. wrote:
>
>
> On 2018-07-20 05:00 AM, Javier Guerra Giraldez wrote:

[...]

> (Code is a language like any other, and can be read like any other.

Sorry, definitely not. Human languages have evolved to allow effective
communication between people.

Computer languages are not an *effective* mean of communication for
people in the vast majority of cases. Especially in cases where an
algorithm is clever enough or the problem is subtle enough.

If they were an effective mean of communication, there won't be
countless studies and practices in SW engineering on how to express the
*intent* of code in a way *easily* understandable for humans (literate
programming anyone).

There is a well-founded rationale behind many corporate guidelines about
*requiring* developers to put an adequate amount of comments in their
code. Of course *quantity* doesn't always corresponds to *quality* of
comments, but you get the point (*effective* code commenting *is* and
highly valuable skill for a developer and it is more art than science).


>I
> find this an useful skill to have, as it helps with reverse engineering
> and tracking down bugs, and it also helps with reading my and other
> ppl's posts.)
>
>

I agree with you on this point and it's good for you if you have such a
skill, but don't expect people to have the same fluency in
*communicating ideas* through code.

Moreover, the way you express things in code is dependent on your way of
thinking, which is not evident in the code (the same *intent* could be
fulfilled by a different algorithm when written by another developer).

Computer languages (especially imperative ones) are not designed to
express *intent*, whereas in human matters *intent* is sometimes the
focus of the discussion at hand.

Coders proficient with a computer language can read and understand
*what* a piece of code does and most probably *how*. The big problem
analyzing someone else's (non-trivial) code is usually *why* they do
that that way.