block-scope finalization

classic Classic list List threaded Threaded
86 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Javier Guerra Giraldez
On Mon, Nov 23, 2015 at 12:07 PM, Viacheslav Usov <[hidden email]> wrote:
> I have thought more about my original proposal, which was to introduce a new
> keyword that makes a variable subject to deterministic finalization.
>
> The chief difficulty is that an object referenced by such a variable can be
> assigned to another variable (stored in a table, captured in a closure), and
> vice versa.


the "easy" solution is that those characteristics should be tied to
the value, not the variable.

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Roberto Ierusalimschy
In reply to this post by Viacheslav Usov
> The chief difficulty is that an object referenced by such a variable can be
> assigned to another variable (stored in a table, captured in a closure),
> and vice versa. And, if that is not prevented, then the other variable may
> end up holding a reference to a finalized object. Today's resurrection
> could also result in that, but today resurrection is fully controlled by
> the finalizer and hence its library's writer; changing that will make a lot
> of existing libraries invalid.

I think it is a good practice to allow objects to be "finalized"
multiple times (that is, to allow their __gc metamethod to be called
multiple times) and also to allow references to finalized objects with
sensible behavior.

First, if the library does not protect metatables, any code can get the
__gc metamethod and call it explicitly. Second, it is a good practice
(well, I think it is) to provide an explicit way to "close" an object
that needs finalization.

The io library, for instance, does that.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Viacheslav Usov
In reply to this post by Javier Guerra Giraldez
On Mon, Nov 23, 2015 at 6:15 PM, Javier Guerra Giraldez <[hidden email]> wrote:

> the "easy" solution is that those characteristics should be tied to the value, not the variable.

The problem is that userdata and tables are not handled as values; they are handled as references. All the variables that have userdata or table "values" do in fact hold references to the underlying objects. I do not think we can seriously discuss changing this aspect of the language.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Viacheslav Usov
In reply to this post by Roberto Ierusalimschy
On Mon, Nov 23, 2015 at 6:25 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> First, if the library does not protect metatables, any code can get the __gc metamethod and call it explicitly.

That requires a deliberate effort so whoever does that will face the consequences knowingly. And, as you say, it is still possible to protect the metatable from that. The problem with my original proposal was that it could result in this unknowingly, and without a way to prevent that.

> Second, it is a good practice (well, I think it is) to provide an explicit way to "close" an object that needs finalization.

I do not disagree.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Coda Highland
In reply to this post by Viacheslav Usov
On Mon, Nov 23, 2015 at 9:07 AM, Viacheslav Usov <[hidden email]> wrote:

> I have thought more about my original proposal, which was to introduce a new
> keyword that makes a variable subject to deterministic finalization.
>
> The chief difficulty is that an object referenced by such a variable can be
> assigned to another variable (stored in a table, captured in a closure), and
> vice versa. And, if that is not prevented, then the other variable may end
> up holding a reference to a finalized object. Today's resurrection could
> also result in that, but today resurrection is fully controlled by the
> finalizer and hence its library's writer; changing that will make a lot of
> existing libraries invalid.
>
> At this point it is tempting to say "let's make it impossible to assign
> block-scope vars to other vars, etc"; after a minute's reflection it is
> clear that such block-scope variables would be useless. So that approach
> won't work. We should not finalize anything while it is still being
> referenced.
>
> As far as I can tell, there is only one well-known method for deterministic
> finalization: reference counting. It is also well known that it does not
> work when references can be circular. However, a combination of reference
> counting AND garbage collection will ensure that finalization WILL happen,
> and will happen deterministically if a certain coding paradigm is followed,
> such as only storing reference-counted objects in local vars. The
> deterministic condition can probably be relaxed to cover a larger set of use
> cases, but local-vars-only would already be pretty good.
>
> I realize that ref counting will introduce certain overhead; but before we
> even go there, is there any fundamental reason why ref counting + GC cannot
> work in Lua? Note that such hybrid solutions exist in other languages, such
> as Python [1].
>
> Cheers,
> V.
>
> [1] https://docs.python.org/2/extending/extending.html#reference-counts
>

I think that we're actually looking at an XY issue here.

Deterministic finalization is a means to a particular end, but what is
that desired end? Ultimately: We want a specific function to be called
when control exits a block, no matter how it exits. In C++, we do this
using block scoping on a local variable. In Python, we have "finally:"
blocks and we have context managers and the "with" keyword; Java has
finally and try-with-parameters for the same.

In Lua, we COULD achieve this with some fairly ugly pcall hackery:

function finally(method, after)
  var result = table.pack(method)
  -- Note: Like in C#/Java/Python, an exception in the finally block
  -- results in the rest of the block being skipped and any previous
  -- exception to be lost.
  after()
  if result[1] then
    return table.unpack(result, 2, result.n)
  else
    error(result[2], 0)
  end
end

finally(function()
  -- do stuff here
end, function()
  -- finalize here
)

You could even create a context handler:

function with(...)
  var args = table.pack(...)
  var method = args[args.n]
  finally(method(table.unpack(args, 1, args.n-1)),
    function()
      for i in 1, args.n-1 do
        args[i]:close()
      end
    end
  )
end

with(io.open("filename.txt", "r"), function(f)
  -- do stuff with f
end)

Like in Python or Java, if you hold on to a reference to the context
handler object, it's still a valid object, but it'll have already had
its close() method called.

I... actually rather like this with() function I've written here. I
was GOING to use this code as an example of where we could try to
improve on things, but in the end I wrote something that actually
looks pretty darn nice. (It might be somewhat more efficient to do
this using the C API due to the nature of varargs stack manipulation,
but it does WORK in Lua.)

That said, I haven't actually TESTED any of the above code, and it
wouldn't even work in Lua 5.1 without some modification (which is what
most of my code targets due to LuaJIT), but I think the concept is
pretty solid, and I do like the varargs tricks making it pretty
flexible.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Philipp Janda
In reply to this post by Roberto Ierusalimschy
First of all, thanks for your answers, Roberto.

Am 23.11.2015 um 18:25 schröbte Roberto Ierusalimschy:

>> The chief difficulty is that an object referenced by such a variable can be
>> assigned to another variable (stored in a table, captured in a closure),
>> and vice versa. And, if that is not prevented, then the other variable may
>> end up holding a reference to a finalized object. Today's resurrection
>> could also result in that, but today resurrection is fully controlled by
>> the finalizer and hence its library's writer; changing that will make a lot
>> of existing libraries invalid.
>
> I think it is a good practice to allow objects to be "finalized"
> multiple times (that is, to allow their __gc metamethod to be called
> multiple times) and also to allow references to finalized objects with
> sensible behavior.
>
> First, if the library does not protect metatables, any code can get the
> __gc metamethod and call it explicitly.

IMHO, protecting the metatable is a good idea anyway so that no-one
removes the `__gc` metamethod and you run out of resources somewhere else.

> Second, it is a good practice (well, I think it is) to provide an
> explicit way to "close" an object that needs finalization.
>
> The io library, for instance, does that.

It's easy for the io library, because it stores pointers. Downside is
that the garbage collector isn't aware of the extra memory, and memory
fragmentation could be higher since there are multiple allocations for
each userdata now. For every non-pointer userdata you'd have to add an
extra field to indicate the state of finalization.
But now every (meta-)method has to check that the userdata is still
valid, and most Lua code that uses those (meta-)methods has to check
again unless it is ok with a simple getter/setter throwing an error.
Also there are cases where explicitly "closing" an object is unsafe,
e.g. if another object holds a pointer to that object. You can ensure
the correct `__gc` order by storing references in the appropriate
uservalue tables, but invalidating dependent objects is harder --
especially if the dependencies might change at runtime. Concrete
examples from the last three libraries I created bindings for are
renderers and textures in libSDL, memory pools and any other APR object
in the Apache Portable Runtime library, and Fl_Input_Choice and its
Fl_Input and Fl_Menu_Button subwidgets in FLTK.

Maybe the hypothetical "block" variable should only call `__gc` if the
metatable is not protected (hoping that the author has taken proper
care). Alternatively, a new (meta-)method could be introduced.

>
> -- Roberto
>

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Javier Guerra Giraldez
In reply to this post by Coda Highland
On Mon, Nov 23, 2015 at 2:38 PM, Coda Highland <[hidden email]> wrote:
> Deterministic finalization is a means to a particular end, but what is
> that desired end? Ultimately: We want a specific function to be called
> when control exits a block, no matter how it exits. In C++, we do this
> using block scoping on a local variable. In Python, we have "finally:"
> blocks and we have context managers and the "with" keyword; Java has
> finally and try-with-parameters for the same.

totally concur with you on this.  far too many people say (not only
about Lua) something on the lines of "professional programmers do
RAII, therefore static languages are the only professional tools".
(spotting other non-sequitur are left as an exercise)

but the real feature is clean up at block exit.  personally, I find
Python-like "with" block nicer than "finally" clauses.


> You could even create a context handler:
>
> function with(...)
>   var args = table.pack(...)
>   var method = args[args.n]
>   finally(method(table.unpack(args, 1, args.n-1)),
>     function()
>       for i in 1, args.n-1 do
>         args[i]:close()
>       end
>     end
>   )
> end
>
> with(io.open("filename.txt", "r"), function(f)
>   -- do stuff with f
> end)

note that this handler assumes that the only cleanup needed is to call
:close() on relevant values.  If the user needs something more
generic, the syntax uglifyies significantly.




--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Eike Decker
In reply to this post by Philipp Janda

2015-11-23 20:58 GMT+01:00 Philipp Janda <[hidden email]>:
First of all, thanks for your answers, Roberto.

Am 23.11.2015 um 18:25 schröbte Roberto Ierusalimschy:
The chief difficulty is that an object referenced by such a variable can be
assigned to another variable (stored in a table, captured in a closure),
and vice versa. And, if that is not prevented, then the other variable may
end up holding a reference to a finalized object. Today's resurrection
could also result in that, but today resurrection is fully controlled by
the finalizer and hence its library's writer; changing that will make a lot
of existing libraries invalid.

I think it is a good practice to allow objects to be "finalized"
multiple times (that is, to allow their __gc metamethod to be called
multiple times) and also to allow references to finalized objects with
sensible behavior.

First, if the library does not protect metatables, any code can get the
__gc metamethod and call it explicitly.

IMHO, protecting the metatable is a good idea anyway so that no-one removes the `__gc` metamethod and you run out of resources somewhere else.

Second, it is a good practice (well, I think it is) to provide an
explicit way to "close" an object that needs finalization.

The io library, for instance, does that.

It's easy for the io library, because it stores pointers. Downside is that the garbage collector isn't aware of the extra memory, and memory fragmentation could be higher since there are multiple allocations for each userdata now. For every non-pointer userdata you'd have to add an extra field to indicate the state of finalization.
But now every (meta-)method has to check that the userdata is still valid, and most Lua code that uses those (meta-)methods has to check again unless it is ok with a simple getter/setter throwing an error. Also there are cases where explicitly "closing" an object is unsafe, e.g. if another object holds a pointer to that object. You can ensure the correct `__gc` order by storing references in the appropriate uservalue tables, but invalidating dependent objects is harder -- especially if the dependencies might change at runtime. Concrete examples from the last three libraries I created bindings for are renderers and textures in libSDL, memory pools and any other APR object in the Apache Portable Runtime library, and Fl_Input_Choice and its Fl_Input and Fl_Menu_Button subwidgets in FLTK.

Maybe the hypothetical "block" variable should only call `__gc` if the metatable is not protected (hoping that the author has taken proper care). Alternatively, a new (meta-)method could be introduced.


-- Roberto


Philipp




I don't think that using an garbage collector finalizer is the right way to go with a mark and sweep collector (even if it is the only way I see right now). 
I've encountered the problem with transactions now in Lua, PHP, Java and C++. It was always desired that a function is called when the execution leaves the scope - in order and deterministic. In Lua and Java it was a pain, while in C++ and PHP it would be fairly straight forward to use objects allocated on the stack that would deal with leaving the scope of interest and calling a function. In PHP it would work because of the reference counting strategy for object collection - showing an advantage over mark and sweep collectors: Object garbage collector finalizers getting called right away when an object is nulled (given that the object on the stack is the only reference).

Thinking about this issue a while (and thinking to start a discussion on the mailing list on this topic) I haven't really had a good idea; But I was thinking that having some bytecode scope guards would be an option. Something like

-----
do 
  local fp = assert(io.open("hello.txt"))
 -- ...
finally
 if fp ~= nil then fp:close()
end
-----

This would not deal with coroutine yielding though. Though I think that could be worked around by adding hooks to the yield / resume functions, like

-----
do 
  local fp = assert(io.open("hello.txt"))
  local hander =  coroutine.addHandler {
    yield = function() fp:close() end;
    resume = function() fp = assert(io.open("hello.txt")) end
  }
  ...
finally
  coroutine.removeHandler(handler)
end
-----

So I think having a finally block option would greatly improve the overall situation, however the OOP finalizer technique would still offer the advantage of being able to cope with resource deinitialization on the class definition level. 

In the end, I think I'd be more happy if Lua had an (optional) reference counting based GC that would execute finalizers in a deterministic way. 

Cheers,
Eike


Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Coda Highland
In reply to this post by Javier Guerra Giraldez
On Mon, Nov 23, 2015 at 12:09 PM, Javier Guerra Giraldez
<[hidden email]> wrote:
> but the real feature is clean up at block exit.  personally, I find
> Python-like "with" block nicer than "finally" clauses.

Depends on what you're doing with it. "with" blocks are nicer for
objects that know how to clean themselves up. "finally" clauses are
nicer if you need to write ad-hoc code in the cleanup block. Obviously
it's always possible to write either in terms of the other (and I
demonstrated this already) but if I had to choose one of the two... I
slightly favor finally's power-user leanings. (That said, I'd probably
use "with" if I were writing something for end-user scripting that
needed reliable cleanup, because for the simple cases it's harder to
screw up.)

>> with(io.open("filename.txt", "r"), function(f)
>>   -- do stuff with f
>> end)
>
> note that this handler assumes that the only cleanup needed is to call
> :close() on relevant values.  If the user needs something more
> generic, the syntax uglifyies significantly.

Which is why I provided finally(). It's not as visually attractive as
a native finally syntax, but it's not too bad. You can declare your
locals in advance of the call to finally() and populate them inside
the body, then those locals are still in scope in the finally block.

If you REALLY had to, you could always write { close = function() ...
end }. That's not HORRENDOUSLY ugly.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Philipp Janda
In reply to this post by Javier Guerra Giraldez
Am 23.11.2015 um 21:09 schröbte Javier Guerra Giraldez:
> On Mon, Nov 23, 2015 at 2:38 PM, Coda Highland <[hidden email]> wrote:
>> Deterministic finalization is a means to a particular end, but what is
>> that desired end? Ultimately: We want a specific function to be called
>> when control exits a block, no matter how it exits. In C++, we do this
>> using block scoping on a local variable. In Python, we have "finally:"
>> blocks and we have context managers and the "with" keyword; Java has
>> finally and try-with-parameters for the same.
>
> totally concur with you on this.

It's good enough for most cases and very easy to implement. (Btw., I
have fixed that for you:)

     local function _finally( after, ok, ... )
       after()
       if ok then
         return ...
       else
         error( (...), 0 )
       end
     end

     function finally( main, after )
       return _finally( after, pcall( main ) )
     end


> far too many people say (not only about Lua) something on the lines
> of "professional programmers do RAII, therefore static languages are
> the only professional tools".

That's nonsense. Many static languages don't have RAII.
Joking aside, so far I have not found a better tool for generic resource
management than RAII. Advantages of RAII over finally/using/with: You
need only one language feature (destructors) instead of two (finalizers
and finally), so the language is less complex/bloated (see C++!). And
more importantly you only write your resource management code once and
not multiple times scattered around in your source code.


Philipp



Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Coda Highland
On Mon, Nov 23, 2015 at 1:25 PM, Philipp Janda <[hidden email]> wrote:

> Am 23.11.2015 um 21:09 schröbte Javier Guerra Giraldez:
>>
>> On Mon, Nov 23, 2015 at 2:38 PM, Coda Highland <[hidden email]>
>> wrote:
>>>
>>> Deterministic finalization is a means to a particular end, but what is
>>> that desired end? Ultimately: We want a specific function to be called
>>> when control exits a block, no matter how it exits. In C++, we do this
>>> using block scoping on a local variable. In Python, we have "finally:"
>>> blocks and we have context managers and the "with" keyword; Java has
>>> finally and try-with-parameters for the same.
>>
>>
>> totally concur with you on this.
>
>
> It's good enough for most cases and very easy to implement. (Btw., I have
> fixed that for you:)
>
>     local function _finally( after, ok, ... )
>       after()
>       if ok then
>         return ...
>       else
>         error( (...), 0 )
>       end
>     end
>
>     function finally( main, after )
>       return _finally( after, pcall( main ) )
>     end
>

Yeah, I realized I accidentally deleted the pcall() out of my
implementation during an edit. ^^() And the worker function does do a
good job of cleaning up the varargs, so thanks for that, that's
probably way faster (especially in LuaJIT).

>> far too many people say (not only about Lua) something on the lines
>> of "professional programmers do RAII, therefore static languages are
>> the only professional tools".
>
>
> That's nonsense. Many static languages don't have RAII.
> Joking aside, so far I have not found a better tool for generic resource
> management than RAII. Advantages of RAII over finally/using/with: You need
> only one language feature (destructors) instead of two (finalizers and
> finally), so the language is less complex/bloated (see C++!). And more
> importantly you only write your resource management code once and not
> multiple times scattered around in your source code.

That's a fallacious claim. You DO need two language features to
implement RAII -- destructors and deterministic cleanup. Destructors
with nondeterministic cleanup is exactly what Lua has right now with
the __gc metamethod.

Saying that C++ is less complex/bloated is also a pretty flimsy claim,
considering that the typical criticism of C++ is that it's
overengineered and complicated, and that the typical praise of Lua is
its minimalism.

And as for where resource management goes, C++ code involves writing
resource management ALL OVER THE PLACE. You have to pay attention to
scoping and you have to either manually free heap-allocated memory or
explicitly use a container that does it for you. That's a far cry from
keeping the code centralized.

And I say these things from the perspective of a C++ lover! C++ is my
favorite language and until recently I wrote the majority of my code
in it.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Philipp Janda
Am 23.11.2015 um 22:46 schröbte Coda Highland:

> On Mon, Nov 23, 2015 at 1:25 PM, Philipp Janda <[hidden email]> wrote:
>>
>> That's nonsense. Many static languages don't have RAII.
>> Joking aside, so far I have not found a better tool for generic resource
>> management than RAII. Advantages of RAII over finally/using/with: You need
>> only one language feature (destructors) instead of two (finalizers and
>> finally), so the language is less complex/bloated (see C++!). And more
>> importantly you only write your resource management code once and not
>> multiple times scattered around in your source code.
>
> That's a fallacious claim. You DO need two language features to
> implement RAII -- destructors and deterministic cleanup. Destructors
> with nondeterministic cleanup is exactly what Lua has right now with
> the __gc metamethod.

If you want to count deterministic cleanup you also have to count the
garbage collector, so it's two features compared to three.

>
> Saying that C++ is less complex/bloated is also a pretty flimsy claim,
> considering that the typical criticism of C++ is that it's
> overengineered and complicated, and that the typical praise of Lua is
> its minimalism.

The C++ part was a joke (obviously, I thought). I'm aware that C++ is
considered complex and bloated (and rightly so). But RAII is not the
problem there (that part was meant seriously).

>
> And as for where resource management goes, C++ code involves writing
> resource management ALL OVER THE PLACE. You have to pay attention to
> scoping and you have to either manually free heap-allocated memory or
> explicitly use a container that does it for you. That's a far cry from
> keeping the code centralized.

Manually free'ing heap-allocated memory is *not* RAII. To get the
benefits you actually have to use it. C++ doesn't force you to use RAII
(and it probably shouldn't -- it's too low-level for that), so you might
be right about resource management in C++ code, but I still think I'm
right about resource management with RAII.

>
> And I say these things from the perspective of a C++ lover! C++ is my
> favorite language and until recently I wrote the majority of my code
> in it.
>
> /s/ Adam
>

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Coda Highland
On Mon, Nov 23, 2015 at 2:16 PM, Philipp Janda <[hidden email]> wrote:

> Am 23.11.2015 um 22:46 schröbte Coda Highland:
>>
>> On Mon, Nov 23, 2015 at 1:25 PM, Philipp Janda <[hidden email]> wrote:
>>>
>>>
>>> That's nonsense. Many static languages don't have RAII.
>>> Joking aside, so far I have not found a better tool for generic resource
>>> management than RAII. Advantages of RAII over finally/using/with: You
>>> need
>>> only one language feature (destructors) instead of two (finalizers and
>>> finally), so the language is less complex/bloated (see C++!). And more
>>> importantly you only write your resource management code once and not
>>> multiple times scattered around in your source code.
>>
>>
>> That's a fallacious claim. You DO need two language features to
>> implement RAII -- destructors and deterministic cleanup. Destructors
>> with nondeterministic cleanup is exactly what Lua has right now with
>> the __gc metamethod.
>
>
> If you want to count deterministic cleanup you also have to count the
> garbage collector, so it's two features compared to three.

No, you don't have to count the garbage collector, because the other
scheme doesn't REQUIRE a garbage collector. You could implement it
using whatever memory management / cleanup scheme you want, including
a deterministic one. (After all, C++ has finally; alternatively,
refcounting is deterministic.) But you can't implement RAII unless you
have deterministic cleanup.

>> Saying that C++ is less complex/bloated is also a pretty flimsy claim,
>> considering that the typical criticism of C++ is that it's
>> overengineered and complicated, and that the typical praise of Lua is
>> its minimalism.
>
>
> The C++ part was a joke (obviously, I thought). I'm aware that C++ is
> considered complex and bloated (and rightly so). But RAII is not the problem
> there (that part was meant seriously).

The way I read it is that RAII is less complex/bloated than finally
clauses because it only requires one language feature instead of two,
and then you called out C++ as if in support of that point; I didn't
perceive humorous intent in highlighting C++ there.

>> And as for where resource management goes, C++ code involves writing
>> resource management ALL OVER THE PLACE. You have to pay attention to
>> scoping and you have to either manually free heap-allocated memory or
>> explicitly use a container that does it for you. That's a far cry from
>> keeping the code centralized.
>
> Manually free'ing heap-allocated memory is *not* RAII. To get the benefits
> you actually have to use it. C++ doesn't force you to use RAII (and it
> probably shouldn't -- it's too low-level for that), so you might be right
> about resource management in C++ code, but I still think I'm right about
> resource management with RAII.

You're right that RAII allows you to avoid littering your code with
cleanup boilerplate -- proper use of RAII is a lot better than try {
open(); } finally { close(); } all over the place. But RAII doesn't
magically free you from cleanup boilerplate; it just moves it from the
point of use to the point of declaration. Every class you create has
to explicitly include its own cleanup code, and there are cases
(particularly when there are interdependencies between states) where
black-boxed RAII causes headaches. C++ eventually realized this
headache and added a finally construct to the language, because
sometimes cleaning up requires more than just calling destructors in
reverse order.

This is why I say that if I had to choose just ONE of the two, I'd
prefer finally over deterministic destructors -- while it's possible
to implement either in terms of the other, the headache of dealing
with the cases that are less-naturally expressed using destructors
outweighs the code-brevity benefit of RAII.

In my ideal language, I have both.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

William Ahern
In reply to this post by Philipp Janda
On Mon, Nov 23, 2015 at 11:16:58PM +0100, Philipp Janda wrote:
> Am 23.11.2015 um 22:46 schröbte Coda Highland:
<snip>

> >And as for where resource management goes, C++ code involves writing
> >resource management ALL OVER THE PLACE. You have to pay attention to
> >scoping and you have to either manually free heap-allocated memory or
> >explicitly use a container that does it for you. That's a far cry from
> >keeping the code centralized.
>
> Manually free'ing heap-allocated memory is *not* RAII. To get the benefits
> you actually have to use it. C++ doesn't force you to use RAII (and it
> probably shouldn't -- it's too low-level for that), so you might be right
> about resource management in C++ code, but I still think I'm right about
> resource management with RAII.

You may need automatic, exception-safe destructors to implement RAII as it
was literally described by Strousoup. But an RAII-like pattern can be
applied to any language. Notice that in RAII you're principally concerned
with three things:

1) Making sure that the lifetime of any resource is bound to the lifetime of
an automatic, scoped variable, usually via a container (smart pointer, etc).

2) Per the namesake, you allocate necessary resources during initialization.
In C++ that means resource allocation (files, memory) should generally occur
in the constructor of a more abstract object.

3) Conversely, resource deallocation should generally occur [indirectly]
from destructors, not haphazardly from various methods (performance
optimizations notwithstanding).

The whole point of RAII is that if you want to remain exception safe, you
want to minimize the number of possible states your object(s) can have,
which in turns minimizes the number of critical regions and thus the number
of possible control flow paths you need to worry about.

If you keep this discipline, you'll find that almost all your resources are
neatly nested as a simple tree with its root being some controlling object
that your application is centered around--for example, a connection context.
What you shouldn't see often are a bunch of arbitrary resources like files
living directly on the stack. Admittedly, with automatic destructors you
don't need to be as strict in this regard, especially when using a container
library that already wraps low-level objects like file handles. But the
orginal purpose of RAII was largely a discpline for how to wrap low-level OS
resources into a C++ container in an exception safe manner. Doing everything
in constructors meant fewer places you needed to catch exceptions and
manually unwind the state of resources not directly bound to the managed C++
stack.

Thus, if one follows an RAII-like discipline in C or Lua, then the places
where you need to _explicitly_ call a destructor like :close should be far
fewer. Most destructors will be called recursively via other destructors.
Which means the _actual_ burden should be substantially less than one might
otherwise think in the absence of lexically scoped destructors. (This isn't
a solution to the problem. I'm just pointing out that the cost of the
problem can be substantially diminished.)

In C++, your constructors can theoretically instantiate subobjects on the
stack and move them to object members before the constructor returns. But
usually you don't: you assign them directly to the member fields. In Lua you
would do the same thing. Instead of

        local fh = io.open"/some/path"
        local self = {}
        self.fh = fh
        return setmetatable(self, mt)

you should do something like

        local self = setmetatable({ }, mt)
        self.fh = io.open"/some/path"
        return self

or maybe even

        local self = setmetatable({ fh = false }, mt) --> preallocate slot
        ...

That way if an exception occurs, the lifetime of fh is bound to self. You
don't have to worry about calling :close on all the subresources
independently, only self:close.

There are all kinds of ways to get fancy with this. And there are various
corner cases to worry about. But the real point is to try to keep resources
nested in a neat hierarchy so that you usually need only worry about
explicit destruction in a relatively small number of places outside
destructors/finalizers. There may still be gaps where a reference to a
resource is lost, but at least they'll be fewer. If you follow an RAII-like
pattern (early, non-lazy allocation of resources), then dealing with
resource exhaustion should be less problematic in general.

If you have a long lived application where resource exhaustion (memory, file
descriptors) is a less theoretical concern, presumably it's using some sort
of event loop. I often step the Lua GC regularly from a timer. That way
there's an upper bound on how long it takes unreferenced resources to be
reclaimed, regardless of load.

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Patrick Donnelly
In reply to this post by Coda Highland
On Mon, Nov 23, 2015 at 3:41 PM, Coda Highland <[hidden email]> wrote:

> On Mon, Nov 23, 2015 at 12:09 PM, Javier Guerra Giraldez
> <[hidden email]> wrote:
>> but the real feature is clean up at block exit.  personally, I find
>> Python-like "with" block nicer than "finally" clauses.
>
> Depends on what you're doing with it. "with" blocks are nicer for
> objects that know how to clean themselves up. "finally" clauses are
> nicer if you need to write ad-hoc code in the cleanup block. Obviously
> it's always possible to write either in terms of the other (and I
> demonstrated this already) but if I had to choose one of the two... I
> slightly favor finally's power-user leanings. (That said, I'd probably
> use "with" if I were writing something for end-user scripting that
> needed reliable cleanup, because for the simple cases it's harder to
> screw up.)

I'd like to point out that Lua doesn't actually need a new compound
block syntax (like Python's "with") because it does real lexical
scoping. Python needs a new control block because the duration of a
local variable lasts for the entire function. So it is necessary for
the language to have a way to express a duration syntactically using
the with statement.

For Lua, we can simply annotate the local variable to indicate its
value should be cleaned up when the local scope ends. Something like
this:

function readall(bar)
   fixed local file = io.open(bar)
   return assert(file:read "a") -- file is closed on error or via return
end

I used a "fixed" keyword because the duration of the local's **value**
is "fixed". This is with consideration that the local may escape the
block as an upvalue to a closure. [If I'm being honest, I couldn't
find a better keyword I liked in the dictionary.] If you change the
local's value to something else, you get what you deserve.

This would execute either the existing __gc metamethod or something
new. It may even be useful to define functions that run when the scope
ends:

function foo()
  for i = 1, 2 do
    fixed local function() print("i = ", i) end
  end --> "i = 1\ni = 2\n"
end

--
Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

steve donovan
In reply to this post by Javier Guerra Giraldez
On Mon, Nov 23, 2015 at 7:15 PM, Javier Guerra Giraldez
<[hidden email]> wrote:
> the "easy" solution is that those characteristics should be tied to
> the value, not the variable.

Exactly. If there is a new builtin function for registering values for
cleanup on scope exit, then that builtin could also register
functions, a la Go's defer.

Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Viacheslav Usov
In reply to this post by Coda Highland
On Mon, Nov 23, 2015 at 8:38 PM, Coda Highland <[hidden email]> wrote:

> I think that we're actually looking at an XY issue here.

> Deterministic finalization is a means to a particular end, but what is that desired end? Ultimately: We want a specific function to be called when control exits a block, no matter how it exits.

At the lowest level of abstraction, one could say that. But I do not like the idea that Lua programs need to be using lowest levels of abstraction and micromanagement that comes with that.

If we forget, for a second, "external" resources. such as files, then Lua's internal resources (memory) are managed automatically. The user does not have to deal with that at all, at least in principle. Why should that be different for external resources? The only reason the user has to be involved now is because the language does not provide any means of deterministic finalization to library writers. If we decorate Lua with additional low level means of resource management, they should primarily be means available to library writers; we should not make the user even MORE involved.

On the other hand, if those new means require library writers to follow some new paradigm, then we cannot expect they will be universally adopted. We will have a mess not unlike what C++ is, with its multiple resource management paradigms. The proposed reference counting mechanism ensures that neither users nor library writers need to do anything new. We just improve the behaviour of all the existing code, and let users write simpler yet more efficient code in future.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Viacheslav Usov
In reply to this post by Patrick Donnelly
On Tue, Nov 24, 2015 at 3:10 AM, Patrick Donnelly <[hidden email]> wrote:
 
> For Lua, we can simply annotate the local variable to indicate its value should be cleaned up when the local scope ends.

This is exactly what I proposed originally. But after a week of thinking my conclusion is that this is too weird, because it breaks a major assumption, See the details in my earlier message and the subsequent exchange with Roberto.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Peter Pimley
In reply to this post by Coda Highland
I'm not backing any horse in this particular race, but I think about "resources" quite a bit, and the absence of RAII in other languages always makes me feel uneasy.

On 23 November 2015 at 23:23, Coda Highland <[hidden email]> wrote:

But RAII doesn't
magically free you from cleanup boilerplate; it just moves it from the
point of use to the point of declaration. Every class you create has
to explicitly include its own cleanup code[...]

It frees the *caller* from boilerplate code.  The destructor only has to be written once and it is written by the library author, who is hopefully in the best position to do so correctly.  Compare this with requiring every user, at *every point of use* to remember to "using" (C#) or worse still to "finally" (Java, etc) call call close()... or was it release(), or deinit(), or something more complex?

The worst part is that things usually appear to work if the caller forgets to protect themselves.  Taking filehandles as the textbook example, it's far easier to forget to "using" or "finally" than it is to abuse a std::ifstream in order to leak the underlying resource.
 
C++ eventually realized this
headache and added a finally construct to the language

Is that a Visual Studio thing?  I'm sure it's not standard.

Peter
Reply | Threaded
Open this post in threaded view
|

Re: block-scope finalization

Viacheslav Usov
In reply to this post by Coda Highland
On Tue, Nov 24, 2015 at 12:23 AM, Coda Highland <[hidden email]> wrote:

> C++ eventually realized this headache and added a finally construct to the language, because sometimes cleaning up requires more than just calling destructors in reverse order.

There is no finally in standard C++, and the consensus of the standard committee is it is not required [1]. Precisely because deterministic destruction makes it redundant.

finally is a Microsoft Visual C/C++ (MVC) extension to the language - not just to C++, but also to C - and it mirrors the "structured exception handling" (SEH) facility that was (still is) a feature of Windows NT, which was itself being developed in MVC, mostly in C, not C++, with SEH. That development happened in late 80's/early 90's, and was at about the same time when the C++ exception mechanism was being standardized, so it is a different mechanism.

Cheers,
V.


12345