block-scope finalization

[Viacheslav Usov]

This may have  been discussed earlier; if so, kindly point me to the previous discussions.

Lua uses garbage collection for everything. User-defined objects can use metatables to perform appropriate finalization. For example, Lua's built-in io library use the __gc field in its files' metatable to close the underlying OS file, even if the user does not call the file:close function. That works very well, except that finalization is non-deterministic, i.e., it is impossible to predict when it will happen. 

Using the example of files, this may be problematic, because the file remains open unpredictably long, which may interfere with the other uses of the file. It could be said that if determinism is important, the user must ensure that file:close is called. Unfortunately, taking into account that there can be some very complicated logic between io.open and file:close, which may also raise Lua errors, this could lead to extremely unwieldy code.

This problem is not specific to Lua and probably exists in every GC-collected environment, so there are some established ways of dealing with it. In C#, for example, this is done via the keyword 'using', which establishes a scope, upon exiting which (including exiting via an exception), the object is finalized. Example from https://msdn.microsoft.com/en-us/library/yh598w02.aspx

using (Font font1 = new Font("Arial", 10.0f)) 
{
    byte charset = font1.GdiCharSet;
}

Syntactically, this could be transferred to Lua using a new keyword, such as block, instead of local, to define a local variable that must be finalized when exiting the block (including error exits). The finalization will use the __gc methamethod, if present, and subsequent garbage collection proceeds in the usual manner. For example:

block file = io.open('file')

The use of a new keyword seems necessary so that the user could control whether this block-scope finalization is to happen, because it will be unsuitable when the object is supposed to outlive its block. This could probably be also controlled by some clever metatable manipulation; but that will make it an 'advanced' feature, limiting its adoption.

Comments?

Cheers,

V.

[Jerome Vuarand]

2015-11-13 10:19 GMT+00:00 Viacheslav Usov <[hidden email]>:

> This problem is not specific to Lua and probably exists in every
> GC-collected environment, so there are some established ways of dealing with
> it. In C#, for example, this is done via the keyword 'using', which
> establishes a scope, upon exiting which (including exiting via an
> exception), the object is finalized. Example from
> https://msdn.microsoft.com/en-us/library/yh598w02.aspx
>
> using (Font font1 = new Font("Arial", 10.0f))
> {
>     byte charset = font1.GdiCharSet;
> }
>
>
> Syntactically, this could be transferred to Lua using a new keyword, such as
> block, instead of local, to define a local variable that must be finalized
> when exiting the block (including error exits). The finalization will use
> the __gc methamethod, if present, and subsequent garbage collection proceeds
> in the usual manner. For example:
>
> block file = io.open('file')
>
> Comments?

What about yields? You cannot close the file on yield, because the
yield might be caused by nested sub-calls out of your control. And you
cannot "not close" the file, because the thread might never be
resumed.

This shows there is no universal solution to the problem. The code
inside the block must cooperate with the code outside.

[Viacheslav Usov]

On Fri, Nov 13, 2015 at 11:50 AM, Jerome Vuarand <[hidden email]> wrote:

> What about yields?

A yield does not exit the current block, so finalization won't result.

[...]

> And you cannot "not close" the file, because the thread might never be resumed.

In that case, the usual garbage collection will occur.

> This shows there is no universal solution to the problem.

By the same token, one could say that a program can just block infinitely, in which case garbage collection won't happen at all. Does that negate the usefulness of garbage collection? In my opinion, blocks-scope finalization addresses a large class of design patterns, giving the programmer simple declarative language constructs instead of hopefully bug-free code and comments that hopefully describe the intent [1]. It does not free the programmer from the need to have a sound design.

Cheers,

V.

[1] Consider the example here: http://keplerproject.github.io/luasql/doc/us/examples.html . This is the epilogue:

-- close everything
cur:close() -- already closed because all the result set was consumed
con:close()
env:close()

Can one be sure just by looking at this code that it really does what it should? Or perhaps we have forgotten something? And why do we still try to close something that is commented to have already been closed?

In fact, it can easily be seen that the program is not entirely correct, because it has a number of assert() calls that may raise errors and the epilogue won't be reached, leaving certain things not closed. Try to correct that program so that it always closes everything; then what I meant by "unwieldy" will be immediately evident. And that is just a "basic use" example, not a real-world program.

[Paul K-2]

Hi Viacheslav,

> This may have  been discussed earlier; if so, kindly point me to the previous discussions.

There have been several related discussions/write-ups, for example, on
using finalize/guard
(http://lua-users.org/lists/lua-l/2008-02/msg00243.html) and
deterministic cleanup
(http://john.neggie.net/2009/lua/resource_finalization).

Paul.

On Fri, Nov 13, 2015 at 2:19 AM, Viacheslav Usov <[hidden email]> wrote:

> This may have  been discussed earlier; if so, kindly point me to the
> previous discussions.
>
> Lua uses garbage collection for everything. User-defined objects can use
> metatables to perform appropriate finalization. For example, Lua's built-in
> io library use the __gc field in its files' metatable to close the
> underlying OS file, even if the user does not call the file:close function.
> That works very well, except that finalization is non-deterministic, i.e.,
> it is impossible to predict when it will happen.
>
> Using the example of files, this may be problematic, because the file
> remains open unpredictably long, which may interfere with the other uses of
> the file. It could be said that if determinism is important, the user must
> ensure that file:close is called. Unfortunately, taking into account that
> there can be some very complicated logic between io.open and file:close,
> which may also raise Lua errors, this could lead to extremely unwieldy code.
>
> This problem is not specific to Lua and probably exists in every
> GC-collected environment, so there are some established ways of dealing with
> it. In C#, for example, this is done via the keyword 'using', which
> establishes a scope, upon exiting which (including exiting via an
> exception), the object is finalized. Example from
> https://msdn.microsoft.com/en-us/library/yh598w02.aspx
>
> using (Font font1 = new Font("Arial", 10.0f))
> {
>     byte charset = font1.GdiCharSet;
> }
>
>
> Syntactically, this could be transferred to Lua using a new keyword, such as
> block, instead of local, to define a local variable that must be finalized
> when exiting the block (including error exits). The finalization will use
> the __gc methamethod, if present, and subsequent garbage collection proceeds
> in the usual manner. For example:
>
> block file = io.open('file')
>
> The use of a new keyword seems necessary so that the user could control
> whether this block-scope finalization is to happen, because it will be
> unsuitable when the object is supposed to outlive its block. This could
> probably be also controlled by some clever metatable manipulation; but that
> will make it an 'advanced' feature, limiting its adoption.
>
> Comments?
>
> Cheers,
> V.

[Soni "They/Them" L.]

On 13/11/15 08:19 AM, Viacheslav Usov wrote:

> This may have  been discussed earlier; if so, kindly point me to the
> previous discussions.
>
> Lua uses garbage collection for everything. User-defined objects can
> use metatables to perform appropriate finalization. For example, Lua's
> built-in io library use the __gc field in its files' metatable to
> close the underlying OS file, even if the user does not call the
> file:close function. That works very well, except that finalization is
> non-deterministic, i.e., it is impossible to predict when it will happen.
>
> Using the example of files, this may be problematic, because the file
> remains open unpredictably long, which may interfere with the other
> uses of the file. It could be said that if determinism is important,
> the user must ensure that file:close is called. Unfortunately, taking
> into account that there can be some very complicated logic between
> io.open and file:close, which may also raise Lua errors, this could
> lead to extremely unwieldy code.
>
> This problem is not specific to Lua and probably exists in every
> GC-collected environment, so there are some established ways of
> dealing with it. In C#, for example, this is done via the keyword
> 'using', which establishes a scope, upon exiting which (including
> exiting via an exception), the object is finalized. Example from
> https://msdn.microsoft.com/en-us/library/yh598w02.aspx
>
> using  (Font font1 =new  Font("Arial", 10.0f))
> {
>      byte  charset = font1.GdiCharSet;
> }

using(Font.new("Arial", 10.0), function(font1)
   local charset = font1.GdiCharSet
end)

Where using() runs the function in a coroutine and hooks errors in order
to finalize the font.

local function cleanup(ret)
   collectgarbage()
   collectgarbage() -- twice to make sure it's collected
   return table.unpack(ret, 1, ret.n)
end

function using(...)
   local f = select(-1, ...)
   local co = coroutine.create(f)
   local ret = table.pack(co.resume(...)) -- or something
   local errmsg
   local yielded = -- process coroutine.yield() or something
   -- etc
   while co.status() ~= "dead" do
     ret = table.pack(co.resume(table.unpack(yielded)))
     local status = table.remove(ret, 1) -- remove ret[1], which
contains the status
     if status then
       -- process coroutine.yield() or something
     else
       errmsg = table.remove(ret, 1) -- remove ret[1] again, which now
contains the error message
     end
   end
   if co.status() == "dead" and errmsg then
     return cleanup(ret) -- pop `...` from the call stack
   end
end

>
> Syntactically, this could be transferred to Lua using a new keyword,
> such as block, instead of local, to define a local variable that must
> be finalized when exiting the block (including error exits). The
> finalization will use the __gc methamethod, if present, and subsequent
> garbage collection proceeds in the usual manner. For example:
>
> block file = io.open('file')
>
> The use of a new keyword seems necessary so that the user could
> control whether this block-scope finalization is to happen, because it
> will be unsuitable when the object is supposed to outlive its block.
> This could probably be also controlled by some clever metatable
> manipulation; but that will make it an 'advanced' feature, limiting
> its adoption.
>
> Comments?
>
> Cheers,
> V.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.

[Patrick Donnelly]

On Fri, Nov 13, 2015 at 11:16 AM, Soni L. <[hidden email]> wrote:

>
>
> On 13/11/15 08:19 AM, Viacheslav Usov wrote:
>>
>> This may have  been discussed earlier; if so, kindly point me to the
>> previous discussions.
>>
>> Lua uses garbage collection for everything. User-defined objects can use
>> metatables to perform appropriate finalization. For example, Lua's built-in
>> io library use the __gc field in its files' metatable to close the
>> underlying OS file, even if the user does not call the file:close function.
>> That works very well, except that finalization is non-deterministic, i.e.,
>> it is impossible to predict when it will happen.
>>
>> Using the example of files, this may be problematic, because the file
>> remains open unpredictably long, which may interfere with the other uses of
>> the file. It could be said that if determinism is important, the user must
>> ensure that file:close is called. Unfortunately, taking into account that
>> there can be some very complicated logic between io.open and file:close,
>> which may also raise Lua errors, this could lead to extremely unwieldy code.
>>
>> This problem is not specific to Lua and probably exists in every
>> GC-collected environment, so there are some established ways of dealing with
>> it. In C#, for example, this is done via the keyword 'using', which
>> establishes a scope, upon exiting which (including exiting via an
>> exception), the object is finalized. Example from
>> https://msdn.microsoft.com/en-us/library/yh598w02.aspx
>>
>> using  (Font font1 =new  Font("Arial", 10.0f))
>> {
>>      byte  charset = font1.GdiCharSet;
>> }
>
> using(Font.new("Arial", 10.0), function(font1)
>   local charset = font1.GdiCharSet
> end)
>
> Where using() runs the function in a coroutine and hooks errors in order to
> finalize the font.
>
> local function cleanup(ret)
>   collectgarbage()
>   collectgarbage() -- twice to make sure it's collected
>   return table.unpack(ret, 1, ret.n)
> end

This double collectgarbage() is both expensive and evil. Don't
micromanage the collector. If your program logic relies on GC
behavior, you're doing it wrong.

> function using(...)
>   local f = select(-1, ...)
>   local co = coroutine.create(f)
>   local ret = table.pack(co.resume(...)) -- or something
>   local errmsg
>   local yielded = -- process coroutine.yield() or something
>   -- etc
>   while co.status() ~= "dead" do
>     ret = table.pack(co.resume(table.unpack(yielded)))
>     local status = table.remove(ret, 1) -- remove ret[1], which contains the
> status
>     if status then
>       -- process coroutine.yield() or something
>     else
>       errmsg = table.remove(ret, 1) -- remove ret[1] again, which now
> contains the error message
>     end
>   end
>   if co.status() == "dead" and errmsg then
>     return cleanup(ret) -- pop `...` from the call stack
>   end
> end

A better solution might have called a required cleanup method on the
first argument of using(...). [Or even getmetatable(o):__gc()...]

Personally, I would like language support for cleanup of objects on
errors / out-of-block jumps. It's becoming an increasingly common
feature in languages: "with" in Python, "using" in C# (OP), "defer" in
Golang (which unfortunately only executes deferred calls until
return), etc. I don't like bundling error handling with
object/resource cleanup. It leads to the common situation of programs
skipping error handling ("because I want to pass the error up to the
calller anyway") and letting the GC eventually clean up the mess.

--
Patrick Donnelly

[Soni "They/Them" L.]

On 13/11/15 04:23 PM, Patrick Donnelly wrote:

> On Fri, Nov 13, 2015 at 11:16 AM, Soni L. <[hidden email]> wrote:
>>
>> On 13/11/15 08:19 AM, Viacheslav Usov wrote:
>>> This may have  been discussed earlier; if so, kindly point me to the
>>> previous discussions.
>>>
>>> Lua uses garbage collection for everything. User-defined objects can use
>>> metatables to perform appropriate finalization. For example, Lua's built-in
>>> io library use the __gc field in its files' metatable to close the
>>> underlying OS file, even if the user does not call the file:close function.
>>> That works very well, except that finalization is non-deterministic, i.e.,
>>> it is impossible to predict when it will happen.
>>>
>>> Using the example of files, this may be problematic, because the file
>>> remains open unpredictably long, which may interfere with the other uses of
>>> the file. It could be said that if determinism is important, the user must
>>> ensure that file:close is called. Unfortunately, taking into account that
>>> there can be some very complicated logic between io.open and file:close,
>>> which may also raise Lua errors, this could lead to extremely unwieldy code.
>>>
>>> This problem is not specific to Lua and probably exists in every
>>> GC-collected environment, so there are some established ways of dealing with
>>> it. In C#, for example, this is done via the keyword 'using', which
>>> establishes a scope, upon exiting which (including exiting via an
>>> exception), the object is finalized. Example from
>>> https://msdn.microsoft.com/en-us/library/yh598w02.aspx
>>>
>>> using  (Font font1 =new  Font("Arial", 10.0f))
>>> {
>>>       byte  charset = font1.GdiCharSet;
>>> }
>> using(Font.new("Arial", 10.0), function(font1)
>>    local charset = font1.GdiCharSet
>> end)
>>
>> Where using() runs the function in a coroutine and hooks errors in order to
>> finalize the font.
>>
>> local function cleanup(ret)
>>    collectgarbage()
>>    collectgarbage() -- twice to make sure it's collected
>>    return table.unpack(ret, 1, ret.n)
>> end
> This double collectgarbage() is both expensive and evil. Don't
> micromanage the collector. If your program logic relies on GC
> behavior, you're doing it wrong.

In Lua, you sometimes have .close(), :close(), :disconnect() and even
.disconnect().

How do you handle all the different ways to do that stuff? You just
force a __gc.

>
>> function using(...)
>>    local f = select(-1, ...)
>>    local co = coroutine.create(f)
>>    local ret = table.pack(co.resume(...)) -- or something
>>    local errmsg
>>    local yielded = -- process coroutine.yield() or something
>>    -- etc
>>    while co.status() ~= "dead" do
>>      ret = table.pack(co.resume(table.unpack(yielded)))
>>      local status = table.remove(ret, 1) -- remove ret[1], which contains the
>> status
>>      if status then
>>        -- process coroutine.yield() or something
>>      else
>>        errmsg = table.remove(ret, 1) -- remove ret[1] again, which now
>> contains the error message
>>      end
>>    end
>>    if co.status() == "dead" and errmsg then
>>      return cleanup(ret) -- pop `...` from the call stack
>>    end
>> end
> A better solution might have called a required cleanup method on the
> first argument of using(...). [Or even getmetatable(o):__gc()...]

__metatable issue.

>
> Personally, I would like language support for cleanup of objects on
> errors / out-of-block jumps. It's becoming an increasingly common
> feature in languages: "with" in Python, "using" in C# (OP), "defer" in
> Golang (which unfortunately only executes deferred calls until
> return), etc. I don't like bundling error handling with
> object/resource cleanup. It leads to the common situation of programs
> skipping error handling ("because I want to pass the error up to the
> calller anyway") and letting the GC eventually clean up the mess.
>

Unlike them, Lua doesn't actually need it.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.

[Tom N Harris]

On Friday, November 13, 2015 11:19:35 AM Viacheslav Usov wrote:
> This may have  been discussed earlier; if so, kindly point me to the
> previous discussions.

Resource Acquisition Is Initialization
http://lua-users.org/wiki/ResourceAcquisitionIsInitialization

Pure Lua RAII. This was my conclusion after experimenting with various patch-
based solutions below.
http://www.whoopdedo.org/doku/lua/finalizers

[Patch] Finalization of function objects
http://lua-users.org/lists/lua-l/2008-01/msg00138.html
and
http://lua-users.org/lists/lua-l/2009-07/msg00486.html
http://lua-users.org/lists/lua-l/2009-08/msg00000.html

--
tom <[hidden email]>

[Viacheslav Usov]

On Fri, Nov 13, 2015 at 5:16 PM, Soni L. <[hidden email]> wrote:

> Where using() runs the function in a coroutine and hooks errors in order to finalize the font.

This is very clever, but:

1. As already commented here, running the entire garbage collector, twice, is expensive.

2. It is fairly common, as in the LuaSQL example, that such scoped objects depend on each other. With C#'s using, one can write code such as

using (var a = f(), b = a.foo(), c = b.bar()) { ... }

The important part of that is that if, for example, foo() or bar() throws an exception, then everything in the parentheses that was instantiated before the exception will be finalized. With the proposed 'block' keyword, that would also be a natural outcome.

With your using() function, however, that is impossible to achieve. Even the basic part, a = f(), b = a:for(), c = b:bar() is already not expressible. But even if it were, somehow, then any error raised in foo() or bar() would not result in deterministic finalization. So with this approach, I essentially need to have a using() call per each object needing deterministic finalization, with a corresponding functional decomposition of my logic. Having to decompose my logic (and wrap parts of it in pcall calls) was the problem that I wanted to solve, so your approach does not really solve the problem. I find it much more natural and comprehensible to write

do

block a = f()

block b = a:foo()

block c = b:bar()

-- do something with a, b,c

end

rather than

using(

    f(),

    function(a)

        using(

            a:foo(),

            function(b)

                 using(b:bar(),

                     function(c)

                         -- do something

                     end

                  )

            end

        )

    end

)

(I hope the syntax above is well-formed; having to say this just stresses again why a better approach is needed)

Cheers,

V.

[Philipp Janda]

Am 16.11.2015 um 12:47 schröbte Viacheslav Usov:

> On Fri, Nov 13, 2015 at 5:16 PM, Soni L. <[hidden email]> wrote:
>
>> Where using() runs the function in a coroutine and hooks errors in order
> to finalize the font.
>
> This is very clever, but:
>
> 1. As already commented here, running the entire garbage collector, twice,
> is expensive.
>
> 2. It is fairly common, as in the LuaSQL example, that such scoped objects
> depend on each other. With C#'s using, one can write code such as
>
> using (var a = f(), b = a.foo(), c = b.bar()) { ... }
>
> The important part of that is that if, for example, foo() or bar() throws
> an exception, then everything in the parentheses that was instantiated
> before the exception will be finalized. With the proposed 'block' keyword,
> that would also be a natural outcome.

If Lua had a hook that is called whenever an error is about to be
thrown, you could do something like the following:

     do
       local destructors

       local function error_hook()
         local d = destructors
         destructors = {}
         for i = #d, 1, -1 do
           d[ i ]()
         end
       end

       function on_error( f )
         debug.sethook( error_hook, "e" )
         local n = #destructors+1
         destructors[ n ] = function( ... )
           destructors[ n ] = nil
           return f( ... )
         end
         return destructors[ n ]
       end
     end


     do
       local a, b, c
       local cleanup = on_error( function()
         if c then c:destroy() end
         if b then b:clear() end
         if a then a:close() end
       end )
       a = f()
       b = a:foo()
       c = b:bar()
       -- do something with a, b, c
       -- ...
       cleanup() -- run cleanup function if no error has been thrown
     end


Maybe that would be good enough (and I think it is relatively easy to
implement). A yield hook would probably be useful as well ...

> [...]
>
> Cheers,
> V.
>

Philipp

[Andrew Starks]

On Fri, Nov 13, 2015 at 4:19 AM, Viacheslav Usov <[hidden email]> wrote:
> Comments?

First, in a dynamic language, it is hard for me to imagine that
something like this wouldn't effect the hot paths within Lua.

Second, I wonder if this issue is limited exclusively to userdata,
given that this is where big huge opaque objects that obfuscate their
memory impact from the garbage collector. If so, then maybe that would
change the approach to solving the problem.

Conversely, if it is not, it seems like there would be no way to do
this cleanly, without replacing the garbage collector, or simply
running a bunch of times, which in some use cases would partially
eliminate the purpose.

Finally, a comment: This appears to be something that would/should
effect many people, but that not many people think it effects them
(me). That is, I'm sure it does effect me and this thread has led me
to study RAII, so thanks!

Andrew Starks

[Viacheslav Usov]

On Mon, Nov 16, 2015 at 3:11 PM, Philipp Janda <[hidden email]> wrote:

> If Lua had a hook that is called whenever an error is about to be thrown, you could do something like the following:

xpcall/pcall are basically such a hook. The real issue is not having those hooks; it is that their use results in a program that is not elegant (and usually quite arcane), which no one enjoys writing (the previously cited LuaSQL example is a case in point). For example, consider this part in your code:

      local a, b, c
      local cleanup = on_error( function()
        if c then c:destroy() end
        if b then b:clear() end
        if a then a:close() end
      end )
      a = f()
      b = a:foo()
      c = b:bar()

It violates quite blatantly the usual wisdom in Lua that one initializes locals right in the their declaration, not six feet lines under. And it forces the user to do what the compiler/runtime could do much better: track what has been finalized and what has not.

Cheers,

V.

[Viacheslav Usov]

On Mon, Nov 16, 2015 at 4:13 PM, Andrew Starks <[hidden email]> wrote:

> First, in a dynamic language, it is hard for me to imagine that something like this wouldn't effect the hot paths within Lua.

I understand that nothing is for free, but before we invoke the performance argument, it would be nice to measure the true effect on performance.

Besides, deterministic finalization could improve performance, because having a block-scope variable very likely means that it won't be used again, which the GC could use as a hint for some advanced optimization.

> Second, I wonder if this issue is limited exclusively to userdata, given that this is where big huge opaque objects that obfuscate their memory impact from the garbage collector. If so, then maybe that would change the approach to solving the problem.

I do not think I wrote anything that is specific to userdata. If there is something else, say a table, with a non-trivial finalizer to it, then the proposed approach would also take care of that.

> Conversely, if it is not, it seems like there would be no way to do this cleanly, without replacing the garbage collector, or simply running a bunch of times, which in some use cases would partially eliminate the purpose.

I do not think I understand why what I proposed cannot be done. This is done in a few garbage collected languages, so we are not inventing something new here.

> Finally, a comment: This appears to be something that would/should effect many people, but that not many people think it effects them (me). That is, I'm sure it does effect me and this thread has led me to study RAII, so thanks!

Your comment about being enlightened about RAII might explain why not many people [1] complain about that. Not knowing that an elegant solution to a problem exists may just keep people thinking that all they have is a bunch of inferior options. My background is C++, so RAII is my second nature and not having it hurts. Another reason this is important for me is because I am using Lua as a way to script activities in system, with potentially a large number of concurrently executing scripts, where those scripts could be written by end users who are not necessarily experts in Lua or any programming language for that matter, and I can safely predict that they will be leaking objects, both because of simply not calling their close() methods and because of errors that prevent the close() methods from being called. We can of course educate our users, but it is one thing to say "use block foo = bar()", and quite another to say "decompose your program functionally so that every acquisition of a resource that needs to be finalized is wrapped in a pcall and finalization is invoked regardless of errors".

Cheers,

V.

[1] With all the links to previous discussion/works on this issue, I would say this is a fairly well known issue.

[Philipp Janda]

Am 16.11.2015 um 16:46 schröbte Viacheslav Usov:

> On Mon, Nov 16, 2015 at 3:11 PM, Philipp Janda <[hidden email]> wrote:
>
>> If Lua had a hook that is called whenever an error is about to be thrown,
> you could do something like the following:
>
> xpcall/pcall are basically such a hook. The real issue is not having those
> hooks; it is that their use results in a program that is not elegant (and
> usually quite arcane), which no one enjoys writing (the previously cited
> LuaSQL example is a case in point). For example, consider this part in your
> code:
>
>        local a, b, c
>        local cleanup = on_error( function()
>          if c then c:destroy() end
>          if b then b:clear() end
>          if a then a:close() end
>        end )
>        a = f()
>        b = a:foo()
>        c = b:bar()
>
> It violates quite blatantly the usual wisdom in Lua that one initializes
> locals right in the their declaration, not six feet lines under. And it
> forces the user to do what the compiler/runtime could do much better: track
> what has been finalized and what has not.

What about:

     local t = transaction( function( a, b, c )
       if c then c:destroy() end
       if b then b:clear() end
       if a then a:close() end
     end )
     local a = t( f() )
     local b = t( a:foo() )
     local c = t( b:bar() )
     -- do something with a, b, c
     -- ...
     t:commit() -- or t:rollback(), or t:cleanup()

No locals declared before initialization, and you could provide a
default rollback function that invokes `__gc` metamethods on all stored
values (although I find that rather ugly and unsafe).

>
> Cheers,
> V.
>

Philipp

[Viacheslav Usov]

On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda <[hidden email]> wrote:

What about:

    local t = transaction( function( a, b, c )
      if c then c:destroy() end
      if b then b:clear() end
      if a then a:close() end
    end )
    local a = t( f() )
    local b = t( a:foo() )
    local c = t( b:bar() )
    -- do something with a, b, c
    -- ...
    t:commit() -- or t:rollback(), or t:cleanup()

No locals declared before initialization, and you could provide a default rollback function that invokes `__gc` metamethods on all stored values (although I find that rather ugly and unsafe).

I do not think I understand how that will work. I understand that the transaction thing, when initialized, gets a finalizer, which is a function of some arguments, and those arguments will be finalized. But how do you specify which argument is which?

Cheers,

V.

[Coda Highland]

On Tue, Nov 17, 2015 at 6:01 AM, Viacheslav Usov <[hidden email]> wrote:

> On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda <[hidden email]> wrote:
>>
>> What about:
>>
>>     local t = transaction( function( a, b, c )
>>       if c then c:destroy() end
>>       if b then b:clear() end
>>       if a then a:close() end
>>     end )
>>     local a = t( f() )
>>     local b = t( a:foo() )
>>     local c = t( b:bar() )
>>     -- do something with a, b, c
>>     -- ...
>>     t:commit() -- or t:rollback(), or t:cleanup()
>>
>> No locals declared before initialization, and you could provide a default
>> rollback function that invokes `__gc` metamethods on all stored values
>> (although I find that rather ugly and unsafe).
>
>
> I do not think I understand how that will work. I understand that the
> transaction thing, when initialized, gets a finalizer, which is a function
> of some arguments, and those arguments will be finalized. But how do you
> specify which argument is which?
>
> Cheers,
> V.

It looks like transaction() would return some sort of curried function.

/s/ Adam

[Philipp Janda]

Am 17.11.2015 um 15:01 schröbte Viacheslav Usov:

> On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda <[hidden email]> wrote:
>
>> What about:
>>
>>      local t = transaction( function( a, b, c )
>>        if c then c:destroy() end
>>        if b then b:clear() end
>>        if a then a:close() end
>>      end )
>>      local a = t( f() )
>>      local b = t( a:foo() )
>>      local c = t( b:bar() )
>>      -- do something with a, b, c
>>      -- ...
>>      t:commit() -- or t:rollback(), or t:cleanup()
>>
>> No locals declared before initialization, and you could provide a default
>> rollback function that invokes `__gc` metamethods on all stored values
>> (although I find that rather ugly and unsafe).
>>
>
> I do not think I understand how that will work. I understand that the
> transaction thing, when initialized, gets a finalizer, which is a function
> of some arguments, and those arguments will be finalized. But how do you
> specify which argument is which?

Currently I use customized (x)pcall functions like you suggested instead
of (or actually in addition to) finalizers. Each call to one of those
functions creates a stack of transactions, and `transaction()` pushes a
transaction object to the current stack. A transaction is just a simple
array, so `t( f() )` will append the value returned by `f()` to the
array. When an error occurs, or when `t:rollback()` or `t:cleanup()` is
called, the elements are `unpack()`ed and passed to the cleanup function
(so the order of the arguments and the `t()` calles must match).
`t:commit()`/`t:rollback()` pop the transaction from the stack (if it's
the most recent).

You can see the current state of my experiments here[1].

>
> Cheers,
> V.
>

Philipp

   [1]:  https://gist.github.com/siffiejoe/2c01b690bfdbc34b85a6

[Philipp Janda]

Am 17.11.2015 um 17:50 schröbte Philipp Janda:

> Am 17.11.2015 um 15:01 schröbte Viacheslav Usov:
>> On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda <[hidden email]> wrote:
>>
>>> What about:
>>>
>>>      local t = transaction( function( a, b, c )
>>>        if c then c:destroy() end
>>>        if b then b:clear() end
>>>        if a then a:close() end
>>>      end )
>>>      local a = t( f() )
>>>      local b = t( a:foo() )
>>>      local c = t( b:bar() )
>>>      -- do something with a, b, c
>>>      -- ...
>>>      t:commit() -- or t:rollback(), or t:cleanup()
>>>
>
> You can see the current state of my experiments here[1].
>
>    [1]:  https://gist.github.com/siffiejoe/2c01b690bfdbc34b85a6
>

I'm currently trying to protect the rollback function (the one passed to
`transaction()` in the snippet above) from memory errors. I realize that
all bets are off if you allocate new Lua objects (strings, tables, table
fields, userdata, functions, ...), but even a simple function call could
allocate memory to grow the Lua stack, and thus raise an error. I
thought about using a separate coroutine for calling the rollback
function and `luaL_checkstack` to ensure a certain stack size. I can
also preallocate a certain number of `CallInfo` elements by calling a
helper function recursively multiple times in this coroutine and
yielding from the inner-most call.

Now my questions:

* Are there any other implicit memory allocations when running Lua code?

* Is there some way to protect the allocated `CallInfo` elements once
the recursive calls have returned, or do I have to disable the GC
temporarily?

* When is the GC triggered anyway? Only when new memory is allocated
(and by calls to `lua_gc()`)?

* Are there any other types of errors that non-faulty Lua code could throw?


TIA,
Philipp

[Roberto Ierusalimschy]

> Now my questions:
>
> * Are there any other implicit memory allocations when running Lua code?

Not that I remember.


> * Is there some way to protect the allocated `CallInfo` elements
> once the recursive calls have returned, or do I have to disable the
> GC temporarily?

You have to stop the GC (but see next item).


> * When is the GC triggered anyway? Only when new memory is allocated
> (and by calls to `lua_gc()`)?

Yes.


> * Are there any other types of errors that non-faulty Lua code could throw?

Not that I remember. (It is not by chance that Lua has a different code
for memory errors.)

-- Roberto

[Viacheslav Usov]

I have thought more about my original proposal, which was to introduce a new keyword that makes a variable subject to deterministic finalization.

The chief difficulty is that an object referenced by such a variable can be assigned to another variable (stored in a table, captured in a closure), and vice versa. And, if that is not prevented, then the other variable may end up holding a reference to a finalized object. Today's resurrection could also result in that, but today resurrection is fully controlled by the finalizer and hence its library's writer; changing that will make a lot of existing libraries invalid.

At this point it is tempting to say "let's make it impossible to assign block-scope vars to other vars, etc"; after a minute's reflection it is clear that such block-scope variables would be useless. So that approach won't work. We should not finalize anything while it is still being referenced.

As far as I can tell, there is only one well-known method for deterministic finalization: reference counting. It is also well known that it does not work when references can be circular. However, a combination of reference counting AND garbage collection will ensure that finalization WILL happen, and will happen deterministically if a certain coding paradigm is followed, such as only storing reference-counted objects in local vars. The deterministic condition can probably be relaxed to cover a larger set of use cases, but local-vars-only would already be pretty good.

I realize that ref counting will introduce certain overhead; but before we even go there, is there any fundamental reason why ref counting + GC cannot work in Lua? Note that such hybrid solutions exist in other languages, such as Python [1].

Cheers,

V.

[1] https://docs.python.org/2/extending/extending.html#reference-counts