|
12345
|
This may have been discussed earlier; if so, kindly point me to the previous discussions.
Lua uses garbage collection for everything. User-defined objects can use metatables to perform appropriate finalization. For example, Lua's built-in io library use the __gc field in its files' metatable to close the underlying OS file, even if the user does not call the file:close function. That works very well, except that finalization is non-deterministic, i.e., it is impossible to predict when it will happen.
Using the example of files, this may be problematic, because the file remains open unpredictably long, which may interfere with the other uses of the file. It could be said that if determinism is important, the user must ensure that file:close is called. Unfortunately, taking into account that there can be some very complicated logic between io.open and file:close, which may also raise Lua errors, this could lead to extremely unwieldy code.
This problem is not specific to Lua and probably exists in every GC-collected environment, so there are some established ways of dealing with it. In C#, for example, this is done via the keyword 'using', which establishes a scope, upon exiting which (including exiting via an exception), the object is finalized. Example from https://msdn.microsoft.com/en-us/library/yh598w02.aspx
using (Font font1 = new Font("Arial", 10.0f))
{
byte charset = font1.GdiCharSet;
}
Syntactically, this could be transferred to Lua using a new keyword, such as block, instead of local, to define a local variable that must be finalized when exiting the block (including error exits). The finalization will use the __gc methamethod, if present, and subsequent garbage collection proceeds in the usual manner. For example:
block file = io.open('file')
The use of a new keyword seems necessary so that the user could control whether this block-scope finalization is to happen, because it will be unsuitable when the object is supposed to outlive its block. This could probably be also controlled by some clever metatable manipulation; but that will make it an 'advanced' feature, limiting its adoption.
Comments?
Cheers, V.
|
|
2015-11-13 10:19 GMT+00:00 Viacheslav Usov < [hidden email]>:
> This problem is not specific to Lua and probably exists in every
> GC-collected environment, so there are some established ways of dealing with
> it. In C#, for example, this is done via the keyword 'using', which
> establishes a scope, upon exiting which (including exiting via an
> exception), the object is finalized. Example from
> https://msdn.microsoft.com/en-us/library/yh598w02.aspx>
> using (Font font1 = new Font("Arial", 10.0f))
> {
> byte charset = font1.GdiCharSet;
> }
>
>
> Syntactically, this could be transferred to Lua using a new keyword, such as
> block, instead of local, to define a local variable that must be finalized
> when exiting the block (including error exits). The finalization will use
> the __gc methamethod, if present, and subsequent garbage collection proceeds
> in the usual manner. For example:
>
> block file = io.open('file')
>
> Comments?
What about yields? You cannot close the file on yield, because the
yield might be caused by nested sub-calls out of your control. And you
cannot "not close" the file, because the thread might never be
resumed.
This shows there is no universal solution to the problem. The code
inside the block must cooperate with the code outside.
|
|
Hi Viacheslav,
> This may have been discussed earlier; if so, kindly point me to the previous discussions.
There have been several related discussions/write-ups, for example, on
using finalize/guard
( http://lua-users.org/lists/lua-l/2008-02/msg00243.html) and
deterministic cleanup
( http://john.neggie.net/2009/lua/resource_finalization).
Paul.
On Fri, Nov 13, 2015 at 2:19 AM, Viacheslav Usov < [hidden email]> wrote:
> This may have been discussed earlier; if so, kindly point me to the
> previous discussions.
>
> Lua uses garbage collection for everything. User-defined objects can use
> metatables to perform appropriate finalization. For example, Lua's built-in
> io library use the __gc field in its files' metatable to close the
> underlying OS file, even if the user does not call the file:close function.
> That works very well, except that finalization is non-deterministic, i.e.,
> it is impossible to predict when it will happen.
>
> Using the example of files, this may be problematic, because the file
> remains open unpredictably long, which may interfere with the other uses of
> the file. It could be said that if determinism is important, the user must
> ensure that file:close is called. Unfortunately, taking into account that
> there can be some very complicated logic between io.open and file:close,
> which may also raise Lua errors, this could lead to extremely unwieldy code.
>
> This problem is not specific to Lua and probably exists in every
> GC-collected environment, so there are some established ways of dealing with
> it. In C#, for example, this is done via the keyword 'using', which
> establishes a scope, upon exiting which (including exiting via an
> exception), the object is finalized. Example from
> https://msdn.microsoft.com/en-us/library/yh598w02.aspx>
> using (Font font1 = new Font("Arial", 10.0f))
> {
> byte charset = font1.GdiCharSet;
> }
>
>
> Syntactically, this could be transferred to Lua using a new keyword, such as
> block, instead of local, to define a local variable that must be finalized
> when exiting the block (including error exits). The finalization will use
> the __gc methamethod, if present, and subsequent garbage collection proceeds
> in the usual manner. For example:
>
> block file = io.open('file')
>
> The use of a new keyword seems necessary so that the user could control
> whether this block-scope finalization is to happen, because it will be
> unsuitable when the object is supposed to outlive its block. This could
> probably be also controlled by some clever metatable manipulation; but that
> will make it an 'advanced' feature, limiting its adoption.
>
> Comments?
>
> Cheers,
> V.
|
|
On 13/11/15 08:19 AM, Viacheslav Usov wrote:
> This may have been discussed earlier; if so, kindly point me to the
> previous discussions.
>
> Lua uses garbage collection for everything. User-defined objects can
> use metatables to perform appropriate finalization. For example, Lua's
> built-in io library use the __gc field in its files' metatable to
> close the underlying OS file, even if the user does not call the
> file:close function. That works very well, except that finalization is
> non-deterministic, i.e., it is impossible to predict when it will happen.
>
> Using the example of files, this may be problematic, because the file
> remains open unpredictably long, which may interfere with the other
> uses of the file. It could be said that if determinism is important,
> the user must ensure that file:close is called. Unfortunately, taking
> into account that there can be some very complicated logic between
> io.open and file:close, which may also raise Lua errors, this could
> lead to extremely unwieldy code.
>
> This problem is not specific to Lua and probably exists in every
> GC-collected environment, so there are some established ways of
> dealing with it. In C#, for example, this is done via the keyword
> 'using', which establishes a scope, upon exiting which (including
> exiting via an exception), the object is finalized. Example from
> https://msdn.microsoft.com/en-us/library/yh598w02.aspx>
> using (Font font1 =new Font("Arial", 10.0f))
> {
> byte charset = font1.GdiCharSet;
> }
using(Font.new("Arial", 10.0), function(font1)
local charset = font1.GdiCharSet
end)
Where using() runs the function in a coroutine and hooks errors in order
to finalize the font.
local function cleanup(ret)
collectgarbage()
collectgarbage() -- twice to make sure it's collected
return table.unpack(ret, 1, ret.n)
end
function using(...)
local f = select(-1, ...)
local co = coroutine.create(f)
local ret = table.pack(co.resume(...)) -- or something
local errmsg
local yielded = -- process coroutine.yield() or something
-- etc
while co.status() ~= "dead" do
ret = table.pack(co.resume(table.unpack(yielded)))
local status = table.remove(ret, 1) -- remove ret[1], which
contains the status
if status then
-- process coroutine.yield() or something
else
errmsg = table.remove(ret, 1) -- remove ret[1] again, which now
contains the error message
end
end
if co.status() == "dead" and errmsg then
return cleanup(ret) -- pop `...` from the call stack
end
end
>
> Syntactically, this could be transferred to Lua using a new keyword,
> such as block, instead of local, to define a local variable that must
> be finalized when exiting the block (including error exits). The
> finalization will use the __gc methamethod, if present, and subsequent
> garbage collection proceeds in the usual manner. For example:
>
> block file = io.open('file')
>
> The use of a new keyword seems necessary so that the user could
> control whether this block-scope finalization is to happen, because it
> will be unsuitable when the object is supposed to outlive its block.
> This could probably be also controlled by some clever metatable
> manipulation; but that will make it an 'advanced' feature, limiting
> its adoption.
>
> Comments?
>
> Cheers,
> V.
--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.
|
|
On Fri, Nov 13, 2015 at 11:16 AM, Soni L. < [hidden email]> wrote:
>
>
> On 13/11/15 08:19 AM, Viacheslav Usov wrote:
>>
>> This may have been discussed earlier; if so, kindly point me to the
>> previous discussions.
>>
>> Lua uses garbage collection for everything. User-defined objects can use
>> metatables to perform appropriate finalization. For example, Lua's built-in
>> io library use the __gc field in its files' metatable to close the
>> underlying OS file, even if the user does not call the file:close function.
>> That works very well, except that finalization is non-deterministic, i.e.,
>> it is impossible to predict when it will happen.
>>
>> Using the example of files, this may be problematic, because the file
>> remains open unpredictably long, which may interfere with the other uses of
>> the file. It could be said that if determinism is important, the user must
>> ensure that file:close is called. Unfortunately, taking into account that
>> there can be some very complicated logic between io.open and file:close,
>> which may also raise Lua errors, this could lead to extremely unwieldy code.
>>
>> This problem is not specific to Lua and probably exists in every
>> GC-collected environment, so there are some established ways of dealing with
>> it. In C#, for example, this is done via the keyword 'using', which
>> establishes a scope, upon exiting which (including exiting via an
>> exception), the object is finalized. Example from
>> https://msdn.microsoft.com/en-us/library/yh598w02.aspx>>
>> using (Font font1 =new Font("Arial", 10.0f))
>> {
>> byte charset = font1.GdiCharSet;
>> }
>
> using(Font.new("Arial", 10.0), function(font1)
> local charset = font1.GdiCharSet
> end)
>
> Where using() runs the function in a coroutine and hooks errors in order to
> finalize the font.
>
> local function cleanup(ret)
> collectgarbage()
> collectgarbage() -- twice to make sure it's collected
> return table.unpack(ret, 1, ret.n)
> end
This double collectgarbage() is both expensive and evil. Don't
micromanage the collector. If your program logic relies on GC
behavior, you're doing it wrong.
> function using(...)
> local f = select(-1, ...)
> local co = coroutine.create(f)
> local ret = table.pack(co.resume(...)) -- or something
> local errmsg
> local yielded = -- process coroutine.yield() or something
> -- etc
> while co.status() ~= "dead" do
> ret = table.pack(co.resume(table.unpack(yielded)))
> local status = table.remove(ret, 1) -- remove ret[1], which contains the
> status
> if status then
> -- process coroutine.yield() or something
> else
> errmsg = table.remove(ret, 1) -- remove ret[1] again, which now
> contains the error message
> end
> end
> if co.status() == "dead" and errmsg then
> return cleanup(ret) -- pop `...` from the call stack
> end
> end
A better solution might have called a required cleanup method on the
first argument of using(...). [Or even getmetatable(o):__gc()...]
Personally, I would like language support for cleanup of objects on
errors / out-of-block jumps. It's becoming an increasingly common
feature in languages: "with" in Python, "using" in C# (OP), "defer" in
Golang (which unfortunately only executes deferred calls until
return), etc. I don't like bundling error handling with
object/resource cleanup. It leads to the common situation of programs
skipping error handling ("because I want to pass the error up to the
calller anyway") and letting the GC eventually clean up the mess.
--
Patrick Donnelly
|
|
On 13/11/15 04:23 PM, Patrick Donnelly wrote:
> On Fri, Nov 13, 2015 at 11:16 AM, Soni L. < [hidden email]> wrote:
>>
>> On 13/11/15 08:19 AM, Viacheslav Usov wrote:
>>> This may have been discussed earlier; if so, kindly point me to the
>>> previous discussions.
>>>
>>> Lua uses garbage collection for everything. User-defined objects can use
>>> metatables to perform appropriate finalization. For example, Lua's built-in
>>> io library use the __gc field in its files' metatable to close the
>>> underlying OS file, even if the user does not call the file:close function.
>>> That works very well, except that finalization is non-deterministic, i.e.,
>>> it is impossible to predict when it will happen.
>>>
>>> Using the example of files, this may be problematic, because the file
>>> remains open unpredictably long, which may interfere with the other uses of
>>> the file. It could be said that if determinism is important, the user must
>>> ensure that file:close is called. Unfortunately, taking into account that
>>> there can be some very complicated logic between io.open and file:close,
>>> which may also raise Lua errors, this could lead to extremely unwieldy code.
>>>
>>> This problem is not specific to Lua and probably exists in every
>>> GC-collected environment, so there are some established ways of dealing with
>>> it. In C#, for example, this is done via the keyword 'using', which
>>> establishes a scope, upon exiting which (including exiting via an
>>> exception), the object is finalized. Example from
>>> https://msdn.microsoft.com/en-us/library/yh598w02.aspx>>>
>>> using (Font font1 =new Font("Arial", 10.0f))
>>> {
>>> byte charset = font1.GdiCharSet;
>>> }
>> using(Font.new("Arial", 10.0), function(font1)
>> local charset = font1.GdiCharSet
>> end)
>>
>> Where using() runs the function in a coroutine and hooks errors in order to
>> finalize the font.
>>
>> local function cleanup(ret)
>> collectgarbage()
>> collectgarbage() -- twice to make sure it's collected
>> return table.unpack(ret, 1, ret.n)
>> end
> This double collectgarbage() is both expensive and evil. Don't
> micromanage the collector. If your program logic relies on GC
> behavior, you're doing it wrong.
In Lua, you sometimes have .close(), :close(), :disconnect() and even
.disconnect().
How do you handle all the different ways to do that stuff? You just
force a __gc.
>
>> function using(...)
>> local f = select(-1, ...)
>> local co = coroutine.create(f)
>> local ret = table.pack(co.resume(...)) -- or something
>> local errmsg
>> local yielded = -- process coroutine.yield() or something
>> -- etc
>> while co.status() ~= "dead" do
>> ret = table.pack(co.resume(table.unpack(yielded)))
>> local status = table.remove(ret, 1) -- remove ret[1], which contains the
>> status
>> if status then
>> -- process coroutine.yield() or something
>> else
>> errmsg = table.remove(ret, 1) -- remove ret[1] again, which now
>> contains the error message
>> end
>> end
>> if co.status() == "dead" and errmsg then
>> return cleanup(ret) -- pop `...` from the call stack
>> end
>> end
> A better solution might have called a required cleanup method on the
> first argument of using(...). [Or even getmetatable(o):__gc()...]
__metatable issue.
>
> Personally, I would like language support for cleanup of objects on
> errors / out-of-block jumps. It's becoming an increasingly common
> feature in languages: "with" in Python, "using" in C# (OP), "defer" in
> Golang (which unfortunately only executes deferred calls until
> return), etc. I don't like bundling error handling with
> object/resource cleanup. It leads to the common situation of programs
> skipping error handling ("because I want to pass the error up to the
> calller anyway") and letting the GC eventually clean up the mess.
>
Unlike them, Lua doesn't actually need it.
--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.
|
|
Am 16.11.2015 um 12:47 schröbte Viacheslav Usov:
> On Fri, Nov 13, 2015 at 5:16 PM, Soni L. < [hidden email]> wrote:
>
>> Where using() runs the function in a coroutine and hooks errors in order
> to finalize the font.
>
> This is very clever, but:
>
> 1. As already commented here, running the entire garbage collector, twice,
> is expensive.
>
> 2. It is fairly common, as in the LuaSQL example, that such scoped objects
> depend on each other. With C#'s using, one can write code such as
>
> using (var a = f(), b = a.foo(), c = b.bar()) { ... }
>
> The important part of that is that if, for example, foo() or bar() throws
> an exception, then everything in the parentheses that was instantiated
> before the exception will be finalized. With the proposed 'block' keyword,
> that would also be a natural outcome.
If Lua had a hook that is called whenever an error is about to be
thrown, you could do something like the following:
do
local destructors
local function error_hook()
local d = destructors
destructors = {}
for i = #d, 1, -1 do
d[ i ]()
end
end
function on_error( f )
debug.sethook( error_hook, "e" )
local n = #destructors+1
destructors[ n ] = function( ... )
destructors[ n ] = nil
return f( ... )
end
return destructors[ n ]
end
end
do
local a, b, c
local cleanup = on_error( function()
if c then c:destroy() end
if b then b:clear() end
if a then a:close() end
end )
a = f()
b = a:foo()
c = b:bar()
-- do something with a, b, c
-- ...
cleanup() -- run cleanup function if no error has been thrown
end
Maybe that would be good enough (and I think it is relatively easy to
implement). A yield hook would probably be useful as well ...
> [...]
>
> Cheers,
> V.
>
Philipp
|
|
On Fri, Nov 13, 2015 at 4:19 AM, Viacheslav Usov < [hidden email]> wrote:
> Comments?
First, in a dynamic language, it is hard for me to imagine that
something like this wouldn't effect the hot paths within Lua.
Second, I wonder if this issue is limited exclusively to userdata,
given that this is where big huge opaque objects that obfuscate their
memory impact from the garbage collector. If so, then maybe that would
change the approach to solving the problem.
Conversely, if it is not, it seems like there would be no way to do
this cleanly, without replacing the garbage collector, or simply
running a bunch of times, which in some use cases would partially
eliminate the purpose.
Finally, a comment: This appears to be something that would/should
effect many people, but that not many people think it effects them
(me). That is, I'm sure it does effect me and this thread has led me
to study RAII, so thanks!
Andrew Starks
|
|
Am 16.11.2015 um 16:46 schröbte Viacheslav Usov:
> On Mon, Nov 16, 2015 at 3:11 PM, Philipp Janda < [hidden email]> wrote:
>
>> If Lua had a hook that is called whenever an error is about to be thrown,
> you could do something like the following:
>
> xpcall/pcall are basically such a hook. The real issue is not having those
> hooks; it is that their use results in a program that is not elegant (and
> usually quite arcane), which no one enjoys writing (the previously cited
> LuaSQL example is a case in point). For example, consider this part in your
> code:
>
> local a, b, c
> local cleanup = on_error( function()
> if c then c:destroy() end
> if b then b:clear() end
> if a then a:close() end
> end )
> a = f()
> b = a:foo()
> c = b:bar()
>
> It violates quite blatantly the usual wisdom in Lua that one initializes
> locals right in the their declaration, not six feet lines under. And it
> forces the user to do what the compiler/runtime could do much better: track
> what has been finalized and what has not.
What about:
local t = transaction( function( a, b, c )
if c then c:destroy() end
if b then b:clear() end
if a then a:close() end
end )
local a = t( f() )
local b = t( a:foo() )
local c = t( b:bar() )
-- do something with a, b, c
-- ...
t:commit() -- or t:rollback(), or t:cleanup()
No locals declared before initialization, and you could provide a
default rollback function that invokes `__gc` metamethods on all stored
values (although I find that rather ugly and unsafe).
>
> Cheers,
> V.
>
Philipp
|
|
On Tue, Nov 17, 2015 at 6:01 AM, Viacheslav Usov < [hidden email]> wrote:
> On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda < [hidden email]> wrote:
>>
>> What about:
>>
>> local t = transaction( function( a, b, c )
>> if c then c:destroy() end
>> if b then b:clear() end
>> if a then a:close() end
>> end )
>> local a = t( f() )
>> local b = t( a:foo() )
>> local c = t( b:bar() )
>> -- do something with a, b, c
>> -- ...
>> t:commit() -- or t:rollback(), or t:cleanup()
>>
>> No locals declared before initialization, and you could provide a default
>> rollback function that invokes `__gc` metamethods on all stored values
>> (although I find that rather ugly and unsafe).
>
>
> I do not think I understand how that will work. I understand that the
> transaction thing, when initialized, gets a finalizer, which is a function
> of some arguments, and those arguments will be finalized. But how do you
> specify which argument is which?
>
> Cheers,
> V.
It looks like transaction() would return some sort of curried function.
/s/ Adam
|
|
Am 17.11.2015 um 15:01 schröbte Viacheslav Usov:
> On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda < [hidden email]> wrote:
>
>> What about:
>>
>> local t = transaction( function( a, b, c )
>> if c then c:destroy() end
>> if b then b:clear() end
>> if a then a:close() end
>> end )
>> local a = t( f() )
>> local b = t( a:foo() )
>> local c = t( b:bar() )
>> -- do something with a, b, c
>> -- ...
>> t:commit() -- or t:rollback(), or t:cleanup()
>>
>> No locals declared before initialization, and you could provide a default
>> rollback function that invokes `__gc` metamethods on all stored values
>> (although I find that rather ugly and unsafe).
>>
>
> I do not think I understand how that will work. I understand that the
> transaction thing, when initialized, gets a finalizer, which is a function
> of some arguments, and those arguments will be finalized. But how do you
> specify which argument is which?
Currently I use customized (x)pcall functions like you suggested instead
of (or actually in addition to) finalizers. Each call to one of those
functions creates a stack of transactions, and `transaction()` pushes a
transaction object to the current stack. A transaction is just a simple
array, so `t( f() )` will append the value returned by `f()` to the
array. When an error occurs, or when `t:rollback()` or `t:cleanup()` is
called, the elements are `unpack()`ed and passed to the cleanup function
(so the order of the arguments and the `t()` calles must match).
`t:commit()`/`t:rollback()` pop the transaction from the stack (if it's
the most recent).
You can see the current state of my experiments here[1].
>
> Cheers,
> V.
>
Philipp
[1]: https://gist.github.com/siffiejoe/2c01b690bfdbc34b85a6
|
|
Am 17.11.2015 um 17:50 schröbte Philipp Janda:
> Am 17.11.2015 um 15:01 schröbte Viacheslav Usov:
>> On Mon, Nov 16, 2015 at 6:28 PM, Philipp Janda < [hidden email]> wrote:
>>
>>> What about:
>>>
>>> local t = transaction( function( a, b, c )
>>> if c then c:destroy() end
>>> if b then b:clear() end
>>> if a then a:close() end
>>> end )
>>> local a = t( f() )
>>> local b = t( a:foo() )
>>> local c = t( b:bar() )
>>> -- do something with a, b, c
>>> -- ...
>>> t:commit() -- or t:rollback(), or t:cleanup()
>>>
>
> You can see the current state of my experiments here[1].
>
> [1]: https://gist.github.com/siffiejoe/2c01b690bfdbc34b85a6>
I'm currently trying to protect the rollback function (the one passed to
`transaction()` in the snippet above) from memory errors. I realize that
all bets are off if you allocate new Lua objects (strings, tables, table
fields, userdata, functions, ...), but even a simple function call could
allocate memory to grow the Lua stack, and thus raise an error. I
thought about using a separate coroutine for calling the rollback
function and `luaL_checkstack` to ensure a certain stack size. I can
also preallocate a certain number of `CallInfo` elements by calling a
helper function recursively multiple times in this coroutine and
yielding from the inner-most call.
Now my questions:
* Are there any other implicit memory allocations when running Lua code?
* Is there some way to protect the allocated `CallInfo` elements once
the recursive calls have returned, or do I have to disable the GC
temporarily?
* When is the GC triggered anyway? Only when new memory is allocated
(and by calls to `lua_gc()`)?
* Are there any other types of errors that non-faulty Lua code could throw?
TIA,
Philipp
|
|
> Now my questions:
>
> * Are there any other implicit memory allocations when running Lua code?
Not that I remember.
> * Is there some way to protect the allocated `CallInfo` elements
> once the recursive calls have returned, or do I have to disable the
> GC temporarily?
You have to stop the GC (but see next item).
> * When is the GC triggered anyway? Only when new memory is allocated
> (and by calls to `lua_gc()`)?
Yes.
> * Are there any other types of errors that non-faulty Lua code could throw?
Not that I remember. (It is not by chance that Lua has a different code
for memory errors.)
-- Roberto
|
12345
|