To-be-close confusion...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

To-be-close confusion...

Paul Ducklin
I think that the 5.4.0 reference manual ought to:

* Add __close and __gc into the list of metamethod events so that all valid metamethods are clearly documented in one place.

* Explain the difference between __close and __gc, and clarify that closing a variable doesn’t finalize it.

(That was not at all clear to me until someone mentioned it here. When you think of “closing” something, the obvious examples are files and sockets, and when you close them you have finished with them and their resources can be reclaimed at once.)

* Give a simple example of how each might be used. (I know it’s a reference manual not a training book but there are useful examples elsewhere in it and they help a lot.)

The biggest problem I have with “toclose” is that having come across it in the manual I still have no real idea why it’s there or what I can do with it that I can’t already do with a finalizer.

It’s also not that clear how to use it, given that the manual says that to-be-closed variables are constants. My thought is that seems like a great feature for things like files and sockets as a way of recovering cleanly from errors - but I’ve never thought of files and sockets as “constant variables” because they are all about side effects and continuously changing state.

In short: why do I need <toclose>, what do I do with it, and how do I use it properly?

Reply | Threaded
Open this post in threaded view
|

Re: To-be-close confusion...

Sean Conner
It was thus said that the Great Paul Ducklin once stated:
>
> It’s also not that clear how to use it, given that the manual says that
> to-be-closed variables are constants. My thought is that seems like a
> great feature for things like files and sockets as a way of recovering
> cleanly from errors - but I’ve never thought of files and sockets as
> “constant variables” because they are all about side effects and
> continuously changing state.

  In Lua, variables and values are two distinct things.  In Lua, I can do:

        x = 'one'
        x = 2
        x = { 'three' }
        x = io.stdout
        x = print

That's because a variable (in this case, 'x') is a name to which you can
attach a value---values have types,  not variables.  When you do:

        local <toclose> x = ...

  This makes the variable 'x' constant---you cannot change what value it
has, but the value itself may change (a table, a userdata, etc).

> In short: why do I need <toclose>, what do I do with it, and how do I use
> it properly?

  Given the following example code:

        local function readfile(name)
          local <toclose> f = io.open(name)
          if not f then return end
          local res = {}
          res.date = parse_date(f)
          if not res.date then return end
          res.name = parse_name(f)
          if not res.name then return end
          res.data = parse_data(f)
          if not res.data then return end
          return res
        end

  Each place where we return, the file f will be closed (because it's marked
as <toclose> and we're leaving the current scope).  There's nothing
inherenetly *wrong* if you don't have <toclose>, as eventually the file will
be closed when garbage collection kicks in, but that might be awhile, and in
the meantime, the number of open files you can have is one less, which could
cause an open failure down the road.  To solve this pre-Lua 5.4, you have to
explicitely close the file along each error path:

        local function readfile(name)
          local f = io.open(name)
          if not f then return end
          local res = {}
          res.date = parse_date(f)
          if not res.date  then
            f:close()
            return
          end
          res.name = parse_name(f)
          if not res.name then
            f:close()
            return
          end
          res.data = parse_data(f)
          if not res.data  then
            f:close()
            return
          end
          f:close()
          return res
        end

  Or perhaps this:
 
  local function readfile(name)
   local function readdata(f,res)
     res.date = parse_date(f)
     if not res.date then return false end
     res.name = parse_name(f)
     if not res.name then return false end
     res.data = parse_data(f)
     return res.data ~= nil
   end
   
   local f = io.open(name)
   local res = {}
   local good = readdata(f,res)
   f:close()
   if good then return res end
  end

  You generally want to use <toclose> for resources other than memory where
waiting for garbage collection may be too long.  And remember, this is a
contrived example.  Imaging two files being processed, or the function being
longer, or more possible error paths (or all the above).

  This if this as coroutines---you can go a very long time without having to
understand, or even use, coroutines, but yet they're there for when you need
them.

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: To-be-close confusion...

David Heiko Kolf-2
Sean Conner wrote:
>   Given the following example code:
>
> local function readfile(name)
>  local <toclose> f = io.open(name)
>  if not f then return end

>  local res = {}
>  res.date = parse_date(f)
>  if not res.date then return end
>  res.name = parse_name(f)
>  if not res.name then return end
>  res.data = parse_data(f)
>  if not res.data then return end
>  return res
> end

I'd like to extend your example to include the error message from
io.open, in which case the current <toclose> syntax requires an extra
assignment:

        local function readfile(name)
          local openres, errmsg, errcode = io.open(name)
          local <toclose> f = openres
          if not f then return nil, errmsg, errcode end

          -- the rest I left unchanged:
          local res = {}
          res.date = parse_date(f)
          if not res.date then return end
          res.name = parse_name(f)
          if not res.name then return end
          res.data = parse_data(f)
          if not res.data then return end
          return res
        end

I think it is necessary to include full error handling in examples to
highlight the real use cases of the new syntax -- which I'd like to see
extended to "local <toclose> f, errmsg, errcode = io.open(name)".

Best regards,

David

(I hope I don't sound like a broken record yet.)

Reply | Threaded
Open this post in threaded view
|

Re: To-be-close confusion...

Francisco Olarte
In reply to this post by Paul Ducklin
Paul:

On Sat, Jun 15, 2019 at 10:37 PM Paul Ducklin <[hidden email]> wrote:
> I think that the 5.4.0 reference manual ought to:
> * Add __close and __gc into the list of metamethod events so that all valid metamethods are clearly documented in one place.

That I agree with, just yesterday I did a FTS on the 5.3 docs for __gc
and was surprised to hit it just AFTER the metamethods list but not IN
the list. IMO just a mention in the list, even if it is forward link,
would be helpful.



> * Explain the difference between __close and __gc, and clarify that closing a variable doesn’t finalize it.
> (That was not at all clear to me until someone mentioned it here. When you think of “closing” something, the obvious examples are files and sockets, and when you close them you have finished with them and their resources can be reclaimed at once.)

Don't be so sure. In my case, not in Lua but in similar environments,
I have file classes which can be reopened and socket classes which
keep the remote address and byte/packet counts after close, and both
of them are auto-closed on the destructor. And due to this I have
"closer" objects which are similar to <toclose> in purpose ( In C++ I
have "closer x(closeable)" which closes it when going out of scope (
even if closeable is not destroyed ) ). What I mean is you can have
objects which are closed when no longer used but can be partially
useable after close, so you might be interested in open-do whatever
needs it open-close to free SOME resources-do some things which do not
need whole open object - release.

> * Give a simple example of how each might be used. (I know it’s a reference manual not a training book but there are useful examples elsewhere in it and they help a lot.)
> The biggest problem I have with “toclose” is that having come across it in the manual I still have no real idea why it’s there or what I can do with it that I can’t already do with a finalizer.

You can make sure the object is closed as soon as you finish using it.
In the file/socket example, I  would make an idempotent and safe close
method ( safe meaning non error throwing ) and make both __close and
__gc forward to it ( with some extra processing if needed, probably
just aliasing in the file/socket case ).

In a simple script I would just open and operate on files, let the GC
close it when needed.

OTOH if I had a couple thousands coroutines operating on a long lived
server I would try to use <toclose> to avoid hitting "too manyopen
files".

> It’s also not that clear how to use it, given that the manual says that to-be-closed variables are constants. My thought is that seems like a great feature for things like files and sockets as a way of recovering cleanly from errors - but I’ve never thought of files and sockets as “constant variables” because they are all about side effects and continuously changing state.

Bear in mind in lua variables/values are not the same, and a const
variable is more like a C++ reference. In C++ is usual to get files
and similar stuffed in (non const) references ( they are not the same,
as <toclose> can be nil ( documented to be just ignored when exiting
scope ).

> In short: why do I need <toclose>, what do I do with it, and how do I use it properly?

I'm planning to use it for resource management. In your file example,
I can just stash the result of an open() call in a toclose var, making
it return a closeable or nil on error, then test and use it and pass
it around and be sure it will be promptly closed when I exit the
scope, and not some time after that. As an exmpale, in one of my
programs I have an IVR where each call is controlled by a lua state.
When I initialize it, it has to read several files. If I relied on
__gc to close them, they may be open for a long time, as when
initialization finishes it is normal for the IVR to not enter the lua
state until end of call, and this would lead to big OS file handle
usage. Presently I'm careful to pcall() the init code and :close all
of them, having <toclose> will eliminate some tedious to write and
easy to get wrong code.

<toclose> can basically be used to emulate RAII stuff, in languages
which can do it like C++ or the equivalent try/finally which languages
like Java ( in which all objects are dynamic, so no RAII ) use.

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: To-be-close confusion...

Paul Ducklin
In reply to this post by Sean Conner
The answer below is a good explanatory example - but it depends on a (new) user knowing or being able to find out that Lua’s “file” object has a __close metamethod set by default when io.open() returns.

I couldn’t find reference to that in the Reference Manual - in other words there is no easy way for someone to find out that your example of autoclosing files is a valid use case for <toclose>. (IIRC the only way to snoop on the metatable of a userdata object is via the debug module.)

A reasonable user might infer, from your example, that “toclose” can reliably be assumed to work with any file-like object that is traditionally closed with an obj:close() call, but that’s not necessarily true, because the semantics and the API of the  __close metamethod are not the same as most library’s :close methods, and <toclose> doesn’t call :close.

Suggestions: [1] the manual should list __close more prominently as a first-class metamethod along with already well-known examples such as _call and _add; [2] the manual should include a simple example of its use; [3] any library objects that support “toclose” - such as Lua files - should be officially labelled as “toclosable” in their own documentation.

Oh, [4] I suspect “autoclose” is a clearer name that “toclose”. 

IMO it denotes that the object gets a “close” metamethod called automatically more clearly that “toclose”, and it avoids the need for the clumsy hyphenated adjectival term “to-be-closed” (which doesn’t convey that the close is automatic anyway).  Like this:

“A local variable tagged with the attribute <autoclose> will automatically have its __close metamethod called as soon as it goes out of scope.    

This is a convenient and reliable way for Lua objects to clean up after themselves automatically without waiting for garbage collection, which could happen some time after the variable goes out of scope. 

One example of an object that can usefully be used with <autoclose> is a Lua file, as returned by io.open() [q.v.]”



It’s also not that clear how to use it, given that the manual says that
to-be-closed variables are constants. My thought is that seems like a
great feature for things like files and sockets as a way of recovering
cleanly from errors - but I’ve never thought of files and sockets as
“constant variables” because they are all about side effects and
continuously changing state.


 You generally want to use <toclose> for resources other than memory where
waiting for garbage collection may be too long.  And remember, this is a
contrived example.  Imaging two files being processed, or the function being
longer, or more possible error paths (or all the above).

 -spc

Reply | Threaded
Open this post in threaded view
|

Re: To-be-close confusion...

Andrew Starks-2
In reply to this post by Paul Ducklin

On 6/15/19, 3:37 PM, "[hidden email] on behalf of Paul Ducklin" <[hidden email] on behalf of [hidden email]> wrote:

    I think that the 5.4.0 reference manual ought to:
   
    * Add __close and __gc into the list of metamethod events so that all valid metamethods are clearly documented in one place.
   
    * Explain the difference between __close and __gc, and clarify that closing a variable doesn’t finalize it.
   
    (That was not at all clear to me until someone mentioned it here. When you think of “closing” something, the obvious examples are files and sockets, and when you close them you have finished with them and their resources can be reclaimed at once.)
   
    * Give a simple example of how each might be used. (I know it’s a reference manual not a training book but there are useful examples elsewhere in it and they help a lot.)
   
    The biggest problem I have with “toclose” is that having come across it in the manual I still have no real idea why it’s there or what I can do with it that I can’t already do with a finalizer.
   
    It’s also not that clear how to use it, given that the manual says that to-be-closed variables are constants. My thought is that seems like a great feature for things like files and sockets as a way of recovering cleanly from errors - but I’ve never thought of files and sockets as “constant variables” because they are all about side effects and continuously changing state.
   
    In short: why do I need <toclose>, what do I do with it, and how do I use it properly?

------

For myself, every time I wanted to use __gc, I found that I couldn't, because what I wanted to do was dispose of some big resource that was outside of Lua's memory allocation / management. For example, I may have a UD object pointing at a big buffer created in C, and I need to control when that buffer is disposed of, even if I don't care about when the table containing the UD is GC'd.

I've been following the naming discussion and it strikes me that "__tocolose" is probably just as good as anything else. I thought that "__scoped" seemed good at first, but then thought that it has potential to infer that other variables aren't scoped. We could do "__tocloseatendofscoperegardlessoferror" but that's a lot of typing.

 The distinction between GC and closing is good to keep in mind:

1: The value that the variable points to is not necessarily collected. Other variables can still point to it and prevent collection.
2: You can close off external (expensive) resources at the end of the scope in a simple, deterministic fashion.

This appears to solve a huge problem for a relatively common and important use case. Here I thought Lua had nothing more to add. :)

-Andrew


-- Andrew Starks