(not) handling new programming idioms with grace

classic Classic list List threaded Threaded
95 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> If Roberto wants to go with a scope-qualified local, at least it's a
> slippery slope to laying a new control structure on top of it :)
> Though it will be important that exit is called on the locals in the
> reverse order of declaration.

We are quite familiar with these "slippery slopes". It is in fact a
hard argument to not add scope-qualified local. We usually add a
feature to solve a problem, not to create new ones.

We had something (scope-qualified locals) that seemed to be enough.
Now we have this list of problems:

- It does not create a new scope by itself;
- It is not visible enough;
- It still does not solve the problem for global variables;
- It cannot supress exceptions.


-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
On Wed, Jul 18, 2018 at 6:19 PM Roberto Ierusalimschy <[hidden email]> wrote:

> We had something (scope-qualified locals) that seemed to be enough.
> Now we have this list of problems:

The trouble with it (and I advocated for it a while ago) is that it introduces a new paradigm, yet it does not solve the original problem fully.

Lua does the right thing about its "native garbage" collecting it lazily and transparently from the user.

"Lazily" seems wrong for (some) "other garbage", but should that mean "transparently" is also wrong?

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Dibyendu Majumdar
In reply to this post by Roberto Ierusalimschy
On 18 July 2018 at 17:19, Roberto Ierusalimschy <[hidden email]> wrote:
> We had something (scope-qualified locals) that seemed to be enough.
> Now we have this list of problems:
>
> - It does not create a new scope by itself;
> - It is not visible enough;
> - It still does not solve the problem for global variables;
> - It cannot supress exceptions.
>

Perhaps something simpler could be done? Such as ask a function to be
called on scope exit (normally or via exception) - similar to defer
statement in Go?

Then user code can do what it likes in that function - responsibility
of Lua ends with invoking the function.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Gé Weijers
In reply to this post by Roberto Ierusalimschy

On Wed, Jul 18, 2018 at 9:19 AM Roberto Ierusalimschy <[hidden email]> wrote:
We had something (scope-qualified locals) that seemed to be enough.
Now we have this list of problems:

- It does not create a new scope by itself;
- It is not visible enough;
- It still does not solve the problem for global variables;
- It cannot supress exceptions.

The better is the enemy of the good here. I've seen many programs that leak memory, even though they were written in a language that uses a garbage collector (forgot to set that object member to nil?). Garbage collection is a 95% solution, you have to know how it works and sometimes you have to tweak things.

The problem of leaking resources is a real one, and the implementation of a generational garbage collector in Lua 5.4 may make them worse by making the object owning the resource extremely long-lived once it leaves the nursery.  Scoped locals, a "with ... as" clause, or an equivalent construct can make the programmer's job easier, but there is no 100% solution. A 100% solution would solve the halting problem :-)


--
--

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Doug Currie
In reply to this post by Dibyendu Majumdar
On 18 July 2018 at 17:19, Roberto Ierusalimschy <[hidden email]> wrote:
We had something (scope-qualified locals) that seemed to be enough.
Now we have this list of problems: 

- It does not create a new scope by itself;
- It is not visible enough;
- It still does not solve the problem for global variables;
- It cannot supress exceptions.
 
None of these things are compelling me to reject scope-qualified locals as a very versatile and useful mechanism

On Wed, Jul 18, 2018 at 11:55 AM Roberto Ierusalimschy <[hidden email]> wrote:
It is not by chance that Lua avoids too many syntactical constructs.
They are hard to be represented in the C API.

(I was considering suggesting dynamic-wind, or unwind-protect from Lisp, but ...)
This is a powerful argument, and convinced me that a new language construct is not the answer. 

e

On Wed, Jul 18, 2018 at 1:13 PM Dibyendu Majumdar <[hidden email]> wrote:
On 18 July 2018 at 17:19, Roberto Ierusalimschy <[hidden email]> wrote:
> We had something (scope-qualified locals) that seemed to be enough.
> Now we have this list of problems:
>
> - It does not create a new scope by itself;
> - It is not visible enough;
> - It still does not solve the problem for global variables;
> - It cannot supress exceptions.
>

Perhaps something simpler could be done? Such as ask a function to be
called on scope exit (normally or via exception) - similar to defer
statement in Go?

Then user code can do what it likes in that function - responsibility
of Lua ends with invoking the function.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Sean Conner
In reply to this post by Dibyendu Majumdar
It was thus said that the Great Dibyendu Majumdar once stated:

> On 18 July 2018 at 17:19, Roberto Ierusalimschy <[hidden email]> wrote:
> > We had something (scope-qualified locals) that seemed to be enough.
> > Now we have this list of problems:
> >
> > - It does not create a new scope by itself;
> > - It is not visible enough;
> > - It still does not solve the problem for global variables;
> > - It cannot supress exceptions.
> >
>
> Perhaps something simpler could be done? Such as ask a function to be
> called on scope exit (normally or via exception) - similar to defer
> statement in Go?
>
> Then user code can do what it likes in that function - responsibility
> of Lua ends with invoking the function.

  It could be done now, although not automatically:

        function hoover(...)
          for i = 1 , select('#',...) do
            local obj = select(i,...)
            local mt  = getmetatable(obj)
            if mt and mt.__gc then
              mt.__gc(obj)
            end
          end
        end

        do
          local expensive = foo_user_obj()
          local cheap     = {}
          local medium    = setmetatable({},{ __gc = function(o) end })

          hoover(expensive,cheap,medium)
        end

  -spc (Must ... avoid ... proposing ... new ... metamethod ... )


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Andrew Starks-2
In reply to this post by Gé Weijers


On Wed, Jul 18, 2018 at 12:18 PM, Gé Weijers <[hidden email]> wrote:

On Wed, Jul 18, 2018 at 9:19 AM Roberto Ierusalimschy <[hidden email]> wrote:
We had something (scope-qualified locals) that seemed to be enough.
Now we have this list of problems:

- It does not create a new scope by itself;
- It is not visible enough;
- It still does not solve the problem for global variables;
- It cannot supress exceptions.

The better is the enemy of the good here. I've seen many programs that leak memory, even though they were written in a language that uses a garbage collector (forgot to set that object member to nil?). Garbage collection is a 95% solution, you have to know how it works and sometimes you have to tweak things.

The problem of leaking resources is a real one, and the implementation of a generational garbage collector in Lua 5.4 may make them worse by making the object owning the resource extremely long-lived once it leaves the nursery.  Scoped locals, a "with ... as" clause, or an equivalent construct can make the programmer's job easier, but there is no 100% solution. A 100% solution would solve the halting problem :-)


--
--



And this is where I'm not seeing what others are seeing.

Is performance the issue or order?

If order is not the issue, then calling collectgarbage("collect") after setting your objects to `nil` would be all that is required. Is that correct?

If performance is the issue, then I what is the data on that?

From my vantage point, it looks like the problem to solve is:

There is no simple and robust way to destroy objects that:
 1: have allocated resource that cannot remain allocated for indeterminate amounts of time
 2: need to be destroyed in a specified order
 3: to do so even when something goes wrong, every time

Is that the problem to be solved? Who is the target for this feature? A person informally acquainted with programming or someone writing Lua libraries?

--
Andrew Starks
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Sean Conner
It was thus said that the Great Andrew Starks once stated:

>
> There is no simple and robust way to destroy objects that:
>  1: have allocated resource that cannot remain allocated for indeterminate
> amounts of time
>  2: need to be destroyed in a specified order
>  3: to do so even when something goes wrong, every time
>
> Is that the problem to be solved? Who is the target for this feature? A
> person informally acquainted with programming or someone writing Lua
> libraries?

  Yes.

  A few months ago I wrote some code to recursively dive into directories,
using something like:

        fsys = require "org.conman.fsys"

        function dive(path)
          fsys.chdir(path)
          for entry in fsys.dir() do
            local info = fsys.stat(entry)
            if info.type == 'file' then
              do_something_with(entry)
            elseif info.type == 'dir' then
              dive(entry)
            end
          end
        end

        dive(os.getenv "HOME")

  I was exhausting file descriptors before my program finished [1].  Upon
investigation, it was because iterating over a directory uses a file
descriptor, and because GC had yet to kick in, I had a bazillion open
directories.  I had to change the iterator to close and release the
directory upon finishing instead of relying upon the GC.  And this is in
what I consider library code.

  -spc

[1] fsys.dir() is an iterator that returns filename in the given
        directory (defaults to '.' if not given one).  It will ignore the
        '.' (current directory) and '..' (parent directory) entries becase
        in my use case, those are things I never want to see.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
>   A few months ago I wrote some code to recursively dive into directories,
> using something like:
>
> [...]
>
>   I was exhausting file descriptors before my program finished [1].  Upon
> investigation, it was because iterating over a directory uses a file
> descriptor, and because GC had yet to kick in, I had a bazillion open
> directories.  I had to change the iterator to close and release the
> directory upon finishing instead of relying upon the GC.  And this is in
> what I consider library code.

I think a few rules could greatly improve things. Unfortunately, I was
not following them :-)

- As Tomas pointed out, iterators should release their resources as soon
as they reach the last element. Sadly, my implementation of a directory
iterator in PiL does not do that; but the implementation of io.lines in
the standard library does :-)

- Routines that create resources could do an "emergency collection"
in case the creation fails. For instance, I have just added the
following code to the IO library:

static FILE *trytoopen (lua_State *L, const char *path, const char *mode) {
  FILE *f = fopen(path, mode);
#if defined(EMFILE)  /* EMFILE is not ANSI C */
  if (f == NULL && errno == EMFILE) {  /* too many open files? */
    lua_gc(L, LUA_GCCOLLECT);  /* try to close some files with a GC */
    f = fopen(path, mode);  /* try to open again */
  }
#else
  (void)L;  /* to avoid warnings */
#endif
  return f;
}


These rules put no burden on library users, only on library implementers.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Patrick Donnelly
In reply to this post by Roberto Ierusalimschy
On Wed, Jul 18, 2018 at 9:19 AM, Roberto Ierusalimschy
<[hidden email]> wrote:

>> If Roberto wants to go with a scope-qualified local, at least it's a
>> slippery slope to laying a new control structure on top of it :)
>> Though it will be important that exit is called on the locals in the
>> reverse order of declaration.
>
> We are quite familiar with these "slippery slopes". It is in fact a
> hard argument to not add scope-qualified local. We usually add a
> feature to solve a problem, not to create new ones.
>
> We had something (scope-qualified locals) that seemed to be enough.
> Now we have this list of problems:
>
> - It does not create a new scope by itself;
> - It is not visible enough;

These two are related. I think people are trying to apply Python's
solution to this as a model for "something that has worked". I want to
point out the reasons why Python requires a control structure but Lua
would not: Python's locals are function-scoped. Declaring a local in
Python means it doesn't go out of scope until the function returns.
For this reason, Python requires an explicit control structure to
define the lifetime of the variable. Lua does not have this
limitation.

The extra "scope" makes the variable lifetime unnecessarily verbose.
This is not something Lua should adopt. Do we require every local to
have explicit scope:

do
  local foo = 0
  do
    local bar = 1
    ...
  end
end

? No.

> - It still does not solve the problem for global variables;

There are more appropriate idioms for this that were adopted by .e.g
C++. In the case of a mutex:

static std::mutex m;

int Foo::bar() {
   std::lock_guard<std::mutex> lock(m);
   ...
}

i.e. you can define the scope by creating a new wrapper object in a
scoped local.

--
Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Tim Hill
In reply to this post by Roberto Ierusalimschy


On Jul 18, 2018, at 8:55 AM, Roberto Ierusalimschy <[hidden email]> wrote:

A good use I see for "local scoped" is in C functions, which often have
a hard time to properly free resources. With something like "local scoped",
it would be enough one single function to mark a stack position as
a scoped variable, to be finalized when the function exits.

+1 to this. Handling resource management in C is pretty cumbersome if the function calls any Lua function that might throw an exception (which is most).

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
On Wed, Jul 18, 2018 at 9:21 PM Tim Hill <[hidden email]> wrote:

> +1 to this. Handling resource management in C is pretty cumbersome if the function calls any Lua function that might throw an exception (which is most).

Making life easier for C developers is a noble goal, but it should not be a constraint for the primary objective, viz., deterministic finalisation for Lua programs.

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Sean Conner
It was thus said that the Great Viacheslav Usov once stated:
> On Wed, Jul 18, 2018 at 9:21 PM Tim Hill <[hidden email]> wrote:
>
> > +1 to this. Handling resource management in C is pretty cumbersome if the
> > function calls any Lua function that might throw an exception (which is
> > most).
>
> Making life easier for C developers is a noble goal, but it should not be a
> constraint for the primary objective, viz., deterministic finalisation for
> Lua programs.

  Yes, but the resources that require cleanup if leaving scope are typically
written in C ...

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
On Wed, Jul 18, 2018 at 10:04 PM Sean Conner <[hidden email]> wrote:

>   Yes, but the resources that require cleanup if leaving scope are typically written in C ...

... and cleaned up via the __gc metamethod. I do not think deterministic finalisation has to break that.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
In reply to this post by Andrew Starks-2
On Wed, Jul 18, 2018 at 7:58 PM Andrew Starks <[hidden email]> wrote:

> Is performance the issue or order?

Deterministic finalisation, defined as finalisation immediately after a value becomes unreachable in a program, is trivially possible by running the full GC cycle after each VM instruction. No halting problem to solve here.

The only issue is performance.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Hakki Dogusan-3
In reply to this post by Sean Conner
Hi,

On 18-07-2018 20:26, Sean Conner wrote:

> It was thus said that the Great Dibyendu Majumdar once stated:
>> On 18 July 2018 at 17:19, Roberto Ierusalimschy <[hidden email]> wrote:
>>> We had something (scope-qualified locals) that seemed to be enough.
>>> Now we have this list of problems:
>>>
>>> - It does not create a new scope by itself;
>>> - It is not visible enough;
>>> - It still does not solve the problem for global variables;
>>> - It cannot supress exceptions.
>>>
>>
>> Perhaps something simpler could be done? Such as ask a function to be
>> called on scope exit (normally or via exception) - similar to defer
>> statement in Go?
>>
>> Then user code can do what it likes in that function - responsibility
>> of Lua ends with invoking the function.
>
>    It could be done now, although not automatically:
>
> function hoover(...)
>  for i = 1 , select('#',...) do
>    local obj = select(i,...)
>    local mt  = getmetatable(obj)
>    if mt and mt.__gc then
>      mt.__gc(obj)
>    end
>  end
> end
>
> do
>  local expensive = foo_user_obj()
>  local cheap     = {}
>  local medium    = setmetatable({},{ __gc = function(o) end })


Could a "scope" arg for __gc solve "the immediate release resource"
problem?

ie:
local medium = setmetatable({},{
   __gc = function(o, scope)
            if scope == "local" then ... end
          end })


>
>  hoover(expensive,cheap,medium)
> end
>
>    -spc (Must ... avoid ... proposing ... new ... metamethod ... )
>
>
>

--
Regards,
Hakki Dogusan

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
In reply to this post by Roberto Ierusalimschy
On Thu, Jul 19, 2018 at 12:08 AM, Roberto Ierusalimschy <[hidden email]> wrote:
That would be much harder to implement. Also, this functionality seems a
little "off topic". I thought the idea was related to resource
finalization/clean up. I don't see why resource finalization would
be able to supress an exception, which often has nothing to do with
the resource.


It's tied to Lua not having try-catch syntax (as opposed to pcall), allowing:

  do
      local scoped function catch(e)
          -- conditional catch...
      end
      -- try block...
  end

which is superior to pcall() since the try block doesn't need to be wrapped in a function.  As we know, that idiom breaks return, goto, etc. which are important to keeping the main code path simple.

Agreed it's lower priority than the other benefits on the table.  As long as the exit function can discern error or not, and exit call order is the reverse of declaration, I think we've got a very good value.


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

szbnwer@gmail.com
hi all! :)


i cant see the real life need for this feature, so is there anything
that really cant be solved without it? :/
it totally feels like an xy problem (
https://en.wikipedia.org/wiki/XY_problem ), but if its not, then just
an easier way for something (possibly rare) that adds complexity to
the language that people will like to use and will make everything
else in general more complex, thats not really a good stuff i think...

+1 for collectgarbage against size, assert for failsafe code - for
solving whatever by the way of lua; and the __gc(o, scope) proposal,
it would be cool to force erasing specific stuffs!! :)

btw if freeing resources is the bottleneck of an application, that is
serious enough to handle those resources from c, being hidden behind a
simple interface...


bests for all! :)

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Hisham
In reply to this post by Andrew Starks-2
On 16 July 2018 at 14:12, Andrew Starks <[hidden email]> wrote:

>
>
> On Mon, Jul 16, 2018 at 4:02 AM, Pierre Chapuis <[hidden email]>
> wrote:
>>
>> On Mon, Jul 16, 2018, at 03:00, Andrew Starks wrote:
>>
>> What about calling collectgarbage(“collect”)?
>>
>>
>> In some system I did that in a global error handler (top-level pcall) to
>> avoid problems. However that forces collection of all garbage, not a single
>> object. I wouldn't do it after a for loop, for instance. Also that doesn't
>> work (or isn't easy to write) if you call `return`.
>>
>> --
>> Pierre Chapuis
>
>
>
> I remain curious. Has this been tested and found too slow before?

Yes. I can't provide a reference right now but relying on
collectgarbage() to force release of non-memory resources via __gc at
the necessary times has proven to be unusable for my workload pattern.
It incurred a significant performance hit due to unnecessary memory
traversal.

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> > I remain curious. Has this been tested and found too slow before?
>
> Yes. I can't provide a reference right now but relying on
> collectgarbage() to force release of non-memory resources via __gc at
> the necessary times has proven to be unusable for my workload pattern.
> It incurred a significant performance hit due to unnecessary memory
> traversal.

I am still curious whether that could work as I suggested, calling
__gc as a kind of emergency collection, only when some resource
allocation fails. For instance, in the case of file handles in Linux,
that would mean ~1 full GC for every 1000 streams created, not
one GC per stream.

-- Roberto

12345