(not) handling new programming idioms with grace

classic Classic list List threaded Threaded
95 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> On Wed, Jul 18, 2018 at 11:55 PM, Roberto Ierusalimschy <
> [hidden email]> wrote:
>
> > It is not by chance that Lua avoids too many syntactical constructs.
> > They are hard to be represented in the C API.
> >
>
> If "with" is syntax sugar on top of scoped vars then a representation in
> the C API isn't needed.
>
> The benefits of "with" syntax are related to readability:  1) it's clear by
> code structure that a resource is open, and 2) named variables can be
> elided in many cases, e.g.:

As hard as I try, I don't see why 'with foo() as x' is clearer than
'local scoped x = foo()'.  Both have a very clear and unique mark
('with' vs 'scoped'). Moreover, 'with' has different meanings in
some other languages, so it can be even more confusing.

'with foo()' might be slightly better than 'local scoped _ = foo()',
but I don't think it is worth creating a new construct (plus a new
reserved word) just to avoid the '_'.

As I already explained, the more you insist on 'with', the less I like
the general idea. If 'with' is really superior to 'local scoped', there
is something I am not understanding correctly about the whole stuff. I
will not try to solve a problem I am not understanding.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Javier Guerra Giraldez
On 25 July 2018 at 17:16, Roberto Ierusalimschy <[hidden email]> wrote:
> As hard as I try, I don't see why 'with foo() as x' is clearer than
> 'local scoped x = foo()'.  Both have a very clear and unique mark
> ('with' vs 'scoped'). Moreover, 'with' has different meanings in
> some other languages, so it can be even more confusing.

I don't think it's clearer or less confusing.  I do think it's more visible.

Being visible has a few advantages on itself, for code reviews it's
more obvious, for beginners (reading code examples) its easier to
realize when it's being used and what is the scope of the variable.

on the other hand, the `scoped` mark is a lot less intrusive, which
has some other advantages, such as:

 -  its easier to add to existing code, simply adding some determinism
without changing the semantics.
 -  variables don't have to be at the top of the scope, so you can add
more "finalizers" without indenting too far from the left margin
 -  carries a lot less baggage, so you don't confuse with other
"withs" from other languages.

if this was a democracy, my vote would go to the `scoped` mark.


--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
In reply to this post by Roberto Ierusalimschy
On Thu, Jul 26, 2018, 12:56 AM Roberto Ierusalimschy <[hidden email]> wrote:
This is correct.

Of course, if a new metamethod is under consideration.  It sounded like you were preferring a plain callable.

The creation of a scoped variable needs to allocate memory, and therefore
it could fail without creating the scoped variable!

Would it be reasonable to only allow scoped var assignment at declaration, and not add the var to the stack if nil?

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> The creation of a scoped variable needs to allocate memory, and therefore
> >
> it could fail without creating the scoped variable!
> >
>
> Would it be reasonable to only allow scoped var assignment at declaration,
> and not add the var to the stack if nil?

I don't think so. As I said, the variable must be allocated before
the creation of the resource; if we create the resource and then
the variable allocation fails, the resource won't be closed. But before
the creation of the resource we cannot know whether it will be nil.

-- Roberto


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Gé Weijers
In reply to this post by William Ahern


On Fri, Jul 20, 2018 at 1:04 PM William Ahern <[hidden email]> wrote:
On Fri, Jul 20, 2018 at 12:05:33PM -0700, Gé Weijers wrote:
> If Lua is embedded in a larger program that uses files or other types of
> resources elsewhere you end up having to call the emergency Lua GC from who
> knows where in the code. This may be a 3rd party library, or even a system
> shared library/DLL that you have no control over. A failure to allocate a
> file handle somewhere deep in library code may be unrecoverable.

The same applies to malloc failure. Should Lua also not bother invoking the
emergency collector when malloc fails?

Poor library QoI doesn't justify crippling the runtime or better written
libraries. Rather, the feature should stand on its own merits.

If 'malloc' fails (or you run out of file handles) just when a program is trying to open a file 'fopen' will in all likelihood fail, and return NULL.

This code is from liolib.c:

  p->f = fopen(fname, mode);
  if (p->f == NULL)
    luaL_error(L, "cannot open file '%s' (%s)", fname, strerror(errno));

The emergency garbage collector is not going to help there. This would require modifying the Lua interpreter to add a recovery loop around every library call that could potentially fail because you run out of memory (or file handles). Unless that's done for 100% of calls that can fail (inside and outside the interpreter) your code can fail in unexpected ways.

BTW: on some systems 'malloc' doesn't fail, but a (possibly unrelated) program is killed to free up resources (look up the Linux kernel's "oom kiiler").

A feature that works most of the time just makes failures less likely, which increases the risk of problems going undetected long enough that broken code gets deployed. I like programs that fail early, and fail deterministically if they have any flaws. I hope that such a feature is made optional.

So for me:  +1 on the "local scoped" proposal, -1 on the "let's try the garbage collector if things fail" idea.


--

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
In reply to this post by Roberto Ierusalimschy
On Thu, Jul 26, 2018 at 12:16 AM, Roberto Ierusalimschy <[hidden email]> wrote:
As I already explained, the more you insist on 'with', the less I like
the general idea. If 'with' is really superior to 'local scoped', there
is something I am not understanding correctly about the whole stuff. I
will not try to solve a problem I am not understanding.
 
I'm far for insisting on "with".  Everything I've written on scope hooks since the gems chapter footnote has presented scoped variables as viable.

I raised "with" again here because it's purely syntax sugar given scoped vars, and doesn't implicate the C API.  If the sugar doesn't look like a good value from this vantage point, I can respect that view.


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
In reply to this post by Roberto Ierusalimschy
On Thu, Jul 26, 2018 at 8:35 AM, Roberto Ierusalimschy <[hidden email]> wrote:
if we create the resource and then
the variable allocation fails, the resource won't be closed.

If the Lua core must take care against this out of memory situation, it follows that any library using scoped vars must take equal care.

But this seems like a tall order in general.  You mean there must be no malloc from the time a resource is created and its corresponding Lua object is assigned to a scoped var.  Does Lua (or LuaJIT) even define when the core or stdlib might allocate memory?  For example, iterators can be complicated and it seems likely the resource object could get wrapped in a closure, etc. before reaching the scoped var.  And what if the iterator contains multiple resources-- is it possible to ensure either both or none are allocated, and cleanup ownership is properly transferred from the iterator constructor to the for loop?
 
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> > if we create the resource and then
> > the variable allocation fails, the resource won't be closed.
>
> If the Lua core must take care against this out of memory situation, it
> follows that any library using scoped vars must take equal care.
>
> But this seems like a tall order in general.  You mean there must be no
> malloc from the time a resource is created and its corresponding Lua object
> is assigned to a scoped var.  Does Lua (or LuaJIT) even define when the
> core or stdlib might allocate memory?  For example, iterators can be
> complicated and it seems likely the resource object could get wrapped in a
> closure, etc. before reaching the scoped var.  And what if the iterator
> contains multiple resources-- is it possible to ensure either both or none
> are allocated, and cleanup ownership is properly transferred from the
> iterator constructor to the for loop?

Does Python's 'with' handle this is some different way?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
On Fri, Jul 27, 2018 at 3:14 AM, Roberto Ierusalimschy <[hidden email]> wrote:
Does Python's 'with' handle this is some different way?

When wrapping a resource within an object, isn't the first order of business to implement a proper __gc finalizer?  Then if someone forgets to explicitly close it, or forgets to put the "scoped" qualifier on var, or there's an out of memory exception when creating the scoped var, it will eventually be cleaned up by garbage collection.  Specifically for the out-of-memory scenario, it's rare enough that this should suffice rather than having every use of scoped var do a paranoid dance.

It would be the same in Lua or Python.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Wim Couwenberg-6
In reply to this post by John Belmonte
> As hard as I try, I don't see why 'with foo() as x' is clearer than
> 'local scoped x = foo()'.  Both have a very clear and unique mark
> ('with' vs 'scoped'). Moreover, 'with' has different meanings in
> some other languages, so it can be even more confusing.

(Sorry if this breaks the mail thread, I only subscribe to digest mails.)

If I understand the “local scoped” proposal correctly then at least “local scoped” and Python’s “with” have different semantics. Python does *not* bind context management to a local variable name. For example you can omit the variable name or assign a different value to it in its scope, it will not affect the behavior. Instead the context manager is linked to the scope itself (behind the scenes as it were).

In terms of Lua, this would more closely resemble a “scoped” expression like this:

scoped <expression>

or

local x = scoped <expression>

Here “scoped <expression>” would bind the expression to the current scope. The local name x in the second example is completely unrelated to the context management of the expression.

Another difference then could be that the order of __enter__/__exit__ calls is quite clear in Python while it could be not so clear for such a “scoped expression” in constructs like:

local x = foo(scoped bar(), scoped baz())

Cheers,
Wim



Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Patrick Donnelly
In reply to this post by Roberto Ierusalimschy


On Wed, Jul 25, 2018, 8:56 AM Roberto Ierusalimschy <[hidden email]> wrote:
> new code. Fortunately, it doesn't 
One problem with this proposal (or the similar one with a 4th parameter)
is that we will have to create the scoped variable in all loops.
Consider this code:

     do
       local f, s, var = explist
       local <mark> s = s -- added!
       while true do
         ...

The creation of a scoped variable needs to allocate memory, and therefore
it could fail without creating the scoped variable!
So, I think the correct code should be more like this:

     do
       local <mark> _s = nil -- added!
       local f, s, var = explist
       _s = s -- added!
       while true do
         ...

Anyway, Lua will have to allocate a scoped variable for every loop,
independently whether it needs it or not.

I'm not seeing why the scoped local (?) needs to be "allocated". Are you actually meaning a malloc? I would have thought in the implementation that the local would be annotated somehow similar to closure upvalues that are pulled off the function stack during return or error. Except the scoped local never needs to be pulled off the stack because when the scope ends for any reason, the value it stores is __exit()ed. (It is probably nonsensical for a local to be scoped and an upvalues.)
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> I'm not seeing why the scoped local (?) needs to be "allocated". Are you
> actually meaning a malloc? I would have thought in the implementation that
> the local would be annotated somehow similar to closure upvalues that are
> pulled off the function stack during return or error. Except the scoped
> local never needs to be pulled off the stack because when the scope ends
> for any reason, the value it stores is __exit()ed. (It is probably
> nonsensical for a local to be scoped and an upvalues.)

Closure upvales are allocated (malloc'ed) in the current implementation.

You are right that we could put some kind of annotation directly in the
stack, but that is not how the simple implementation being considered
until now works. (Moreover, I think any annotation in the stack would
add more problems than it solves.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Hisham
In reply to this post by Roberto Ierusalimschy
On 21 July 2018 at 09:27, Roberto Ierusalimschy <[hidden email]> wrote:
>> If the emergency collection became the standard behavior of io.open I
>> suspect that "beware that opening a file in Lua may trigger a full GC"
>> would become one of those dreaded folk-wisdom performance tips.
>
> In a generational collector, any allocation can trigger a major
> collection. io.open allocates memory (to create the stream handle),
> so this dreaded folk-wisdom performance tip is bound to happen anyway.
> (Or maybe not?)

Is a major collection in the generational GC comparable to a full
collection as triggered by collectgarbage()? I thought it would be
more like a regular collection from the incremental GC (with the minor
collections being faster than those).

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Javier Guerra Giraldez
On 15 August 2018 at 00:55, Hisham <[hidden email]> wrote:
> Is a major collection in the generational GC comparable to a full
> collection as triggered by collectgarbage()? I thought it would be
> more like a regular collection from the incremental GC (with the minor
> collections being faster than those).


in my mental model (plz correct!)  a collection "step" is comparable
in both GC designs, the main difference is when they consider that
they've done enough steps and switch to the next phase.  in the "old"
GC it's when the whole tree has been traversed, while in generational
GC it's when the next objects are "too old", unless this happened to
be a major cycle, then it keeps on as previously.

that still leaves the question: would `collectgarbage('collect')`
always do a major collection, or just a full cycle of whatever size is
currently scheduled?



--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> On 15 August 2018 at 00:55, Hisham <[hidden email]> wrote:
> > Is a major collection in the generational GC comparable to a full
> > collection as triggered by collectgarbage()? I thought it would be
> > more like a regular collection from the incremental GC (with the minor
> > collections being faster than those).
>
>
> in my mental model (plz correct!)  a collection "step" is comparable
> in both GC designs, the main difference is when they consider that
> they've done enough steps and switch to the next phase.  in the "old"
> GC it's when the whole tree has been traversed, while in generational
> GC it's when the next objects are "too old", unless this happened to
> be a major cycle, then it keeps on as previously.

The generational collector is not incremental. It does not have "steps";
it always performs a complete collection each time it is invoked. What
changes is what objects it traverses and sweeps (only new objects in
minor collections and all of them in major collections).

A major collection in generational mode is the same as a full collection
triggered by collectgarbage(). This is also the same as a "regular"
collection from the incremental GC, except that in incremental mode this
"regular" (or "full") collection is done incrementally, interleaved with
program execution.

Usually, a minor collection is much faster than a full ("regular")
collection, but slower than a single step from the incremental collector.


> that still leaves the question: would `collectgarbage('collect')`
> always do a major collection, or just a full cycle of whatever size is
> currently scheduled?

It always performs a full collection, which is equivalent to enough steps
in incremental mode or a major collection in generational mode.

Moreover, an emergency collection can happen in any mode, whenever an
allocation fails. It is equivalent to a full collection, except that
it does not run finalizers. (They are queued to be run in the next
opportunity.)

-- Roberto

12345