(not) handling new programming idioms with grace

classic Classic list List threaded Threaded
95 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> Indeed this was discussed before - I thought then and now the problem
> is that deterministic cleanup is difficult when errors are thrown. I
> do not recall seeing a solution to that - I re-read your replies just
> now, and the solution did not stand out to me.

It wasn't in my replies:

  http://lua-users.org/lists/lua-l/2015-11/msg00335.html

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Dibyendu Majumdar
On 16 July 2018 at 19:25, Roberto Ierusalimschy <[hidden email]> wrote:
>> Indeed this was discussed before - I thought then and now the problem
>> is that deterministic cleanup is difficult when errors are thrown. I
>> do not recall seeing a solution to that - I re-read your replies just
>> now, and the solution did not stand out to me.
>
> It wasn't in my replies:
>
>   http://lua-users.org/lists/lua-l/2015-11/msg00335.html
>

Okay thanks - I obviously did not understand this. Will try to work it out.


Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Jay Carlson
In reply to this post by Roberto Ierusalimschy
On 2018-07-16, at 1:54 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> However, a main issue of any "deterministic cleanup" in Lua is its
> interaction with coroutines. A coroutine can enter a block and never
> leave it.

Yeah, last time the proposal was, "Don't let that happen, and if you do, at least the garbage collector will get to it eventually."

Is there some stylized/restricted/wrapped way to use coroutines such that this doesn't happen, or is caught near the point of error? Like, if you always use something like nurseries for your coroutine needs, can the damage be limited? In that kind of model, direct use of _G.coroutine would be like use of _G.debug.

--
Jay
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
In reply to this post by Roberto Ierusalimschy
On Mon, Jul 16, 2018 at 7:55 PM Roberto Ierusalimschy <[hidden email]> wrote:

> We do. All this has been discussed some time ago; see [1].

I would also like to reference a more recent discussion [2].

A point that may insufficiently have been stressed there is that deterministic cleanup is a desirable property even for non-locally scoped objects.

Especially keeping the status quo for "global by default" (as in a recent discussion), it is hard to avoid this conclusion.

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Petite Abeille
In reply to this post by Roberto Ierusalimschy


> On Jul 16, 2018, at 8:25 PM, Roberto Ierusalimschy <[hidden email]> wrote:
>
> It wasn't in my replies:

Out of curiosity, when replying, why don’t you ever* attribute your quotations?

* Seldomly, e.g. on 05 May 1997 10:49:00 -0300, you actually quoted Mark Ian Barlow <[hidden email]> by name in regards to "lua 3.0a tag methods”. Go figure.


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Dibyendu Majumdar
In reply to this post by Jay Carlson
Hi,

On 16 July 2018 at 19:50, Jay Carlson <[hidden email]> wrote:
> On 2018-07-16, at 1:54 PM, Roberto Ierusalimschy <[hidden email]> wrote:
>
>> However, a main issue of any "deterministic cleanup" in Lua is its
>> interaction with coroutines. A coroutine can enter a block and never
>> leave it.
>
> Yeah, last time the proposal was, "Don't let that happen, and if you do, at least the garbage collector will get to it eventually."
>

I am not sure if this is a problem? If you think of coroutine as a new
thread with its own stack, then any cleanup in the original thread
need not be called unless the coroutine returned to the original
thread.  Have I missed the point here?

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Soni "They/Them" L.
In reply to this post by Pierre Chapuis


On 2018-07-16 09:21 AM, Pierre Chapuis wrote:

> On Mon, Jul 16, 2018, at 03:17, Daurnimator wrote:
>
>> What doesn't nicely translate to lua is what to do in the `yield` case:
>>
>> local function foo()
>>      with bar do
>>          coroutine.yield(baz) -- maybe baz contains something based on `bar`
>>      end
>> end
>> local f = coroutine.wrap(foo)
>> f()
>> if xyz then -- only *maybe* does the coroutine get resumed
>>      f()
>> end
> I was unsure how that was dealt with in Python. This code:
>
>      class Foo(object):
>
>          def __init__(self):
>              pass
>
>          def __enter__(self):
>              print("enter")
>              return self
>
>          def __exit__(self, *args):
>              print("exit")
>
>      foo = Foo()
>
>      def my_gen_f():
>          while True:
>              with foo:
>                  yield True
>
>      def f():
>          my_gen = my_gen_f()
>          next(my_gen)
>          print("after")
>
>      f()
>
>      print("finally")
>
> prints:
>
>      enter
>      after
>      exit
>      finally
>
> This is unrelated to the GC, disabling it does not change the output.
>

Run it on pypy. Keep in mind cpython does refcounting, which cannot be
disabled.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

kurapica
In reply to this post by John Belmonte

I do provide a simple code structure to my teammates so they can focus on the db logic:

 

```

local ctx = MyDatabase()

 

with(ctx)(function()

local entity = ctx.Users:Add{ name = form.name }

entity.create = Date.Now

ctx:SaveChanges()

 

self:Redirect(“/user/profile/” .. entity.id)

end, function(err)

    self:Redirect(“/user/logout”)

end)

```

 

The “with” can have several objects with Open/Close method, so they can be called at the start and the end point. The second function is the error handler, if it’s omit, the “error” will be used so it can be handled by the global error handler.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Daurnimator
In reply to this post by Pierre Chapuis
On 16 July 2018 at 22:21, Pierre Chapuis <[hidden email]> wrote:

> On Mon, Jul 16, 2018, at 03:17, Daurnimator wrote:
>
>> What doesn't nicely translate to lua is what to do in the `yield` case:
>>
>> local function foo()
>>     with bar do
>>         coroutine.yield(baz) -- maybe baz contains something based on `bar`
>>     end
>> end
>> local f = coroutine.wrap(foo)
>> f()
>> if xyz then -- only *maybe* does the coroutine get resumed
>>     f()
>> end
>
> I was unsure how that was dealt with in Python. This code:
>
>     class Foo(object):
>
>         def __init__(self):
>             pass
>
>         def __enter__(self):
>             print("enter")
>             return self
>
>         def __exit__(self, *args):
>             print("exit")
>
>     foo = Foo()
>
>     def my_gen_f():
>         while True:
>             with foo:
>                 yield True
>
>     def f():
>         my_gen = my_gen_f()
>         next(my_gen)
>         print("after")
>
>     f()
>
>     print("finally")
>
> prints:
>
>     enter
>     after
>     exit
>     finally
>
> This is unrelated to the GC, disabling it does not change the output.
>

Looks like PEP-0533 mostly fixes the issue in python.
https://stackoverflow.com/questions/41881731/is-it-safe-to-combine-with-and-yield-in-python#comment70959811_41881820
suggests that the GC is still involved.


I'm wondering if `with` in lua could just have more methods than
:open() and :close().
e.g. :yield() and :resume() if a `with` block is yielded past.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
In reply to this post by Roberto Ierusalimschy
On Tue, Jul 17, 2018 at 1:54 AM, Roberto Ierusalimschy <[hidden email]> wrote:
We do. All this has been discussed some time ago; see [1].

I proposed exactly that in 2007 [1], so I'd welcome it.

But as a minim requirement for many of the use cases I've been advocating over the years (e.g. scope guards), the exit function needs to know if the scope was exited by error--  please pass the error object as a function argument.  Also it would be useful if the exit function could control suppression of the exception in flight (e.g. by having a requirement that it explicitly re-raise).  These are all covered in that post as well.


Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
On Tue, Jul 17, 2018 at 4:54 PM John Belmonte <[hidden email]> wrote:

> I proposed exactly that in 2007 [1], so I'd welcome it.

Which I repeated in 8 years, unaware of your prior work (which led to the message referenced by Roberto earlier).

Unfortunately, Lua is not much like C++ or D. Here is an "official" example from LuaSQL:

-- load driver
local driver = require "luasql.postgres"
-- create environment object
env = assert (driver.postgres())
-- connect to data source
con = assert (env:connect("luasql-test"))
-- reset our table
res = con:execute"DROP TABLE people"
res = assert (con:execute[[
  CREATE TABLE people(
    name  varchar(50),
    email varchar(50)
  )
]])
-- add a few elements
list = {
  { name="Jose das Couves", email="[hidden email]", },
  { name="Manoel Joaquim", email="[hidden email]", },
  { name="Maria das Dores", email="[hidden email]", },
}
for i, p in pairs (list) do
  res = assert (con:execute(string.format([[
    INSERT INTO people
    VALUES ('%s', '%s')]], p.name, p.email)
  ))
end
-- retrieve a cursor
cur = assert (con:execute"SELECT name, email from people")
-- print all rows, the rows will be indexed by field names
row = cur:fetch ({}, "a")
while row do
  print(string.format("Name: %s, E-mail: %s", row.name, row.email))
  -- reusing the table of results
  row = cur:fetch (row, "a")
end
-- close everything
cur:close() -- already closed because all the result set was consumed
con:close()
env:close()

Observe that there is only one local variable. The rest are globals. And before somebody says "this is bad code that no one will write", I see this same code copied over and over in production, pretty much verbatim (easily making DB servers or Lua hosts run out of memory).

Here is another example from the same source:

function rows (connection, sql_statement)
  local cursor = assert (connection:execute (sql_statement))
  return function ()
    return cursor:fetch()
  end
end

The variable is local; but it is an upvalue, too, so it can only be collected when the closure is finalised, which means the return value of rows() needs to be a scoped variable. But here is how it is supposed to be used:

env = assert (require"luasql.mysql".mysql())
con = assert (env:connect"my_db")
for id, name, address in rows (con, "select * from contacts") do
  print (string.format ("%s: %s", name, address))
end

That can certainly be rewritten with a scoped declaration, but the point, like in the simpler example above, is that this goes against Lua's entire written corpus and oral tradition. People still find 10-15 year old snippets of code online and argue that "your system is broken" when they do not work, and that is AFTER they are given the latest edition of PiL and pointed to the latest docs. A new obscure way to declare variables isn't likely to fare much better.

This is why, in my opinion, deterministic finalisation should just kick in fully transparently whenever something requiring it is created. It is opting OUT that should be left to the user.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Dibyendu Majumdar
In reply to this post by John Belmonte
On 17 July 2018 at 15:53, John Belmonte <[hidden email]> wrote:

> On Tue, Jul 17, 2018 at 1:54 AM, Roberto Ierusalimschy
> <[hidden email]> wrote:
>>
>> We do. All this has been discussed some time ago; see [1].
>
>
> I proposed exactly that in 2007 [1], so I'd welcome it.
>
> [1] http://lua-users.org/lists/lua-l/2007-05/msg00354.html
>

Very interesting. It made me think that some of the current features
in Lua might be due to ideas from various people - it would be nice if
this was listed somewhere - maybe in the Wiki? It is always good to be
able to see the history of the development of a feature.

Thanks for posting the link to the your 2007 post.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

John Belmonte
In reply to this post by Viacheslav Usov
On Tue, Jul 17, 2018 at 11:56 PM, Viacheslav Usov <[hidden email]> wrote:
That can certainly be rewritten with a scoped declaration, but the point, like in the simpler example above, is that this goes against Lua's entire written corpus and oral tradition. People still find 10-15 year old snippets of code online and argue that "your system is broken" when they do not work, and that is AFTER they are given the latest edition of PiL and pointed to the latest docs. A new obscure way to declare variables isn't likely to fare much better.

I think it's a fair argument.  Users will forget to put the proper qualifier on the variable, which will result in subtly wrong runtime behavior.  It reminds me of the async/await implementation in Python.  Apparently if you forget to qualify your async function call with the await keyword, Python will just happily do nothing (since async functions simply return a coroutine).

In my old post, I actually argue for "with .. as" over a new type of local.  That control structure seems well-proven at this point, there are so many use cases.  I don't know if there's some collective ego here about "doing something like Python", but it would be a shame to not consider it seriously.

If Roberto wants to go with a scope-qualified local, at least it's a slippery slope to laying a new control structure on top of it :)
Though it will be important that exit is called on the locals in the reverse order of declaration.

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
> I think it's a fair argument.  Users will forget to put the proper
> qualifier on the variable, which will result in subtly wrong runtime
> behavior.  It reminds me of the async/await implementation in Python.
> Apparently if you forget to qualify your async function call with the await
> keyword, Python will just happily do nothing (since async functions simply
> return a coroutine).
>
> In my old post, I actually argue for "with .. as" over a new type of
> local.  That control structure seems well-proven at this point, there are
> so many use cases.  I don't know if there's some collective ego here about
> "doing something like Python", but it would be a shame to not consider it
> seriously.

I did not get your point here. Why a user that would forget to properly
qualify its 'local' would not forget to use 'with'? What are the
differences between 'local scoped' vs 'with' except the syntax?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Tomás Guisasola-2
In reply to this post by Viacheslav Usov
Hi Viacheslav

As one of the maintainers of LuaSQL, I have to explain two characteristics of LuaSQL drivers with respect to this thread.

Observe that there is only one local variable. The rest are globals.
Yes, this was by choice: the example was developed with the command line interpreter in mind.  That code was not supposed to be copied as a pattern.  Anyway, I'll try to improve it or at least to make it clear.
 
function rows (connection, sql_statement)
  local cursor = assert (connection:execute (sql_statement))
  return function ()
    return cursor:fetch()
  end
end

The variable is local; but it is an upvalue, too, so it can only be collected when the closure is finalised, which means the return value of rows() needs to be a scoped variable.
I think all LuaSQL drivers were carefully written to release the resources when the cursor reach the end of the result set.
 
But here is how it is supposed to be used:

env = assert (require"luasql.mysql".mysql())
con = assert (env:connect"my_db")
for id, name, address in rows (con, "select * from contacts") do
  print (string.format ("%s: %s", name, address))
end
Unless there is an error during the for-loop, the resources will be release at the end regardless of the garbage collector.

Note that I am just pointing out some details in the "defense" of LuaSQL.  I not against your argument in general.

Regards,
Tomás
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Javier Guerra Giraldez
In reply to this post by Roberto Ierusalimschy
On 18 July 2018 at 15:20, Roberto Ierusalimschy <[hidden email]> wrote:
> I did not get your point here. Why a user that would forget to properly
> qualify its 'local' would not forget to use 'with'? What are the
> differences between 'local scoped' vs 'with' except the syntax?

it's precisely the syntax.   a 'with' block is always an extra block,
even if it covers the whole loop or function block (it usually
doesn't).  it's very visible, and so hard to miss.   if a novice
programmer sees the `with open('fname') as file: xxxx`  and likes it,
they will probably use it very intentionally.


i do like that feature, but not the syntax.  i too think that scopes
are scopes, don't have to be "special scopes".   still, for things
that might "seem to work until they don't"  (which is common around
resource finalization), explicit and "in your face" syntax has its
advantages.


--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Viacheslav Usov
In reply to this post by Tomás Guisasola-2
On Wed, Jul 18, 2018 at 4:47 PM Tomás Guisasola <[hidden email]> wrote:

> Yes, this was by choice: the example was developed with the command line interpreter in mind.  That code was not supposed to be copied as a pattern.  Anyway, I'll try to improve it or at least to make it clear.

Well, it is copied and it is fact of life. This is actually not Lua nor LuaSQL specific, people just do it and it is hard to blame them.

I did not want to imply that your code was bad, even though for use in production (in a system that runs Lua code repeatedly without exiting) it needs to be heavily modified to make sure resources are always released no matter what.

What I did want to say is the modification required to make that code "robust" would be an an exercise in careful and paranoid programming not unlike that practised in C with respect to dynamic memory allocation, which seems quite at odds with the nature of Lua.

> I think all LuaSQL drivers were carefully written to release the resources when the cursor reach the end of the result set.

That'a a pretty big assumption. Just recently I stumbled upon a big-name commercial ODBC driver that was ignoring the equivalent of env:close() and leaking multiple megabytes of memory as  a result. That is definitely not a problem in LuaSQL, but it illustrates that an expectation of high quality might be mistaken.

> Unless there is an error during the for-loop, the resources will be release at the end regardless of the garbage collector.

"Unless there is an error" is precisely where the problem is; in practice, those non-memory resources are used when interacting with some "external" systems, when errors just happen, and the complexity of error handling that releases those resources is rarely anticipated by anyone. And when done "properly", it easily dominates the complexity of the "main" logic.

Cheers,
V. 
Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
In reply to this post by Javier Guerra Giraldez
> On 18 July 2018 at 15:20, Roberto Ierusalimschy <[hidden email]> wrote:
> > I did not get your point here. Why a user that would forget to properly
> > qualify its 'local' would not forget to use 'with'? What are the
> > differences between 'local scoped' vs 'with' except the syntax?
>
> it's precisely the syntax.   a 'with' block is always an extra block,
> even if it covers the whole loop or function block (it usually
> doesn't).  it's very visible, and so hard to miss.   if a novice
> programmer sees the `with open('fname') as file: xxxx`  and likes it,
> they will probably use it very intentionally.

It is not by chance that Lua avoids too many syntactical constructs.
They are hard to be represented in the C API.

A good use I see for "local scoped" is in C functions, which often have
a hard time to properly free resources. With something like "local scoped",
it would be enough one single function to mark a stack position as
a scoped variable, to be finalized when the function exits.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Dirk Laurie-2
2018-07-18 17:55 GMT+02:00 Roberto Ierusalimschy <[hidden email]>:

>> On 18 July 2018 at 15:20, Roberto Ierusalimschy <[hidden email]> wrote:
>> > I did not get your point here. Why a user that would forget to properly
>> > qualify its 'local' would not forget to use 'with'? What are the
>> > differences between 'local scoped' vs 'with' except the syntax?
>>
>> it's precisely the syntax.   a 'with' block is always an extra block,
>> even if it covers the whole loop or function block (it usually
>> doesn't).  it's very visible, and so hard to miss.   if a novice
>> programmer sees the `with open('fname') as file: xxxx`  and likes it,
>> they will probably use it very intentionally.
>
> It is not by chance that Lua avoids too many syntactical constructs.
> They are hard to be represented in the C API.
>
> A good use I see for "local scoped" is in C functions, which often have
> a hard time to properly free resources. With something like "local scoped",
> it would be enough one single function to mark a stack position as
> a scoped variable, to be finalized when the function exits.

At the Lua level, maybe

with(local_variable_or_upvalue[,level])

where 'level' behaves as in 'error'?

Reply | Threaded
Open this post in threaded view
|

Re: (not) handling new programming idioms with grace

Roberto Ierusalimschy
In reply to this post by John Belmonte
> [...].  Also it would be useful if the exit function could control
> suppression of the exception in flight (e.g. by having a requirement that
> it explicitly re-raise).

That would be much harder to implement. Also, this functionality seems a
little "off topic". I thought the idea was related to resource
finalization/clean up. I don't see why resource finalization would
be able to supress an exception, which often has nothing to do with
the resource.

-- Roberto

12345