Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

Dirk Laurie-2
2018-03-14 8:04 GMT+02:00 Russell Haley <[hidden email]>:

>> for i,v in pairs(u) do print(i,v) end
> 1       one
> 2       two
> 3       four
> 4       nil
> 5       five
>> #t
> 5
>> t[4] = undef
>> for i,v in pairs(t) do print(i,v) end
> 1       one
> 2       two
> 3       four
> 5       five
>> #t
> 3  --<<<<<< I Expected 4 here?

The above results were verified with 'lua' built on Ubuntu 16.04 LTS
from inside the src directory by

   make -e MYFLAGS=-DLUA_NILINTABLE linux-readline

The feature seems to implemented in the following way:

1. type 'nil' now has a subtype LUA_TEMPTY.
2. There are two values of type 'nil', one of each subtype, with
corresponding predefined objects nil and empty (my name; it is
actually 'luaH_emptyobject'). However, if LUA_NILINTABLE is not set,
these objects are the same.
3. t[i] = nil stores empty, which is just nil if LUA_NILINTABLE is not set.
4. t[i] = undef stores nil regardless.
5. A macro isempty() is defined which distinguishes (or not, depending
on LUA_NILINTABLE) between nil and empty.
6. The difference seems not to be directly visible in the C API (no
lua_isempty).
7. Most (but not all) cases, especially in the support for tables,
where previously there was a test for a nil value now use isempty()
instead. In particular, the function which searches for a boundary
(i.e. the default for '#') does so.
8. The table library is written in the API, so cannot test for empty.
It is affected indirectly because it uses #.

We can now argue if we like whether it would not have been less
confusing there was no 'undef', and 'empty' was a second predefined
value of tye 'nil', with t[i] = nil and t[i] = empty meaning what they
seem to — but that would break nearly all existing programs in Lua.

Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

pocomane
On Wed, Mar 14, 2018 at 3:11 PM, Roberto Ierusalimschy
<[hidden email]> wrote:
> That is true. But you cannot pass 'undef' to a function or return
> an 'undef' from a function (anymore than you can pass 'while' to a
> function).

Sorry Roberto, my previous comment was a bit confusing. The point was:

1) The LUA_NILINTABLE patch uses undef for table content deletion and
existence check, but not in the only other place we need nil: function
arguments and returns
2) I do not see any easy way to use undef in that place without make
it a value [1], i.e. falling back to the nil solution.
3) So if the undef will stay, it will always be in a mixed solution:
nil for functions, undef for tables. I found this scenaio to be ugly
due to its complexity when compared with the current situation.

At this point I want to ask to the list: does someone see an easy
solution for optional arguments/returns that does not involve a nil
value?

In that case I would be very interested to see a nil-less lua
experiment (or a keyworded-nil lua, if you prefer call it that way)!

[1] Well, actually one can guess a syntax like `func(a, , b)` or
`return a, , b`. But what `a,b,c = func()` should means when one of
the return values is missing?

Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

Axel Kittenberger
Maybe think of it this way (as I understand it) 'undef' isn't a value and it isn't a non-value either, it's nothing. it's just a new keyword, a syntactically somewhat less intuative way of defining a 'delete' operator on the one hand and a "key 'in' table" operator on the other hand to test it's existance (even if nil)

I don't care much about syntax, tough. Semantically I consider it a hugh improvement.

Kind regards, Axel
Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

pocomane
On Wed, Mar 14, 2018 at 4:02 PM, Axel Kittenberger <[hidden email]> wrote:
> I don't care much about syntax, tough. Semantically I consider it a hugh
> improvement.
>

Me too, IF I can use it everywhere. Not just in tables.

Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

Axel Kittenberger
Me too, IF I can use it everywhere. Not just in tables.

If it instead would be a new 'delete' and 'in' operator, you wouldn't have this complaint, you can't use it otherwhere. IMO it's still just the confusion you think of it as some kind of new value, which is understandable, but it isn't.
Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

pocomane
On Wed, Mar 14, 2018 at 4:10 PM, Axel Kittenberger <[hidden email]> wrote:
> If it instead would be a new 'delete' and 'in' operator, you wouldn't have
> this complaint, you can't use it otherwhere. IMO it's still just the
> confusion you think of it as some kind of new value, which is
> understandable, but it isn't.


I am perfectly aware it is not a value but a keyword (I explicitly
state it in my previous messages). My only concern is about having two
different way to indicate "Emptyness": a value one for the functions
(nil) and a keyword one for the tables (undef).

Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

pocomane
However, with undef around, is there still any reason to not allow nil
in table keys?

Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

Hisham
In reply to this post by Axel Kittenberger
On 14 March 2018 at 12:10, Axel Kittenberger <[hidden email]> wrote:
>> Me too, IF I can use it everywhere. Not just in tables.
>
> If it instead would be a new 'delete' and 'in' operator, you wouldn't have
> this complaint, you can't use it otherwhere. IMO it's still just the
> confusion you think of it as some kind of new value, which is
> understandable, but it isn't.

I agree: the "looks like a value but isn't a value" nature of undef is
going to quickly go to the top of the list of "common pitfalls" if it
ever becomes part of the core language.

(Plus, having an expression of the form `<thing1> == <thing2>` which
is true but `<thing2> == <thing1>` isn't even syntactially valid is...
bound to produce some head-scratching, to put it mildly.)

The idea that the `undef` global variable is nil in Lua <5.4 and then
many programs could have the same behavior if the programmer's careful
is a clever hack, but the confusion these misleading syntactic forms
bring to the language are just not worth it IMO.

If you're writing code where you're concerned with 5.3-and-lower
compatibility, you'll have to stick with 5.3-and-lower semantics for
tables and #, and using undef won't give you anything (if anything,
you'll have to be careful that no single use of # ever relies on
undef's 5.4 behavior -- which means undef becomes useless and just
another 5.3-nil). Otherwise, if you don't care about 5.3-and-lower
compat, you'd be better off with a less confusing and more consistent
syntax, using actual operators.

I do value a lot the possibility of being able to keep writing code
that's "5.1 and up" compatible (LuaRocks is like that!). But I'd
prefer to have a syntax that doesn't work at all on 5.1-5.3 than to
have 5.4 code that runs on 5.3 (and vice-versa) but then silently
behaves differently.

I understand that the current work version's proposal of `undef` as a
semi-value is a great effort for addressing the holes-in-tables
question while keeping "5.1 and up" possible. But writing code with
`undef` for "5.1 and up" will be painful both ways: if anywhere you
forget and assume 5.4 behavior for #, your code's silently broken on
5.1-5.3; if anywhere you forget and "unset" a key with nil (which you
can do indirectly, e.g. with table.move(), so no, grepping the sources
isn't enough), you're growing a table forever and silently making a
memory leak in 5.4. I'm afraid cross-version coding in that world
would become a minefield.

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

KHMan
On 3/16/2018 12:27 AM, Hisham wrote:

> On 14 March 2018 at 12:10, Axel Kittenberger wrote:
>>> Me too, IF I can use it everywhere. Not just in tables.
>>
>> If it instead would be a new 'delete' and 'in' operator, you wouldn't have
>> this complaint, you can't use it otherwhere. IMO it's still just the
>> confusion you think of it as some kind of new value, which is
>> understandable, but it isn't.
>
> I agree: the "looks like a value but isn't a value" nature of undef is
> going to quickly go to the top of the list of "common pitfalls" if it
> ever becomes part of the core language.
>
> (Plus, having an expression of the form `<thing1> == <thing2>` which
> is true but `<thing2> == <thing1>` isn't even syntactially valid is...
> bound to produce some head-scratching, to put it mildly.)
>
> The idea that the `undef` global variable is nil in Lua <5.4 and then
> many programs could have the same behavior if the programmer's careful
> is a clever hack, but the confusion these misleading syntactic forms
> bring to the language are just not worth it IMO.
>
> If you're writing code where you're concerned with 5.3-and-lower
> compatibility, you'll have to stick with 5.3-and-lower semantics for
> tables and #, and using undef won't give you anything (if anything,
> you'll have to be careful that no single use of # ever relies on
> undef's 5.4 behavior -- which means undef becomes useless and just
> another 5.3-nil). Otherwise, if you don't care about 5.3-and-lower
> compat, you'd be better off with a less confusing and more consistent
> syntax, using actual operators.
>
> I do value a lot the possibility of being able to keep writing code
> that's "5.1 and up" compatible (LuaRocks is like that!). But I'd
> prefer to have a syntax that doesn't work at all on 5.1-5.3 than to
> have 5.4 code that runs on 5.3 (and vice-versa) but then silently
> behaves differently.
>
> I understand that the current work version's proposal of `undef` as a
> semi-value is a great effort for addressing the holes-in-tables
> question while keeping "5.1 and up" possible. But writing code with
> `undef` for "5.1 and up" will be painful both ways: if anywhere you
> forget and assume 5.4 behavior for #, your code's silently broken on
> 5.1-5.3; if anywhere you forget and "unset" a key with nil (which you
> can do indirectly, e.g. with table.move(), so no, grepping the sources
> isn't enough), you're growing a table forever and silently making a
> memory leak in 5.4. I'm afraid cross-version coding in that world
> would become a minefield.

+1

It's like a neat syntax trick.

More and more non-intuitive things to remember about table
behaviour? Now I am being reminded about the 1001 details about
coding objects, templates, etc in C++. Ewwwwwww.

--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia



Reply | Threaded
Open this post in threaded view
|

Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

Roberto Ierusalimschy
In reply to this post by Hisham
> I do value a lot the possibility of being able to keep writing code
> that's "5.1 and up" compatible (LuaRocks is like that!). But I'd
> prefer to have a syntax that doesn't work at all on 5.1-5.3 than to
> have 5.4 code that runs on 5.3 (and vice-versa) but then silently
> behaves differently.

I don't see how 't[i]=undef' would behave differently with the new or
the old semantics. What will behave differently is 't[i]=nil' (after
all, this is the whole point of the change in the behavior), which
we should avoid.


> [...] if anywhere you forget and "unset" a key with nil (which you
> can do indirectly, e.g. with table.move(), so no, grepping the sources
> isn't enough), [...]

Just for curiosity, can you give me a real-case example of unseting a
key with nil indirectly through table.move?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Fwd: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available

Hisham
Roberto,

For some reason the "From:" address changed in my message and the
email didn't go through the list, sorry. I did reply right away with a
real-world example. Follows below.

-- Hisham

---------- Forwarded message ----------
Date: 15 March 2018 at 20:22
Subject: Re: Nils in tables in Lua 5.4.0 (worki1) (Was: [ANN] ...now available
To: Lua mailing list <[hidden email]>


On Thu, Mar 15, 2018, 15:59 Roberto Ierusalimschy
<[hidden email]> wrote:

>
> > I do value a lot the possibility of being able to keep writing code
> > that's "5.1 and up" compatible (LuaRocks is like that!). But I'd
> > prefer to have a syntax that doesn't work at all on 5.1-5.3 than to
> > have 5.4 code that runs on 5.3 (and vice-versa) but then silently
> > behaves differently.
>
> I don't see how 't[i]=undef' would behave differently with the new or
> the old semantics. What will behave differently is 't[i]=nil' (after
> all, this is the whole point of the change in the behavior), which
> we should avoid.


#t will silently work differently. But the point is that Lua code
doesn't live in a bubble. When I write Lua 5.3 code in a "Lua 5.1 and
up" fashion I assume that I have access to the entire Lua 5.x
ecosystem (from LuaFileSystem onwards). This has been true from 5.1 up
to 5.3.

Up until 5.3, writing code "5.1 and up" simply meant "do not depend on
the latest version's features": no `goto`s, the occasional `if`
statement to switch between `setfenv` and `_ENV`, a little wrapper to
normalize the return values of `os.execute`, that kind of thing. It
was never the case that "you can run code written for Lua 5.(x-1), but
it'll be broken in subtle ways".

If the clever "backwards compatibility" of undef means that I can
write Lua 5.4 code that runs on 5.3... as long as it doesn't make use
of the distinctive 5.4 features (fine) _and never interacts with any
Lua 5.1-5.3 code_ (not fine), then that's of very little use. And
making it look syntactically like it's "Lua 5.1 and up" compatible in
the sense that we've taken it to mean until today is bound to trip
people up (really not fine).

The bottom line is: if these two worlds of code (Lua 5.4 code with
undef and the vast ecosystem of thousands of existing Lua 5.x modules)
will never be able to integrate properly anyway, then it's better to
break the syntax to make sure people never try to run them at the same
time.

> > [...] if anywhere you forget and "unset" a key with nil (which you
> > can do indirectly, e.g. with table.move(), so no, grepping the sources
> > isn't enough), [...]
>
> Just for curiosity, can you give me a real-case example of unseting a
> key with nil indirectly through table.move?

https://github.com/titan-lang/titan/blob/foreign/c-parser/cpp.lua#L451

Anyway, I just gave that one example because I had it from the top of
my head. But it's trivial to think up other realistic examples. Take
any assignment of the form `t[k] = f()` where function f may or may
not return a value. Saying that such a drastic change would be
greppable is understating the problem by a huge amount.

-- Hisham