Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

Anton Jordaan

>> Italo Maio wrote:
>> Perl has auto-vivification and it creates way more problems than it
>> solves. Also agree this should not be part of the language and the
>> error when trying to index a null value is a good thing.
I am not very familiar with Perl, but AFAIK, Perl's auto-vivification
automatically creates array elements even when just *reading* from the
array.
This is not what I suggest Lua should do. Lua should only create tables
when *assigning*. When reading, just return nil and don't create anything.


> Soni "They/Them" L. wrote:
>> > v = (pcall(function() t[a][b][c][d][e] end) or nil) and
>> t[a][b][c][d][e]
This will work, and there are indeed multiple ways to code v =
t[a][b][c][d][e] to avoid throwing an error.
To the best of my knowledge, the code example that posted (with lots of
"and"s), executes the fastest (and is also relatively easily readable).


> Coda Highland wrote:
>> This comes up very, very frequently.
>>
>> This is absolutely a non-starter for inclusion in stock Lua, but for
>> your own programs there are a number of solutions that have been
>> proposed and/or implemented.
Could you elaborate why this is a non-starter?

I have seen a number of solutions (including solutions that use
metatables to define dynamic arrays, which are useful only when
assigning) and also offered my own preferred code examples,
which manage do the job in my own programs, but none of these
workarounds are as simple -- and consistent -- as writing v =
t[a][b][c][d][e].


-- Anton

Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

Philippe Verdy-2
If such behavior is added, it should depend on a table's
meta-property, i.e. a special value in its metatable.
Autocreation can also create problems in Lua.
Or may be there's a need to subclass the "nil" value: reading an
unassigned position in a table with this property could return this
special "niltable" value (whose type/class would be the same as nil)
but which differs from "nil" by the fact that "niltable" is indexable
like a table but will not throw an error when subindexing it for
further reads or writes and would behave as a no-op.
Basically "niltable" and "nil" would still be equal for "==" but
distinct for identity, and they would both by of type "nil":

  nil==niltable and typeof(nil)=="nil" and typeof(niltable)=="nil".

And unlike "nil", "niltable" would have a metatable containing the
array-indexing meta-operator functions. "niltable" would not be
modifiable/mutable at all.
A table that must not throw an error, and "niltable" itself, would
just both have to contain the same meta-operators to override the
default error-throwing behavior.

Le dim. 1 mars 2020 à 16:28, Anton Jordaan <[hidden email]> a écrit :

>
>
> >> Italo Maio wrote:
> >> Perl has auto-vivification and it creates way more problems than it
> >> solves. Also agree this should not be part of the language and the
> >> error when trying to index a null value is a good thing.
> I am not very familiar with Perl, but AFAIK, Perl's auto-vivification
> automatically creates array elements even when just *reading* from the
> array.
> This is not what I suggest Lua should do. Lua should only create tables
> when *assigning*. When reading, just return nil and don't create anything.
>
>
> > Soni "They/Them" L. wrote:
> >> > v = (pcall(function() t[a][b][c][d][e] end) or nil) and
> >> t[a][b][c][d][e]
> This will work, and there are indeed multiple ways to code v =
> t[a][b][c][d][e] to avoid throwing an error.
> To the best of my knowledge, the code example that posted (with lots of
> "and"s), executes the fastest (and is also relatively easily readable).
>
>
> > Coda Highland wrote:
> >> This comes up very, very frequently.
> >>
> >> This is absolutely a non-starter for inclusion in stock Lua, but for
> >> your own programs there are a number of solutions that have been
> >> proposed and/or implemented.
> Could you elaborate why this is a non-starter?
>
> I have seen a number of solutions (including solutions that use
> metatables to define dynamic arrays, which are useful only when
> assigning) and also offered my own preferred code examples,
> which manage do the job in my own programs, but none of these
> workarounds are as simple -- and consistent -- as writing v =
> t[a][b][c][d][e].
>
>
> -- Anton
>

Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

Francisco Olarte
In reply to this post by Anton Jordaan
Anton:

On Sun, Mar 1, 2020 at 4:28 PM Anton Jordaan <[hidden email]> wrote:
> >> Perl has auto-vivification and it creates way more problems than it
> >> solves. Also agree this should not be part of the language and the
> >> error when trying to index a null value is a good thing.
> I am not very familiar with Perl, but AFAIK, Perl's auto-vivification
> automatically creates array elements even when just *reading* from the
> array.

No, it does not, it just returns undef:

folarte@7of9:~$ perl -e '@x=(1); for(4..6) { print $x[$_]//"x"; }
print " $#x  $^V\n";'
xxx 0  v5.28.1

( For the uninitiated, this creates a list with a single element, 1,
then reads index 4,5,6 printing their content or x if it is undefined,
then prints the index of the last element of the ( 0 based ) array and
the interpreter version ).

And, AFAIK, it's been doing that since the eighties.

It does a similar thing on dictionaries:

folarte@7of9:~$ perl -e '%x=(a => 1); print $x{b}//"x"; print %x,"\n";'
xa1

But unlike lua it stores undef ( the more similar thing to nil it has ):
folarte@7of9:~$ perl -w -e '%x=(a => 1); $x{b}=undef; print %x,"\n";'
Use of uninitialized value $x{"b"} in print at -e line 1.
a1b

And autocreates intermediate slots, as it has lists separated from maps:

folarte@7of9:~$ perl -w -e '@x=(0); $x[3]=3; print @x," $#x\n";'
Use of uninitialized value in print at -e line 1.
Use of uninitialized value in print at -e line 1.
03 3
( note the double warning due to elements 1 and 2 being on the list
but undefined ).

But autovivification can sometimes create arrays/tables. I think it is
when you index an unitialized scalar variable.

To add some opinion, adding a safe navigation operator seems right,
although against keeping lua a simple language, and preserves back
compatibility, but changing the semantics of regular indexing of plain
tables does not seem right to me.

Francisco Olarte.

Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

Coda Highland
In reply to this post by Anton Jordaan
On Sun, Mar 1, 2020 at 9:28 AM Anton Jordaan <[hidden email]> wrote:> Coda Highland wrote:
>> This comes up very, very frequently.
>>
>> This is absolutely a non-starter for inclusion in stock Lua, but for
>> your own programs there are a number of solutions that have been
>> proposed and/or implemented.
Could you elaborate why this is a non-starter?

I have seen a number of solutions (including solutions that use
metatables to define dynamic arrays, which are useful only when
assigning) and also offered my own preferred code examples,
which manage do the job in my own programs, but none of these
workarounds are as simple -- and consistent -- as writing v =
t[a][b][c][d][e].

For read-only use, nil[k] == nil isn't TOO awful, and if the language had been designed that way from the beginning it would probably have been okay. Unfortunately, plenty of existing code depends on the current behavior so it's far, far too late into Lua's existence to be making that change. It would break too many things to convert a meaningful runtime error into something silent, especially when there are alternatives (such as the get() function I demonstrated) that can achieve the same goal without breaking anything.

For automatically creating a hierarchy of tables, on the other hand... The realistic use cases where this would be the best syntax without any drawbacks are fairly few. It would require a more invasive patch to Lua to accomplish it for something that isn't super commonly applicable in practice -- for most programs, you're not going to be making a deep assignment into an arbitrary possibly-undefined table hierarchy. And enabling this kind of behavior by default can not only mask errors, it can also really break things if your data type has certain expectations for how things are supposed to be assembled. Again, there are other ways of achieving the goal that have fewer drawbacks, so doing it in the core just doesn't really make sense.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

nobody
In reply to this post by Philippe Verdy-2
On 01/03/2020 16.46, Philippe Verdy wrote:

> Or may be there's a need to subclass the "nil" value: reading an
> unassigned position in a table with this property could return this
> special "niltable" value (whose type/class would be the same as nil)
> but which differs from "nil" by the fact that "niltable" is indexable
> like a table but will not throw an error when subindexing it for
> further reads or writes and would behave as a no-op.
> Basically "niltable" and "nil" would still be equal for "==" but
> distinct for identity, and they would both by of type "nil":
>
>    nil==niltable and typeof(nil)=="nil" and typeof(niltable)=="nil".
>
> And unlike "nil", "niltable" would have a metatable containing the
> array-indexing meta-operator functions. "niltable" would not be
> modifiable/mutable at all.
> A table that must not throw an error, and "niltable" itself, would
> just both have to contain the same meta-operators to override the
> default error-throwing behavior.

I've made similar things in the past.  By changing __eq to not care
about "primitive types" (number, table, string, …)[1], you can build
"logical types" that are spread across multiple primitive types and
still compare correctly according to the "logical type"'s rules (e.g. a
Complex type that logically contains tables (with .re and .im), plain
numbers (just the real part) and userdata (C complex)[2]; or (here) an
Absent type that contains both plain nil and e.g. tables supposed to act
like nil.)  That solves the general problem once, and makes it so that
you don't need a specific nil-like value just for this and then a
foo-like value for something else…

Add __false / __bool and then your fake nil can also be false.  (And now
finally falsy values can contain information – so you can store where
this particular nil was generated, and thus you can later use that info
to either generate a fancy error message (bar.lua:42: attempt to add a
nil value, generated at foo.lua:23 when accessing table t at field
"foo"), or to retroactively create fields / intermediate data structure
layers and turn things into a valid assignment.  But if you're happy
with fake nils taking the "wrong" (truthy) `if` branch, we don't even
need that…)

type() can be wrapped / monkey patched from inside the language, and
past that point it's fairly hard to keep them apart.  (generic `for`
loops and tables[3] will still be able to tell which one's which, but
that's about it.)

-- nobody


[1]  More specifically, it's back to the 5.2(?) behavior where a == b
calls the metamethod only if both values have the same __eq metamethod,
which prevents accidental asymmetric garbage like

   tt = setmetatable( {}, { __eq = function()  return true   end } )
   ff = setmetatable( {}, { __eq = function()  return false  end } )
   print( tt == ff, ff == tt )
   --> true, false

(you could still do __eq = function() return math.random( 2 ) == 1 end
or explicitly share the metamethod and add logic that creates asymmetry)
and also prevents logically illegal comparisons (so your __eq doesn't
have to also do a type check, beyond what it itself needs) and then
removing the constraint that primitive types have to match… at that
point, it's just

   Nil = setmetatable( {}, { __eq = function() return  true  end } )
   debug.setmetatable( nil, getmetatable( Nil ) )

and primitive equality of the __eq metamethod is what's doing all the
heavy lifting.  (Only fake and real nils have this __eq metamethod, so
only comparisons between them will call this __eq.  They should all be
logically equal, so that's precisely what the function says.)


[2]  If you set __index for numbers to allow (only) .re (returns the
number itself) and .im (returns 0), then it's just

   __eq = function( a, b )  return a.re == b.re and a.im == b.im  end

no matter how the particular values are represented!  (Primitively
comparable values (numbers) still compare according to their rules even
though there's an __eq metamethod, so this doesn't explode.)  I think
that's a very nice, terse solution, and it's actually _more_ readable
than the usual type()-case done by nested `if`s…


[3] If you make it so that _ALL_ newly created tables set the value from
the per-type metatable list as their metatable (ordinarily that's nil,
so unless you set that, nothing really changes), you have the means to
prevent that:  Set a metatable for new tables in the per-type metatable
list that has __newindex, which checks for fake nils & maps them to real
nils / errors out.  Past that point, only generic `for` really cares
about the "real" `nil`.  (Further, if you move the per-type metatable
list from the global state to each coroutine (and they're initialized at
creation from the parent coroutine), then each coroutine can do whatever
it needs without stepping on the toes of the others… at which point you
basically have independent "universes" with different local rules.  That
can already be done with fully independent Lua states, but then you
can't directly send tables or full userdata across…)

Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

Anton Jordaan
In reply to this post by Anton Jordaan
> Sean Conner wrote:
>> Here are some of the issues that need to be addressed for this to work.
>> 1) t[a] = v This is syntactic sugar for t = { [a] = v }.  Then does that mean:
>> local t[a] = v is legal
Yes, local is legal.


>> 2) t[a][b][c][d][e] = v -- when t[a][b] exists
>> An asusmption being made is that a,b,c,d,e are all tables.  They
>> don't have to be.  It could be a userdata that responds to both
>> __index and __newindex to act like a table.  Then this is fine, but
>> what if b*doesn't*  respond to __newindex?
a,b,c,d,e can be any valid table indices.  I assume you meant "what if
t[a][b] doesn't respond to __newindex"?

Since t[a][b][c][d][e] = v is simply syntactic sugar, it should expand
to the following, and will throw the same errors:
(assuming in this case that the interpreter finds that t[a][b] already
exists)

t[a][b][c] = {[d] = {[e] = v}}

If t[a][b] exists but doesn't respond like a table, an error will be thrown.


>> 3) v = t[a][b][c][d][e]
>>
>> v ends up nil.  Which index is nil?
The nil return value will not provide this information.


>> 4) Also, what about: t[a],t[b],t[c] = 1,2,3 How does this play out, syntactic surgar wise?
Although my proposal has more to do with multi-dimensional arrays, the
same can be applied to new one-dimensional tables.
If t is undefined, the above unambiguously means that t is intended to
be a table, so following the principle that
"When writing: automatically assign a single-entry table to each
undefined variable, with the given index the sole entry", we get:
t = {[a] = 1}
t[b] = 2
t[c] = 3



>> 4) t[a] [ t[B] ] [c][d][e] = v
>> If t doesn't exist, then what do we end up with?  If B doesn't
>> exist, what do we end up with?  It's hard to think about.
This should throw an "table index is nil" error, since t[B] is nil, and
therefore an invalid index for assignment.
N.B.:  My proposal is not that nil or NaN should become valid table
indices, and my "syntactic sugar for autovivification" (if I may borrow
the Perl term)
will only apply when the indices are valid, and only when assigning, not
when reading.



>> 5) t[a][0/0][c][d][e] = v
>> The previous question brings this up---what if the index is NaN?
>> Rare, but possible.  Treat it as nil?  Error?
As in question (4), this should throw an error, since it is an
assignment to an invalid index.

Currently, Lua throws an error when assigning to an invalid index (nil
or NaN), but returns nil when reading from the same:
t[0/0] = 1 throws "table index is NaN" error,
v = t[NaN] just returns nil.



Reply | Threaded
Open this post in threaded view
|

Re: Deprecate "Attempt to index a nil value" error; instead, return nil or create table

Philippe Verdy-2
Le mar. 3 mars 2020 à 12:53, Anton Jordaan <[hidden email]> a écrit :
> a,b,c,d,e can be any valid table indices.  I assume you meant "what if
> t[a][b] doesn't respond to __newindex"?
>
> Since t[a][b][c][d][e] = v is simply syntactic sugar,

This is completely false. The real meaning is in fact much like:
  local _ref; _ref = t[a]; _ref = _ref[b]; _ref = _ref[c]; _ref =
_ref[d]; _ref[e] = v

Or better:
  local _ref1 = t.__getindex(a)
  local _ref2 = _ref1.__getindex(b)
  local _ref3 = _ref2.__getindex(c)
  local _ref4 = _ref3.__getindex(d)
  _ref4._setindex(e, v)*

Note that there are references to *different* table objects and that
the syntax t[a][b][c][d][e] is not a single table indexing but
multiple indexings, with all but the last one being dereferenced by
explicit reads.
This is not really a multidimensional table. but a table of tables of tables....

If you want to create a single multidimendional table "t", what you
would need in Lua would be to use a syntax like
  t[mykey(a, b, c, d, e)] = v
by using some companion function to compute a key. That function could
be a metaproperty of the table, stored for example in its metatable.
  getmetatable(t)['__setindex']( [getmetatable(t)['mykey'](a, b, c, d, e)], v)

The current syntax t[a][b][c][d][e] however does not assume the
existence of any custom multidimenional indexing function; so all
tables in Lua are strictly unidimensional, and this syntax then
implies the creation of multiple table objects (one for each plane,
one for each row in each plane, onde of each column in each row in
each plane, etc.). This is clearly inefficient for multidimensional
tables (this is also true for Java or Javascript).

The same syntax in C or C++ however may be efficient if each dimension
is a positive integer in a compact range (from 0 to n-1) whose
cardinality is static and predeclared as part of the type, because it
does not imply any creation of subtable objects to hold extra table
pointers (or pointers and references in C++). When the dimensions are
not constrained, C and C++ also require the use of multiple tables. So
the syntax becomes inefficient as well.

The technic is then to have tables having multi-key indexing. As Lua
has a single key indexing, you have to create it yourself.

What would be good in Lua is to allwo this syntax
   t[a, b, c, d, e]  for a single table with multi-key indexing
instead of:
  t[a][b][c][d][e] for multiple tables with unikey indexing

This technically would require a the multidimensional multikey
indexing function that would create a indexable unique reference that
can be used as the single key in the existing unidimensional Lua
table, both for reading and writing in that multidimensional table. It
is posible to do that by extending the "__getindex", "__setindex"
metafunctions so they accept more than one "index" parameter.
Internally that metafunction may need an internal table to keep a
dictionary of used multidimensional indexes and infer an
unidimensional index position, but this is not always the case (e.g.
when all dimensions are unconstrained ranges but are all integers, you
can interleave the bits in each dimension to create a single integer
value from multiple ones, but the classic multidimensional arrays of
C/C++ use a simple multiplication of each key by a static
cardinal.before adding them, and this is much faster than bit
interleaving (used for example in geographic databases for indexing
longitude/latitude in "quad" indexes).