q: nil in tables

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

q: nil in tables

Stephan Hennig-2
Hi,

I tend to think tables with nil are constant size or grow only.  So
here's a stupid question:

What use-case is there to have nil in tables /and/ to be able to delete
nil from tables?

Best regards,
Stephan Hennig

Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Stephan Hennig-2
Am 17.03.2018 um 17:06 schrieb Stephan Hennig:

> I tend to think tables with nil are constant size or grow only.  So
> here's a stupid question:
>
> What use-case is there to have nil in tables /and/ to be able to delete
> nil from tables?

Mixed two thoughts into one: What use-case is there to have nil in
tables /and/ to be able to delete (any) values from the same tables?

Best regards,
Stephan Hennig

Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Coda Highland
On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email]> wrote:

> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>
>> I tend to think tables with nil are constant size or grow only.  So
>> here's a stupid question:
>>
>> What use-case is there to have nil in tables /and/ to be able to delete
>> nil from tables?
>
> Mixed two thoughts into one: What use-case is there to have nil in
> tables /and/ to be able to delete (any) values from the same tables?
>
> Best regards,
> Stephan Hennig

One realistic example: Imagine a cache table that holds the results of
expensive function calls. The function can return nil, so you have to
be able to store that, and you need to be able to remove entries in it
as they become stale.

Without the ability to delete keys distinctly from storing nil, you
would have to use a singleton to act as a fake nil and then detect
that and replace it with a real nil at runtime. Possible, sure, but a
bit ugly.

Another realistic example: For performance reasons, you're trying to
reuse tables as much as you can. If you need to be able to distinguish
between a nil value and an absent key, then you HAVE to be able to
properly delete keys if you're going to reuse tables.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Stephan Hennig-2
Am 17.03.2018 um 18:08 schrieb Coda Highland:

> On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email]> wrote:
>> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>>
>>> I tend to think tables with nil are constant size or grow only.  So
>>> here's a stupid question:
>>>
>> What use-case is there to have nil in tables /and/ to be able to
>> delete (any) values from the same tables?>
> One realistic example: Imagine a cache table that holds the results of
> expensive function calls. The function can return nil, so you have to
> be able to store that, and you need to be able to remove entries in it
> as they become stale.
>
> Without the ability to delete keys distinctly from storing nil, you
> would have to use a singleton to act as a fake nil and then detect> that and replace it with a real nil at runtime. Possible, sure, but a
> bit ugly.

Or a second fall-back table containing a true value for all keys with a
nil value.

Anyway, when caching values you're pretty much into the metatable
business already.  That is, this use-case has already a requirement of
the same complexity as the Lua-ish way of storing nil in tables via
metatables already provides.  Nice example, but I don't see a pressing
need for a change in the language (undef) from this one.  (I do see a
need for easier handling of nil in tables, though.  But that need can be
satisfied with more metatable mechanisms as has been proposed by others
already.)


> Another realistic example: For performance reasons, you're trying to
> reuse tables as much as you can. If you need to be able to distinguish
> between a nil value and an absent key, then you HAVE to be able to
> properly delete keys if you're going to reuse tables.

A more tough one to discuss away. :-)

For the simple case of finally emptying a table containing nils in
preparation of reuse, a new function table.empty() can be provided that
does exactly that.

If single keys had to be removed during lifetime of a table, this
use-case has the following requirements

  a) performance (no metatables if possible),
  b) storing nil in tables,

This certainly required something like undef.  It is not quite a common
use-case, though, I guess.

Thanks for the examples, Coda!  May be obvious, but I'd just feel better
with tables with nils, but without undef.

Best regards,
Stephan Hennig

Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Russell Haley


On Sat, Mar 17, 2018 at 10:54 AM, Stephan Hennig <[hidden email]> wrote:
Am 17.03.2018 um 18:08 schrieb Coda Highland:
> On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email]> wrote:
>> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>>
>>> I tend to think tables with nil are constant size or grow only.  So
>>> here's a stupid question:
>>>
>> What use-case is there to have nil in tables /and/ to be able to
>> delete (any) values from the same tables?>
> One realistic example: Imagine a cache table that holds the results of
> expensive function calls. The function can return nil, so you have to
> be able to store that, and you need to be able to remove entries in it
> as they become stale.
>
> Without the ability to delete keys distinctly from storing nil, you
> would have to use a singleton to act as a fake nil and then detect> that and replace it with a real nil at runtime. Possible, sure, but a
> bit ugly.

Or a second fall-back table containing a true value for all keys with a
nil value.

Anyway, when caching values you're pretty much into the metatable
business already.  That is, this use-case has already a requirement of
the same complexity as the Lua-ish way of storing nil in tables via
metatables already provides.  Nice example, but I don't see a pressing
need for a change in the language (undef) from this one.  (I do see a
need for easier handling of nil in tables, though.  But that need can be
satisfied with more metatable mechanisms as has been proposed by others
already.)


> Another realistic example: For performance reasons, you're trying to
> reuse tables as much as you can. If you need to be able to distinguish
> between a nil value and an absent key, then you HAVE to be able to
> properly delete keys if you're going to reuse tables.

A more tough one to discuss away. :-)

For the simple case of finally emptying a table containing nils in
preparation of reuse, a new function table.empty() can be provided that
does exactly that.

If single keys had to be removed during lifetime of a table, this
use-case has the following requirements

  a) performance (no metatables if possible),
  b) storing nil in tables,

This certainly required something like undef.  It is not quite a common
use-case, though, I guess.

Thanks for the examples, Coda!  May be obvious, but I'd just feel better
with tables with nils, but without undef.

Best regards,
Stephan Hennig

https://github.com/RussellHaley/lua-persist

For my key-value store library I have to add true/false to enumerations and I have to check for nil and add false to uninitialized values. Not a bid deal, but nil in tables 'feels more natural' to me. If I want holes, I can force them with undef. Otherwise, I can expect a sequence like behaviour, which makes me happy. 

Russ
Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Stephan Hennig-2
Am 17.03.2018 um 19:03 schrieb Russell Haley:

> On Sat, Mar 17, 2018 at 10:54 AM, Stephan Hennig <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Am 17.03.2018 um 18:08 schrieb Coda Highland:
>     > On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email] <mailto:[hidden email]>> wrote:
>     >> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>     >>
>     >>> I tend to think tables with nil are constant size or grow only.  So
>     >>> here's a stupid question:
>     >>>
>     >> What use-case is there to have nil in tables /and/ to be able to
>     >> delete (any) values from the same tables?>
>
> https://github.com/RussellHaley/lua-persist
>
> For my key-value store library I have to add true/false to
> enumerations and I have to check for nil and add false to
> uninitialized values. Not a bid deal, but nil in tables 'feels more
> natural' to me.
I'm all for tables with nils.


> If I want holes, I can force them with undef.

Asking for nil in tables naturally asks for nils /and/ holes in the same
table.  But I'm not convinced there's really a need for having both at
the same time.


> Otherwise, I can expect a sequence like behaviour, which makes me
> happy.

Sharing your happiness!

Best regards,
Stephan Hennig

Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Duane Leslie
In reply to this post by Stephan Hennig-2
> On 18 Mar 2018, at 04:08, Coda Highland <[hidden email]> wrote:
>
>> On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email]> wrote:
>>> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>>>
>>> I tend to think tables with nil are constant size or grow only.  So
>>> here's a stupid question:
>>>
>>> What use-case is there to have nil in tables /and/ to be able to delete
>>> nil from tables?
>>
>> Mixed two thoughts into one: What use-case is there to have nil in
>> tables /and/ to be able to delete (any) values from the same tables?
>>
>> Best regards,
>> Stephan Hennig
>
> One realistic example: Imagine a cache table that holds the results of
> expensive function calls. The function can return nil, so you have to
> be able to store that, and you need to be able to remove entries in it
> as they become stale.

I think this is where a tuple type should exist.  The set of arguments (as captured by `...`) and the set of return results (I believe only captured by passing as arguments to another function) are naturally tuples and require the special `select` method to manipulate.  If you stored the empty tuple or the tuple containing only nil this would represent the two possible returns without needing to store `nil` in the table.

Regards,

Duane.
Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

ThePhD
Having nil in tables is also an immense conceptual boon for people who work with the C API and userdata. Having a userdata-specific `nil` was extremely sad for when you wanted to store your userdata in a table, because it messed with usual language semantics of `if t[key] then ... end`. Now, it works out of the box and both pairs() and ipairs() behaves just fine, and `nil` can be used as a proper `nullptr`-alike value!

On Sat, Mar 17, 2018 at 5:58 PM, Duane Leslie <[hidden email]> wrote:
> On 18 Mar 2018, at 04:08, Coda Highland <[hidden email]> wrote:
>
>> On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email]> wrote:
>>> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>>>
>>> I tend to think tables with nil are constant size or grow only.  So
>>> here's a stupid question:
>>>
>>> What use-case is there to have nil in tables /and/ to be able to delete
>>> nil from tables?
>>
>> Mixed two thoughts into one: What use-case is there to have nil in
>> tables /and/ to be able to delete (any) values from the same tables?
>>
>> Best regards,
>> Stephan Hennig
>
> One realistic example: Imagine a cache table that holds the results of
> expensive function calls. The function can return nil, so you have to
> be able to store that, and you need to be able to remove entries in it
> as they become stale.

I think this is where a tuple type should exist.  The set of arguments (as captured by `...`) and the set of return results (I believe only captured by passing as arguments to another function) are naturally tuples and require the special `select` method to manipulate.  If you stored the empty tuple or the tuple containing only nil this would represent the two possible returns without needing to store `nil` in the table.

Regards,

Duane.

Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Sean Conner
It was thus said that the Great ThePhD once stated:
> Having nil in tables is also an immense conceptual boon for people who work
> with the C API and userdata. Having a userdata-specific `nil` was extremely
> sad for when you wanted to store your userdata in a table, because it
> messed with usual language semantics of `if t[key] then ... end`. Now, it
> works out of the box and both pairs() and ipairs() behaves just fine, and
> `nil` can be used as a proper `nullptr`-alike value!

  I'm not following, and I'm someone who uses the C API and userdata.  Can
you give an example of the problem?

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

ThePhD
The example of the problem comes from a design decision I made when dealing with Userdata, particular pointers to structs. Consider the following:

     struct thing {};
     thing v;
     thing* array_of_things[3] = {&v, NULL, &v};

Then, you serialize this into a table in Lua. The typical implementation would be to simply iterate through the array, and then push the value of the item at `array_of_things[i]` into Lua, as a userdata, and put that into a table. this makes the following happen:

     print( c_api_created_table[1] , c_api_created_table[2] ,  c_api_created_table[3] )
     ---- 0BE57DAD, 000000000,  0BE57DAD, or something

However, users who do this would also expect the following to work decently well:

     if c_api_created_table[2] then
          print('totally here') -- prints 'totally here', but that's not what the user expects!
     else
          print('totally NOT here')
     end

The problem is that pushing a NULL pointer as a userdata value doesn't play nice with `nil` semantics and truthiness in Lua. `c_api_created_table[2] == nil`  also would not play ball very nicely: it results in false, which sends everyone on a wild ride. It is also impossible to override the equals operator for such a thing, due to Lua's metatable rules for __eq [1]. You have 2 choices: either create a Sentinel Null proxy value, OR you push nil when you detect NULL pointers.

In order to play nice with Lua semantics and Prior Art[2] in these cases, detecting C or C++ NULL and appropriately doing a `lua_pushnil` is how many libraries, wrappers and frameworks have behaved for a long while (though some others did choose a sentinel USERDATA_NIL approach as well).

Doing that, however resulted in ANOTHER problem: tables under pre-Lua 5.4work1 with nils would essentially cut a sequence "short" [3]. This made it impossible to use regular tables with Userdata Pointers But With Nil In the Mix. ipairs and all those class of functions -- and many wrappers that took Lua's definition of sequences to heart and emulated its behavior -- all would fail[4].

Under the new paradigm of allowing "nil", everything Just Works™ now. The user gets their expectation that `nil` gets pushed for NULL userdata pointers, and sequences no longer have to be broken because of taking this design decision. It's a really good win-win.

[1] -  Emphasis my own: "__eq: the equal (==) operation. Behavior similar to the addition operation, except that Lua will try a metamethod only when the values being compared are either both tables or both full userdata and they are not primitively equal. The result of the call is always converted to a boolean."
[2] - https://github.com/ThePhD/sol2/issues/289
[3] - https://github.com/ThePhD/sol2/issues/383
[4] - This reminds me that I have to add some custom handling to my table serialization routines. I was going to start checking if the type had a metatable, and if that metatable has a `__len` entry, to use that to get a size hint which will help for proper iteration, even in the presence of `nil`. This would help patch away some of the issues of iterating through a custom container that may contain `nil`...


On Sat, Mar 17, 2018 at 7:29 PM, Sean Conner <[hidden email]> wrote:
It was thus said that the Great ThePhD once stated:
> Having nil in tables is also an immense conceptual boon for people who work
> with the C API and userdata. Having a userdata-specific `nil` was extremely
> sad for when you wanted to store your userdata in a table, because it
> messed with usual language semantics of `if t[key] then ... end`. Now, it
> works out of the box and both pairs() and ipairs() behaves just fine, and
> `nil` can be used as a proper `nullptr`-alike value!

  I'm not following, and I'm someone who uses the C API and userdata.  Can
you give an example of the problem?

  -spc



Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

ThePhD
Oops, I goofed up some of the footnotes. The "all would fail[4]" should be "all would fail[3]", and the [4] should just be a P.S., aha.

On Sun, Mar 18, 2018 at 4:28 AM, ThePhD <[hidden email]> wrote:
The example of the problem comes from a design decision I made when dealing with Userdata, particular pointers to structs. Consider the following:

     struct thing {};
     thing v;
     thing* array_of_things[3] = {&v, NULL, &v};

Then, you serialize this into a table in Lua. The typical implementation would be to simply iterate through the array, and then push the value of the item at `array_of_things[i]` into Lua, as a userdata, and put that into a table. this makes the following happen:

     print( c_api_created_table[1] , c_api_created_table[2] ,  c_api_created_table[3] )
     ---- 0BE57DAD, 000000000,  0BE57DAD, or something

However, users who do this would also expect the following to work decently well:

     if c_api_created_table[2] then
          print('totally here') -- prints 'totally here', but that's not what the user expects!
     else
          print('totally NOT here')
     end

The problem is that pushing a NULL pointer as a userdata value doesn't play nice with `nil` semantics and truthiness in Lua. `c_api_created_table[2] == nil`  also would not play ball very nicely: it results in false, which sends everyone on a wild ride. It is also impossible to override the equals operator for such a thing, due to Lua's metatable rules for __eq [1]. You have 2 choices: either create a Sentinel Null proxy value, OR you push nil when you detect NULL pointers.

In order to play nice with Lua semantics and Prior Art[2] in these cases, detecting C or C++ NULL and appropriately doing a `lua_pushnil` is how many libraries, wrappers and frameworks have behaved for a long while (though some others did choose a sentinel USERDATA_NIL approach as well).

Doing that, however resulted in ANOTHER problem: tables under pre-Lua 5.4work1 with nils would essentially cut a sequence "short" [3]. This made it impossible to use regular tables with Userdata Pointers But With Nil In the Mix. ipairs and all those class of functions -- and many wrappers that took Lua's definition of sequences to heart and emulated its behavior -- all would fail[4].

Under the new paradigm of allowing "nil", everything Just Works™ now. The user gets their expectation that `nil` gets pushed for NULL userdata pointers, and sequences no longer have to be broken because of taking this design decision. It's a really good win-win.

[1] -  Emphasis my own: "__eq: the equal (==) operation. Behavior similar to the addition operation, except that Lua will try a metamethod only when the values being compared are either both tables or both full userdata and they are not primitively equal. The result of the call is always converted to a boolean."
[2] - https://github.com/ThePhD/sol2/issues/289
[3] - https://github.com/ThePhD/sol2/issues/383
[4] - This reminds me that I have to add some custom handling to my table serialization routines. I was going to start checking if the type had a metatable, and if that metatable has a `__len` entry, to use that to get a size hint which will help for proper iteration, even in the presence of `nil`. This would help patch away some of the issues of iterating through a custom container that may contain `nil`...


On Sat, Mar 17, 2018 at 7:29 PM, Sean Conner <[hidden email]> wrote:
It was thus said that the Great ThePhD once stated:
> Having nil in tables is also an immense conceptual boon for people who work
> with the C API and userdata. Having a userdata-specific `nil` was extremely
> sad for when you wanted to store your userdata in a table, because it
> messed with usual language semantics of `if t[key] then ... end`. Now, it
> works out of the box and both pairs() and ipairs() behaves just fine, and
> `nil` can be used as a proper `nullptr`-alike value!

  I'm not following, and I'm someone who uses the C API and userdata.  Can
you give an example of the problem?

  -spc




Reply | Threaded
Open this post in threaded view
|

Re: q: nil in tables

Stephan Hennig-2
In reply to this post by Stephan Hennig-2
Am 17.03.2018 um 18:54 schrieb Stephan Hennig:

> Am 17.03.2018 um 18:08 schrieb Coda Highland:
>> On Sat, Mar 17, 2018 at 11:55 AM, Stephan Hennig <[hidden email]> wrote:
>>> Am 17.03.2018 um 17:06 schrieb Stephan Hennig:
>>>
>>>> I tend to think tables with nil are constant size or grow only.  So
>>>> here's a stupid question:
>>>>
>> Another realistic example: For performance reasons, you're trying to
>> reuse tables as much as you can. If you need to be able to distinguish
>> between a nil value and an absent key, then you HAVE to be able to
>> properly delete keys if you're going to reuse tables.
>
> A more tough one to discuss away. :-)
>
> For the simple case of finally emptying a table containing nils in
> preparation of reuse, a new function table.empty() can be provided that
> does exactly that.
>
> If single keys had to be removed during lifetime of a table, this
> use-case has the following requirements
>
>   a) performance (no metatables if possible),
>   b) storing nil in tables,
>
> This certainly required something like undef.

When tables are reused to prevent gc to kick in too often when producing
a fast number of short-lived tables in rapid succession, those tables
are probably of constant size.  (Of course, there are still other
reasons for table reuse.)


> It is not quite a common use-case, though, I guess.>
> Thanks for the examples, Coda!  May be obvious, but I'd just feel better
> with tables with nils, but without undef.
>
> Best regards,
> Stephan Hennig