# Quest: real world "Lua array with holes" usage

71 messages
1234
Open this post in threaded view
|

## Quest: real world "Lua array with holes" usage

 In face of recent (long) discussion about "Lua arrays", I would request a simple few lines of real world examples where the use of "Lua arrays with holes" is significant. I known that, intuitively, the length operator # should return the "size" of the array, but I'm not able to find significant use cases where "holes" are, in fact, a real problem other than a *simple* algorithm redesign.For my particular use case I made heavy use of "sparse things", from which Lua tables are awesome, where all computations are based on the keys, namely, with pairs() or indexing the relevant key. Also, I use "proper sequences" with Lua tables. In this case, table.pack and table.unpack are symmetric, and the relevant computation is given by the "order" of the keys, namely, with table.insert, table.remove, table.move etc that is by default, in this case, simple to understand. I never ever used a "Lua array with holes", where the ordering, or existence, of the keys is indeed relevant for computation. Precluding the use of __len or misbehaviour of  #.Could you help me giving a (real world) counterexample?-- Rodrigo Azevedo Moreira da Silva
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 > For my particular use case I made heavy use of "sparse things", from which > Lua tables are awesome, where all computations are based on the keys, > namely, with pairs() or indexing the relevant key. Also, I use "proper > sequences" with Lua tables. In this case, table.pack and table.unpack are > symmetric, and the relevant computation is given by the "order" of the > keys, namely, with table.insert, table.remove, table.move etc that is by > default, in this case, simple to understand. I never ever used a "Lua array > with holes", where the ordering, or existence, of the keys is indeed > relevant for computation. Precluding the use of __len or misbehaviour of  #. > > Could you help me giving a (real world) counterexample? The only real-world case that bothers me is when you want to collect all the results from a function call, something like {f()}. If there is an error, the standard protocol is to return (nil, msg), which would create an array with holes. (Currently I think the correct way to fix this would be to return (false, msg), not to change the behavior of lists.) -- Roberto
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Rodrigo Azevedo On 21 July 2016 at 08:13, Rodrigo Azevedo <[hidden email]> wrote: > In face of recent (long) discussion about "Lua arrays", I would request a > simple few lines of real world examples where the use of "Lua arrays with > holes" is significant. > I known that, intuitively, the length operator # should return the "size" of > the array, but I'm not able to find significant use cases where "holes" are, > in fact, a real problem other than a *simple* algorithm redesign. > > For my particular use case I made heavy use of "sparse things", from which > Lua tables are awesome, where all computations are based on the keys, > namely, with pairs() or indexing the relevant key. Also, I use "proper > sequences" with Lua tables. In this case, table.pack and table.unpack are > symmetric, and the relevant computation is given by the "order" of the keys, > namely, with table.insert, table.remove, table.move etc that is by default, > in this case, simple to understand. I never ever used a "Lua array with > holes", where the ordering, or existence, of the keys is indeed relevant for > computation. Precluding the use of __len or misbehaviour of  #. > > Could you help me giving a (real world) counterexample? > > -- > Rodrigo Azevedo Moreira da Silva The place it usually comes up for me is when keeping arguments across a callback-style api: local function foo(...)     local args = {...}     do_something_async(function()         -- called when done         my_other_function(x, table.unpack(args))     end) end If the argument list contains a `nil` then the above will fail for some inputs. Hence you end up needing to use `table.pack` or `select("#", ...)`
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Roberto Ierusalimschy On 21/07/16 14:09, Roberto Ierusalimschy wrote: > The only real-world case that bothers me is when you want to collect > all the results from a function call, something like {f()}. If there > is an error, the standard protocol is to return (nil, msg), which > would create an array with holes. Yes, that's the only way I get holes in arrays. OTOH, that's happening a lot less since I decided to assert  all functions that can return nil,... unless I specifically know what to do in that case. Anyway, it's pretty easy to check ret[1]==nil to see if the call was successful. > (Currently I think the correct way to fix this would be to return > (false, msg), not to change the behavior of lists.) -- Roberto I have doubts on this, because false is a reasonable value to get from a function... Returning nil is clearer because the meaning is "could not produce a value to return". Otherwise you end using a explicit return for status pcall-like, all the time. I believe the main problem with nil in arrays stems not from the language, but from the design of programs: there is a tendency to use nil for nil-like concepts in the business logic, and that's when things get broken. A database NULL is not nil, is a singleton {}. The same for "free_slot", "nodataacquired" and whatever. Nil should be reserver for language-level concepts, like "the function failed to get a result" or "this variable has no value assigned". Jorge
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Rodrigo Azevedo On Thu, Jul 21, 2016 at 12:13:40PM -0300, Rodrigo Azevedo wrote: > In face of recent (long) discussion about "Lua arrays", I would request a > simple few lines of real world examples where the use of "Lua arrays with > holes" is significant. The first use case that came to mind was JSON deserializers, where the obvious mapping of null to nil causes problems with arrays. But I think that's irritating mostly because superficially it _seems_ like such a trivial (and trivially fixable) mismatch between Lua and JavaScript type semantics, but as the thread has shown it's not trivial at all. I wouldn't use this case to argue for change in Lua.
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 Hi there is really a significant difference: Whatever you assign to an array (or object) member in JavaScript, it stays in the array/object. Assign null, or even undefined, doesn't matter; it's there and you can enumerate it. You need to explicitly delete() it. This is kind f confusing if you come from C++, in JavaScript delete() does not kill the data, it kills the variable; the data may be referenced else where and survive. I do not really like the doubled feature undefined - null either. -- Oliver Am 21.07.2016 um 21:28 schrieb William Ahern: > On Thu, Jul 21, 2016 at 12:13:40PM -0300, Rodrigo Azevedo wrote: >> In face of recent (long) discussion about "Lua arrays", I would request a >> simple few lines of real world examples where the use of "Lua arrays with >> holes" is significant. > The first use case that came to mind was JSON deserializers, where the > obvious mapping of null to nil causes problems with arrays. > > But I think that's irritating mostly because superficially it _seems_ like > such a trivial (and trivially fixable) mismatch between Lua and JavaScript > type semantics, but as the thread has shown it's not trivial at all. I > wouldn't use this case to argue for change in Lua. > >
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by William Ahern > The first use case that came to mind was JSON deserializers, where the > obvious mapping of null to nil causes problems with arrays. That's the whole point: one probably shouldn't not force the semantics of one language into another. In this case, and others, it's simpler and clearer to define         null = {} once and use it where needed. if you must skip over these when traversing a table, then that's what your semantics dictates and you must do it. Just do not try to do this with nil, which has its own, clear semantics in Lua.
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 On Thu, Jul 21, 2016 at 6:08 PM, Luiz Henrique de Figueiredo <[hidden email]> wrote: >> The first use case that came to mind was JSON deserializers, where the >> obvious mapping of null to nil causes problems with arrays. > > That's the whole point: one probably shouldn't not force the semantics > of one language into another. In this case, and others, it's simpler > and clearer to define >         null = {} > once and use it where needed. if you must skip over these when traversing > a table, then that's what your semantics dictates and you must do it. > Just do not try to do this with nil, which has its own, clear semantics > in Lua. I think part of the reason people try to use Lua's `nil` in ways like JSON's `null` or Python's `None` is because, in a context where you know you won't get the Boolean `false`, `if var then ...` is an easy way to check if `var` is a useful value or not, as `nil`, `null`, and `None` all evaluate to false in a Boolean context. The problem arises when one tries to create a singleton to represent this lack of a useful value. Python provides the `__bool__` method (in 3.x; `__nonzero__` in Python 2.x) to customize the behavior of an object when used in a Boolean context. Lua does not provide any such metamethod, for either tables or userdata, so any user-created singleton is automatically true in a Boolean context. This means that the idiom `if var then ...` no longer works, and an explicit (in)equality check against the singleton is required, creating uglier code. Perhaps what Lua *really* needs to solve this problem, then, is a __bool metamethod that, if present, is called when the object is used in a boolean context (such as `if`, `and`, and `or`) and returns either true or false, replacing the default true. Then one can take these `null` singletons and slap a `{__bool = function(t) return false end}` metatable on them, allowing them to gain the benefits of `nil` without the drawback of `nil`. What does everyone else think about a __bool metamethod?
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Roberto Ierusalimschy > On Jul 21, 2016, at 10:09 AM, Roberto Ierusalimschy <[hidden email]> wrote: > >> For my particular use case I made heavy use of "sparse things", from which >> Lua tables are awesome, where all computations are based on the keys, >> namely, with pairs() or indexing the relevant key. Also, I use "proper >> sequences" with Lua tables. In this case, table.pack and table.unpack are >> symmetric, and the relevant computation is given by the "order" of the >> keys, namely, with table.insert, table.remove, table.move etc that is by >> default, in this case, simple to understand. I never ever used a "Lua array >> with holes", where the ordering, or existence, of the keys is indeed >> relevant for computation. Precluding the use of __len or misbehaviour of  #. >> >> Could you help me giving a (real world) counterexample? > > The only real-world case that bothers me is when you want to collect all > the results from a function call, something like {f()}. If there is an > error, the standard protocol is to return (nil, msg), which would create > an array with holes. (Currently I think the correct way to fix this > would be to return (false, msg), not to change the behavior of lists.) > > -- Roberto > But that would still leave the general case of {f()} unsolved :( —Tim
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Jonathan Goble It was thus said that the Great Jonathan Goble once stated: > On Thu, Jul 21, 2016 at 6:08 PM, Luiz Henrique de Figueiredo > <[hidden email]> wrote: > >> The first use case that came to mind was JSON deserializers, where the > >> obvious mapping of null to nil causes problems with arrays. > > > > That's the whole point: one probably shouldn't not force the semantics > > of one language into another. In this case, and others, it's simpler > > and clearer to define > >         null = {} > > once and use it where needed. if you must skip over these when traversing > > a table, then that's what your semantics dictates and you must do it. > > Just do not try to do this with nil, which has its own, clear semantics > > in Lua. > > I think part of the reason people try to use Lua's `nil` in ways like > JSON's `null` or Python's `None` is because, in a context where you > know you won't get the Boolean `false`, `if var then ...` is an easy > way to check if `var` is a useful value or not, as `nil`, `null`, and > `None` all evaluate to false in a Boolean context.   In my CBOR [1] library [2], it returns nil for 'null' and 'undefined' by default, but that can be overridden by the user (in fact, just about everything can be overridden by the user) so to me, that's more a limitation of the JSON module than of actual problems with semantics.  If you care enough, you'll make the distinction. > The problem arises when one tries to create a singleton to represent > this lack of a useful value. Python provides the `__bool__` method (in > 3.x; `__nonzero__` in Python 2.x) to customize the behavior of an > object when used in a Boolean context. Lua does not provide any such > metamethod, for either tables or userdata, so any user-created > singleton is automatically true in a Boolean context. This means that > the idiom `if var then ...` no longer works, and an explicit > (in)equality check against the singleton is required, creating uglier > code. > > Perhaps what Lua *really* needs to solve this problem, then, is a > __bool metamethod that, if present, is called when the object is used > in a boolean context (such as `if`, `and`, and `or`) and returns > either true or false, replacing the default true. Then one can take > these `null` singletons and slap a `{__bool = function(t) return false > end}` metatable on them, allowing them to gain the benefits of `nil` > without the drawback of `nil`. > > What does everyone else think about a __bool metamethod?   A __boolean metamethod was mentioned?  Proposed?  I'm not sure, but in any case, one was mentioned here:         http://lua-users.org/lists/lua-l/2011-12/msg00621.htmlbut it seems like it was for something else.   -spc (Keeper of the metamethod proposals ... ) [1] Concise Binary Object Representation, RFC-7049 [2] https://github.com/spc476/CBOR
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Tim Hill On Fri, Jul 22, 2016 at 1:54 AM, Tim Hill <[hidden email]> wrote: > But that would still leave the general case of {f()} unsolved :( We always have table.pack(f()) ... Especially when pack/unpack become symmetrical.
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 table.pack is very good, but I use ".n" field for other purposes. Then, I'm (seldom) using something likepack_len = function(...)        local len = select('#',...)        return setmetatable({...},{                        __len = function() return len end,                        __newindex = function(t,k,v)                                if math.type(k) == 'integer' then                                        len = math.max(len,k)                                        rawset(t,k,v)                                end                        end                })endt = pack_len(nil,3,nil)print(#t, table.unpack(t))t[10] = 3.1415print(#t, table.unpack(t))t[3] = 5t[1] = 2t[2] = 7print(#t, table.unpack(t))t[#t+1] = 0.5772for i=1,#t do print(i,t[i]) endreturn3    nil    3    nil10    nil    3    nil    nil    nil    nil    nil    nil    nil    3.141510    2    7    5    nil    nil    nil    nil    nil    nil    3.14151    22    73    54    nil5    nil6    nil7    nil8    nil9    nil10    3.141511    0.5772as expected.This keep symmetry with table.pack, as well as others table.something, and give me an unbounded "sparse array" with well defined length.There is something wrong with this implementation?2016-07-22 4:06 GMT-03:00 steve donovan :On Fri, Jul 22, 2016 at 1:54 AM, Tim Hill <[hidden email]> wrote: > But that would still leave the general case of {f()} unsolved :( We always have table.pack(f()) ... Especially when pack/unpack become symmetrical. -- Rodrigo Azevedo Moreira da Silva
Open this post in threaded view
|

## Re: __bool / __false (was: Quest: real world "Lua array with holes" usage)

Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Rodrigo Azevedo Am 21.07.2016 17:13 schrieb "Rodrigo Azevedo" <[hidden email]>: > > [...] > > Could you help me giving a (real world) counterexample? No. :-) I've been bitten *once* by # doing something unexpected (and debugging for some hours then) but that was due to a logical error further down in the call stack. I tend to skip all the threads regarding the "issue" with a smile. Matthias PS. Sorry for this unhelpful reply, but I really like Lua the way it is...
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Jorge Visca On 21 July 2016 at 14:58, Jorge <[hidden email]> wrote: > > > On 21/07/16 14:09, Roberto Ierusalimschy wrote: >> >> The only real-world case that bothers me is when you want to collect all >> the results from a function call, something like {f()}. If there is an >> error, the standard protocol is to return (nil, msg), which would create an >> array with holes. > > Yes, that's the only way I get holes in arrays. OTOH, that's happening a lot > less since I decided to assert  all functions that can return nil,... unless > I specifically know what to do in that case. Anyway, it's pretty easy to > check ret[1]==nil to see if the call was successful. > >> (Currently I think the correct way to fix this would be to return (false, >> msg), not to change the behavior of lists.) -- Roberto > > I have doubts on this, because false is a reasonable value to get from a > function... Returning nil is clearer because the meaning is "could not > produce a value to return". Otherwise you end using a explicit return for > status pcall-like, all the time. I agree with Jorge. Also, sometimes when returning multiple things, one of these things might be nil, so it's not only the error situation that produces a nil value upon return, although it is of course the most common. So, unfortunately just changing the policy from nil-on-error to false-on-error still wouldn't allow us to just forget about table.pack(). -- Hisham
Open this post in threaded view
|

## Re: __bool / __false (was: Quest: real world "Lua array with holes" usage)

 In reply to this post by nobody On Jul 22, 2016 10:11 AM, "nobody" <[hidden email]> wrote: > >  - a boolean or nil: __false is true -> value is false >                      __false is false (or nil) -> value is true > (The true/false order was chosen such that you're not jumping around in > the truth table: (__false = nil) ~ (__false = false) whereas it would > probably be (__bool = nil) !~ (__bool = false).) While I understand the reasoning, the double negation here is confusing. I'd find __bool much easier to understand: if it's a boolean, that's the truth value; if it's a function, call it to determine the truth value; if it's nil, the standard logic applies (anything that isn't `false` or `nil` is true). If we were designing a new language, I'd be tempted to say it should raise an error when comparing a value which is not a boolean to one that is (where having __bool counts as being a boolean).
Open this post in threaded view
|

## Re: Quest: real world "Lua array with holes" usage

 In reply to this post by Hisham > I agree with Jorge. Also, sometimes when returning multiple things, > one of these things might be nil, so it's not only the error situation > that produces a nil value upon return, although it is of course the > most common. So, unfortunately just changing the policy from > nil-on-error to false-on-error still wouldn't allow us to just forget > about table.pack(). Of course any function can return nil among its returns, but maybe that might be considered a bad practice. My point is that, if nil is representing the absence of a value, it is weird not to have a third value but to have a fourth one. If nil is representing something else, probably it shouldn't be (as pointed out by others already). ("is weird" ~ "it might be considered weird") -- Roberto
Open this post in threaded view
|

## The Undefined Country (was Re: Quest: real world "Lua array with holes" usage)

 It was thus said that the Great Roberto Ierusalimschy once stated: > > I agree with Jorge. Also, sometimes when returning multiple things, > > one of these things might be nil, so it's not only the error situation > > that produces a nil value upon return, although it is of course the > > most common. So, unfortunately just changing the policy from > > nil-on-error to false-on-error still wouldn't allow us to just forget > > about table.pack(). > > Of course any function can return nil among its returns, but maybe > that might be considered a bad practice. > > My point is that, if nil is representing the absence of a value, it > is weird not to have a third value but to have a fourth one. If nil > is representing something else, probably it shouldn't be (as pointed > out by others already).   Javascript has an "undefined" state, in addition to a "null" state. Perhaps Lua should have one as well.  To me, "nil" means "the absence of a value" while "undef" means "this never had a value to begin with".   Proposal:  undef.  It works just like nil does now, but "nil" (the new nil behavior) is that it *is* allowed in sequences.           t = {}         x = t.one -- x is "undef"         local zork -- zork is "undef"         t = { 1 , 2 , nil , 4 }         y = #t -- 4, because there are four items in t         z = t[5] -- z is "undef", falls outside sequence         function foo(...)           local args = table.pack(...) -- no 'n' field to worry about           local a,b,c,d,e = table.unpack(t)           -- a is 1           -- b is 2           -- c is nil           -- d is 4           -- e is undef           if e then             print(e)           elseif e == nil then             print("e is nil")           elseif e == undef then             print("e never had a chance")           end         end   It seems, to me, to solve all the current issues Lua has with arrays.   -spc (There's probably stuff I'm missing though ... )
Open this post in threaded view
|

## Re: The Undefined Country (was Re: Quest: real world "Lua array with holes" usage)

 On Jul 24, 2016 3:27 PM, "Sean Conner" <[hidden email]> wrote: > > It was thus said that the Great Roberto Ierusalimschy once stated: > > > I agree with Jorge. Also, sometimes when returning multiple things, > > > one of these things might be nil, so it's not only the error situation > > > that produces a nil value upon return, although it is of course the > > > most common. So, unfortunately just changing the policy from > > > nil-on-error to false-on-error still wouldn't allow us to just forget > > > about table.pack(). > > > > Of course any function can return nil among its returns, but maybe > > that might be considered a bad practice. > > > > My point is that, if nil is representing the absence of a value, it > > is weird not to have a third value but to have a fourth one. If nil > > is representing something else, probably it shouldn't be (as pointed > > out by others already). > >   Javascript has an "undefined" state, in addition to a "null" state. > Perhaps Lua should have one as well.  To me, "nil" means "the absence of a > value" while "undef" means "this never had a value to begin with". > howsabout NaV (Not a Value) by analogy to NaN?  "Undefined value" seems kind of an oxymoron to me.  NaV could mean sth like "its a value, kinda sorta, but one that cannot be represented,  not even by nil".