# Lua 5.2 Length Operator and tables (bug?)

76 messages
1234
Open this post in threaded view
|

## Lua 5.2 Length Operator and tables (bug?)

 Hello, I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6> t = {10, 20, 30, nil, 50} > print(#t) 5 > t["x"] = nil > print(#t) 3 > t = {10, 20, 30, nil, 50} > print(#t) 5 > t["x"] = "hello" > print(#t) 3 > t = {10, 20, nil, 40} > print(#t) 4 > t["x"] = nil > print(#t) 4 > t = {10, 20, nil, 40, 50} > print(#t) 5 > t["x"] = nil > print(#t) 5 > t = {[1]=10,[2]=20,[3]=30,[4]=nil,[5]=50} > print(#t) 3 > t = {[1]=10,[2]=20,[3]=nil,[4]=40,[5]=50} > print(#t) 5 My point in this is that if #t returns a particular value n, it becomes surprising if each key of {1..n} does not index a non nil value in the table.  But in any case, I'd expect defined behavior rather than undefined behavior.  I can not wrap my head around how the above behavior might be defined. Smells like a bug to me. The other bit of strangeness represented above of setting a value of non numerical index having interaction with the length operator seems to go against documentation also, but that is likely just some internal implementation details leaking out by the existence of the previous issue. For a further (related?) look try variations of print(table.unpack(table.pack(a,b[..,z]))) where some of the parameters are nil.  Sometimes the unpack list contains the same nil parameters, others not. chris
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 Op 16 april 2012 10:31 heeft  <[hidden email]> het volgende geschreven: > > I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. > > Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6> This subject has been discussed to exhaustion many times.  Please visit    http://lua-users.org/lists/lua-l/and search for the topic "Possible bug related to table sizes".
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Dirk Laurie-2 On Apr 16, 2012, at 1:58 AM, Dirk Laurie wrote: > Op 16 april 2012 10:31 heeft  <[hidden email]> het volgende geschreven: > >> >> I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. >> >> Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6>> > > This subject has been discussed to exhaustion many times.  Please visit >   http://lua-users.org/lists/lua-l/> and search for the topic "Possible bug related to table sizes". > This is a subtle point.  I see that: function p() for i,v in ipairs(a) do print (i,v) end end > a ={1,2,3,4} > p() 1 1 2 2 3 3 4 4 > a[3]=nil > =#a 4 > p() 1 1 2 2 > From the manual: The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n CAN be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t CAN be any of the indices that directly precedes a nil value (that is, it may consider any such nil value as the end of the array)" With emphasis on the CAN added.   My interpretation is that this is a reflection of the Lua parser's method of storing a table internally as two objects; a regular array of some optimum size that is a power of two and some sort of hashed list.  Am I correct that # is the size of the array part of a table?  Clearly it cannot be used to figure out what ipairs might do.   Is there any way other than probing as Carl did to find out how Lua has organized a table or even to predict what ipairs will do?  Perhaps the message is dont ask because the parser is free to reorganize a table whenever it wants which suggests that # has very limited usefulness.  It seems that if a table has valid  elements from index 1 to n # will return n but the reverse is not true so # cannot be used to learn if a table includes a fully populated array. Jose Torre-Bueno [hidden email]
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 On 16 April 2012 11:35, Jose Torre-Bueno <[hidden email]> wrote: > > Is there any way other than probing as Carl did to find out how Lua has organized a table or even to predict what ipairs will do? > >  Perhaps the message is dont ask because the parser is free to reorganize a table whenever it wants which suggests that # has very limited usefulness.  It seems that if a table has valid  elements from index 1 to n # will return n but the reverse is not true so # cannot be used to learn if a table includes a fully populated array. Attempting to define explicitly undefined behaviour is just not a good idea. It will easily lead you down the road of portability and compatibility problems. Regards, Matthew
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Chris Rempel On Apr 16, 2012, at 10:31 AM, [hidden email] wrote: > For a further (related?) look try variations of print(table.unpack(table.pack(a,b[..,z]))) where some of the parameters are nil.  Sometimes the unpack list contains the same nil parameters, others not. Perhaps worthwhile mentioning that pack/unpack are mostly useful in terms of varargs. table.pack holds an additional 'n' field, to keep track of how many effective elements there is: http://www.lua.org/manual/5.2/manual.html#pdf-table.packtable.pack is equivalent to the following Lua code: local function pack( ... )   return { n = select( '#', ... ), ... } end print( pack( nil, nil, nil ).n ) > 3 To properly unpack a, hmmm, packed table, you need to explicitly specify it's length: local varargs = table.pack( nil, nil, nil ) print( #varargs ) print( varargs.n ) print( table.unpack( varargs, 1, varargs.n ) ) > 0 > 3 > nil nil nil
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Jose Torre-Bueno I was wondering what is the advantage in not enforcing a table without an array part to return a len of 0. Honest question, not a critique. Same thing for why 0 and negative numbers  (and numbers with fractional part as well) could change the behaviour of len. I would have found more intuitive considering only strictly positive integers, i.e. reason only on 1...n. On Apr 16, 2012 11:36 AM, "Jose Torre-Bueno" <[hidden email]> wrote: On Apr 16, 2012, at 1:58 AM, Dirk Laurie wrote: > Op 16 april 2012 10:31 heeft  <[hidden email]> het volgende geschreven: > >> >> I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. >> >> Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6 >> > > This subject has been discussed to exhaustion many times.  Please visit >   http://lua-users.org/lists/lua-l/ > and search for the topic "Possible bug related to table sizes". > This is a subtle point.  I see that: function p() for i,v in ipairs(a) do print (i,v) end end > a ={1,2,3,4} > p() 1       1 2       2 3       3 4       4 > a[3]=nil > =#a 4 > p() 1       1 2       2 > >From the manual: The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n CAN be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t CAN be any of the indices that directly precedes a nil value (that is, it may consider any such nil value as the end of the array)" With emphasis on the CAN added. My interpretation is that this is a reflection of the Lua parser's method of storing a table internally as two objects; a regular array of some optimum size that is a power of two and some sort of hashed list.  Am I correct that # is the size of the array part of a table?  Clearly it cannot be used to figure out what ipairs might do. Is there any way other than probing as Carl did to find out how Lua has organized a table or even to predict what ipairs will do?  Perhaps the message is dont ask because the parser is free to reorganize a table whenever it wants which suggests that # has very limited usefulness.  It seems that if a table has valid  elements from index 1 to n # will return n but the reverse is not true so # cannot be used to learn if a table includes a fully populated array. Jose Torre-Bueno [hidden email]
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 Sorry about the noise on the second part of my post, at least for 5.2 (but in 5.1 there is no mention of positive integers) On Apr 16, 2012 12:27 PM, "Krunal Rao" <[hidden email]> wrote: I was wondering what is the advantage in not enforcing a table without an array part to return a len of 0. Honest question, not a critique. Same thing for why 0 and negative numbers  (and numbers with fractional part as well) could change the behaviour of len. I would have found more intuitive considering only strictly positive integers, i.e. reason only on 1...n. On Apr 16, 2012 11:36 AM, "Jose Torre-Bueno" <[hidden email]> wrote: On Apr 16, 2012, at 1:58 AM, Dirk Laurie wrote: > Op 16 april 2012 10:31 heeft  <[hidden email]> het volgende geschreven: > >> >> I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. >> >> Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6 >> > > This subject has been discussed to exhaustion many times.  Please visit >   http://lua-users.org/lists/lua-l/ > and search for the topic "Possible bug related to table sizes". > This is a subtle point.  I see that: function p() for i,v in ipairs(a) do print (i,v) end end > a ={1,2,3,4} > p() 1       1 2       2 3       3 4       4 > a[3]=nil > =#a 4 > p() 1       1 2       2 > >From the manual: The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n CAN be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t CAN be any of the indices that directly precedes a nil value (that is, it may consider any such nil value as the end of the array)" With emphasis on the CAN added. My interpretation is that this is a reflection of the Lua parser's method of storing a table internally as two objects; a regular array of some optimum size that is a power of two and some sort of hashed list.  Am I correct that # is the size of the array part of a table?  Clearly it cannot be used to figure out what ipairs might do. Is there any way other than probing as Carl did to find out how Lua has organized a table or even to predict what ipairs will do?  Perhaps the message is dont ask because the parser is free to reorganize a table whenever it wants which suggests that # has very limited usefulness.  It seems that if a table has valid  elements from index 1 to n # will return n but the reverse is not true so # cannot be used to learn if a table includes a fully populated array. Jose Torre-Bueno [hidden email]
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Krunal Rao > I was wondering what is the advantage in not enforcing a table without an > array part to return a len of 0. The reason why # does not return 0 or raise an error for tables that are not sequences is that checking whether this holds can be expensive and so # does something even for tables that are not sequences, though the returned value is not useful. Even if a table is a sequence, part of it may be stored in the hash part and so the naive idea of maintaining the size of the array part does not work (and this book keeping would be expensive as well). All this has been discussed before at length :-)
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 Thanks and sorry, I will research better next time :p On Apr 16, 2012 12:48 PM, "Luiz Henrique de Figueiredo" <[hidden email]> wrote: > I was wondering what is the advantage in not enforcing a table without an > array part to return a len of 0. The reason why # does not return 0 or raise an error for tables that are not sequences is that checking whether this holds can be expensive and so # does something even for tables that are not sequences, though the returned value is not useful. Even if a table is a sequence, part of it may be stored in the hash part and so the naive idea of maintaining the size of the array part does not work (and this book keeping would be expensive as well). All this has been discussed before at length :-)
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Dirk Laurie-2 Dirk Laurie gmail.com> writes: > > Op 16 april 2012 10:31 heeft   gmx.com> het volgende geschreven: > > > > > I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. > > > > Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6> > > > This subject has been discussed to exhaustion many times.  Please visit >    http://lua-users.org/lists/lua-l/> and search for the topic "Possible bug related to table sizes". Thanks for the response.  I had searched the list archives, perhaps with the wrong terms as the only thread I came upon that seemed relevant was the thread discussion the documentation of this section for 5.2 which wasn't very enlightening.  Although now that I've read the follow up posts here, it clarifies the finer points that the 5.2 documentation is trying to dance around. In any case, I did as you suggested: http://lua-users.org/cgi-bin/namazu.cgi? query=Possible+bug+related+to+table+sizes&idxname=lua-l which resulted in no matches. However, using other search terms based on this thread's discussion I have now found many related posts.  Thanks. chris
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Jose Torre-Bueno Jose Torre-Bueno cox.net> writes: > On Apr 16, 2012, at 1:58 AM, Dirk Laurie wrote: > > Op 16 april 2012 10:31 heeft   gmx.com> het volgende geschreven: > >> I'm trying to reconcile documentation to the surprising behavior of the length operator on a table. > >> > >> Please refer to http://www.lua.org/manual/5.2/manual.html#3.4.6> >> > From the manual: > The length of a table t is defined to be any integer index n such that > t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n CAN be > zero. For a regular array, with non-nil values from 1 to a given n, > its length is exactly that n, the index of its last value. If the > array has "holes" (that is, nil values between other non-nil values), > then #t CAN be any of the indices that directly precedes a nil value > (that is, it may consider any such nil value as the end of the array)" You quoted the 5.1 manual, not 5.2.  However, it does seem that the 5.1 manual text is somewhat clearer than 5.2, if "CAN" were to be elaborated on. chris
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Luiz Henrique de Figueiredo Luiz Henrique de Figueiredo tecgraf.puc-rio.br> writes: > > > I was wondering what is the advantage in not enforcing a table without an > > array part to return a len of 0. > > The reason why # does not return 0 or raise an error for tables that are > not sequences is that checking whether this holds can be expensive and > so # does something even for tables that are not sequences, though the > returned value is not useful. > > Even if a table is a sequence, part of it may be stored in the hash part > and so the naive idea of maintaining the size of the array part does not > work (and this book keeping would be expensive as well). > > All this has been discussed before at length I would be interested to understand the intended purpose of the length operator for tables then.  As I see it, you have to know that the table is a sequence in order to find the length operator useful. If you know it is a sequence, that means you could have already tracked the length manually (which is the suggested course of action, according to the luafaq.org, when you know it is *not* a sequence). As the table.* api appear to be useful for working on tables made of sequences, I would venture to pose that a table.len() function be more appropriate exposure of this functionality, and less ambiguous of purpose. chris
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Matthew Wild Matthew Wild gmail.com> writes: > Attempting to define explicitly undefined behaviour is just not a good > idea. It will easily lead you down the road of portability and > compatibility problems. Could someone state what the *defined* behavior is? The best I can come up with is: The length operator, when used on a table, will only provide the length of the table, if and only if, the table's only positive numeric indices's are a sequence of {1..n}.  Non numeric indices's may also exist.  The existence of any other numeric indices's invalidate the result of the length operator.  As such you must know before hand whether or not a table conforms to these constraints in order to make valid use of the length operator on a table. Can anyone add to or improve upon that? Thanks, chris
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Chris Rempel chris <[hidden email]> writes: > I would be interested to understand the intended purpose of the > length operator for tables then.  As I see it, you have to know that > the table is a sequence in order to find the length operator useful. Yes (but of course, it's the usual case that one does know how one's data structures are structured ... :) > If you know it is a sequence, that means you could have already > tracked the length manually Sure, you _could_, but why would you want to, when the # operator does the right thing, and is more convenient (and in many cases more efficient)? -miles -- "Suppose He doesn't give a shit?  Suppose there is a God but He just doesn't give a shit?"  [George Carlin]
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Chris Rempel > :)  Thanks for attempting to clarify it, as for me "clearly" was not so clear. > To me the documentation stated that if the list was not a sequence that the > length of the table is not defined, meaning that it is nil.  As that is not what > my testing showed me, instead it showed rather aberrant behavior which caused > confusion.  Slightly better wording would have been to say that in such a case > the result of the length operator is 'undefined'. In spec jargon, that's exactly what "undefined" means. If it returned nil, that would be a DEFINED result. /s/ Adam
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 In reply to this post by Chris Rempel On 17 April 2012 02:29, chris <[hidden email]> wrote: > Matthew Wild gmail.com> writes: >> On 16 April 2012 09:31,   gmx.com> wrote: >> > My point in this is that if #t returns a particular value n, it becomes >> > surprising if each key of {1..n} does not index a non nil value in the >> > table. >> >> It might be surprising, until you read the documentation. > > Reading the documentation obviously did not clarify it for me, hence my post. :( > >> > But in any case, I'd expect defined behavior rather than undefined >> > behavior.  I can not wrap my head around how the above behavior might >> > be defined. >> >> Re-read the first sentence of the documentation you linked, especially >> the words "is only defined if". The documentation clearly says that >> the length of non-sequence tables is *not* defined. That means you >> cannot rely on it to give any particular value. It's not defined to be >> random either though, so don't use it for a random number generator :) > > :)  Thanks for attempting to clarify it, as for me "clearly" was not so clear. > To me the documentation stated that if the list was not a sequence that the > length of the table is not defined, meaning that it is nil.  As that is not what > my testing showed me, instead it showed rather aberrant behavior which caused > confusion.  Slightly better wording would have been to say that in such a case > the result of the length operator is 'undefined'. Aha! Now I understand your issue. I can see how you might read the text that way. It's tricky to re-word the documentation, as I think "is undefined if" could cause confusion just as easily as "is only defined if". I think it would say "returns nil" if it really returned nil, but saying that it can return any value isn't so easy. Regards, Matthew
Open this post in threaded view
|

## Re: Lua 5.2 Length Operator and tables (bug?)

 At 06:44 PM 4/16/2012, Matthew Wild wrote:  >On 17 April 2012 02:29, chris <[hidden email]> wrote:  >> Matthew Wild gmail.com> writes:  >>> On 16 April 2012 09:31, gmx.com> wrote:  >>> > My point in this is that if #t returns a particular value n,  >>> > it becomes surprising if each key of {1..n} does not index a  >>> > non nil value in the table.  >>>  >>> It might be surprising, until you read the documentation.  >>  >> Reading the documentation obviously did not clarify it for me,  >> hence my post. :(  >>  >> ....  >> To me the documentation stated that if the list was not a  >> sequence that the length of the table is not defined, meaning  >> that it is nil. As that is not what my testing showed me, instead  >> it showed rather aberrant behavior which caused confusion. Slightly  >> better wording would have been to say that in such a case the result  >> of the length operator is 'undefined'.  >  >Aha! Now I understand your issue. I can see how you might read the  >text that way. At least it is defined in one sense: You can be confident that the expression (t[#t+1] == nil) is true for all tables t, regardless of whether t holds a valid sequence or not. Note that you cannot assume that (t[#t] ~= nil) holds, because if t holds an empty sequence, #t is expected to be zero. Ross Berteig                               [hidden email] Cheshire Engineering Corp.           http://www.CheshireEng.com/