[Proposal] ## operator

[Soni "They/Them" L.]

(IRC logs)

<SoniEx2> I wish Lua had a way to count keys in a table
<SoniEx2> so for example {1,2,3} would be 3, {[0]=0,1,2,3} would be 4, {A = 1, B = 2, C = 3} would be 3
<SoniEx2> etc
<SoniEx2> I can use __newindex to count key adds
<SoniEx2> but there's no __remindex or something
<SoniEx2> and it wouldn't help with what I wanna do
<SoniEx2> sure, I can use pairs() and a counter but that's slow :/
<kansuu> should be possible with proxy table?
<SoniEx2> kansuu, won't help when you're taking in random tables
<kansuu> nope, then you'd need to count the number of entries first anyway
<mniip> except the part where lua already knows the value internally
<SoniEx2> see? this is why lua needs to count when ppl set things and stuff
<kansuu> it doesn't _need_ to, it'd just be an optimization
<kansuu> for your specific use case
<SoniEx2> if setting to nil and it wasn't already nil, decrement, if setting to non-nil and it wasn't non-nil, increment
<SoniEx2> then provide a table.nkeys()
<SoniEx2> or a ## operator

[Marco Mastropaolo]

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

On Fri, Nov 21, 2014 at 4:23 PM, Thiago L. <[hidden email]> wrote:

(IRC logs)

<SoniEx2> I wish Lua had a way to count keys in a table
<SoniEx2> so for example {1,2,3} would be 3, {[0]=0,1,2,3} would be 4, {A = 1, B = 2, C = 3} would be 3
<SoniEx2> etc
<SoniEx2> I can use __newindex to count key adds
<SoniEx2> but there's no __remindex or something
<SoniEx2> and it wouldn't help with what I wanna do
<SoniEx2> sure, I can use pairs() and a counter but that's slow :/
<kansuu> should be possible with proxy table?
<SoniEx2> kansuu, won't help when you're taking in random tables
<kansuu> nope, then you'd need to count the number of entries first anyway
<mniip> except the part where lua already knows the value internally
<SoniEx2> see? this is why lua needs to count when ppl set things and stuff
<kansuu> it doesn't _need_ to, it'd just be an optimization
<kansuu> for your specific use case
<SoniEx2> if setting to nil and it wasn't already nil, decrement, if setting to non-nil and it wasn't non-nil, increment
<SoniEx2> then provide a table.nkeys()
<SoniEx2> or a ## operator

[Soni "They/Them" L.]

On 21/11/14 01:37 PM, Marco Mastropaolo wrote:

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

On Fri, Nov 21, 2014 at 4:23 PM, Thiago L. <[hidden email]> wrote:

(IRC logs)

<SoniEx2> I wish Lua had a way to count keys in a table
<SoniEx2> so for example {1,2,3} would be 3, {[0]=0,1,2,3} would be 4, {A = 1, B = 2, C = 3} would be 3
<SoniEx2> etc
<SoniEx2> I can use __newindex to count key adds
<SoniEx2> but there's no __remindex or something
<SoniEx2> and it wouldn't help with what I wanna do
<SoniEx2> sure, I can use pairs() and a counter but that's slow :/
<kansuu> should be possible with proxy table?
<SoniEx2> kansuu, won't help when you're taking in random tables
<kansuu> nope, then you'd need to count the number of entries first anyway
<mniip> except the part where lua already knows the value internally
<SoniEx2> see? this is why lua needs to count when ppl set things and stuff
<kansuu> it doesn't _need_ to, it'd just be an optimization
<kansuu> for your specific use case
<SoniEx2> if setting to nil and it wasn't already nil, decrement, if setting to non-nil and it wasn't non-nil, increment
<SoniEx2> then provide a table.nkeys()
<SoniEx2> or a ## operator

Hmm... then a ! operator?

[Marco Mastropaolo]

Sure - it wasn't a critique to the request itself, just at the proposed implementation.

I personally think it's a good request as I think it covers needs when tables are used as sets or maps (specially sets).

Honestly, if I was to choose, I'd go with the table.keycount() method instead of an operator (and maybe adding a bunch of methods operating on sets/maps as they are quite common, like union() or intersection()).

My 2 cents,

-- Marco

On Fri, Nov 21, 2014 at 4:40 PM, Thiago L. <[hidden email]> wrote:

On 21/11/14 01:37 PM, Marco Mastropaolo wrote:

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

On Fri, Nov 21, 2014 at 4:23 PM, Thiago L. <[hidden email]> wrote:

(IRC logs)

<SoniEx2> I wish Lua had a way to count keys in a table
<SoniEx2> so for example {1,2,3} would be 3, {[0]=0,1,2,3} would be 4, {A = 1, B = 2, C = 3} would be 3
<SoniEx2> etc
<SoniEx2> I can use __newindex to count key adds
<SoniEx2> but there's no __remindex or something
<SoniEx2> and it wouldn't help with what I wanna do
<SoniEx2> sure, I can use pairs() and a counter but that's slow :/
<kansuu> should be possible with proxy table?
<SoniEx2> kansuu, won't help when you're taking in random tables
<kansuu> nope, then you'd need to count the number of entries first anyway
<mniip> except the part where lua already knows the value internally
<SoniEx2> see? this is why lua needs to count when ppl set things and stuff
<kansuu> it doesn't _need_ to, it'd just be an optimization
<kansuu> for your specific use case
<SoniEx2> if setting to nil and it wasn't already nil, decrement, if setting to non-nil and it wasn't non-nil, increment
<SoniEx2> then provide a table.nkeys()
<SoniEx2> or a ## operator

Hmm... then a ! operator?

[Andrew Starks]

On Fri, Nov 21, 2014 at 9:40 AM, Thiago L. <[hidden email]> wrote:

On 21/11/14 01:37 PM, Marco Mastropaolo wrote:

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

On Fri, Nov 21, 2014 at 4:23 PM, Thiago L. <[hidden email]> wrote:

(IRC logs)

<SoniEx2> I wish Lua had a way to count keys in a table
<SoniEx2> so for example {1,2,3} would be 3, {[0]=0,1,2,3} would be 4, {A = 1, B = 2, C = 3} would be 3
<SoniEx2> etc
<SoniEx2> I can use __newindex to count key adds
<SoniEx2> but there's no __remindex or something
<SoniEx2> and it wouldn't help with what I wanna do
<SoniEx2> sure, I can use pairs() and a counter but that's slow :/
<kansuu> should be possible with proxy table?
<SoniEx2> kansuu, won't help when you're taking in random tables
<kansuu> nope, then you'd need to count the number of entries first anyway
<mniip> except the part where lua already knows the value internally
<SoniEx2> see? this is why lua needs to count when ppl set things and stuff
<kansuu> it doesn't _need_ to, it'd just be an optimization
<kansuu> for your specific use case
<SoniEx2> if setting to nil and it wasn't already nil, decrement, if setting to non-nil and it wasn't non-nil, increment
<SoniEx2> then provide a table.nkeys()
<SoniEx2> or a ## operator

Hmm... then a ! operator?

I have to admit to wanting something along these lines from time to time. Usually, I want to know the number of non-numeric keys, but knowing the total count is probably better.

I think I remember someone saying that the trouble with this is that it's not trivial to implement it in a way that wasn't the same speed as a while loop. Maybe weak values had something to do with it?

--Andrew

[Luiz Henrique de Figueiredo]

> <SoniEx2> I wish Lua had a way to count keys in a table

This question often appears here and in
http://stackoverflow.com/questions/tagged/lua and it always strikes me:
why would you need to count entries in a table? It seems to me that
this is rarely required, and that it is probably an instance of
The XY Problem <http://xyproblem.info/> and trying to use in Lua using
paradigms from other languages. Perhaps I'm missing something.

This requests also springs from a misconception that Lua already
stores this value internally anyway, which it doesn't.

If you really need to count entries in a table you can use:

function number_of_entries(t)
        local n=0
        local k=nil
        while true do
                k=next(t,k)
                if k==nil then return n end
                n=n+1
        end
end

But there is really no need for an *operator* for that.

[Soni "They/Them" L.]

On 21/11/14 01:58 PM, Luiz Henrique de Figueiredo wrote:

>> <SoniEx2> I wish Lua had a way to count keys in a table
> This question often appears here and in
> http://stackoverflow.com/questions/tagged/lua and it always strikes me:
> why would you need to count entries in a table? It seems to me that
> this is rarely required, and that it is probably an instance of
> The XY Problem <http://xyproblem.info/> and trying to use in Lua using
> paradigms from other languages. Perhaps I'm missing something.
>
> This requests also springs from a misconception that Lua already
> stores this value internally anyway, which it doesn't.
>
> If you really need to count entries in a table you can use:
>
> function number_of_entries(t)
> local n=0
> local k=nil
> while true do
> k=next(t,k)
> if k==nil then return n end
> n=n+1
> end
> end
>
> But there is really no need for an *operator* for that.
>

Well say you have 2 functions, one for heavy tables and another for
light tables, and you wanna pick the right one for the right job. The
current way is to use:

function nkeys(t)
   local c = 0
   for x in next, t do
     c = c + 1
   end
   return c
end

But if you're the Lua internals, you could instead do this:

*something somewhere calls rawset*
1. if key exists and value is nil, then decrement
2. else if key doesn't exist and value is not nil, then increment
*do normal rawset stuff*

*GC collection of weak keys/values*
*GC detects a weak key/value to destroy*
1. call rawset with key and nil
*GC does normal GC stuff*

Which would be much more efficient.

[Axel Kittenberger]

it always strikes me why would you need to count entries in a table? It seems to me that
this is rarely required, and that it is probably an instance of
The XY Problem <http://xyproblem.info/> and trying to use in Lua using
paradigms from other languages. Perhaps I'm missing something.

 

people are working in different fields and encounter different problems. In my opinion any "this isn't often required" argument is at most ignorance.

I had this issue also, for me it was simply keeping track of the count using a metatable.

This requests also springs from a misconception that Lua already

stores this value internally anyway, which it doesn't.

This is the good answer, making a native implementation is not going into a much faster thing than coding it in Lua. Otherwise if one is worried much of this in an inner perfomance critical loop. One can just copy/paste numusearray() and numusehash() from ltable.c into  a seperate c file (they are declared static there), count them together and export that function as a lua library call. Easy extension without need to change the core.

Kind regards, Axel

[Axel Kittenberger]

PS: I'd be interested in an experimental perfomance comparison between the native numuse*() calls and the next() loop in Lua. But not interested enough to do it myself :-)

[Roberto Ierusalimschy]

> In my opinion any "this isn't often required" argument is at most
> ignorance.

And at least knowledge?

-- Roberto

[Axel Kittenberger]

You're knowledge about other peoples fields and the issues they are regulary facing?

Anyway, I agree such operator is not a good idea, simply because the count isn't stored in the core. If it would, I'd say instead by all means make that integer accessible to the script, no big harm.

However, the resoning ought to be, what is a strong concept and well doable with the current code base without bloating it? In this case it is not.

Not, "I don't need that in my field of work, so I just suppose other don't as well." Or do you have a scientific method on operationalising how often other people need this or that with their way of using Lua?

On Fri, Nov 21, 2014 at 5:47 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> In my opinion any "this isn't often required" argument is at most
> ignorance.

And at least knowledge?

-- Roberto

[Andrew Starks]

On Fri, Nov 21, 2014 at 11:02 AM, Axel Kittenberger <[hidden email]> wrote:

You're knowledge about other peoples fields and the issues they are regulary facing?

Anyway, I agree such operator is not a good idea, simply because the count isn't stored in the core. If it would, I'd say instead by all means make that integer accessible to the script, no big harm.

However, the resoning ought to be, what is a strong concept and well doable with the current code base without bloating it? In this case it is not.

Not, "I don't need that in my field of work, so I just suppose other don't as well." Or do you have a scientific method on operationalising how often other people need this or that with their way of using Lua?

On Fri, Nov 21, 2014 at 5:47 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> In my opinion any "this isn't often required" argument is at most
> ignorance.

And at least knowledge?

-- Roberto

Alex,

I read the comment in the same way that I did a similar statement made in Programming in Lua, which I believe was regarding Linked Lists.

    An iterator to traverse a linked list is another interesting example of a

    stateless iterator. (As we already mentioned, linked lists are not frequent in

    Lua, but sometimes we need them.)

Which is not to say that getting the count is not necessary, only that there are other ways of modeling the problem that Lua is better suited for.

Is that different than how you understood it?

-Andrew

[Tony Papadimitriou]

I didn’t know you could do ##, but then again I didn’t think about it until you mentioned it.  I actually use the length of a number (as string) quite often.

 

So, being shorter than what I currently do, I tried ## and I have to ask (not you necessarily, whoever knows), why an error?

 

x = '1234567890'
print(#x)
print(#(#x..''))    -- this works (prints 2, the length of 10 as string)
print(##x)          -- this fails "attempt to get length of a number value"

If numbers and strings can be mixed in Lua (as in the example that works above), why is ##a an error if x is a number, and Lua does not implicitly convert the number to a string to have its length taken, rather than give an error?  The length of a number is a meaningful thing.  Of course one could say it should return the length as the number of bytes required to stored the number but this is the less likely interpretation one would expect.

 

TIA

 

From: [hidden email]

Sent: Friday, November 21, 2014 5:37 PM

To: [hidden email]

Subject: Re: [Proposal] ## operator

 

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

 

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

 

[Soni "They/Them" L.]

On 21/11/14 04:57 PM, [hidden email] wrote:

I didn’t know you could do ##, but then again I didn’t think about it until you mentioned it.  I actually use the length of a number (as string) quite often.

 

So, being shorter than what I currently do, I tried ## and I have to ask (not you necessarily, whoever knows), why an error?

 

x = '1234567890'
print(#x)
print(#(#x..''))    -- this works (prints 2, the length of 10 as string)
print(##x)          -- this fails "attempt to get length of a number value"

If numbers and strings can be mixed in Lua (as in the example that works above), why is ##a an error if x is a number, and Lua does not implicitly convert the number to a string to have its length taken, rather than give an error?  The length of a number is a meaningful thing.  Of course one could say it should return the length as the number of bytes required to stored the number but this is the less likely interpretation one would expect.

 

TIA

 

From: [hidden email]

Sent: Friday, November 21, 2014 5:37 PM

To: [hidden email]

Subject: Re: [Proposal] ## operator

 

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

 

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

 

Because # is not a string method.

[Axel Kittenberger]

print(#(#x..''))    -- this works (prints 2, the length of 10 as string)
print(##x)          -- this fails "attempt to get length of a number 

..'' is what you're missing in the second line. In the prior you are turning the number to a string with decimal encoding and counting then its length.

print(#(#x)) will not work either out of the same reason

[Steven Degutis]

Quoth SoniEx2:
> Well say you have

What is the actual, root, core, user-facing problem you're trying to solve?

[Soni "They/Them" L.]

On 21/11/14 05:24 PM, Steven Degutis wrote:
> Quoth SoniEx2:
>> Well say you have
> What is the actual, root, core, user-facing problem you're trying to solve?
>
I have 2 functions, one for heavy tables, another for light tables, I
wanna be able to merge both together for (memory) efficiency.

--
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.

[Tony Papadimitriou]

Well, # is the length operator and it’s meant to return the length of whatever follows, provided what follows has a concept of length/size (e.g., table, string), so one -- I being that one, I guess :) -- would expect if a number follows it would return its length (of its common printable representation), rather than give error.  In other words, I would expect to #number to be equivalent to #(#number..’’)

In any case, in a ‘loosely typed language’ giving an error when there is a non-error alternative behavior does not serve any useful purpose, does it?

 

Obviously, it’s not a big issue as it’s easy to overcome, but it seems (to me, at least) not the best possible behavior for #number

 

(Another thing that I would expect is the IN operator to work like in Python when used with strings.  But that’s another discussion...)

 

From: [hidden email]

Sent: Friday, November 21, 2014 9:04 PM

To: [hidden email]

Subject: Re: ## operator (error)

 

 

On 21/11/14 04:57 PM, [hidden email] wrote:

I didn’t know you could do ##, but then again I didn’t think about it until you mentioned it.  I actually use the length of a number (as string) quite often.

 

So, being shorter than what I currently do, I tried ## and I have to ask (not you necessarily, whoever knows), why an error?

 

x = '1234567890'
print(#x)
print(#(#x..''))    -- this works (prints 2, the length of 10 as string)
print(##x)          -- this fails "attempt to get length of a number value"

If numbers and strings can be mixed in Lua (as in the example that works above), why is ##a an error if x is a number, and Lua does not implicitly convert the number to a string to have its length taken, rather than give an error?  The length of a number is a meaningful thing.  Of course one could say it should return the length as the number of bytes required to stored the number but this is the less likely interpretation one would expect.

 

TIA

 

From: [hidden email]

Sent: Friday, November 21, 2014 5:37 PM

To: [hidden email]

Subject: Re: [Proposal] ## operator

 

Probably a scenario which is totally unlikely and broken on so many other aspects.. but this code

t = { 1, 2, 3 };
setmetatable(t, { __len = function(tbl) return tostring(rawlen(tbl)); end });
x = ##t;

-- x = 1

 

actually works and would break if ## would be implemented.

As I said, totally unlikely and super-dirty to starts of, but at the moment the parser treats (correctly) ## as two executions of the # operator and if a __len meta returns a string or table, it's actually "working" code (for some definition of working).

-- Marco

 

Because # is not a string method.

[Steven Degutis]

>> Quoth SoniEx2:
>> What is the actual, root, core, user-facing problem you're trying to
>> solve?
>>
> I have 2 functions, one for heavy tables, another for light tables, I wanna
> be able to merge both together for (memory) efficiency.

If your use-case values memory efficiency more than CPU efficiency,
then why not just count the keys using pairs() and an increment
variable, and get the count that way? You could put this into its own
function and even store that as `table.nkeys` if you'd like.

[Dirk Laurie-2]

2014-11-21 21:37 GMT+02:00  <[hidden email]>:
> Well, # is the length operator and it’s meant to return the length of
> whatever follows, provided what follows has a concept of length/size (e.g.,
> table, string), so one -- I being that one, I guess :) -- would expect if a
> number follows it would return its length (of its common printable
> representation), rather than give error.  In other words, I would expect to
> #number to be equivalent to #(#number..’’)

Nothing stops you from saying

> debug.setmetatable(1,{__len=function(n) return #tostring(n) end})
> #math.pi
15

> (Another thing that I would expect is the IN operator to work like in Python
> when used with strings.  But that’s another discussion...)

`in` is not an operator any more than `end` is. It is a token in the syntax
of the generic `for`.