Some conservative Lua metamethod extension proposals

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Some conservative Lua metamethod extension proposals

Sergey Zakharchenko
Hello list,

I have some half-baked ideas on conservative but potentially useful
adjustments to metamethod protocols. No Lua implementation code
changes required (just yet). It's all about convention. "We cannot
wait for Lua creators' favour, our aim is to help ourselves" (slightly
evil grin).

* Keys-only __pairs semantics.

If __pairs is invoked wth a second argument equal to the special value
false, the caller is not interested in values (only in keys), so the
returned iterator may returned any value, including nil, or no value
at all, for them.

* Key-adjusting __newindex semantics.

If __newindex is invoked with a special fourth argument (what kind of?
equal to false?), then the metamethod is expected (but not required)
to perform insertion (a la table.insert) if the vale is non-nil, or
removal (a la table.remove) if the value is nil. This may or may not
include:
- adjusting indices of other values;
- raising an error if adjustment is not possible (e.g. the key is a
string and already used) or there is no such element to remove;
- supporting nil as key;
- returning the (actual) key used.

Tell me if you find these interesting and worth developing further.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Dirk Laurie-2
2018-03-23 10:05 GMT+02:00 Sergey Zakharchenko <[hidden email]>:

>
> Hello list,
>
> I have some half-baked ideas on conservative but potentially useful
> adjustments to metamethod protocols. No Lua implementation code
> changes required (just yet). It's all about convention. "We cannot
> wait for Lua creators' favour, our aim is to help ourselves" (slightly
> evil grin).
>
> * Keys-only __pairs semantics.
>
> If __pairs is invoked wth a second argument equal to the special value
> false, the caller is not interested in values (only in keys), so the
> returned iterator may returned any value, including nil, or no value
> at all, for them.

You can already write "for k in pairs(tbl,false) do ... end".

> * Key-adjusting __newindex semantics.
>
> If __newindex is invoked with a special fourth argument (what kind of?
> equal to false?), then the metamethod is expected (but not required)
> to perform insertion (a la table.insert) if the vale is non-nil, or
> removal (a la table.remove) if the value is nil. This may or may not
> include:
> - adjusting indices of other values;
> - raising an error if adjustment is not possible (e.g. the key is a
> string and already used) or there is no such element to remove;
> - supporting nil as key;
> - returning the (actual) key used.

I can't figure out what the idea is. Give an example.

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Sergey Zakharchenko
Dirk,

>> If __pairs is invoked wth a second argument equal to the special value
>> false, the caller is not interested in values (only in keys), so the
>> returned iterator may returned any value, including nil, or no value
>> at all, for them.
>
> You can already write "for k in pairs(tbl,false) do ... end".

Sure I can. However, certain kinds of (heavily meta) tables may have a
high cost of value retrieval, which this proposal aims to eliminate.

>> * Key-adjusting __newindex semantics.
>>
>> If __newindex is invoked with a special fourth argument (what kind of?
>> equal to false?), then the metamethod is expected (but not required)
>> to perform insertion (a la table.insert) if the vale is non-nil, or
>> removal (a la table.remove) if the value is nil. This may or may not
>> include:
>> - adjusting indices of other values;
>> - raising an error if adjustment is not possible (e.g. the key is a
>> string and already used) or there is no such element to remove;
>> - supporting nil as key;
>> - returning the (actual) key used.
>
> I can't figure out what the idea is. Give an example.
>

One simple use case is an alternate method of making
table.insert/table.remove meta-aware in a really general manner and
without resorting to the 5.3 sequence of __index/__newindex calls
which are slower while being too specific.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Philipp Janda
Am 23.03.2018 um 09:41 schröbte Sergey Zakharchenko:

> Dirk,
>
>>> If __pairs is invoked wth a second argument equal to the special value
>>> false, the caller is not interested in values (only in keys), so the
>>> returned iterator may returned any value, including nil, or no value
>>> at all, for them.
>>
>> You can already write "for k in pairs(tbl,false) do ... end".
>
> Sure I can. However, certain kinds of (heavily meta) tables may have a
> high cost of value retrieval, which this proposal aims to eliminate.

When using the `pairs` function, I expect getting pairs. An iterator
factory for keys only would better be called `keys()`.
Implementation-wise it could look for a `__keys` metamethod for those
performance critical cases and fall back to the `pairs` protocol but
dropping the second value.

Philipp


Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Egor Skriptunoff-2
On Sat, Mar 24, 2018 at 3:24 AM, Philipp Janda  wrote:
Am 23.03.2018 um 09:41 schröbte Sergey Zakharchenko:
Dirk,

If __pairs is invoked wth a second argument equal to the special value
false, the caller is not interested in values (only in keys), so the
returned iterator may returned any value, including nil, or no value
at all, for them.

You can already write "for k in pairs(tbl,false) do ... end".

Sure I can. However, certain kinds of (heavily meta) tables may have a
high cost of value retrieval, which this proposal aims to eliminate.

When using the `pairs` function, I expect getting pairs. An iterator factory for keys only would better be called `keys()`. Implementation-wise it could look for a `__keys` metamethod for those performance critical cases and fall back to the `pairs` protocol but dropping the second value.

for k in pairs(tbl, 'k') do
for v in pairs(tbl, 'v') do
for k, v in pairs(tbl, 'kv') do  -- the default value of second argument
Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Dirk Laurie-2
2018-03-24 7:08 GMT+02:00 Egor Skriptunoff <[hidden email]>:

>
> On Sat, Mar 24, 2018 at 3:24 AM, Philipp Janda  wrote:
>>
>> Am 23.03.2018 um 09:41 schröbte Sergey Zakharchenko:
>>>
>>> Dirk,
>>>
>>>>> If __pairs is invoked wth a second argument equal to the special value
>>>>> false, the caller is not interested in values (only in keys), so the
>>>>> returned iterator may returned any value, including nil, or no value
>>>>> at all, for them.
>>>>
>>>>
>>>> You can already write "for k in pairs(tbl,false) do ... end".
>>>
>>>
>>> Sure I can. However, certain kinds of (heavily meta) tables may have a
>>> high cost of value retrieval, which this proposal aims to eliminate.
>>
>>
>> When using the `pairs` function, I expect getting pairs. An iterator factory for keys only would better be called `keys()`. Implementation-wise it could look for a `__keys` metamethod for those performance critical cases and fall back to the `pairs` protocol but dropping the second value.
>
>
> for k in pairs(tbl, 'k') do
> for v in pairs(tbl, 'v') do
> for k, v in pairs(tbl, 'kv') do  -- the default value of second argument

Neat, especially the second one — but the idiom 'for _,v in pairs(tbl)
do' is so well entrenched .

IIUC, the OP's "conservative" proposal is merely that 'pairs' should
not discard extra arguments, i.e. it should trigger __pairs(tbl,...)
rather than just _pairs(tbl).

Since 'pairs' is just an entry in the base libary, one can monkey-patch it.

local _keep_pairs = pairs
pairs = function(tbl,...)
  local meta = getmetatable(tbl)
  if meta then meta = meta.__pairs end
  if meta then return meta(tbl,...)
  else return next,tbl
  end
end

Why change 'pairs' in standard Lua if this is so easy?

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Sergey Zakharchenko
In reply to this post by Philipp Janda

Philipp,

> When using the `pairs` function, I expect getting pairs. An iterator factory for keys only would better be called `keys()`. Implementation-wise it could look for a `__keys` metamethod for those performance critical cases and fall back to the `pairs` protocol but dropping the second value.

Not sure we should introduce a separate metamethod for that, would be nice to check memory usage/processing overhead in these cases. BTW while we're discussing all of this, pairs() seems to pass only 1 argument to __pairs, but I didn't suggest changing *that*.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Sergey Zakharchenko
In reply to this post by Dirk Laurie-2

Dirk,
> > for k in pairs(tbl, 'k') do
> > for v in pairs(tbl, 'v') do
> > for k, v in pairs(tbl, 'kv') do  -- the default value of second argument
>
> Neat, especially the second one

Though it doesn't lend itself to similarly efficient implementation and can't really provide real performance benefits; for _,v in pairs() is just fine.

> — but the idiom 'for _,v in pairs(tbl)
> do' is so well entrenched .

And undeservedly so.

> IIUC, the OP's "conservative" proposal is merely that 'pairs' should
> not discard extra arguments, i.e. it should trigger __pairs(tbl,...)
> rather than just _pairs(tbl).

Note I never wrote about changing meaning of any existing Lua core functions. Reread original email if that's not obvious.

> Why change 'pairs' in standard Lua if this is so easy?

See above.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Dirk Laurie-2
2018-03-24 8:11 GMT+02:00 Sergey Zakharchenko <[hidden email]>:

>> IIUC, the OP's "conservative" proposal is merely that 'pairs' should
>> not discard extra arguments, i.e. it should trigger __pairs(tbl,...)
>> rather than just _pairs(tbl).
>
> Note I never wrote about changing meaning of any existing Lua core
> functions.

True, you do not suggest changing what 'pairs' does in the absence of
a metamethod. In fact, I wrote that interpretation  precisely to
emphasize that the proposal that 'pairs' should have a second argument
did not come from you.

> Reread original email if that's not obvious.

Maybe you should take your own advice here. The original says clearly:
"If __pairs is invoked wth a second argument ..."

You can't change the way '__pairs' is called without changing 'pairs'
itself. The manual states: "pairs (t) ... If t has a metamethod
__pairs, calls it with t as argument." I see no way of implementing
your proposal that will not involve a change to "pairs (t[,opt]) ...
If t has a metamethod __pairs, calls it with t and opt as arguments."

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Sergey Zakharchenko

Dirk,

> You can't change the way '__pairs' is called without changing 'pairs'
> itself. The manual states: "pairs (t) ... If t has a metamethod
> __pairs, calls it with t as argument." I see no way of implementing
> your proposal that will not involve a change to "pairs (t[,opt]) ...
> If t has a metamethod __pairs, calls it with t and opt as arguments."

Does pairs() have exclusive rights on calling __pairs()? (Obviously no).

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Egor Skriptunoff-2
In reply to this post by Dirk Laurie-2
On Sat, Mar 24, 2018 at 9:01 AM, Dirk Laurie wrote:
2018-03-24 7:08 GMT+02:00 Egor Skriptunoff:
>
> On Sat, Mar 24, 2018 at 3:24 AM, Philipp Janda  wrote:
>>
>> Am 23.03.2018 um 09:41 schröbte Sergey Zakharchenko:
>>>
>>> Dirk,
>>>
>>>>> If __pairs is invoked wth a second argument equal to the special value
>>>>> false, the caller is not interested in values (only in keys), so the
>>>>> returned iterator may returned any value, including nil, or no value
>>>>> at all, for them.
>>>>
>>>>
>>>> You can already write "for k in pairs(tbl,false) do ... end".
>>>
>>>
>>> Sure I can. However, certain kinds of (heavily meta) tables may have a
>>> high cost of value retrieval, which this proposal aims to eliminate.
>>
>>
>> When using the `pairs` function, I expect getting pairs. An iterator factory for keys only would better be called `keys()`. Implementation-wise it could look for a `__keys` metamethod for those performance critical cases and fall back to the `pairs` protocol but dropping the second value.
>
>
> for k in pairs(tbl, 'k') do
> for v in pairs(tbl, 'v') do
> for k, v in pairs(tbl, 'kv') do  -- the default value of second argument

Neat, especially the second one — but the idiom 'for _,v in pairs(tbl)
do' is so well entrenched .


ipairs is actually not needed if we could use
for k,v in pairs(tbl, 'ikv') do  -- Integer Keys and Values
instead of
for k,v in ipairs(tbl) do

for v in pairs(tbl, 'iv') do  -- Integer Values
is the equivalent of
for _,v in ipairs(tbl) do


Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Egor Skriptunoff-2
On Sat, Mar 24, 2018 at 10:51 AM, Egor Skriptunoff wrote:
On Sat, Mar 24, 2018 at 9:01 AM, Dirk Laurie wrote:
2018-03-24 7:08 GMT+02:00 Egor Skriptunoff:
>
> On Sat, Mar 24, 2018 at 3:24 AM, Philipp Janda  wrote:
>>
>> Am 23.03.2018 um 09:41 schröbte Sergey Zakharchenko:
>>>
>>> Dirk,
>>>
>>>>> If __pairs is invoked wth a second argument equal to the special value
>>>>> false, the caller is not interested in values (only in keys), so the
>>>>> returned iterator may returned any value, including nil, or no value
>>>>> at all, for them.
>>>>
>>>>
>>>> You can already write "for k in pairs(tbl,false) do ... end".
>>>
>>>
>>> Sure I can. However, certain kinds of (heavily meta) tables may have a
>>> high cost of value retrieval, which this proposal aims to eliminate.
>>
>>
>> When using the `pairs` function, I expect getting pairs. An iterator factory for keys only would better be called `keys()`. Implementation-wise it could look for a `__keys` metamethod for those performance critical cases and fall back to the `pairs` protocol but dropping the second value.
>
>
> for k in pairs(tbl, 'k') do
> for v in pairs(tbl, 'v') do
> for k, v in pairs(tbl, 'kv') do  -- the default value of second argument

Neat, especially the second one — but the idiom 'for _,v in pairs(tbl)
do' is so well entrenched .


ipairs is actually not needed if we could use
for k,v in pairs(tbl, 'ikv') do  -- Integer Keys and Values
instead of
for k,v in ipairs(tbl) do

for v in pairs(tbl, 'iv') do  -- Integer Values
is the equivalent of
for _,v in ipairs(tbl) do


More options may be passed in second argument:

for k, v in pairs(tbl, 'ikvb') do  -- Integer Keys, Values Backwards
(the same as "for k = #tbl, 1, -1 do")

for k, v in pairs(tbl, 'ikv0') do  -- Integer Keys, Values for zero-based arrays
(the same as "local key = 0; while tbl[key] ~= nil do local k, v = key, tbl[key]; ...; key = key + 1 end")
Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

steve donovan
On Sat, Mar 24, 2018 at 10:01 AM, Egor Skriptunoff
<[hidden email]> wrote:
> More options may be passed in second argument:

And then we have the 'passing options as strings' problem: very
flexible but easy to break.

I don't feel the great need, personally - we all know `pairs` and its
limitations.

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Dirk Laurie-2
In reply to this post by Sergey Zakharchenko
2018-03-24 9:18 GMT+02:00 Sergey Zakharchenko <[hidden email]>:

>> You can't change the way '__pairs' is called without changing 'pairs'
>> itself. The manual states: "pairs (t) ... If t has a metamethod
>> __pairs, calls it with t as argument." I see no way of implementing
>> your proposal that will not involve a change to "pairs (t[,opt]) ...
>> If t has a metamethod __pairs, calls it with t and opt as arguments."
>
> Does pairs() have exclusive rights on calling __pairs()? (Obviously no).

Should we extend that logic to __len, __add, __concat etc, or is there
something special about __pairs?

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Sergey Zakharchenko

Dirk,

> > Does pairs() have exclusive rights on calling __pairs()? (Obviously no).
>
> Should we extend that logic to __len, __add, __concat etc, or is there
> something special about __pairs?

Why not? E.g. __len could be placed on tables for forward compatibility and called explicitly in Lua 5.1 in some sort of an lpairs() iterator.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Philipp Janda
In reply to this post by Sergey Zakharchenko
Am 24.03.2018 um 07:06 schröbte Sergey Zakharchenko:
> Philipp,

Hi!

>
>> When using the `pairs` function, I expect getting pairs. An iterator
>> factory for keys only would better be called `keys()`. Implementation-wise
>> it could look for a `__keys` metamethod for those performance critical
>> cases and fall back to the `pairs` protocol but dropping the second value.
>
> Not sure we should introduce a separate metamethod for that, would be nice
> to check memory usage/processing overhead in these cases. BTW while we're
> discussing all of this, pairs() seems to pass only 1 argument to __pairs,
> but I didn't suggest changing *that*.

Not sure I can follow the second part. I thought about something like

     do
       local function firstonly( f, ... )
         return function( s, var )
           return (f( s, var ))
         end, ...
       end

       function keys( obj )
         local mt = getmetatable( obj )
         if type( mt ) == "table" and
            type( mt.__keys ) == "function" then
           return mt.__keys( obj )
         else
           return firstonly( pairs( obj ) )
         end
       end
     end

which reuses the `pairs` protocol as is but only returns the keys.

I agree on the first part though. It seems that in other languages you
would look for some method to call while in Lua we use a metamethod
instead if we want customized behavior.

>
> Best regards,
>

Philipp


Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

John Hind
In reply to this post by Sergey Zakharchenko
>Date: Sat, 24 Mar 2018 09:05:59 +0200
>From: Dirk Laurie <[hidden email]>
>
>2018-03-24 8:11 GMT+02:00 Sergey Zakharchenko
><[hidden email]>:
>
>>>IIUC, the OP's "conservative" proposal is merely that 'pairs' should
>>>not discard extra arguments, i.e. it should trigger __pairs(tbl,...)
>>>rather than just _pairs(tbl).
>>
>>Note I never wrote about changing meaning of any existing Lua core
>>functions.
>
>True, you do not suggest changing what 'pairs' does in the absence of
>a metamethod. In fact, I wrote that interpretation  precisely to
>emphasize that the proposal that 'pairs' should have a second argument
>did not come from you.
>
>>Reread original email if that's not obvious.
>
>Maybe you should take your own advice here. The original says clearly:
>"If __pairs is invoked wth a second argument ..."
>
>You can't change the way '__pairs' is called without changing 'pairs'
>itself. The manual states: "pairs (t) ... If t has a metamethod
>__pairs, calls it with t as argument." I see no way of implementing
>your proposal that will not involve a change to "pairs (t[,opt]) ...
>If t has a metamethod __pairs, calls it with t and opt as arguments."
>
>
I'm going to get it printed on a T-Shirt (or maybe a bumper sticker):

"'pairs' and 'ipairs' are not syntax, dammit!"

'pairs' and 'ipairs' are bog-standard Lua functions and Lua already has
perfectly good generic mechanisms for overloading functions. Metamethods
for overloading specific functions are unnecessary bloat. Metamethods
are only needed to overload syntax, and IMHO should be reserved and
restricted for this purpose. If you want to overload 'pairs' for your
class, implement your own version as a method and have your users write:

for k, v in instanceofmyclass:pairs() do ... end

If you want to improve this, make the metamethod a proper syntax
overload like my '__iter' proposal allowing your users to write:

for k, v in instanceofmyclass do ... end

My proposal does not preclude using the method approach instead (or as
well) if you want multiple iterators, and the method approach allows you
to parameterise the iterator method any way you want.


Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Sergey Zakharchenko

John,

> for k, v in instanceofmyclass:pairs() do ... end
> for k, v in instanceofmyclass do ... end

I'm after code that's quite a bit more generic than this, and in particular supports plain tables.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re[2]: Some conservative Lua metamethod extension proposals

John Hind
First, sorry if I am missing some things you said, but this damn message system has had a bug for more than a decade now whereby some of the digest messages come through as hex gobblygook and I can only read them 24 hours or more later on the web archive (or if you copy me in on an individual message like you kindly did in this case)!

But I really do not understand where you are coming from. If you want a different iteration behaviour that applies to ALL tables, what is the benefit of making it a metamethod? Either sell that behaviour for the library version of 'pairs', or just write your own version with a different name. ('monkey-patching' is not only objectionable because of the brittle techniques used to achieve it, but because it causes unexpected and undocumented behaviour for users)

Also, my '__iter' metamethod proposal DOES support plain tables too - it defaults to an internal implementation functionally equivalent to 'pairs'. In other news, 'ipairs' is redundant with or without a metamethod since we have numeric for!

------ Original Message ------
From: "Sergey Zakharchenko" <[hidden email]>
Sent: 24/03/2018 09:50:15
Subject: Re: Some conservative Lua metamethod extension proposals

John,

> for k, v in instanceofmyclass:pairs() do ... end
> for k, v in instanceofmyclass do ... end

I'm after code that's quite a bit more generic than this, and in particular supports plain tables.

Best regards,

--
DoubleF

Reply | Threaded
Open this post in threaded view
|

Re: Some conservative Lua metamethod extension proposals

Dirk Laurie-2
In reply to this post by Sergey Zakharchenko
2018-03-24 10:40 GMT+02:00 Sergey Zakharchenko <[hidden email]>:
> Dirk,
>
>> > Does pairs() have exclusive rights on calling __pairs()? (Obviously no).
>>
>> Should we extend that logic to __len, __add, __concat etc, or is there
>> something special about __pairs?
>
> Why not? E.g. __len could be placed on tables for forward compatibility and
> called explicitly in Lua 5.1 in some sort of an lpairs() iterator.

I would find that very confusing.

Metamethods are functions called in a situation where otherwise Lua
would signal an error. They are listed in the manual in §2.4 –
"Metatables and Metamethods".

__pairs, __gc and __tostring are not listed there, because they are
not real metamethods. They are functions stored in the metatable whose
names start with '__'.

To call them metamethods is an abuse of terminology. It is much
shorter that for example "a __gc field in the metatable", and
admittedly even the manual is guilty of using it, but it leads people
to expect [1] that metamethods can be used to customize the behaviour
of Lua. They can't. They are fallbacks.

This is not to say that you can't use the same function to be called
by a user program and also as a metemethod. I do it all the time. But
then I give them different names. Functions in Lua are first-class
values, and the property is there to be exploited.

[1] I was a Lua user for several months before I discovered that
__index and __newindex are not called every time you index an array,
they are only called when the key does not exist.

12