Type Metatables for Table and Userdata - Powerpatch

classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Type Metatables for Table and Userdata - Powerpatch

John Hind

Please take a look at my new PowerPatch at http://lua-users.org/wiki/LuaPowerPatches. This allows the Table and Userdata types to have type metatables without losing their current per-object ones. This new mechanism is then used to expose the Table Library as methods of Table objects in the same way the String Library is exposed for String objects. (Logically, 'pairs', 'ipairs' and 'next' should also be exposed as methods, but I have not done that in the patch.) For very little cost, this makes Lua more consistent and also adds a powerful new generalised mechanism for others to build on.

 

While there, you could maybe take another look at my 'Self-iterating Objects' patch further down the same page. Having used this extensively myself over the last few years, I am more than ever convinced this is a worthwhile improvement.

 

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Rena

On 2012-12-05 6:46 AM, "John Hind" <[hidden email]> wrote:
>
> Please take a look at my new PowerPatch at http://lua-users.org/wiki/LuaPowerPatches. This allows the Table and Userdata types to have type metatables without losing their current per-object ones. This new mechanism is then used to expose the Table Library as methods of Table objects in the same way the String Library is exposed for String objects. (Logically, 'pairs', 'ipairs' and 'next' should also be exposed as methods, but I have not done that in the patch.) For very little cost, this makes Lua more consistent and also adds a powerful new generalised mechanism for others to build on.
>

This makes all kinds of sense. Of course there should be a debug.settypemt as well.

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Wolfgang Pupp-2
John Hind wrote:
> This allows the Table and Userdata types to have type metatables without
> losing their current per-object ones. This new mechanism is then used to
> expose the Table Library as methods of Table objects in the same way the
> String Library is exposed for String objects.

I like this idea a lot! How does this work, exactly- if an object already
has an __index function, is a lookup for a generic method only performed
when the __index- function returns nil (errors?), or not at all?
"Sparse" (frequent lookup of nonexistent keys) tables also become slightly
more expensive with this change, I suppose?

> While there, you could maybe take another look at my
> 'Self-iterating Objects' patch further down the same page.

I think this is the most convincing proposal for a Lua-syntax change I've
seen this far. I've observed Lua-beginners trying to do table-iteration
with exactly this pattern repeatedly ("for k, v in t do ...")- even to me,
after years of Lua-exposure, this still looks more intuitive.

IMHO, omitting the "pairs(.)" in a for-loop for this syntax is both more
concise *and* comprehensible (for a beginner), while "(__)ipairs" is simply
not useful enough (hardly useful *at all* AFAICT) to warrant maintaining
the current rules.

I strongly believe that this would make Lua syntax more "natural" and
would really like to hear peoples opinion on this because it looks too
good to let discussion about it die before it even started...

--Wolfgang

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Rena

On 2012-12-05 12:07 PM, "Wolfgang Pupp" <[hidden email]> wrote:
>
> I like this idea a lot! How does this work, exactly- if an object already
> has an __index function, is a lookup for a generic method only performed
> when the __index- function returns nil (errors?), or not at all?

I want to say the ideal behaviour is that the type metatable is not used at all if an individual metatable is present. This gives the most flexibility: if you want to fall back to the type metatable, you can easily do so in your __index metamethod. But it seems potentially more error-prone too, since you have to remember to add that fallback (and add it to existing code).

To try to clarify...
t, m = {}, {}
m.__index = function(t, k) return nil end
assert(t.pairs == pairs)
setmetatable(t, m)
assert(t.pairs == nil)
m.__index = function(t, k) return getmetatable({})[k] end
assert(t.pairs == pairs)

Another option is to have another field in the metatable telling whether to use the type metatable as a fallback or not, with false meaning no and anything else meaning yes (including nil, so it's default behaviour).

E.g.:
t, m = {}, {}
m.__index = function(t, k) return nil end
assert(t.pairs == pairs)
setmetatable(t, m)
assert(m.__fallback == nil)
assert(t.pairs == pairs)
m.__fallback = false
assert(t.pairs == nil)

I think either method is powerful, but potentially confusing...

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Sven Olsen
In reply to this post by John Hind

While there, you could maybe take another look at my 'Self-iterating Objects' patch further down the same page. Having used this extensively myself over the last few years, I am more than ever convinced this is a worthwhile improvement.


Self-iterating objects is a handy patch, I like it as well.  

But when you add it to 5.2, having both __iter and __pairs seems unnecessary and confusing.  I think it would make more sense to unify the two.  i.e., change the event name from "__iter" to "__pairs".

-Sven
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Sven Olsen
One more little question about the self-iterating patch.  I found this line in the patch notes:
 
This will break some subtle code tricks such as using the __call event to "fake" self-iteration.

This certainly sounds like something all patch users should keep in mind, but I don't understand what sorts of "subtle tricks" are being referenced. 

Even with the patch enabled, it appears that you can still use __call as an iterator function.  For example, this still works:

for k,v in setmetatable({},{__call=function() return pairs({3,2,1}) end})() do
   print(k,v)
end

And that's about as dangerous a trick as I can think of.  What sorts of patterns does the patch break?

-Sven
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Claire Lewis
In reply to this post by John Hind
So hang on, would this mean that, as a host app or module writer, I can no longer rely on this to work:
 
  t = { 1,2,3 }
  print(t[2])
  -> 2
 
if some other module or user script has gone and installed a ‘clever’ type metatable?
 
I realise this is currently the case with string, but there’s a lot less incentive to do clever things with that.
 
Like I said, I like the concept, but I wonder if there is some way this could be made _ENV bound, or similar, to avoid nasty side-effects in other modules.
 
- Claire.
 
 
Sent: Wednesday, December 05, 2012 10:16 PM
Subject: Type Metatables for Table and Userdata - Powerpatch
 

Please take a look at my new PowerPatch at http://lua-users.org/wiki/LuaPowerPatches. This allows the Table and Userdata types to have type metatables without losing their current per-object ones. This new mechanism is then used to expose the Table Library as methods of Table objects in the same way the String Library is exposed for String objects. (Logically, 'pairs', 'ipairs' and 'next' should also be exposed as methods, but I have not done that in the patch.) For very little cost, this makes Lua more consistent and also adds a powerful new generalised mechanism for others to build on.

 

While there, you could maybe take another look at my 'Self-iterating Objects' patch further down the same page. Having used this extensively myself over the last few years, I am more than ever convinced this is a worthwhile improvement.

 

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2221 / Virus Database: 2629/5437 - Release Date: 12/04/12

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Rena

On 2012-12-05 7:36 PM, "Claire Lewis" <[hidden email]> wrote:
>
> So hang on, would this mean that, as a host app or module writer, I can no longer rely on this to work:
>  
>   t = { 1,2,3 }
>   print(t[2])
>   -> 2
>  
> if some other module or user script has gone and installed a ‘clever’ type metatable?
>  
> I realise this is currently the case with string, but there’s a lot less incentive to do clever things with that.
>  
> Like I said, I like the concept, but I wonder if there is some way this could be made _ENV bound, or similar, to avoid nasty side-effects in other modules.
>  
> - Claire.
>  
>  
It's a metatable, so __index would still work the same as it always has.

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Peter Loveday
> It's a metatable, so __index would still work the same as it always has.
Understood, but what I’m saying is, if a module or user script has set a type metatable on table, then any naively constructed table in unrelated code (ie t = {} ) suddenly may not behave as expected.
 
- Claire.
 
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Luiz Henrique de Figueiredo
In reply to this post by Claire Lewis
> So hang on, would this mean that, as a host app or module writer, I can no longer rely on this to work:
>
>   t = { 1,2,3 }
>   print(t[2])
>   -> 2
>
> if some other module or user script has gone and installed a ???clever??? type metatable?

The problems of having *one* event handler for all values of each type
led us to move from fallbacks to metatables. See section 6.8 of our HOPL
paper: http://www.lua.org/doc/hopl.pdf

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

steve donovan
In reply to this post by Claire Lewis
On Thu, Dec 6, 2012 at 2:36 AM, Claire Lewis <[hidden email]> wrote:
> So hang on, would this mean that, as a host app or module writer, I can no
> longer rely on this to work:
> if some other module or user script has gone and installed a ‘clever’ type
> metatable?

It's already true ;)  debug.setmetatable can attach a new metatable to
any type.  Now, no _sensible_ library author would do this, but the
temptation to use clever stuff in code gets strong.

Lua people are less tolerant of monkey-patching than the Rubyists.
Sure, you can add your own methods to the string table, Penlight even
has an option to do "require 'pl.stringx'.import()" and bring in
additional Python-style string functions.  But increasingly I don't
think this is such a good idea anyway, since these are global
modifications. So restricting to _ENV in some way would be a useful
way to sandbox cleverness on a per-module basis.

On the subject of iterators:  I missed 'for k,v in t' when 5.1 finally
removed it, for the sensible reason that the syntax did not make it
clear whether we're doing pairs() or ipairs(). But by now I think
people are clear that this ought to mean pairs() only. So an __iter
metamethod makes sense, and a default table metatable could have that
enabled.  But then there's also the common need to iterate over the
values of a collection, without the keys (especially 'arrays')...

steve d.

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

John Hind
In reply to this post by John Hind

Sven Olsen <sven2718 <at> gmail.com>

>One more little question about the self-iterating patch.  I found this line in the patch notes:

>This will break some subtle code tricks such as using the __call event to "fake" self-iteration.

 

>This certainly sounds like something all patch users should keep in mind, but I don't understand what >sorts of "subtle tricks" are being referenced.

 

Before I implemented this patch I used to do:

 

tab = {"John","Hind"}

setmetatable(tab, {__call=pairs})

for k,v in tab() do print(k,v) end

 

I thought this would be broken (it may have been in earlier Lua versions) but re-testing it today it seems to be fine. To maintain compatibility, I test the type of the first of the three parameters of the generic for opcode. If it's a function I proceed with the original code, otherwise I look for the '__iter' metamethod. The original code does not check the parameters, it just launches into the first iteration which fails if the first parameter cannot be called. So the difference is between 'being callable' and 'being a function'. I think you'd need to have a 'pairs' analog which returned a 'next' analog that was a callable object but not a function to break it - I cannot think of a reasonable usage case for that. My caveat may be over-cautious!

 

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

John Hind
In reply to this post by John Hind

Wolfgang Pupp <wolfgang.pupp <at> gmail.com> wrote:

 

>I like this idea a lot! How does this work, exactly- if an object already

>has an __index function, is a lookup for a generic method only performed

>when the __index- function returns nil (errors?), or not at all?

 

The mechanism is a bit simpler than that. There is still 'really' only the object metatable. The type metatable pointer is copied from the Lua state into the object automatically on object creation (so think of it as a default). A newly created object uses the type metatable, but you can replace this for the object using setmetatable. You can even disable the type metatable for a specific object using setmetatable(tab, nil). getmetatable(tab) immediately after object creation will return the type metatable allowing you to delegate to it from the '__index' metamethod of your own object metatable if that's what you want to do.

 

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

John Hind
In reply to this post by John Hind

Rena said:

>This makes all kinds of sense. Of course there should be a debug.settypemt as well.

 

You are right - I added this to the powerpatch today. Also makes it a lot easier to test!

 

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

spir ☣
In reply to this post by Luiz Henrique de Figueiredo
On 06/12/2012 02:17, Luiz Henrique de Figueiredo wrote:

>> So hang on, would this mean that, as a host app or module writer, I can no longer rely on this to work:
>>
>>    t = { 1,2,3 }
>>    print(t[2])
>>    -> 2
>>
>> if some other module or user script has gone and installed a ???clever??? type metatable?
>
> The problems of having *one* event handler for all values of each type
> led us to move from fallbacks to metatables. See section 6.8 of our HOPL
> paper: http://www.lua.org/doc/hopl.pdf

Has the following been considered: that the implementation first checks whether
a table _itself_ holds a "metafield", before checking a possible mt? This would
allow 2 options:
* individual table elements of a prog to have proper behaviour w/o needing a mt
   example: customize the display of a table 'present_time'
* tables with a common mt used as custom type to still be specialisable
   example: among specimens of 'Point', a few show in a distinct manner (each
proper)

This would also make some things easier. First and mainly, explaining and
understanding the feature without too deep meta & abstract reasoning (it's just
frightening for many, I guess). Also, meta-meta levels when touching metatypes
or other kinds of root objects may be easier to setup [1].

(As an aside, this would also justify the special format of metafield names,
else when beeing on proper metatables there is no risk of accidental name clash,
imo.)

However, this would make tables special compared to other types (eg strings
cannot themselves hold metafields...).

Denis

[1] No meta-meta[-meta]-table... but for this issue, I may miss a point, too.
Still, when trying to formalise even the simplest regular custom-type system
(also for array-tables or such clearly defined uses of tables) I step on the
issue of the root object and its own mt beeing apart, with the occasional "loop
in gettable".

Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Andrew Starks

I'm struggling with the problem that this is solving. What thing does it do that lua does not already do?

Consider that strings are wrapped in an object where they get the string library's methods. This makes sense because it's hard to imagine using strings in any other way, except as strings. (Even here, it is instructive that unicode characters break some of string library's intent.)

However, tables are a generic interface for everything. A "by default" hardwiring of the table library puts too much emphasis on those methods and the use of a table as a dictionary or indexed array.

In the same way, making "for k, v in t do..." has the same problem. In the name of a more "natural" syntax, to my thinking, you've obfuscated the fact that a function that returns a function that knows how to iterate "t" is being called. In this one case,  passing a function to "for", but not "t" (which could be callable), but "t"'s "__iter" function, or if not that, then its "__pairs" or "__ipairs?". 

"pairs", "ipairs" or some other iterator function made it explicit that I was iterating the object in a certain way. I could also say "for k, v in myobj:tier() do..." and accomplish the same thing, more clearly. (For that reason, I like the idea of writing explicit iterators, rather than redefining ipairs/pairs, but I'm probably in the minority on that one)

This proposal doesn't break anything (much?), and it does makes the code a bit shorter. I can't see how it makes anything clearer.

My $0.02.
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Sven Olsen

This proposal doesn't break anything (much?), and it does makes the code a bit shorter. I can't see how it makes anything clearer.

I admit, I'm unsure what to make of the type metatables idea.  I don't see a compelling argument for it -- but, maybe there's some important application that I'm missing. 

I do think there are plenty of situations where John's table shorthand will clarify the programmer's intent.  Unfortunately, it simultaneously obscures the function of the code.  Maybe it would be better executed as a parser hack, rather than a VM change.  For example, we could add some syntax sugar to convert:

for k,v : t do  ==> for k,v in pairs(t) do

Replacing the "in" token with something else would make it clear that we're not using the standard generic for syntax.  ":" is probably the best-looking existing token, though i suppose it might be mistaken by the reader as having something to do with method calls (from the parser's perspective, however, I don't think there'd be any confusion).

-Sven
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Sven Olsen
In reply to this post by John Hind

I think you'd need to have a 'pairs' analog which returned a 'next' analog that was a callable object but not a function to break it - I cannot think of a reasonable usage case for that. My caveat may be over-cautious!


Right -- of course.  So with the patch on we can't use function objects as iterators.  i.e., this breaks:

local function mypairs(t)
local function __call(self,...)
return next(...)
end
return setmetatable({"wrong table"},{__call=__call}), t
end

local t={1,2,3}

for k,v in mypairs(t) do
   console(k,v)
end

I actually think that's a fairly serious point against the patch.  It's true that it's not a case that would come up often, but, it's a difficult error case to understand.  Some kind of implicit pairs call is a useful shorthand, but I think I'd prefer it as a parser patch.   'for k : t' ==> 'for k in pairs(t)' would feel simpler and cleaner.

-Sven
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Sven Olsen
In reply to this post by Sven Olsen

for k,v : t do  ==> for k,v in pairs(t) do

(in a compulsive moment i wrote up a little patch that does this.  if anyone else wants it let me know.)
Reply | Threaded
Open this post in threaded view
|

Re: Type Metatables for Table and Userdata - Powerpatch

Wolfgang Pupp-2
In reply to this post by Andrew Starks
Andrew Starks wrote:
> In the same way, making "for k, v in t do..." has the same problem. In the
> name of a more "natural" syntax, to my thinking, you've obfuscated the fact
> that a function that returns a function that knows how to iterate "t" is
> being called.

Let me rephrase/clarify: What I argue for is to *extend* Luas generic
for syntax (currently "iterator function syntax") into an "iterator
function OR table iteration"- syntax.
It'd be like giving the language a concept of "traversing a table". By
default, this traversal would behave like pairs (the *only* sensible
choice). But in some (rare!) cases, one might wish to override that
default behavior (via __iter) i.e. when the table contains no real
data because it serves only as an abstraction.
Both table iteration overriding and "raw" traversal would still happen
via conventional iterator functions.

Points in favour:
 - table traversal is a simpler concept than iterator functions-
(partial) separation would be beneficial at the very least for
beginners.
 - pairs needs to be a "magical" function anyway (can't be implemented
without "cheating"), and it basically *never* appears outside of the
for-statement (it is never needed "as function"- only for what it
achieves inside a for-loop).

The fact that an iterator function might be called for the "table
iteration" syntax is an implementation detail, not an obfuscation.

> I like the idea of writing explicit iterators, rather than
> redefining ipairs/pairs, but I'm probably in the minority on that one)

I strongly agree with you on this. The meaning of iteration (be it via
__pairs or __iter) should only be overridden IMO when the new behavior
is absolutely the *only* one that makes sense. Being explicit never
hurts; but having to call "pairs" for every table traversal is
*boilerplate* code IMHO, not being explicit.

Sven Olsen wrote:
> Unfortunately, it ("for k,v in t do ..." syntax) simultaneously obscures the function of the code.

I do not understand what you mean- could you elaborate on this?

--Wolfgang

123