__method metamethod

[Greg Fitzgerald]

I wonder if Lua might consider adding a __method metamethod, which
could be used for OO programming and as an alternative to the colon
operator.

For example, if you wanted to do OO programming without using the
colon syntax using Lua 5.2, you might write:

local funcs = {
    inc = function(self)
        self.a = self.a + 1
        return self
    end
}

local function new(t)
   local o = t or {a = 0}
   return setmetatable(o, {
      __index = function(t, k)
          return function(...) return funcs[k](t, ...) end
      end
   })
end

assert(new().inc().inc().inc().a == 3)


This style, when compared to using the colon operator, has the nice
property that the parentheses are optional.  You can write:

local f = new().inc
assert(f().a == 1)

But performance here is far worse than using the colon operator.  In
the assertion above, 3 closures are created, whereas there are zero if
using the colon operator.

If, however, Lua offered a __method metamethod that matched the syntax
"t.k(...)", then we could add the following to our metatable:

    __method = function(t, k, ...)
       return funcs[k](t, ...)
    end

Like using the colon operator, the following code would create no closures:

    new().inc().inc().inc()

Thoughts?

Thanks,
Greg

[Coda Highland]

On Fri, Jul 19, 2013 at 3:38 PM, Greg Fitzgerald <[hidden email]> wrote:

> I wonder if Lua might consider adding a __method metamethod, which
> could be used for OO programming and as an alternative to the colon
> operator.
>
> For example, if you wanted to do OO programming without using the
> colon syntax using Lua 5.2, you might write:
>
> local funcs = {
>     inc = function(self)
>         self.a = self.a + 1
>         return self
>     end
> }
>
> local function new(t)
>    local o = t or {a = 0}
>    return setmetatable(o, {
>       __index = function(t, k)
>           return function(...) return funcs[k](t, ...) end
>       end
>    })
> end
>
> assert(new().inc().inc().inc().a == 3)
>
>
> This style, when compared to using the colon operator, has the nice
> property that the parentheses are optional.  You can write:
>
> local f = new().inc
> assert(f().a == 1)
>
> But performance here is far worse than using the colon operator.  In
> the assertion above, 3 closures are created, whereas there are zero if
> using the colon operator.
>
> If, however, Lua offered a __method metamethod that matched the syntax
> "t.k(...)", then we could add the following to our metatable:
>
>     __method = function(t, k, ...)
>        return funcs[k](t, ...)
>     end
>
> Like using the colon operator, the following code would create no closures:
>
>     new().inc().inc().inc()
>
> Thoughts?
>
> Thanks,
> Greg
>

My first thought is: What's wrong with the colon operator? What
problem does this suggestion actually solve?

/s/ Adam

[Javier Guerra Giraldez]

On Fri, Jul 19, 2013 at 5:46 PM, Coda Highland <[hidden email]> wrote:
> My first thought is: What's wrong with the colon operator? What
> problem does this suggestion actually solve?


as i understand it, it's not (mostly) about the color or dot
operators, but about being able to use a bound method as a value.

a slightly better proposal could be:  don't modify the current
operators, but add this syntax:

local f = o:m

as syntax sugar to:

local f = function(...) return o.m(o, ...)  end

i think it's better than Greg's proposal because it doesn't magically
changes the meaning of an existing operator (dot), instead uses a
currently invalid syntax.

of course, Greg assumed that being in the core would somehow avoid
closure creation, but i doubt that would be simple.  Still, it
postpones it until (if) it's needed, so it wouldn't have any impact on
any code that doesn't need it.

said that, i don't see the need, it's not so ugly to define a
"bind(o,m)" utility:

function bind(o,m)
    return function(...) return o[m](o,...) end
end

then, you could get a bound method as:

local f = bind(o,'methodname')

--
Javier

[Greg Fitzgerald]

On Fri, Jul 19, 2013 at 4:02 PM, Javier Guerra Giraldez
<[hidden email]> wrote:
> local f = o:m
>
> as syntax sugar to:
>
> local f = function(...) return o.m(o, ...)  end

That would be a good addition too.


> i think it's better than Greg's proposal because it doesn't magically
> changes the meaning of an existing operator (dot)

That's not exactly fair.  It magically changes the dot operator in the
same way the __index metamethod magically changes the dot operator.
Both are tucked under the hood.  From the user's perspective, the same
code could have just as easily been written with no metamethods at
all:

local funcs = {
    inc = function(self)
        self.a = self.a + 1
        return self
    end
}

local function new(t)
   local o = t or {a = 0}
   for k,f in pairs(funcs) do
      o[k] = function(...) return f(o, ...) end
   end
   return o
end

assert(new().inc().inc().inc().a == 3)


Using the __index metamethod is a handy way of exposing a Lua library
with the familiar syntax of JavaScript, Python, Java, etc.  Using this
proposed __method metamethod is a technique for getting that syntax
without the performance regression.

-Greg

[Coda Highland]

On Fri, Jul 19, 2013 at 5:31 PM, Greg Fitzgerald <[hidden email]> wrote:

> On Fri, Jul 19, 2013 at 4:02 PM, Javier Guerra Giraldez
> <[hidden email]> wrote:
>> local f = o:m
>>
>> as syntax sugar to:
>>
>> local f = function(...) return o.m(o, ...)  end
>
> That would be a good addition too.
>
>
>> i think it's better than Greg's proposal because it doesn't magically
>> changes the meaning of an existing operator (dot)
>
> That's not exactly fair.  It magically changes the dot operator in the
> same way the __index metamethod magically changes the dot operator.
> Both are tucked under the hood.  From the user's perspective, the same
> code could have just as easily been written with no metamethods at
> all:
>
> local funcs = {
>     inc = function(self)
>         self.a = self.a + 1
>         return self
>     end
> }
>
> local function new(t)
>    local o = t or {a = 0}
>    for k,f in pairs(funcs) do
>       o[k] = function(...) return f(o, ...) end
>    end
>    return o
> end
>
> assert(new().inc().inc().inc().a == 3)
>
>
> Using the __index metamethod is a handy way of exposing a Lua library
> with the familiar syntax of JavaScript, Python, Java, etc.  Using this
> proposed __method metamethod is a technique for getting that syntax
> without the performance regression.
>
> -Greg
>

Overloading : to work when not coupled with a parameter list is the
better solution. It's still important to be able to access the unbound
version of the method, so you want x.y to still return the unbound
version, and you don't want to overload the behavior of . any more
than it already is. Meanwhile, : without a parameter list is currently
a syntax error, so adding it would cause no incompatibilities and
minimal overhead (there would still be a __method dispatch on a:b()
where there wasn't one before, but that overhead can be brought down
to just a single comparison by requiring that __method be defined on
the metatable already when setmetatable is called).

That said... I'm not convinced that it would be anything more than
syntactic sugar. Adding "bound functions" that aren't closures to the
language... well, I can see how it could be done with less overhead
than defining a closure, but would it actually make a substantive
difference, and could it be done CLEANLY?

It sounds to me like this is the time to put one's money where one's
mouth is and put together a patch so that it can be tested instead of
merely speculated upon.

/s/ Adam

[Coda Highland]

On Fri, Jul 19, 2013 at 5:51 PM, Coda Highland <[hidden email]> wrote:

> On Fri, Jul 19, 2013 at 5:31 PM, Greg Fitzgerald <[hidden email]> wrote:
>> On Fri, Jul 19, 2013 at 4:02 PM, Javier Guerra Giraldez
>> <[hidden email]> wrote:
>>> local f = o:m
>>>
>>> as syntax sugar to:
>>>
>>> local f = function(...) return o.m(o, ...)  end
>>
>> That would be a good addition too.
>>
>>
>>> i think it's better than Greg's proposal because it doesn't magically
>>> changes the meaning of an existing operator (dot)
>>
>> That's not exactly fair.  It magically changes the dot operator in the
>> same way the __index metamethod magically changes the dot operator.
>> Both are tucked under the hood.  From the user's perspective, the same
>> code could have just as easily been written with no metamethods at
>> all:
>>
>> local funcs = {
>>     inc = function(self)
>>         self.a = self.a + 1
>>         return self
>>     end
>> }
>>
>> local function new(t)
>>    local o = t or {a = 0}
>>    for k,f in pairs(funcs) do
>>       o[k] = function(...) return f(o, ...) end
>>    end
>>    return o
>> end
>>
>> assert(new().inc().inc().inc().a == 3)
>>
>>
>> Using the __index metamethod is a handy way of exposing a Lua library
>> with the familiar syntax of JavaScript, Python, Java, etc.  Using this
>> proposed __method metamethod is a technique for getting that syntax
>> without the performance regression.
>>
>> -Greg
>>
>
> Overloading : to work when not coupled with a parameter list is the
> better solution. It's still important to be able to access the unbound
> version of the method, so you want x.y to still return the unbound
> version, and you don't want to overload the behavior of . any more
> than it already is. Meanwhile, : without a parameter list is currently
> a syntax error, so adding it would cause no incompatibilities and
> minimal overhead (there would still be a __method dispatch on a:b()
> where there wasn't one before, but that overhead can be brought down
> to just a single comparison by requiring that __method be defined on
> the metatable already when setmetatable is called).
>
> That said... I'm not convinced that it would be anything more than
> syntactic sugar. Adding "bound functions" that aren't closures to the
> language... well, I can see how it could be done with less overhead
> than defining a closure, but would it actually make a substantive
> difference, and could it be done CLEANLY?
>
> It sounds to me like this is the time to put one's money where one's
> mouth is and put together a patch so that it can be tested instead of
> merely speculated upon.
>
> /s/ Adam

Upon further reflection, I think __method may not even be necessary.

The proposal seems to say that "local x = y:z" ought to mean "local x
= getmetatable(y).__method(y,z)", analogous to how "local x = y.z"
means "local x = getmetatable(y).__index(y,z)".

But instead, "local x = y:z" could be defined to have behavior
equivalent to "local x = function(...) return
getmetatable(y).__index(y,z)(y, ...) end". (I say "equivalent" because
of course it could be generating a special bound-function object
instead of a closure.) This avoids the need to create a new metatable
entry or new dispatch rules, and it makes "a:b()" perform exactly the
way it already does.

I'm still not convinced that this is NECESSARY, but if it were to be
implemented, I think this would be best.

/s/ Adam

[steve donovan]

On Sat, Jul 20, 2013 at 12:38 AM, Greg Fitzgerald <[hidden email]> wrote:

I wonder if Lua might consider adding a __method metamethod, which
could be used for OO programming and as an alternative to the colon
operator.

But ... we're all used to the colon operator, and we know what it does. We avoid any weird magic and implicit generation of partial functions.  However, I'm partial to 'm:foo' on its own meaning partial application, as it is in Moonscript.

Now, I've always thought that the use of something like __method is to distinguish between things which actually are methods and things which are fields. So when writing a Set class which has a 'add' method, we don't get false positives like 's["set"]` being true.  (in the Penlight Set 'class', the solution was not to use methods but overload the usual operators and provide some functions to operate on sets)

steve d.

[Sven Olsen]

(I say "equivalent" because
of course it could be generating a special bound-function object
instead of a closure.) 

I've been daydreaming about whether or not such a special "bound-function object" might be practical.  A non-allocating implementation would would require 2 registers, one for the object and one for the method name.  After assigning `local x=y:z`, it would be easy to evaluate 'x(1,2,3)', but, things would start to fall apart if you wanted to pass x to another function, return it, or insert it into a table.  You could throw a syntax error in such cases, but, such a limited feature strikes me as more trouble than it would be worth.  

I'm still not convinced that this is NECESSARY, but if it were to be
implemented, I think this would be best.

A special parser hook is probably possible; provided that we're just talking about replacing instances of the invalid syntax 

"x=y:z" with

"function x(...) return y:z(...) end"

I don't think it's a good idea though. The current syntax has the advantage of clarity.  If you need terser code, probably better to write a helper function.  For example:

function metafunctions_to_closures(object)

  local new_object=shallow_copy(object)

  for k,v in pairs(getmetatable(object)) do

     if type(v)=='function' then

         new_object[k] = function(...) return v(new_object,...) end

     end

  end

  return new_object

end

-Sven

[Greg Fitzgerald]

> It sounds to me like this is the time to put one's money where one's
> mouth is and put together a patch so that it can be tested instead of
> merely speculated upon.

Without changing the VM, I did some performance testing using Lua 5.2.
 Using the attached script, you can try:

# __index creates closure
$ perf stat lua main.lua slow
2.24 seconds

# __method bypasses closure creation
$ perf stat lua main.lua fast
1.91 seconds

# bypass helper function
$ perf stat lua main.lua faster
1.12 seconds

# use colon syntax
$ perf stat lua main.lua uberfast
0.86 seconds

As expected, the __method metamethod was a measurable improvement over
using __index and creating closures.  But to my surprise, I had to
cheat quite a bit to get within 20% of the performance of using colon
syntax.  It looks to me like table lookups in Lua are significantly
more expensive than the same lookup using the C API.  Maybe the VM can
get away with a few less typechecks?

My conclusion is that now one really needs to put one's money where
one's mouth is and patch the VM.  I don't have much experience in the
VM code, so if somebody beats me to it, thanks in advance.

-Greg

On Fri, Jul 19, 2013 at 5:51 PM, Coda Highland <[hidden email]> wrote:

> On Fri, Jul 19, 2013 at 5:31 PM, Greg Fitzgerald <[hidden email]> wrote:
>> On Fri, Jul 19, 2013 at 4:02 PM, Javier Guerra Giraldez
>> <[hidden email]> wrote:
>>> local f = o:m
>>>
>>> as syntax sugar to:
>>>
>>> local f = function(...) return o.m(o, ...)  end
>>
>> That would be a good addition too.
>>
>>
>>> i think it's better than Greg's proposal because it doesn't magically
>>> changes the meaning of an existing operator (dot)
>>
>> That's not exactly fair.  It magically changes the dot operator in the
>> same way the __index metamethod magically changes the dot operator.
>> Both are tucked under the hood.  From the user's perspective, the same
>> code could have just as easily been written with no metamethods at
>> all:
>>
>> local funcs = {
>>     inc = function(self)
>>         self.a = self.a + 1
>>         return self
>>     end
>> }
>>
>> local function new(t)
>>    local o = t or {a = 0}
>>    for k,f in pairs(funcs) do
>>       o[k] = function(...) return f(o, ...) end
>>    end
>>    return o
>> end
>>
>> assert(new().inc().inc().inc().a == 3)
>>
>>
>> Using the __index metamethod is a handy way of exposing a Lua library
>> with the familiar syntax of JavaScript, Python, Java, etc.  Using this
>> proposed __method metamethod is a technique for getting that syntax
>> without the performance regression.
>>
>> -Greg
>>
>
> Overloading : to work when not coupled with a parameter list is the
> better solution. It's still important to be able to access the unbound
> version of the method, so you want x.y to still return the unbound
> version, and you don't want to overload the behavior of . any more
> than it already is. Meanwhile, : without a parameter list is currently
> a syntax error, so adding it would cause no incompatibilities and
> minimal overhead (there would still be a __method dispatch on a:b()
> where there wasn't one before, but that overhead can be brought down
> to just a single comparison by requiring that __method be defined on
> the metatable already when setmetatable is called).
>
> That said... I'm not convinced that it would be anything more than
> syntactic sugar. Adding "bound functions" that aren't closures to the
> language... well, I can see how it could be done with less overhead
> than defining a closure, but would it actually make a substantive
> difference, and could it be done CLEANLY?
>
> It sounds to me like this is the time to put one's money where one's
> mouth is and put together a patch so that it can be tested instead of
> merely speculated upon.
>
> /s/ Adam
>

[Sven Olsen]

As expected, the __method metamethod was a measurable improvement over
using __index and creating closures.  But to my surprise, I had to
cheat quite a bit to get within 20% of the performance of using colon
syntax.

I think you're actually selling the colon syntax short in these tests.  You've defined new_colon(t) using:

  return setmetatable(o, {__index = function(t, k)  return funcs[k] end, })

but, most people who use the colon syntax are probably skipping that extra level indirection by using the __index=table special case.  i.e:

  return setmetatable(o, {__index = funcs, })

In my own tests, switching to that syntax speeds up the colon-syntax numbers by about 25%.

I also think there's an important option that you're overlooking here.  I use implicit-closure creation to implement object methods in some of my own code, but, I take some care to avoid repeated creation of new closures.  It's still significantly slower than the colon syntax, but, much of the time, it's far better than just dropping down new closures anytime index is called (on my machine, the approach is about on par with the 'fast' method, assuming the garbage collector is turned off.)

The __method idea you're playing with feels like something that could be a useful hack -- but, I'm worried about some of the strange edge cases it would add to the language.  For example, it's unclear how to handle userdata that have definitions for both __index and __method.  If all you want is a version of the closure-style object syntax with relatively low overhead, I'd suggest looking at some version of lazy closure creation.  It's not as fast as the colon-syntax, but, it's also not nearly as slow as you've been assuming.

-Sven

Here's a lazy implementation that's compatible with your tests:

local function new_lazy(t)

  -- only store cached member functions until the 

  -- gc gets around to freeing them.

  local member_functions=setmetatable({},{__mode="kv"})

  local mt = {}

  

  function mt.__index(t,k)

     -- return t[k], if it exists

    local var = rawget(t,k)

    if var then 

      return var 

    end

    -- check the member function cache

    var = rawget(member_functions,k)

    if var then

      return var 

    end

    -- create a new cached member function

    var = rawget(funcs,k) 

    if var and type(var)=='function' then

      local function self_fun(...)

        return var(t,...)

      end

      rawset(member_functions,k,self_fun)

      return self_fun

    end

    -- otherwise, behave as if __index=funcs

    return funcs[k]

  end

  return setmetatable( t or {a = 0} ,mt)

end

[Mark Hamburg-3]

I've long wanted a few extensions to the behavior around the colon operator:

1. o:m not followed by parameters should do a bind. My inclination is to do the method lookup at the time the binding is created. I wouldn't worry about storage allocation provided no binding was generated on an actual call. This sort of construct is useful whenever building a system with callbacks. The caveat is that it would create strong rather than weak links.

2. o:[ mexpr ]( ... ) would allow for computing the method name in a dispatch. (This is actually a pretty easy compiler change.)

3. The combination of 1 & 2. (Less obviously useful but a natural point of completeness.)

I used to want the colon operator raised to first-class status where it could then have a metamethod, but I never found a formulation that really seemed useful. (Someone can go find my old postings on the topic and point out how much I forget...)

Mark

[Jay Carlson]

On Jul 27, 2013, at 2:47 PM, Mark Hamburg wrote:

> I used to want the colon operator raised to first-class status where it could then have a metamethod, but I never found a formulation that really seemed useful. (Someone can go find my old postings on the topic and point out how much I forget...)

I have a very bad feeling about creating *half* an extra namespace. We might as well make it an lvalue too: __setmethod.

[Coda Highland]

On Sun, Jul 28, 2013 at 4:42 PM, Jay Carlson <[hidden email]> wrote:
> On Jul 27, 2013, at 2:47 PM, Mark Hamburg wrote:
>
>> I used to want the colon operator raised to first-class status where it could then have a metamethod, but I never found a formulation that really seemed useful. (Someone can go find my old postings on the topic and point out how much I forget...)
>
> I have a very bad feeling about creating *half* an extra namespace. We might as well make it an lvalue too: __setmethod.

I still think that __index is sufficient here; I have a very bad
feeling about having a.b and a:b reference different concepts.

/s/ Adam

[Jay Carlson]

On Jul 28, 2013, at 9:50 PM, Coda Highland wrote:

> On Sun, Jul 28, 2013 at 4:42 PM, Jay Carlson <[hidden email]> wrote:
>> On Jul 27, 2013, at 2:47 PM, Mark Hamburg wrote:
>>
>>> I used to want the colon operator raised to first-class status where it could then have a metamethod, but I never found a formulation that really seemed useful. (Someone can go find my old postings on the topic and point out how much I forget...)
>>
>> I have a very bad feeling about creating *half* an extra namespace. We might as well make it an lvalue too: __setmethod.
>
> I still think that __index is sufficient here; I have a very bad
> feeling about having a.b and a:b reference different concepts.

Oh, sure. The next question is how a:b is related to a["b"], and I think the answers get worse and worse until you arrive at a two-namespace Python-object-like table. But reification of *methods* needs some clear reasoning. It's much simpler to look at Mark's (clipped) suggestions of a bare "o:m" as syntactic sugar for

function (obj,method)
  return function (...)
    return obj[method](obj, ...)
  end
end(o,stringify(m))

This doesn't have those philosophical questions lying around; it's just syntax. I dunno how much I'd use that, though. I think this is one of those features you just don't know how much you'd use until you started designing for it, and then of course it's too late.

Actually, I fixed the overly tight binding: my expansion looks up methods by name at invocation rather than at construction. I think the primary goal is coming up with a reasonable syntax for the expression; this is separable from trying to optimize out the table lookup. If you're trying to seek and destroy dynamicism overhead, I think you ought to be poking at the "from table import concat" designs and "static".

Jay

[Coda Highland]

On Mon, Jul 29, 2013 at 11:51 AM, Jay Carlson <[hidden email]> wrote:

> On Jul 28, 2013, at 9:50 PM, Coda Highland wrote:
>
>> On Sun, Jul 28, 2013 at 4:42 PM, Jay Carlson <[hidden email]> wrote:
>>> On Jul 27, 2013, at 2:47 PM, Mark Hamburg wrote:
>>>
>>>> I used to want the colon operator raised to first-class status where it could then have a metamethod, but I never found a formulation that really seemed useful. (Someone can go find my old postings on the topic and point out how much I forget...)
>>>
>>> I have a very bad feeling about creating *half* an extra namespace. We might as well make it an lvalue too: __setmethod.
>>
>> I still think that __index is sufficient here; I have a very bad
>> feeling about having a.b and a:b reference different concepts.
>
> Oh, sure. The next question is how a:b is related to a["b"], and I think the answers get worse and worse until you arrive at a two-namespace Python-object-like table. But reification of *methods* needs some clear reasoning. It's much simpler to look at Mark's (clipped) suggestions of a bare "o:m" as syntactic sugar for
>
> function (obj,method)
>   return function (...)
>     return obj[method](obj, ...)
>   end
> end(o,stringify(m))

I had made the same description several days ago, except I had granted
the possibility of having a "bound function" subtype that isn't a full
closure, which would enable equality checks (do these two variables
refer to the same function bound to the same object? yes? okay,
they're equal).

/s/ Adam

[Mark Hamburg-3]

On Jul 29, 2013, at 11:51 AM, Jay Carlson <[hidden email]> wrote:

> Actually, I fixed the overly tight binding: my expansion looks up methods by name at invocation rather than at construction. I think the primary goal is coming up with a reasonable syntax for the expression; this is separable from trying to optimize out the table lookup.

I've written bind implementations both ways. Whether one cares about delayed method lookup depends on whether you redefine methods on the fly. Whether you should have bind do delayed lookup depends on how closely you want to have o:m mirror o.m.

On the equality testing front, I can see wanting this but I could also see the next step being a desire to have a single value in the universe for o:m so that it could be used as a table key and that would raise the cost of creating such bindings. As a point of reference, these are essentially C# delegates and those, I believe, get constructed anew each time (but I could be wrong on that).

Mark