Lua Needs

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Soni "They/Them" L.
The first reply on this side-chain was not directed at you, but at Sam Pagenkopf.

On Tue, Nov 27, 2018, 19:02 Coda Highland <[hidden email] wrote:
On Tue, Nov 27, 2018 at 2:30 PM Soni L. <[hidden email]> wrote:
>
> Umm, you realize I made Cratera? I thought I made that explicit, but maybe I didn't.
>
> I was replying to someone who had some incorrect ideas about composition.
>
> Sorry for the confusion.

You might have made that explicit and I just missed it. My apologies
on that point.

But you replied to... um... me. And I'm pretty sure I've got my head
on straight when it comes to composition.

So there's definitely some confusion here. Not anything that's an
actual problem, I just think I might have missed a point somewhere.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Coda Highland
On Tue, Nov 27, 2018 at 3:09 PM Soni L. <[hidden email]> wrote:
>
> The first reply on this side-chain was not directed at you, but at Sam Pagenkopf.

Oh, that's really weird. I definitely see that gmail's thread history
agrees with you, but the quote history in that post was my post. Very
odd.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Sam Pagenkopf
I'm very hazy on the benefit of a trait system that relies on self.trait.subtrait.othersubtrait(self). Is it because each trait can remain constant? Why is it more important to have methods in a child operating on a shared pool of variables, when it's possible to put both the variables and methods in children and avoid relying on a large, shared state? It just feels like a very arbitrary separation to me, and goes again the grain of good compositional design.

On Tue, Nov 27, 2018 at 4:12 PM Coda Highland <[hidden email]> wrote:
On Tue, Nov 27, 2018 at 3:09 PM Soni L. <[hidden email]> wrote:
>
> The first reply on this side-chain was not directed at you, but at Sam Pagenkopf.

Oh, that's really weird. I definitely see that gmail's thread history
agrees with you, but the quote history in that post was my post. Very
odd.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Egor Skriptunoff-2
In reply to this post by Coda Highland
On Tue, Nov 27, 2018 at 1:31 AM Coda Highland wrote:
>>>> Lua has little to no support for composition-based OOP
>>> Can you give an example of what you would want here?
>> The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
> What OOP languages do support this "composition-based OOP" ?

Lots of them. C++, Java, C#, Python...

Python's the only one I know of off the top of my head
that can make it look that clean.

How does Python support "composition OOP"?
Do you mean the ability to define __getattr__?
It doesn't look like a "support"; a user has to do all the work manually.
It' very similar to Lua's ability to define __index.

I like the suggestion to make "a:b.c()" a valid Lua syntax.

But the next question from Sony L. would be the following:
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()


Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Coda Highland
On Wed, Nov 28, 2018 at 4:06 AM Egor Skriptunoff
<[hidden email]> wrote:

>
> On Tue, Nov 27, 2018 at 1:31 AM Coda Highland wrote:
>>
>> >>>> Lua has little to no support for composition-based OOP
>> >>> Can you give an example of what you would want here?
>> >> The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
>> > What OOP languages do support this "composition-based OOP" ?
>>
>> Lots of them. C++, Java, C#, Python...
>>
>> Python's the only one I know of off the top of my head
>> that can make it look that clean.
>
>
> How does Python support "composition OOP"?
> Do you mean the ability to define __getattr__?
> It doesn't look like a "support"; a user has to do all the work manually.
> It' very similar to Lua's ability to define __index.

As I said in another post, by default you don't explicitly call out
the component you want to use. Python accomplishes composition via
multiple inheritance ("mixins"), so you'd just write a.c() instead of
a.b.c().

But you're right, being able to tinker with Python's metaclass
functionality is how Python is able to achieve something analogous to
the specific technique under discussion. And yes, it IS like being
able to define __index, except since Python has a native notion of
classes there's a different attitude about doing so.

> I like the suggestion to make "a:b.c()" a valid Lua syntax.
>
> But the next question from Sony L. would be the following:
> How to save intermediate value in this code
> a:b.c()
> a:b.d()
> to rewrite it in optimized way
> local x = a:b
> x.c()
> x.d()

(Nitpick: it would have to be x:c(), not x.c().)

I hinted at the same notion earlier. The problem with doing this at
all is that it's EXPENSIVE -- probably more expensive than just
evaluating the two paths in the first place. You have to allocate an
object to represent the binding, such that using . or : to access its
contents returns b, but using () passes through a. And then you have
to make the function call machinery detect when one of these objects
is being used in order to make sure the right value is passed in the
parameter list. And of course, this allocation will subsequently need
to be garbage-collected... It's really not worth it, especially since
this overhead would apply to ALL uses of : and ALL uses of () even if
it's not needed unless you add EVEN MORE complexity to the parser.

In practice it would be better to write this:

local x = a.b
x.c(a)
x.d(a)

This works with stock Lua or with the proposed a:b.c() syntax patch
and doesn't involve any additional allocations.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Sam Pagenkopf
In reply to this post by Egor Skriptunoff-2
Egor Skripturnoff:
How to save intermediate value in this code
a:b.c()
a:b.d()
to rewrite it in optimized way
local x = a:b
x.c()
x.d()
Okay, but what would you expect from this?
a.b = a:b
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Coda Highland
On Wed, Nov 28, 2018 at 12:18 PM Sam Pagenkopf <[hidden email]> wrote:

>
> Egor Skripturnoff:
>>
>> How to save intermediate value in this code
>> a:b.c()
>> a:b.d()
>> to rewrite it in optimized way
>> local x = a:b
>> x.c()
>> x.d()
>
> Okay, but what would you expect from this?
> a.b = a:b

Using the implementation I was alluding to, that operation would
BASICALLY be a no-op, because the new value of a.b would be a binding
object that exposes the properties of b but gets passed to methods as
if it were a. It would cause issues if you tried to then use a as a
prototype to construct a2, but in and of itself it wouldn't be an
issue.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy
In reply to this post by Egor Skriptunoff-2
Le mer. 28 nov. 2018 à 11:06, Egor Skriptunoff <[hidden email]> a écrit :
But the next question from Sony L. would be the following:
How to save intermediate value in this code
a:b.c()
a:b.d()

This idea a very strange object type in Lua creating an object that references two:
 
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

The way I percieve it is that "x" should be a "proxying" function object (which internally will call function "b"), defined in a closure that holds the value "a". Something like:

local x = function
  local __o___ = a;
  return function(...)
    return b(__o__, ...)
 end
end
x(c)
x(d)

This adds the overhead of one level of function call (even if this is a trailing call that will not consume stack space)
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy


Le jeu. 29 nov. 2018 à 15:22, Philippe Verdy <[hidden email]> a écrit :
Le mer. 28 nov. 2018 à 11:06, Egor Skriptunoff <[hidden email]> a écrit :
But the next question from Sony L. would be the following:
How to save intermediate value in this code
a:b.c()
a:b.d()

This idea a very strange object type in Lua creating an object that references two:
 
to rewrite it in optimized way
local x = a:b
x.c()
x.d()

The way I percieve it is that "x" should be a "proxying" function object (which internally will call function "b"), defined in a closure that holds the value "a". Something like:

Fix:

local x = function(a, b)
  local __o___ = a;
  return function(...)
    return b(__o__, ...)
  end
end
x(c)
x(d)

Sorry: I forgot the (a,b) parameters to the first function... 
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Soni "They/Them" L.
In reply to this post by Coda Highland
In practice it's better to use:

local a = a
a:b.c()
a:d.e()

Because they're components and as such are a property of the object. They may change and as such you should not cache them. Consider them part of the object, rather than separate objects.

It's okay to cache their functions tho, as expected.

On Wed, Nov 28, 2018, 15:52 Coda Highland <[hidden email] wrote:
On Wed, Nov 28, 2018 at 4:06 AM Egor Skriptunoff
<[hidden email]> wrote:
>
> On Tue, Nov 27, 2018 at 1:31 AM Coda Highland wrote:
>>
>> >>>> Lua has little to no support for composition-based OOP
>> >>> Can you give an example of what you would want here?
>> >> The classic Cratera[1] `foo:[component].method()` (note the colon) syntax.
>> > What OOP languages do support this "composition-based OOP" ?
>>
>> Lots of them. C++, Java, C#, Python...
>>
>> Python's the only one I know of off the top of my head
>> that can make it look that clean.
>
>
> How does Python support "composition OOP"?
> Do you mean the ability to define __getattr__?
> It doesn't look like a "support"; a user has to do all the work manually.
> It' very similar to Lua's ability to define __index.

As I said in another post, by default you don't explicitly call out
the component you want to use. Python accomplishes composition via
multiple inheritance ("mixins"), so you'd just write a.c() instead of
a.b.c().

But you're right, being able to tinker with Python's metaclass
functionality is how Python is able to achieve something analogous to
the specific technique under discussion. And yes, it IS like being
able to define __index, except since Python has a native notion of
classes there's a different attitude about doing so.

> I like the suggestion to make "a:b.c()" a valid Lua syntax.
>
> But the next question from Sony L. would be the following:
> How to save intermediate value in this code
> a:b.c()
> a:b.d()
> to rewrite it in optimized way
> local x = a:b
> x.c()
> x.d()

(Nitpick: it would have to be x:c(), not x.c().)

I hinted at the same notion earlier. The problem with doing this at
all is that it's EXPENSIVE -- probably more expensive than just
evaluating the two paths in the first place. You have to allocate an
object to represent the binding, such that using . or : to access its
contents returns b, but using () passes through a. And then you have
to make the function call machinery detect when one of these objects
is being used in order to make sure the right value is passed in the
parameter list. And of course, this allocation will subsequently need
to be garbage-collected... It's really not worth it, especially since
this overhead would apply to ALL uses of : and ALL uses of () even if
it's not needed unless you add EVEN MORE complexity to the parser.

In practice it would be better to write this:

local x = a.b
x.c(a)
x.d(a)

This works with stock Lua or with the proposed a:b.c() syntax patch
and doesn't involve any additional allocations.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy


Le jeu. 29 nov. 2018 à 16:31, Soni L. <[hidden email]> a écrit :
In practice it's better to use:

local a = a
a:b.c()
a:d.e()

Because they're components and as such are a property of the object. They may change and as such you should not cache them. Consider them part of the object, rather than separate objects.

you could have the case

 local a = a
 a:b.d()
 a:c.d()

where the desire is to have a way to select one of the "d" component methods (or properties) independantly of the path (possibly complex and resulting from a function call) from which it is accessed, i.e. something like:

 local a = a
 local x = test and a:b or a:c
 x:d()

The function+closure based solution I proposed also works to hold the two references (the base object, and the target function which is any component accessible from the base object via an unknown path, when this targetr function needs in its first parameter a reference to the base object) for this case:

    local a = a
local x = function(a, b, c)
  local __o___ = a;
  return function(...)
    return (test and b or c)(__o__, ...)
  end
end
x(d)
 
imagine that "d" is a generic "get" or "set" function used to define some property or subproperty of an object. Such concept is used in oop based on facets (I've used it on a L4G programming language where there was no "method" at all, only dynamic properties.

But such concept is not dead and is widely used for implementing UI components, that are exposing only properties to define the behavior and content/visible aspect of the UI component and propagate all side effects in a component tree, without forcing users to really write any line of code, only get/set values of properties, these values having simple types : generally a single string, a numeric or enum value, a color value, or a locale identifier (with its implied BCP 47 fallback semantics, something that strings alone don't have; such value having also other subproperties like the language code, the script code, or sets of other locales that the locale identifer match during fallbacks). Several "Visual" programming languages (and thier associated UI designer) use such concept with the additional benefit that they are generally easy to serialize (objects that have programmable method facets are much harder to serialize correctly and safely)
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Coda Highland
On Thu, Nov 29, 2018 at 11:08 AM Philippe Verdy <[hidden email]> wrote:
>  local a = a
>  local x = test and a:b or a:c
>  x:d()

I mean...

a:[test and 'b' or 'c'].d()

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy
Yes, that's exactly what I understood: you have a variable/conditional "path" to the property of object "a" that interest you, and then "d()" is a method on that property that must be used where you want to it to act on the object "a".

My approach in pure Lua, using a closure to enclose the value of "a", plus a function in Lua to build an intermediate object should work for your case even if you modified it:

    a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample') 

which is also writable in Lua as:

    (function(a)
        local __o__ = a;
        return function(...)
            return  (__o__[test and 'b' or 'c']).d(__o__, ...)
        end
    end)(d,"additional parameters:", {12, 34}, 'sample')
 
See how I added also the other parameters (which are passed to the internal function here using a vararg, but the vararg is not mandatory).

The typical usage would be to create dynamic properties in a OOP language with facets:

    value = (some expression selecting an object):[some expression computing a property name].get();

    (some expression selecting an object):[some expression selecting a property name in the object].set(value);

which can be written as:

   value = (function(o)
        local __o___ =  o;
        return function() return  __o__.get(__o__[some expression selecting a property name in the object]) end
    end)(get, some expression selecting an object);

   (function(o, v)
        local __o___ =  o;
        return function() return  __o__.set(__o__[some expression selecting a property name in the object]) end
    end)(some expression selecting an object, value);

There's many variants possible depending on which part is variable or not or dependant on the object that the "apparently simple" syntax
    a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed this:
    a:b:c.d(e)
which could be understood differently as any one of:
    ((a:b):c).d(e)
    (a:(b:c)).d(e)
    a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final dynamic method named "d" here.


Le ven. 30 nov. 2018 à 02:56, Coda Highland <[hidden email]> a écrit :
On Thu, Nov 29, 2018 at 11:08 AM Philippe Verdy <[hidden email]> wrote:
>  local a = a
>  local x = test and a:b or a:c
>  x:d()

I mean...

a:[test and 'b' or 'c'].d()

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy
For this reason, the generalization of the ":" pseudo-operator of Lua is not easy: it would create lot of overhead at compile-time and runtime to create many closure objects, that are compiled using a static "prototype" which is an  integer-indexed array mapping the numbered upvalues to their bound object, and then used at runtime to bind each actual upvalues to other accessible external objects

The ":" pseudo-operator is a syntaxic sugar of Lua which can always be rewritten using the "." pseudo-operator.
But as well the "." pseudo-operator is a syntax sugar of Lua which can always be rewritten using the "[]" table indexing operator (this one is the only real operator).
Combining the two sugars (pseudo-operators ":" and ".") needs to be limited to avoid ambiguities.

The only safe way (without needing the overhead of instanciating too many additional closures) is to declare local variables in separate statements to precompute the object references you need to pass further down for forming the function call. You can finally always resolve all cases using the "[]" table indexing operator only.

Then use the final "()" function call operator where you'll pass the explicit references by these temporary variables and minimize the number of needed closures (because they are costly on the heap, and may then be quite slow if closures are used extensively in tight loops with many repetitions).

---

Note that each closure is an actual object (a simple indexed array, not a full table with random keys) that is allocated on the heap (then instanciated by binding the references to the actual variables), so it has a runtime cost on the garbage collector, each time we call any Lua function that uses any upvalues or returns a list with more than one value).
This object needs then to be garbage-collected when exiting the closure (after each function call). No such closure needs to be allocated is the function has no use of external variables in its lexical scope and uses only its parameters and the reference to its own metatable which keeps data across all calls from the same thread.

The function's own metatable is then like a "thread local storage" (TLS) that does not need to be allocated at each call; this TLS is used only for data that can be modified, it does not contain static data which is stored in its constant pool, including the static prototype built by the Lua compiler and which is used to instanciate the mapping for the closure of the function to its lexical context). This metatable also contains a reference to its external dynamic environment (not bound directly by the closure). C functions are a bit different because instead of a full table, their environment is an indexed array of userdata, but userdata bound to c functions are also thread local storage, allocated on the heap only once per thread. These TLS avoid the extra cost of closures at each call by limiting a lot the use of heap allocation.

If you have a program that makes many loops across function calls and you see that it uses lot of heap and garbage collector is too much sollicited, it may help to see if you can avoid the closures and use TLS instead, i.e. the environment of functions (which is unique per thread, and kept across calls withoout being allocated and garbage collected at each call like closures).


Le ven. 30 nov. 2018 à 14:48, Philippe Verdy <[hidden email]> a écrit :
Yes, that's exactly what I understood: you have a variable/conditional "path" to the property of object "a" that interest you, and then "d()" is a method on that property that must be used where you want to it to act on the object "a".

My approach in pure Lua, using a closure to enclose the value of "a", plus a function in Lua to build an intermediate object should work for your case even if you modified it:

    a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample') 

which is also writable in Lua as:

    (function(a)
        local __o__ = a;
        return function(...)
            return  (__o__[test and 'b' or 'c']).d(__o__, ...)
        end
    end)(d,"additional parameters:", {12, 34}, 'sample')
 
See how I added also the other parameters (which are passed to the internal function here using a vararg, but the vararg is not mandatory).

The typical usage would be to create dynamic properties in a OOP language with facets:

    value = (some expression selecting an object):[some expression computing a property name].get();

    (some expression selecting an object):[some expression selecting a property name in the object].set(value);

which can be written as:

   value = (function(o)
        local __o___ =  o;
        return function() return  __o__.get(__o__[some expression selecting a property name in the object]) end
    end)(get, some expression selecting an object);

   (function(o, v)
        local __o___ =  o;
        return function() return  __o__.set(__o__[some expression selecting a property name in the object]) end
    end)(some expression selecting an object, value);

There's many variants possible depending on which part is variable or not or dependant on the object that the "apparently simple" syntax
    a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed this:
    a:b:c.d(e)
which could be understood differently as any one of:
    ((a:b):c).d(e)
    (a:(b:c)).d(e)
    a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final dynamic method named "d" here.


Le ven. 30 nov. 2018 à 02:56, Coda Highland <[hidden email]> a écrit :
On Thu, Nov 29, 2018 at 11:08 AM Philippe Verdy <[hidden email]> wrote:
>  local a = a
>  local x = test and a:b or a:c
>  x:d()

I mean...

a:[test and 'b' or 'c'].d()

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy
Note, my use of "TLS" here is a bit abusive, because it is not strictly thread-local: objects referenced in closures or function environemnt is accessible in other threads by communicating them via yield(...) or resume(...), because they are all allocated on the same heap shared by all threads/coroutines created from the first one which is started by the instance of the Lua engine (which runs all its "thread" objects in the same OS thread).

But "TLS" or "thread local storage" must be understood by interpreting the "thread" term as being "Lua thread/coroutine", where each one can have its own separate storage; there's no limitation however for allowing other threads to use and modify these storages if they are exchanged by yield(...) and resume(...), passing their references via the stack of each thread. Stacks are very fast compared to the heap and you can even get much higher performance by using yield() and resume() to invoke threads running a service loop, instead of calling functions and passing references by closures.

Le ven. 30 nov. 2018 à 15:39, Philippe Verdy <[hidden email]> a écrit :
For this reason, the generalization of the ":" pseudo-operator of Lua is not easy: it would create lot of overhead at compile-time and runtime to create many closure objects, that are compiled using a static "prototype" which is an  integer-indexed array mapping the numbered upvalues to their bound object, and then used at runtime to bind each actual upvalues to other accessible external objects

The ":" pseudo-operator is a syntaxic sugar of Lua which can always be rewritten using the "." pseudo-operator.
But as well the "." pseudo-operator is a syntax sugar of Lua which can always be rewritten using the "[]" table indexing operator (this one is the only real operator).
Combining the two sugars (pseudo-operators ":" and ".") needs to be limited to avoid ambiguities.

The only safe way (without needing the overhead of instanciating too many additional closures) is to declare local variables in separate statements to precompute the object references you need to pass further down for forming the function call. You can finally always resolve all cases using the "[]" table indexing operator only.

Then use the final "()" function call operator where you'll pass the explicit references by these temporary variables and minimize the number of needed closures (because they are costly on the heap, and may then be quite slow if closures are used extensively in tight loops with many repetitions).

---

Note that each closure is an actual object (a simple indexed array, not a full table with random keys) that is allocated on the heap (then instanciated by binding the references to the actual variables), so it has a runtime cost on the garbage collector, each time we call any Lua function that uses any upvalues or returns a list with more than one value).
This object needs then to be garbage-collected when exiting the closure (after each function call). No such closure needs to be allocated is the function has no use of external variables in its lexical scope and uses only its parameters and the reference to its own metatable which keeps data across all calls from the same thread.

The function's own metatable is then like a "thread local storage" (TLS) that does not need to be allocated at each call; this TLS is used only for data that can be modified, it does not contain static data which is stored in its constant pool, including the static prototype built by the Lua compiler and which is used to instanciate the mapping for the closure of the function to its lexical context). This metatable also contains a reference to its external dynamic environment (not bound directly by the closure). C functions are a bit different because instead of a full table, their environment is an indexed array of userdata, but userdata bound to c functions are also thread local storage, allocated on the heap only once per thread. These TLS avoid the extra cost of closures at each call by limiting a lot the use of heap allocation.

If you have a program that makes many loops across function calls and you see that it uses lot of heap and garbage collector is too much sollicited, it may help to see if you can avoid the closures and use TLS instead, i.e. the environment of functions (which is unique per thread, and kept across calls withoout being allocated and garbage collected at each call like closures).


Le ven. 30 nov. 2018 à 14:48, Philippe Verdy <[hidden email]> a écrit :
Yes, that's exactly what I understood: you have a variable/conditional "path" to the property of object "a" that interest you, and then "d()" is a method on that property that must be used where you want to it to act on the object "a".

My approach in pure Lua, using a closure to enclose the value of "a", plus a function in Lua to build an intermediate object should work for your case even if you modified it:

    a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample') 

which is also writable in Lua as:

    (function(a)
        local __o__ = a;
        return function(...)
            return  (__o__[test and 'b' or 'c']).d(__o__, ...)
        end
    end)(d,"additional parameters:", {12, 34}, 'sample')
 
See how I added also the other parameters (which are passed to the internal function here using a vararg, but the vararg is not mandatory).

The typical usage would be to create dynamic properties in a OOP language with facets:

    value = (some expression selecting an object):[some expression computing a property name].get();

    (some expression selecting an object):[some expression selecting a property name in the object].set(value);

which can be written as:

   value = (function(o)
        local __o___ =  o;
        return function() return  __o__.get(__o__[some expression selecting a property name in the object]) end
    end)(get, some expression selecting an object);

   (function(o, v)
        local __o___ =  o;
        return function() return  __o__.set(__o__[some expression selecting a property name in the object]) end
    end)(some expression selecting an object, value);

There's many variants possible depending on which part is variable or not or dependant on the object that the "apparently simple" syntax
    a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed this:
    a:b:c.d(e)
which could be understood differently as any one of:
    ((a:b):c).d(e)
    (a:(b:c)).d(e)
    a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final dynamic method named "d" here.


Le ven. 30 nov. 2018 à 02:56, Coda Highland <[hidden email]> a écrit :
On Thu, Nov 29, 2018 at 11:08 AM Philippe Verdy <[hidden email]> wrote:
>  local a = a
>  local x = test and a:b or a:c
>  x:d()

I mean...

a:[test and 'b' or 'c'].d()

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Coda Highland
In reply to this post by Philippe Verdy
On Fri, Nov 30, 2018 at 8:39 AM Philippe Verdy <[hidden email]> wrote:
> For this reason, the generalization of the ":" pseudo-operator of Lua is not easy: it would create lot of overhead at compile-time and runtime to create many closure objects, that are compiled using a static "prototype" which is an  integer-indexed array mapping the numbered upvalues to their bound object, and then used at runtime to bind each actual upvalues to other accessible external objects

I agree, and have agreed since the beginning: the generalization is
inappropriate. Limit it to only using one per function call expression
and don't have it generate any sort of extra binding object; it simply
defines which object in the lookup is the one passed to the "self"
parameter. This is easy, efficient, useful, and most importantly
doesn't introduce overhead for the existing use case.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Soni "They/Them" L.
In reply to this post by Philippe Verdy
I think you're just overcomplicating it tbh

On Fri, Nov 30, 2018, 12:39 Philippe Verdy <[hidden email] wrote:
For this reason, the generalization of the ":" pseudo-operator of Lua is not easy: it would create lot of overhead at compile-time and runtime to create many closure objects, that are compiled using a static "prototype" which is an  integer-indexed array mapping the numbered upvalues to their bound object, and then used at runtime to bind each actual upvalues to other accessible external objects

The ":" pseudo-operator is a syntaxic sugar of Lua which can always be rewritten using the "." pseudo-operator.
But as well the "." pseudo-operator is a syntax sugar of Lua which can always be rewritten using the "[]" table indexing operator (this one is the only real operator).
Combining the two sugars (pseudo-operators ":" and ".") needs to be limited to avoid ambiguities.

The only safe way (without needing the overhead of instanciating too many additional closures) is to declare local variables in separate statements to precompute the object references you need to pass further down for forming the function call. You can finally always resolve all cases using the "[]" table indexing operator only.

Then use the final "()" function call operator where you'll pass the explicit references by these temporary variables and minimize the number of needed closures (because they are costly on the heap, and may then be quite slow if closures are used extensively in tight loops with many repetitions).

---

Note that each closure is an actual object (a simple indexed array, not a full table with random keys) that is allocated on the heap (then instanciated by binding the references to the actual variables), so it has a runtime cost on the garbage collector, each time we call any Lua function that uses any upvalues or returns a list with more than one value).
This object needs then to be garbage-collected when exiting the closure (after each function call). No such closure needs to be allocated is the function has no use of external variables in its lexical scope and uses only its parameters and the reference to its own metatable which keeps data across all calls from the same thread.

The function's own metatable is then like a "thread local storage" (TLS) that does not need to be allocated at each call; this TLS is used only for data that can be modified, it does not contain static data which is stored in its constant pool, including the static prototype built by the Lua compiler and which is used to instanciate the mapping for the closure of the function to its lexical context). This metatable also contains a reference to its external dynamic environment (not bound directly by the closure). C functions are a bit different because instead of a full table, their environment is an indexed array of userdata, but userdata bound to c functions are also thread local storage, allocated on the heap only once per thread. These TLS avoid the extra cost of closures at each call by limiting a lot the use of heap allocation.

If you have a program that makes many loops across function calls and you see that it uses lot of heap and garbage collector is too much sollicited, it may help to see if you can avoid the closures and use TLS instead, i.e. the environment of functions (which is unique per thread, and kept across calls withoout being allocated and garbage collected at each call like closures).


Le ven. 30 nov. 2018 à 14:48, Philippe Verdy <[hidden email]> a écrit :
Yes, that's exactly what I understood: you have a variable/conditional "path" to the property of object "a" that interest you, and then "d()" is a method on that property that must be used where you want to it to act on the object "a".

My approach in pure Lua, using a closure to enclose the value of "a", plus a function in Lua to build an intermediate object should work for your case even if you modified it:

    a:[test and 'b' or 'c'].d("additional parameters:", {12, 34}, 'sample') 

which is also writable in Lua as:

    (function(a)
        local __o__ = a;
        return function(...)
            return  (__o__[test and 'b' or 'c']).d(__o__, ...)
        end
    end)(d,"additional parameters:", {12, 34}, 'sample')
 
See how I added also the other parameters (which are passed to the internal function here using a vararg, but the vararg is not mandatory).

The typical usage would be to create dynamic properties in a OOP language with facets:

    value = (some expression selecting an object):[some expression computing a property name].get();

    (some expression selecting an object):[some expression selecting a property name in the object].set(value);

which can be written as:

   value = (function(o)
        local __o___ =  o;
        return function() return  __o__.get(__o__[some expression selecting a property name in the object]) end
    end)(get, some expression selecting an object);

   (function(o, v)
        local __o___ =  o;
        return function() return  __o__.set(__o__[some expression selecting a property name in the object]) end
    end)(some expression selecting an object, value);

There's many variants possible depending on which part is variable or not or dependant on the object that the "apparently simple" syntax
    a:b.c(d)
does not disambiguate clearly. And this would be even worse if we allowed this:
    a:b:c.d(e)
which could be understood differently as any one of:
    ((a:b):c).d(e)
    (a:(b:c)).d(e)
    a:((b:c).d)(e)
depending on associativity, and which object must be passed to the final dynamic method named "d" here.


Le ven. 30 nov. 2018 à 02:56, Coda Highland <[hidden email]> a écrit :
On Thu, Nov 29, 2018 at 11:08 AM Philippe Verdy <[hidden email]> wrote:
>  local a = a
>  local x = test and a:b or a:c
>  x:d()

I mean...

a:[test and 'b' or 'c'].d()

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy


Le sam. 1 déc. 2018 à 02:10, Soni L. <[hidden email]> a écrit :
I think you're just overcomplicating it tbh

That's your opinion. It's a fact that "." and ":" are syntaxic sugars of Lua and are not real operators. They cannot interact freely But all can be made with closures (for binding the upvalues) and with the "[]" indexing operator (in tables). And a fact as well that "self" is not a keyword but a conventional name of the first parameter of a function that can be called with ":".

"self" is just the *implicit* parameter name used when we DECLARE a function with the ":" syntaxic sugar. We an do everything in Lua without ever using "." or ":", but only using "[]" and passing all parameters explicitly. for this reason "self" is not even a reserved keyword (it can be renamed as we want if we don't use the ":"-enhanced function declaration).

Once you understand it, there are three ways to pass parameters to functions:
- by parameters on stack, which become "registers" just like other local variables declared with "local" in the function body (the fastest method, but limited to ~250 variables): this includes the "self" parameter.
- by the function's metatable, which includes its environment (this is fast too because it is allocated on the heap only once by thread and preserved across multiple calls in the same thread/coroutine: there's no limit on the number of parameters): this includes the access to the "global" environment (actually not global, it's not static like in C but scoped recursively and specific to each thread; it's sort of "TLS" except that multiple threads can communicate objects as all these objects are allocated on the same global heap using the same Allocator, if all threads are created in Lua from the same main thread instanciated in C by the Lua engine)
- by external variables bound into upvalues with the closure (it is the slowest method because the closure is allocated on each call and needs to be garbage collected and it stresses a lot the garbage collector: this woks for returning multiple values from the function)

The closures are interesting and facilitates a lot programming in Lua, but it has a significant runtime cost and stresses the GC at each call.

If there's something to do to the Lua engine is to rework the way upvalues are allocated: they should be on the call stack as much as possible (e.g. for up to ~250 upvalues) and not on the heap. This would dramatically increase the performance of closures and reduce a lot the stress on the GC, they would become as fast as function parameters and local variables, working like "registers" (with negative index), and would simplify also the instruction set for the bytecode and the complexity of the compiler, which would also not need to create static "prototypes" for each closure and make any run-time binding (but this would require to increase a bit the size of stacks per thread (stacks are allocated only once on the heap before the thread starts): this increase would have no cost because we save lot of uses on the heap and would save lot of work that needs to be made by the GC... The global memory footprint would be much lower wioth higher performance too.



Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Dirk Laurie-2
Op Sa. 1 Des. 2018 om 17:08 het Philippe Verdy <[hidden email]> geskryf:

> If there's something to do to the Lua engine is to rework the way upvalues are allocated: they should be on the call stack as much as possible (e.g. for up to ~250 upvalues) and not on the heap.

The way I understand it, upvalues are not on the heap. They are on the
main execution stack below the bottom of the current function's stack
frame.

Reply | Threaded
Open this post in threaded view
|

Re: Lua Needs

Philippe Verdy
Ask yourself what are LUA_TCLOSURE objects (yes in the heap...) and see how they are correlated with the static "prototypes" generated by the compiler which instructs how to bind caller's variables to callee's upvalues. The actual values are on the stack but at unknown position from the callee, so a mapping  LUA_TCLOSURE object is allocated on the heap.

Ask yourself why these closure objects exist: this is because functions may be recursive (and not necessarily by trailing calls), so the variables in closures may refer not to the immediate parent caller frame but to some ancestor frame at arbitrary number of levels in the calls stack). To avoid every access to the closure varaible to pass through a chain, there's a need for a mapping which is created and initialized before each call to rebind each variable for the next inner call (according to the closure's prototype): such object is allocated only if there are external variables used by the function that are not local to the function itself, they are not directly within the stack window.

This explains also because the bytecode needs separate opcodes for accessing registers and upvalues: if they could be directly on the stack, it would be enough to reference upvalues with negative register indexes and then use the stack as a sliding window, like we do in C/C++ call conventions (except that stack indexes in C/C++ are negative for parameters, and positive for local variables, some of the former being cached in actual registers, but still supported by "shadow" variables on the stack, allocated at fixed positions in the stack fram by the compiler or just pushed before the actual parameters and popped to restore these registers after the call).

There's been performance tests that show that closures are not so fast, they can create massive amounts of garbage collected objects (with internal type LUA_TCLOSURE). I think this behavior very curious, and the current implementation that allocates the LUA_TCLOSURE objects on the heap is not the best option, the mapping could be allocated directly in the stack of local variables/registers of the caller (and all these closure objects used by the caller could be merged to a single one, i.e. as the largest closure object needed for inner calls, merged like in an union. The closure objects themselves to not hold any variable value, these are just simple mappings from a small fixed integer sets (between 1 and the number of upvalues of the called function) and variable integers (absolute indexes in the thread's stack where the actual variable is located).

The byte code is not as optimized as it could be: the register numbers are only positive, the upvalue numbers are also only positive, they could forml a single set (positive integers for local registers, negative integers for upvalues, meaning that they are used to index the entries in the closure object to get access to the actual varaible located anywhere on the stack, outside of the immediate parent frame). The generated bytecode is not as optimal as it could be because various operations can only work on registers or constants (like ADD r1,r2,r3) so temporary registers must be allocated by the compiler (let's remember that the number of registers is limited). As well Lua's default engine treat all registers the same, when most of them will work with a single r0 register (an "accumulator") which could be implicit for most instructions, and this would reduce the instruction sizes (currently 32-bit or 64 bits), which is inefficient as it uses too much the CPU L1 data cache.

I'm convinced that the current approach in the existing Lua VM engine and its internal instruction set can be largely improved for better performance, without really changing the language itself, to get better data locality (smaller instruction sizes, less wasted unused bit fields), and elimination of uses of the heap for closures (to dramatically reduce the stress on the garbage collector)



Le sam. 1 déc. 2018 à 16:51, Dirk Laurie <[hidden email]> a écrit :
Op Sa. 1 Des. 2018 om 17:08 het Philippe Verdy <[hidden email]> geskryf:

> If there's something to do to the Lua engine is to rework the way upvalues are allocated: they should be on the call stack as much as possible (e.g. for up to ~250 upvalues) and not on the heap.

The way I understand it, upvalues are not on the heap. They are on the
main execution stack below the bottom of the current function's stack
frame.
123