# [NoW] Deja vu

19 messages
Open this post in threaded view
|

## [NoW] Deja vu

 t = {}for j = 1, 2 do   t[j] = function(x) return 2*x endendprint(t[1] == t[2])Lua 5.1    falseLua 5.2    trueLua 5.3    trueLua 5.4    falseIt seems that the equality between function values has returned back to its Lua 5.1 semantics.1)Why "function values caching" has been removed from Lua 5.4?2)Could we rely on the fact that in Lua 5.4+ any created function is unique value?
Open this post in threaded view
|

## Re: [NoW] Deja vu

 This looks like a bug: even if the functions are declared multiple times (in a loop), they are all in the same context (same closure), and should resolve as one, because even their upvalues will be necessarily the same; here the only thing that changes is the current value of the outer variable j=1,2, whose value if not even used in the function and not even part of its closure (if j was part of the closure and used like " function(x) return 2*x+j end  " the value of j would be kept in that closure in upvalues even after the foor loop exits and the local vairavle j no longer exists: that variable would remain in the closure, and in that case the closure should be duplicated creating two separate functions). For me, as there's a single closure instance, the test should return true in Lua 5.4, like in Lua 5.2 and 5.3; returning to the bad Lua 5.1 behavior is a regression, it just means that we can no longer unify function declarations that are explicitly created in the same closure and not depending on other external variables.Le jeu. 8 août 2019 à 16:25, Egor Skriptunoff <[hidden email]> a écrit :t = {}for j = 1, 2 do   t[j] = function(x) return 2*x endendprint(t[1] == t[2])Lua 5.1    falseLua 5.2    trueLua 5.3    trueLua 5.4    falseIt seems that the equality between function values has returned back to its Lua 5.1 semantics.1)Why "function values caching" has been removed from Lua 5.4?2)Could we rely on the fact that in Lua 5.4+ any created function is unique value?
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Egor Skriptunoff-2 On 2019-08-07 8:28 p.m., Egor Skriptunoff wrote: > t = {} > for j = 1, 2 do >    t[j] = function(x) return 2*x end > end > print(t[1] == t[2]) > > Lua 5.1    false > Lua 5.2    true > Lua 5.3    true > Lua 5.4    false > > It seems that the equality between function values has returned back > to its Lua 5.1 semantics. > > 1) > Why "function values caching" has been removed from Lua 5.4? > > 2) > Could we rely on the fact that in Lua 5.4+ any created function is > unique value? Lua 5.4 has const locals in loops. Each loop iteration gets a new j, which is const. That just means each closure is using a different upvalue, and gets closed on each loop iteration. Try calling them! (Well, I could be wrong, I guess. Correct me if I'm wrong.)
Open this post in threaded view
|

## Re: [NoW] Deja vu

 On 2019-08-08 6:06 p.m., Soni "They/Them" L. wrote: > > > On 2019-08-07 8:28 p.m., Egor Skriptunoff wrote: >> t = {} >> for j = 1, 2 do >>    t[j] = function(x) return 2*x end >> end >> print(t[1] == t[2]) >> >> Lua 5.1    false >> Lua 5.2    true >> Lua 5.3    true >> Lua 5.4    false >> >> It seems that the equality between function values has returned back >> to its Lua 5.1 semantics. >> >> 1) >> Why "function values caching" has been removed from Lua 5.4? >> >> 2) >> Could we rely on the fact that in Lua 5.4+ any created function is >> unique value? > > Lua 5.4 has const locals in loops. Each loop iteration gets a new j, > which is const. > > That just means each closure is using a different upvalue, and gets > closed on each loop iteration. Try calling them! > > (Well, I could be wrong, I guess. Correct me if I'm wrong.) Nvm the previous message, I misread the OP.
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Soni "They/Them" L. >Lua 5.4 has const locals in loops. Each loop iteration gets a new j, >which is const. That is an interesting possibility. That said, the anonymous funcions in the example don't capture any upvalues so in theory there should be no difference.
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Soni "They/Them" L. May be, but this "j" upvalue is not referenced at all in the function; there's then no point at all to create a new separate closure for each distinct value of the const "j".All this causes unnecessary stress on the memory allocator (and later on the garbage collector): imagine that instead of performing 2 loops, you loop one million times, it will then allocate one million objects with needless contents for the distinct j constant never used: that's what I all a regression, which can be severe and can affect serious projects, where there are lot of loops spread everywhere and the regression will create really too many needless distinct closures, causing a severe degradation in terms of memory usage.In fact a compiler should make all efforts possible to reduce what is not needed to include at all in the set of upvalues of all closures (this is what Javascript does by default and it is even part of its core specifications).Then if the generated reduced closures are identical, there's no need at all to duplicate them. In the example given, the closure should always have an empty set of upvalues, so the same function closure can be safely shared: the value of j does not matter at all, it's never accessed, never needed by the function body.So I consider this to be a severe regression bug in Lua 5.4.Le jeu. 8 août 2019 à 23:06, Soni "They/Them" L. <[hidden email]> a écrit : On 2019-08-07 8:28 p.m., Egor Skriptunoff wrote: > t = {} > for j = 1, 2 do >    t[j] = function(x) return 2*x end > end > print(t[1] == t[2]) > > Lua 5.1    false > Lua 5.2    true > Lua 5.3    true > Lua 5.4    false > > It seems that the equality between function values has returned back > to its Lua 5.1 semantics. > > 1) > Why "function values caching" has been removed from Lua 5.4? > > 2) > Could we rely on the fact that in Lua 5.4+ any created function is > unique value? Lua 5.4 has const locals in loops. Each loop iteration gets a new j, which is const. That just means each closure is using a different upvalue, and gets closed on each loop iteration. Try calling them! (Well, I could be wrong, I guess. Correct me if I'm wrong.)
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Egor Skriptunoff-2 > Why "function values caching" has been removed from Lua 5.4? See https://github.com/lua/lua/commit/e8c779736f3029df353038352c14c8ab63728811
Open this post in threaded view
|

## Re: [NoW] Deja vu

 On 2019-08-08 10:10 p.m., Luiz Henrique de Figueiredo wrote: > > Why "function values caching" has been removed from Lua 5.4? > > See https://github.com/lua/lua/commit/e8c779736f3029df353038352c14c8ab63728811> say that to cratera: (look for "INPUT"/"OUTPUT" markers) ----------------------------------------- INPUT -- Cratera-specific tests. Run Lua test suite separately. local t = setmetatable({}, { __tostring=function()return"t"end}) local F = {} local T = {} t.t = t t.tt = t t[T] = t t.f = print t.ff = print t.g = function(self, a) print(self, a[1]) end t[F] = print local _f="f" local _t="t" -- print("------ t:[k]()") -- t:f(1) -- plain old lua -- t:[_f](2) -- simple string key in register -- t:[string.char(string.byte("f"))](3,32,33) -- string key from function -- t:["f".."f"](4) -- string key from concatenation -- t:["f"..string.sub("afun",2,2)](5,52,53) -- concatenation with function result -- t:[(string.sub("afun",2,2))](6,62,63) -- function result in parentheses -- t:[(function()return"f"end)()](7) -- closure in key -- -- be careful with the ambiguous function call!!! -- ;(function()return t end)():[(function()return"f"end)()](8) -- closure in object and in key -- t:[F](9) -- object key -- standard lua tests (compiler/passthrough) do    print("------ standard lua tests (compiler/passthrough)")    local x    t["t"]:f(1) end print("------ t:[k].f()") t:t.f(1) -- string identifier t:[_t].f(2) -- string key in register t:[string.char(string.byte("t"))].f(3,32,33) -- string key from function t:["t".."t"].f(4) -- string key from concatenation t:["t"..string.sub("atable",2,2)].f(5,52,53) -- concatenation with function result t:[(string.sub("atable",2,2))].f(6,62,63) -- function result in parentheses t:[(function()return"t"end)()].f(7) -- closure in key do end(function()return t end)():[(function()return"t"end)()].f(8) -- closure in object and in key, with "end" keyword at the start -- be careful with the ambiguous function call!!! ;(function()return t end)():[(function()return"t"end)()].f(9) -- closure in object and in key, with semicolon at the start t:[T].f(10) -- object key _=(t:[_t].f(11)) -- inside () t:[_t].g {12} -- table call t:[_t].f "13" -- string call entity = {} inventory = {get=false, set=false, size=false} inventory.new=function(size)    local t = {size=function() return size end}    function t.set(e, i, o)      if i <= 0 or i > e:[inventory].size() then error() end      e[inventory][i] = o    end    function t.get(e, i)      if i <= 0 or i > e:[inventory].size() then error() end      return e[inventory][i]    end    return t end inventory.of=function(e) -- helper for passing standalone inventories around    return {get=function(...)return e:[inventory].get(...)end, set=function(...)return e:[inventory].set(...)end, size=function(...)return e:[inventory].size(...)end} end entity[inventory] = inventory.new(5) entity:[inventory].set(1, "Hello World!") print(entity:[inventory].get(1)) for i=1, entity:[inventory].size() do    print(i, entity:[inventory].get(i)) end local myinv = inventory.of(entity) for i=1, myinv.size() do    print("wrapped", i, myinv.get(i)) end ----------------------------------------- OUTPUT   local t = setmetatable ( { } , { __tostring = function ( ) return "t" end } )   local F = { }   local T = { }   t . t = t   t . tt = t   t [ T ] = t   t . f = print   t . ff = print   t . g = function ( self , a ) print ( self , a [ (1) ] ) end   t [ F ] = print   local _f = "f"   local _t = "t"   do   print ( "------ standard lua tests (compiler/passthrough)" )   local x   t [ "t" ] : f ( (1) )   end   print ( "------ t:[k].f()" ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , "t" , "f" , (1) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( _t ) , "f" , (2) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( string . char ( string . byte ( "t" ) ) ) , "f" , (3) , (32) , (33) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( "t" .. "t" ) , "f" , (4) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( "t" .. string . sub ( "atable" , (2) , (2) ) ) , "f" , (5) , (52) , (53) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( ( string . sub ( "atable" , (2) , (2) ) ) ) , "f" , (6) , (62) , (63) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( ( function ( ) return "t" end ) ( ) ) , "f" , (7) )   do end ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( ( function ( ) return t end ) ( ) , ( ( function ( ) return "t" end ) ( ) ) , "f" , (8) )   ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( ( function ( ) return t end ) ( ) , ( ( function ( ) return "t" end ) ( ) ) , "f" , (9) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( T ) , "f" , (10) )   _ = ( ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( t , ( _t ) , "f" , (11) ) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( _t ) , "g" , { (12) } ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   t , ( _t ) , "f" , "13" )   entity = { }   inventory = { get = false , set = false , size = false }   inventory . new = function ( size )   local t = { size = function ( ) return size end }   function t . set ( e , i , o )   if i <= (0) or i > ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( e , ( inventory ) , "size" ) then error ( ) end   e [ inventory ] [ i ] = o   end   function t . get ( e , i )   if i <= (0) or i > ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( e , ( inventory ) , "size" ) then error ( ) end   return e [ inventory ] [ i ]   end   return t   end   inventory . of = function ( e )   return { get = function ( ... ) return ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( e , ( inventory ) , "get" , ... ) end , set = function ( ... ) return ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( e , ( inventory ) , "set" , ... ) end , size = function ( ... ) return ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( e , ( inventory ) , "size" , ... ) end }   end   entity [ inventory ] = inventory . new ( (5) ) ; ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) (   entity , ( inventory ) , "set" , (1) , "Hello World!" )   print ( ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( entity , ( inventory ) , "get" , (1) ) )   for i = (1) , ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( entity , ( inventory ) , "size" ) do   print ( i , ( function(myobj, mytrait, myfunction, ...) return myobj[mytrait][myfunction](myobj, ...) end ) ( entity , ( inventory ) , "get" , i ) )   end   local myinv = inventory . of ( entity )   for i = (1) , myinv . size ( ) do   print ( "wrapped" , i , myinv . get ( i ) )   end
Open this post in threaded view
|

## Re: [NoW] Deja vu

 On 2019-08-09 10:43 a.m., Soni "They/Them" L. wrote: > > > On 2019-08-08 10:10 p.m., Luiz Henrique de Figueiredo wrote: >> > Why "function values caching" has been removed from Lua 5.4? >> >> See >> https://github.com/lua/lua/commit/e8c779736f3029df353038352c14c8ab63728811>> > > say that to cratera: (look for "INPUT"/"OUTPUT" markers) > also, perhaps unrelated, would it be possible to implement this optimization with an internal ephemeron table attached to the global state (the part of lua_State that is shared between coroutines)? does the generational GC handle ephemeron tables with strong values that refer to their weak keys?
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Philippe Verdy On Thu, Aug 8, 2019 at 1:01 PM Philippe Verdy <[hidden email]> wrote:This looks like a bug: even if the functions are declared multiple times (in a loop), they are all in the same context (same closure), and should resolve as one, because even their upvalues will be necessarily the same; here the only thing that changes is the current value of the outer variable j=1,2, whose value if not even used in the function and not even part of its closure (if j was part of the closure and used like " function(x) return 2*x+j end  " the value of j would be kept in that closure in upvalues even after the foor loop exits and the local vairavle j no longer exists: that variable would remain in the closure, and in that case the closure should be duplicated creating two separate functions). If a closure is created in a loop but it does not depend on any local variables inside the loop you can move it out of the loop, and avoid the overhead of creating different but equivalent closures each time. If it does depend on variables declared inside the loop it's a different closure, because it refers to a different instance of the local variable each time you go through the loop.Either way the cache that is used in Lua 5.2 and 5.3 to reuse a closure is probably. Gé-- --Gé
Open this post in threaded view
|

## Re: [NoW] Deja vu

 On Fri, Aug 9, 2019 at 9:33 AM Gé Weijers <[hidden email]> wrote:On Thu, Aug 8, 2019 at 1:01 PM Philippe Verdy <[hidden email]> wrote:This looks like a bug: even if the functions are declared multiple times (in a loop), they are all in the same context (same closure), and should resolve as one, because even their upvalues will be necessarily the same; here the only thing that changes is the current value of the outer variable j=1,2, whose value if not even used in the function and not even part of its closure (if j was part of the closure and used like " function(x) return 2*x+j end  " the value of j would be kept in that closure in upvalues even after the foor loop exits and the local vairavle j no longer exists: that variable would remain in the closure, and in that case the closure should be duplicated creating two separate functions). If a closure is created in a loop but it does not depend on any local variables inside the loop you can move it out of the loop, and avoid the overhead of creating different but equivalent closures each time. If it does depend on variables declared inside the loop it's a different closure, because it refers to a different instance of the local variable each time you go through the loop.Either way the cache that is used in Lua 5.2 and 5.3 to reuse a closure is probably. ... probably not necessarily a necessary feature. I got interrupted, and forgot to complete the sentence :-)-- --Gé
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Egor Skriptunoff-2 This actually seems pretty intuitive to me; in fact, I don't see why the two functions *would* be the same. If you know that it's always the same function, you can move it out of the loop yourself. That's one less thing the compiler will have to think about, meaning simpler code. To me it's the behavior of versions 5.2 and 5.3 that seems like a bug. On 08/08/2019 01:28, Egor Skriptunoff wrote: > t = {} > for j = 1, 2 do >    t[j] = function(x) return 2*x end > end > print(t[1] == t[2]) > > Lua 5.1    false > Lua 5.2    true > Lua 5.3    true > Lua 5.4    false > > It seems that the equality between function values has returned back > to its Lua 5.1 semantics. > > 1) > Why "function values caching" has been removed from Lua 5.4? > > 2) > Could we rely on the fact that in Lua 5.4+ any created function is > unique value?
Open this post in threaded view
|

## Re: [NoW] Deja vu

 On Fri, 9 Aug 2019 at 10:15, Dennis Fischer <[hidden email]> wrote: > To me it's the behavior of versions 5.2 and 5.3 that seems like a bug. in 5.1 and before, any function declaration returns in a new function object.  so they "feel" a bit like tables, where every "{}" creates a new, different table. in 5.2/3, if two functions are really identical, they're one and the same.  so they "feel" a bit like strings, where two identical strings are the same one (ignoring "large strings" (de)optimizations). Of course, tables are mutable, so returning an old one would be total chaos, while strings are immutable, so they behave as pure values. Are functions mutable?  the code itself is immutable, the only mutable things are the upvalues.  But if two functions share the same set, then even if they're distinct, they would "change" together.  So it makes sense to make them indistinct. But so far, this has been considered mostly an optimization, not a design feature.  So if it's more expensive than worth, could be removed.   Or not.  Neither behaviour is a "bug". personally, i do like the 5.2/3 behaviour.  seems more "pure".  but relying on it is asking for trouble.  better be explicit and don't repeat the definition if you want the function to be the same. -- Javier
Open this post in threaded view
|

## Re: [NoW] Deja vu

 Moving functions out of a loop is not as simple as you think: the loop may not be visible at all and come from an invisible outer scope. The new function declaration will be made incide a deep call.If then that function declaration automatically inherits many upvalues that are useless, there will always be a new function created at each call. And this makes no sense.Eliminating all useless upvalues (even if they are in lexical scope) from the set of upvalues is then necessary: if at least this set of upvalues is now empty, it makes no sense at all to create a new closure at each call: a static closure should then be generated always by the compiler, without even having to lookup in a cache if there are other closures with the same set of upvalues (with the same effective reference to the same variable from the same outer scope).Note also that if there are remaining upvalues in that set, the effective value of these variables is still mutable: nothing warranties it will be immutable, so the upvalues are normally variable references, not "constants", except if they are declared this way or by a deep inspection of varaible mutation within the lexical scope.So I still maintain this is a bug for the exposed case: the set of upvalues is empty, the closure should then be static, and it does not need at all any cache.Le ven. 9 août 2019 à 19:30, Javier Guerra Giraldez <[hidden email]> a écrit :On Fri, 9 Aug 2019 at 10:15, Dennis Fischer <[hidden email]> wrote: > To me it's the behavior of versions 5.2 and 5.3 that seems like a bug. in 5.1 and before, any function declaration returns in a new function object.  so they "feel" a bit like tables, where every "{}" creates a new, different table. in 5.2/3, if two functions are really identical, they're one and the same.  so they "feel" a bit like strings, where two identical strings are the same one (ignoring "large strings" (de)optimizations). Of course, tables are mutable, so returning an old one would be total chaos, while strings are immutable, so they behave as pure values. Are functions mutable?  the code itself is immutable, the only mutable things are the upvalues.  But if two functions share the same set, then even if they're distinct, they would "change" together.  So it makes sense to make them indistinct. But so far, this has been considered mostly an optimization, not a design feature.  So if it's more expensive than worth, could be removed.   Or not.  Neither behaviour is a "bug". personally, i do like the 5.2/3 behaviour.  seems more "pure".  but relying on it is asking for trouble.  better be explicit and don't repeat the definition if you want the function to be the same. -- Javier
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Soni "They/Them" L. > On Aug 8, 2019, at 2:06 PM, Soni They/Them L. <[hidden email]> wrote: > > > > On 2019-08-07 8:28 p.m., Egor Skriptunoff wrote: >> t = {} >> for j = 1, 2 do >>    t[j] = function(x) return 2*x end >> end >> print(t[1] == t[2]) >> >> Lua 5.1    false >> Lua 5.2    true >> Lua 5.3    true >> Lua 5.4    false >> >> It seems that the equality between function values has returned back to its Lua 5.1 semantics. >> >> 1) >> Why "function values caching" has been removed from Lua 5.4? >> >> 2) >> Could we rely on the fact that in Lua 5.4+ any created function is unique value? > > Lua 5.4 has const locals in loops. Each loop iteration gets a new j, which is const. > > That just means each closure is using a different upvalue, and gets closed on each loop iteration. Try calling them! > > (Well, I could be wrong, I guess. Correct me if I'm wrong.) > The const-ness is new, but (since at least 5.2) Lua has always treated a for loop iterator variable as newly declared for each loop iteration, so each instance of the closure would get a distinct “j” (though, of course, in this case the inner function does not close j, so the OP issue is still a regression). —Tim
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Egor Skriptunoff-2 > Could we rely on the fact that in Lua 5.4+ any created function is unique > value? See the manual. -- Roberto
Open this post in threaded view
|

## Re: [NoW] Deja vu

 In reply to this post by Luiz Henrique de Figueiredo Luiz Henrique de Figueiredo <[hidden email]> 于2019年8月9日周五 上午9:10写道： > > > Why "function values caching" has been removed from Lua 5.4? > > See https://github.com/lua/lua/commit/e8c779736f3029df353038352c14c8ab63728811> Another benefit is that the proto object is constant now by removing the internal cache (it only refer other proto objects and string objects, and it's a invariant). So we can share the proto pointer to other lua VMs with minimal modifcation (modify GC). -- http://blog.codingnow.com