[Suggestion] Sharing local variables with "load"-ed functions

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Suggestion] Sharing local variables with "load"-ed functions

Egor Skriptunoff-2
Hi!

Some functions are better to be generated and "load"-ed on-the-fly
instead of be fully prepared in the source code.

At the moment, we are forced to use globals or pass separate environment
table to share variables with "load"-ed functions, which is not very handy.
It also implies frequent hash-table lookups, which is not very fast.

To make sharing local variables with "load"-ed functions nice and easy,
a new keyword, a new syntax and a new function can be introduced:

1) New keyword "upvalue" is used to mark some variables as upvalues
in current chunk and to assign consecutive negative upvalue indices to them

2) New syntax "@varname" is used to get "id" for locals and upvalues,
such "id" has the same meaning as values returned by "debug.upvalueid(f,n)"

3) New function "debug.useupvalue(f, n, id)" is a more powerful version
of old function "debug.upvaluejoin":
"debug.upvaluejoin(f1, n1, f2, n2)" is the same as
"debug.useupvalue(f1, n1, debug.upvalueid(f2, n2))"
But we also can write "debug.useupvalue(f, n, @variable_name)"
to use local variable "variable_name" from our current chunk
as an upvalue inside function "f".


Example:

local str = 'original'
do
  local x = 123
  local func = load("upvalue y,z; y = y + 1; print(y); z = 'modified'")
  -- "y" and "z" are upvalues, they are not auto-prefixed by "_ENV."
  -- Negative indices starting with -1 are assigned to "y" and "z"
  -- Function "func" has three upvalues: _ENV, y and z,
  -- which have indices 1, (-1) and (-2) accordingly.
  -- Index (-1) is the same as 3, index (-2) is the same as 2.
  debug.useupvalue(func, -1, @x)
  -- "@x" is an expression, which returns the "id" for variable "x"
  -- our local variable "x" is now used as upvalue with index -1 in "func"
  debug.useupvalue(func, -2, @str)
  -- "@str" is an expression, which returns the "id" for upvalue "str"
  -- our upvalue "str" is now used as upvalue with index -2 in "func"
  func()      -->  124
  print(x)    -->  124
  print(str)  -->  modified
end


-- Egor

Reply | Threaded
Open this post in threaded view
|

Re: [Suggestion] Sharing local variables with "load"-ed functions

Patrick Donnelly
On Sun, Oct 25, 2015 at 11:47 AM, Egor Skriptunoff
<[hidden email]> wrote:

> Hi!
>
> Some functions are better to be generated and "load"-ed on-the-fly
> instead of be fully prepared in the source code.
>
> At the moment, we are forced to use globals or pass separate environment
> table to share variables with "load"-ed functions, which is not very handy.
> It also implies frequent hash-table lookups, which is not very fast.
>
> To make sharing local variables with "load"-ed functions nice and easy,
> a new keyword, a new syntax and a new function can be introduced:

This is very much indulging premature optimization but I'll bite.

The existing functionality (debug.upvaluejoin) is all you need to do
what you want:

- Declare locals to be exported:

local x = 1
export("x", function() return x end) -- **
local y = 2
export("y", function() return y end)

- hook the load function which creates locals with the names of the
exported locals and then joins them to the loaded chunk, something
like:

-- get exported local names
local new_chunk = "local x,y; return function(...)"..chunk.." end"
local f = original_load(new_chunk)()
-- join exported upvalues on f

** It's necessary to create this function otherwise you have no
closure with x as an upvalue

--
Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: [Suggestion] Sharing local variables with "load"-ed functions

Egor Skriptunoff-2
On Sun, Oct 25, 2015 at 8:29 PM, Patrick Donnelly <[hidden email]> wrote:
The existing functionality (debug.upvaluejoin) is all you need to do
what you want:

- Declare locals to be exported:

local x = 1
export("x", function() return x end) -- **
local y = 2
export("y", function() return y end)

- hook the load function which creates locals with the names of the
exported locals and then joins them to the loaded chunk, something
like:

Patrick, thanks for the idea!
It is quite workable, albeit rather anti-aesthetic due to exporting new function body for every local:
  local a, b, c = 1, 2, 3
  export("x", function() return a end)
  export("y", function() return b end)
  export("z", function() return c end)
  local f = load_with_upvalues("print(x + y + z); x, y, z = x+1, y*2, z^3")
Compare with this nice-and-easy thing I wanted:
  local a, b, c = 1, 2, 3
  local f = load_with_upvalues(@a, @b, @c, "upvalue x, y, z; print(x + y + z); x, y, z = x+1, y*2, z^3")

 
local new_chunk = "local x,y; return function(...)"..chunk.." end"
local f = original_load(new_chunk)()
-- join exported upvalues on f

There is one little problem with your solution: if the program is stored as .luac-file with debug info stripped, it is impossible to distinguish between upvalues x and y in function f: both have the same name = "(*no name)" and the same value = nil.
Upvalue indices don't contain useful information also:
"Upvalues have no particular order, as they are active through the whole function. They are numbered in an arbitrary order." (c)Lua manual
To solve this problem, the upvalues should contain its names as values:
local new_chunk = "local x,y = 'x','y'; return function(...)"..chunk.." end"


Let's think about optimization of this clumsy sequence of export() lines:
  local x1, x2, x3, x4, x5, x6, x7
  export("x1", function() return x1 end)
  export("x2", function() return x2 end)
  ...
  export("x7", function() return x7 end)
We can export N variables using only 1+log(N) parameters: 1 string and log(N) functions:
  local x1, x2, x3, x4, x5, x6, x7
  export_all("x1,x2,x3,x4,x5,x6,x7",
    function()x1()x3()x5()x7()end,
    function()x2()x3()x6()x7()end,
    function()x4()x5()x6()x7()end
  )
Now we can detect the correspondence of 7 upvalues (partially shared between 3 functions) to 7 variable names.
But I think the code has gotten even more clumsy...

Any new ideas?
Reply | Threaded
Open this post in threaded view
|

Re: [Suggestion] Sharing local variables with "load"-ed functions

Soni "They/Them" L.


On 25/10/15 07:37 PM, Egor Skriptunoff wrote:

> On Sun, Oct 25, 2015 at 8:29 PM, Patrick Donnelly
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     The existing functionality (debug.upvaluejoin) is all you need to do
>     what you want:
>
>     - Declare locals to be exported:
>
>     local x = 1
>     export("x", function() return x end) -- **
>     local y = 2
>     export("y", function() return y end)
>
>     - hook the load function which creates locals with the names of the
>     exported locals and then joins them to the loaded chunk, something
>     like:
>
>
> Patrick, thanks for the idea!
> It is quite workable, albeit rather anti-aesthetic due to exporting
> new function body for every local:
>   local a, b, c = 1, 2, 3
>   export("x", function() return a end)
>   export("y", function() return b end)
>   export("z", function() return c end)
>   local f = load_with_upvalues("print(x + y + z); x, y, z = x+1, y*2,
> z^3")
> Compare with this nice-and-easy thing I wanted:
>   local a, b, c = 1, 2, 3
>   local f = load_with_upvalues(@a, @b, @c, "upvalue x, y, z; print(x +
> y + z); x, y, z = x+1, y*2, z^3")
>
>     local new_chunk = "local x,y; return function(...)"..chunk.." end"
>     local f = original_load(new_chunk)()
>     -- join exported upvalues on f
>
>
> There is one little problem with your solution: if the program is
> stored as .luac-file with debug info stripped, it is impossible to
> distinguish between upvalues x and y in function f: both have the same
> name = "(*no name)" and the same value = nil.
> Upvalue indices don't contain useful information also:
> "Upvalues have no particular order, as they are active through the
> whole function. They are numbered in an arbitrary order." (c)Lua manual
> To solve this problem, the upvalues should contain its names as values:
> local new_chunk = "local x,y = 'x','y'; return
> function(...)"..chunk.." end"
>
>
> Let's think about optimization of this clumsy sequence of export() lines:
>   local x1, x2, x3, x4, x5, x6, x7
>   export("x1", function() return x1 end)
>   export("x2", function() return x2 end)
>   ...
>   export("x7", function() return x7 end)
> We can export N variables using only 1+log(N) parameters: 1 string and
> log(N) functions:
>   local x1, x2, x3, x4, x5, x6, x7
> export_all("x1,x2,x3,x4,x5,x6,x7",
> function()x1()x3()x5()x7()end,
>     function()x2()x3()x6()x7()end,
> function()x4()x5()x6()x7()end
>   )
> Now we can detect the correspondence of 7 upvalues (partially shared
> between 3 functions) to 7 variable names.
> But I think the code has gotten even more clumsy...
>
> Any new ideas?
https://github.com/SoniEx2/loadx

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: [Suggestion] Sharing local variables with "load"-ed functions

Soni "They/Them" L.


On 25/10/15 07:43 PM, Soni L. wrote:
>
>
> https://github.com/SoniEx2/loadx
>
Well apparently github is down so I'm attaching the file instead...

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


loadx.c (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Suggestion] Sharing local variables with "load"-ed functions

Sean Conner
In reply to this post by Egor Skriptunoff-2
It was thus said that the Great Egor Skriptunoff once stated:

> On Sun, Oct 25, 2015 at 8:29 PM, Patrick Donnelly <[hidden email]>
> wrote:
>
> > The existing functionality (debug.upvaluejoin) is all you need to do
> > what you want:
> >
> > - Declare locals to be exported:
> >
> > local x = 1
> > export("x", function() return x end) -- **
> > local y = 2
> > export("y", function() return y end)
> >
> > - hook the load function which creates locals with the names of the
> > exported locals and then joins them to the loaded chunk, something
> > like:
> >
>
> Patrick, thanks for the idea!
> It is quite workable, albeit rather anti-aesthetic due to exporting new
> function body for every local:
>   local a, b, c = 1, 2, 3
>   export("x", function() return a end)
>   export("y", function() return b end)
>   export("z", function() return c end)
>   local f = load_with_upvalues("print(x + y + z); x, y, z = x+1, y*2, z^3")
> Compare with this nice-and-easy thing I wanted:
>   local a, b, c = 1, 2, 3
>   local f = load_with_upvalues(@a, @b, @c, "upvalue x, y, z; print(x + y +
> z); x, y, z = x+1, y*2, z^3")

  One thing to keep in mind---chunks are functions.  So the following does
work:

        CODE1 = [[
          local x = select(1,...)
          local y = select(2,...)
          local z = select(3,...)
         
          return function() return x,y,z end  
        ]]  
       
        f1 = load(CODE1,"code1","t",{select = select})("one",2,true)
        print(f1())

  This works to get external values into the locals of a chunk, but it
doesn't solve the issue of sharing upvalues among different chunks.  I have
noticed that despite this:

        Upvalues have no particular order, as they are active through the
        whole function. They are numbered in an arbitrary order.

that upvalues are numbered in a consistent order---the order they are
defined in (whether you can depend on this is another matter, but I've found
this to be true for Lua 5.2 and Lua 5.3), so this also works:

        CODE1 = [[ local x,y; return function() return x,y end ]]
        CODE2 = [[ local x,y; return function() return x,y end ]]

        f1 = load(CODE1,"code1","t",{})()
        f2 = load(CODE2,"code2","t",{})()

        debug.upvaluejoin(f1,1,f2,1)
        debug.upvaluejoin(f1,2,f2,2)

        debug.setupvalue(f1,1,14)
        debug.setupvalue(f2,2,"hello")

        print(f1())
        print(f2())
        print()

as does this:

        function load_with_upvalues(code,join,...)              
          local f = load(code)()              
       
          if join then
            local info = debug.getinfo(join)
            for i = 1 , info.nups do
              debug.upvaluejoin(f,i,join,i)
            end
          else
            for i = 1 , select('#',...) do
              local v = select(i,...)
              debug.setupvalue(f,i,v)
            end
          end  
               
          return f
        end
   
        CODE1 = [[ local x,y; return function() return x,y end ]]
        CODE2 = [[ local x,y; return function() return x,y end ]]

        f1 = load_with_upvalues(CODE1,false,14,"hello")              
        f2 = load_with_upvalues(CODE2,f1)              
       
        print(f1())
        print(f2())
        print()

  -spc