Explanation needed for setfenv replacement for Lua 5.2

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
Hi all!

I need some help in understanding some Lua code Sergey Rhozenko posted a
while ago. They are two version of 'setfenv' to use with upcoming Lua
5.2 in order to replace the missing library function. The code is
replicated below.


The lines that I don't understand are those where debug.upvaluejoin is
called. Why is this call needed? Isn't debug.setupvalue enough?

Any help appreciated


----------------------------------------------------
-- 1st version

setfenv = setfenv or function(f, t)
     f = (type(f) == 'function' and f or debug.getinfo(f + 1, 'f').func)
     local name
     local up = 0
     repeat
         up = up + 1
         name = debug.getupvalue(f, up)
     until name == '_ENV' or name == nil
     if name then
         debug.upvaluejoin(f, up, function() return name end, 1)  -- use
unique upvalue
         debug.setupvalue(f, up, t)
     end
end


-- 2nd version
setfenv = setfenv or function(f, t)
     f = (type(f) == 'function' and f or debug.getinfo(f + 1, 'f').func)
     local name
     local up = 0
     repeat
         up = up + 1
         name = debug.getupvalue(f, up)
     until name == '_ENV' or name == nil
     if name then
         debug.upvaluejoin(f, up, function() return t end, 1)  -- use
unique upvalue, set it to f
     end
end


Thanks in advance.

--
Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Peter Cawley
On Wed, May 4, 2011 at 3:00 PM, Lorenzo Donati
<[hidden email]> wrote:
> The lines that I don't understand are those where debug.upvaluejoin is
> called. Why is this call needed? Isn't debug.setupvalue enough?

In 5.1, each closure had an individual environment, and changing the
environment of one closure would not change the environment of any
other closures. On the other hand, upvalues can be shared between
multiple closures, so that changing the value in one closure affects
what other closures subsequently see. Hence to replicate the 5.1
behaviour, you need to ensure that the upvalue that you're setting
isn't shared with any other closures, so the conceptual behaviour is:
debug.splitupvalue(f, up) -- this function doesn't actually exist
debug.setupvalue(f, up, t)
Splitting can be implemented in terms of joining against the upvalues
of a temporary closure, and then setting can be combined with
splitting into a single call to debug.upvaluejoin.

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Edgar Toernig
In reply to this post by Lorenzo Donati-2
Lorenzo Donati wrote:
>
> I need some help in understanding some Lua code Sergey Rhozenko posted a
> while ago. They are two version of 'setfenv' to use with upcoming Lua
> 5.2 in order to replace the missing library function.

I just hope, that people using such code know what they do.
But anyway ...

> The lines that I don't understand are those where debug.upvaluejoin is
> called. Why is this call needed? Isn't debug.setupvalue enough?

>Version 1:
>      if name then
>          debug.upvaluejoin(f, up, function() return name end, 1)
>          debug.setupvalue(f, up, t)
>      end

Version 2:
>      if name then
>          debug.upvaluejoin(f, up, function() return t end, 1)
>      end

The 2nd is just an optimization of the first.  Some background:

Upvalues are heap objects that hold values.  They can be shared between
functions.  setupvalue changes the value the upvalue holds.  All functions
sharing that upvalue will see that change:

  local x=1
  function a() return x end
  function b() return x end
  debug.setupvalue(a, 1, 42)
  print(a(), b()) --> 42 42

Here a and b share a single upvalue that is created for x.  If you
modify the value of the upvalue for "a" (via setupvalue), you will
at the same time change the value "b" will see.


On the other hand, upvaluejoin does not modify the value of an upvalue
but replaces the actual upvalue:

  local x=1
  local y=2
  function a() return x end
  function b() return y end
  debug.upvaluejoin(a, 1, b, 1)
  print(a(), b()) --> 2 2
  y=42
  print(a(), b()) --> 42 42

Here, a's first upvalue (x) is replaced by b's first upvalue (y) so
both now reference y.


Back to the setfenv emulation.  The first version

>          debug.upvaluejoin(f, up, function() return name end, 1)
>          debug.setupvalue(f, up, t)

creates a dummy function with one new upvalue (for name) and transfers
this upvalue to f.  In the next step it sets the value of this new
upvalue to t.

The 2nd version

>          debug.upvaluejoin(f, up, function() return t end, 1)

starts the same, but makes sure, the new new upvalue already holds the
correct value (t).  So the second step is no longer needed.


With that background, one can see that there are usually better ways
to handle the environment in Lua 5.2 - especially if you want to
modify the env for a handful of functions:

--8<--
        global = "global"

        do
            local _ENV = _ENV
            function demo_setenv(t) _ENV = t end
            function demo1() print("demo1", global) end
            function demo2() print("demo2", global) end
        end

        demo1() --> demo1 global
        demo2() --> demo2 global
        demo_setenv{print=print, global="private"}
        demo1() --> demo1 private
        demo2() --> demo2 private
        demo_setenv(_ENV)
        demo1() --> demo1 global
        demo2() --> demo2 global
--8<--

I.E. if you want to change the env for a complete module just add a simple
setenv function to its module table and be done:

        function M.setenv(t) _ENV = t end

Ciao, ET.


PS: I still not sure if a "function environment" is The Right Thing.
I would prefer a simple thread-wide "setglobals" ;-)

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
Peter & Edgar:
Thank you very much for the useful and detailed explanations.

On 04/05/2011 19.30, E. Toernig wrote:
> Lorenzo Donati wrote:
>>
>> I need some help in understanding some Lua code Sergey Rhozenko posted a
>> while ago. They are two version of 'setfenv' to use with upcoming Lua
>> 5.2 in order to replace the missing library function.
>
> I just hope, that people using such code know what they do.
> But anyway ...
>

[snip]


>
> With that background, one can see that there are usually better ways
> to handle the environment in Lua 5.2 - especially if you want to
> modify the env for a handful of functions:
>

Yes, I know. Thanks.

Although I appreciate the improvement in upcoming 5.2 regarding globals
management (_ENV etc.) for common uses, i.e. when the old environment is
to be changed lexically, I still found setfenv/getfenv extremely useful
for a very specific pattern, where the change is to be done dynamically.

Please, consider the following toy code:


---------------------------

-- (simplified) module simulation
-- (just to run the example in a single file)

simulated_require = function()
    local M = {}

    local function DslInterpreter( closure )
       local private_data = {}
       local env = {}
       function env.AddItem( item )
          table.insert( private_data, item )
       end
       function env.PrintResult()
          print( table.concat( private_data, ' ' ) )
       end
       setfenv( closure, env )
       closure()
    end
    M.Interpreter = DslInterpreter

    return M
end


-- client code:

local lib = simulated_require 'mymodule'
local AN_UPVALUE = 'This is a silly DSL!'

lib.Interpreter( function() -- anonymous closure to emulate scoping

    -- DSL fragment
    AddItem 'hello!'
    AddItem( AN_UPVALUE )
    AddItem 'Hope You Enjoy!'

    PrintResult() --> hello! This is a silly DSL! Hope You Enjoy!

end)
--------------------------------


Note that this pattern, to my knowledge, cannot be replaced using 5.2
loadin, since AN_UPVALUE is referenced inside the DSL fragment.

Even in the absence of upvalues, loadin incurs the overhead of
recompiling the chunk, if that chunk must be run in different environments.

Moreover, this pattern allows resusing the closure with a different
environment, if the need arise (e.g., debugging the DSL code with
different bindings, etc.). Or even with a different Interpreter.

Well, if 5.2 provided a function like, for example, callin(env,func)
analogous to loadin, but which takes a closure instead of a string
representing the chunk, I could probably ditch these setfenv/getenv
replacements, since I use them only for this specific pattern (and
variants). Maybe loadin could also be changed to do the right thing if
fed with a closure instead of a string: loadin(env,func)

I doubt that Lua team will introduce such a thing :-), so I must live
with my substitutes (thanks to Sergey Rhozenko!).


P.S: I know that I could pass the environment as an argument to the
anonymous closure like this:

------------------------------------
lib.Interpreter( function( env )

    _ENV = env

    -- DSL fragment
    AddItem 'hello!'
    AddItem( AN_UPVALUE )
    AddItem 'Hope You Enjoy!'

    PrintResult() --> hello! This is a silly DSL! Hope You Enjoy!

end)
------------------------------------


but this is less clean to me: not only the user must remember what to do
with the argument to the anon closure, but it adds cluttering
boilerplate code. The user of the DSL shouldn't be concerned with that
stuff.

Keep in mind that the anon closure could also come from loading a chunk
from an external file (yes, there is 'loadin' for that, but then you
have to do things in two different ways, depending on whether you load a
chunk or use an explicit anon closure).


[snip]

>
> I.E. if you want to change the env for a complete module just add a simple
> setenv function to its module table and be done:
>
> function M.setenv(t) _ENV = t end
>
> Ciao, ET.
>
>
> PS: I still not sure if a "function environment" is The Right Thing.
> I would prefer a simple thread-wide "setglobals" ;-)
>
>
>


Cheers

--
Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Enrico Colombini
In reply to this post by Edgar Toernig
On 04/05/2011 19.30, E. Toernig wrote:
> I just hope, that people using such code know what they do.
> But anyway ...

Thanks for the very good explanation! I always had a slightly foggy idea
of how upvalues worked and was too lazy to find out what upvaluejoin did.

I think that message should go into an intermediate-level Lua FAQ.

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Roberto Ierusalimschy
In reply to this post by Lorenzo Donati-2
> Please, consider the following toy code:
>
> [...]

This is the kind of use that deserves the use of the debug library,
because you explicitly want to break the usual visibility rules.

Note that, for this particular use, you do not need the entire
apparatus of Rhozenko's code. (In particular, you do not need the part
you had not understood.)  A simpler 'setfenv' will do:

-- not tested!
setfenv = function(f, t)
  for up = 1, math.huge do
    local name = debug.getupvalue(f, up)
    if name == '_ENV' then
      debug.setupvalue(f, up, t)
      return
    elseif name == nil then
      return
    end
  end
end

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Roberto Ierusalimschy
>
> Note that, for this particular use, you do not need the entire
> apparatus of Rhozenko's code. (In particular, you do not need the part
> you had not understood.)  A simpler 'setfenv' will do:
>
> [...]

Forget it. I was tired... (Sorry for the noise.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Edgar Toernig
In reply to this post by Lorenzo Donati-2
Just for the record:

> P.S: I know that I could pass the environment as an argument to the
> anonymous closure like this:
> ------------------------------------
> lib.Interpreter( function( env )
>
>     _ENV = env
> ------------------------------------

That would also change the callers env!

Either "local _ENV = env" or simply name the parameter _ENV:

  lib.Interpreter( function(_ENV)

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
On 05/05/2011 0.53, E. Toernig wrote:

> Just for the record:
>
>> P.S: I know that I could pass the environment as an argument to the
>> anonymous closure like this:
>> ------------------------------------
>> lib.Interpreter( function( env )
>>
>>      _ENV = env
>> ------------------------------------
>
> That would also change the callers env!

Ouch! I knew that! I forgot the local keyword!
I was tired too! :-P

That's the problem with "toy code": you risk of simplifying too much
during cut and paste! :-D

>
> Either "local _ENV = env" or simply name the parameter _ENV:
>
>    lib.Interpreter( function(_ENV)

Nice reduction of the boilerplate code. I really didn't think of that!
Thanks!
It is still too boilerplate IMO when explaining how to write a DSL
snippet to a non-programmer (too easy to forget or to write in a wrong
way: _ENV, ENV, __Env, who knows!), but at least it becomes viable when
I write code in the DSL, which is my most common use case (As I said, I
hate boilerplate :-).

It's a matter of separation of concerns: whenever possible I write a
small Lua driver, then express the core logic in a DSL. Together with
"thin layer" OOP I found it is really a powerful technique - leading to
extremely clean code. Even months later, when you maybe forgot how the
driver worked, you look at the DSL and (if it was carefully designed)
you can easly tune/change the business logic without delving into the
possible gory details well hidden in the driver.

And it is an affordable technique too: Lua is fantastic in this respect
- a few dozens of lines and you have a nice DSL interpreter without
needing to know how to write a recursive descent parser!


>
> Ciao, ET.
>
>
>

thanks again!

--
Lorenzo


Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
In reply to this post by Roberto Ierusalimschy
On 04/05/2011 23.58, Roberto Ierusalimschy wrote:
>>
>> Note that, for this particular use, you do not need the entire
>> apparatus of Rhozenko's code. (In particular, you do not need the part
>> you had not understood.)  A simpler 'setfenv' will do:
>>
>> [...]
>
> Forget it. I was tired... (Sorry for the noise.)

Don't worry! You're welcome. :-)
Thank you for reading my (long) posts anyway!

>
> -- Roberto
>
>
>

As the old say goes:

Stupid people don't learn from their mistakes,
smart people do,
but wise ones learn from the mistakes of others.

It's the essence of debugging, if you want! :-D

Cheers!

--
Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Edgar Toernig
In reply to this post by Lorenzo Donati-2
| lib.Interpreter( function(_ENV)
|    AddItem "hello!"
|    AddItem(AN_UPVALUE)
|    AddItem "Hope You Enjoy!"
|
|    PrintResult()  --> hello! This is a silly DSL! Hope You Enjoy!
| end)
 
> It is still too boilerplate IMO when explaining how to write a DSL
> snippet to a non-programmer

Just an idea: How about misusing the for statement?

| for _ENV in lib.Interpreter do
|    AddItem "hello!"
|    AddItem(AN_UPVALUE)
|    AddItem "Hope You Enjoy!"
|
|    PrintResult()  --> hello! This is a silly DSL! Hope You Enjoy!
| end

Still the _ENV but it's readable ;-)

Full code below:
------------------------------------------------------
function Interpreter(_, second_iteration)
    if second_iteration then return end

    local private_data = {}

    return {
        AddItem = function(item) table.insert(private_data, item) end,
        PrintResult = function() print(table.concat(private_data, " ")) end,
    }
end

-- client code

local AN_UPVALUE = "This is a silly DSL!"

for _ENV in Interpreter do
    AddItem "hello!"
    AddItem(AN_UPVALUE)
    AddItem "Hope You Enjoy!"

    PrintResult()  --> hello! This is a silly DSL! Hope You Enjoy!
end
------------------------------------------------------

Ciao, ET.

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
In reply to this post by Roberto Ierusalimschy
On 04/05/2011 23.46, Roberto Ierusalimschy wrote:
>> Please, consider the following toy code:
>>
>> [...]
>
> This is the kind of use that deserves the use of the debug library,
> because you explicitly want to break the usual visibility rules.
>

Yes, definitely!

Clear separation of concerns. Lua is lexically scoped and _ENV & C. is
coherent with that. In this use getfenv/setfenv were a workaround to
emulate dynamic scoping, so I agree their use must be restrained and
marked as "beware - know what you are doing".

[...]

>
> -- Roberto
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
In reply to this post by Edgar Toernig
On 05/05/2011 22.53, E. Toernig wrote:

> | lib.Interpreter( function(_ENV)
> |    AddItem "hello!"
> |    AddItem(AN_UPVALUE)
> |    AddItem "Hope You Enjoy!"
> |
> |    PrintResult()  -->  hello! This is a silly DSL! Hope You Enjoy!
> | end)
>
>> It is still too boilerplate IMO when explaining how to write a DSL
>> snippet to a non-programmer
>
> Just an idea: How about misusing the for statement?
>
> | for _ENV in lib.Interpreter do
> |    AddItem "hello!"
> |    AddItem(AN_UPVALUE)
> |    AddItem "Hope You Enjoy!"
> |
> |    PrintResult()  -->  hello! This is a silly DSL! Hope You Enjoy!
> | end
>
> Still the _ENV but it's readable ;-)

Really smart code! But probably too smart :-D ! As I see it my internal
parser triggers the "what's being iterated here?" exception! :-D
As the old say goes: "Try not to outsmart the
language/compiler/interpreter!" ;-)
But you told me that: "for" is being misused! :-)
Nevertheless, very smart! I wonder if there is a less academic use for
that pattern.

P.S.: (for future reference in the archive) I think now AN_UPVALUE
inside the "for" body is no more an upvalue, but just a local. From the
manual (emphasis mine):

*****
Because of the lexical scoping rules, local variables can be freely
accessed by functions defined inside their scope. A *local variable used
by an inner function* is called an upvalue, or external local variable,
*inside the inner function*.
*****

Since the body of the "for" is not a closure, technically AN_UPVALUE is
not an upvalue there (debug.getupvalue doesn't list it, while
debug.getlocal does).

>
> Full code below:

[...]

>
> Ciao, ET.
>
>
>

Cheers!
--
Lorenzo


Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

steve donovan
In reply to this post by Edgar Toernig
On Thu, May 5, 2011 at 10:53 PM, E. Toernig <[hidden email]> wrote:
> for _ENV in Interpreter do
>    AddItem "hello!"
>    AddItem(AN_UPVALUE)
>    AddItem "Hope You Enjoy!"
>
>    PrintResult()  --> hello! This is a silly DSL! Hope You Enjoy!
> end

That is a very cute piece of code - bordering on abuse, maybe, but it
looks good!

Let me see if I get it:

function env(tbl)
  return function(_,second_iteration)
     if second_iteration then return nil end
     return tbl
  end
end

for _ENV in env {print=print,A=1,B=2} do
   print(A,B)
end

The in statement reappears ...

steve d.

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
On 06/05/2011 15.23, steve donovan wrote:

[...]

> That is a very cute piece of code - bordering on abuse, maybe, but it
> looks good!
>
> Let me see if I get it:
>
> function env(tbl)
>    return function(_,second_iteration)
>       if second_iteration then return nil end
>       return tbl
>    end
> end
>
> for _ENV in env {print=print,A=1,B=2} do
>     print(A,B)
> end
>
> The in statement reappears ...
>
> steve d.
>
>
>

Ok. But besides being a stress test for Lua 5.2 syntax and a proof of
concept, isn't it a complicated way to do the following:

--------------
do local _ENV = {print=print,A=1,B=2}
   print(A,B)
end
-------------

in the end? Or am I missing something?

Cheers.

--
Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

steve donovan
On Fri, May 6, 2011 at 10:05 PM, Lorenzo Donati
<[hidden email]> wrote:
> in the end? Or am I missing something?

You are missing nothing here.  The notation just seemed very cute.

(the env() function could also ensure that we also look up in the
enclosing environment, so it would not be necessary to use print=print
in the table.)

Note also that this kind of one-shot iterator could do something
interesting on the start of the second iteration before returning nil,
like finalizers.

steve d.

Reply | Threaded
Open this post in threaded view
|

Re: Explanation needed for setfenv replacement for Lua 5.2

Lorenzo Donati-2
On 07/05/2011 9.26, steve donovan wrote:
> On Fri, May 6, 2011 at 10:05 PM, Lorenzo Donati
> <[hidden email]>  wrote:
>> in the end? Or am I missing something?
>
> You are missing nothing here.  The notation just seemed very cute.

Always good to know! :-P :-)

>
> (the env() function could also ensure that we also look up in the
> enclosing environment, so it would not be necessary to use print=print
> in the table.)
>
> Note also that this kind of one-shot iterator could do something
> interesting on the start of the second iteration before returning nil,
> like finalizers.
>

Interesting points. thanks!

> steve d.
>
>
>

Cheers!


--
Lorenzo