callstack recursive environment

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

callstack recursive environment

codemonkey Monkey

I"m using LUA to extend a game engine.  It's generally working OK (could wish for static type checking though...), but I've run into something I can't work out, and searching the list archives hasn't turned up an answer for any search I've tried so far.

The behavior I'm after seems like it should be very simple, maybe the most common thing to want, but I can't find a combination of metatable/setfenv behaviors to do it.  I have a bunch of C++ internal game objects.  Each of these C++ objects has a Lua table associated with it which defines a desired Lua global environment.  All I want is to call a Lua fn from C++, and have this environment table be used for the Lua fn I call, and anything that *it* calls, recursively for the entire call stack.  In other words, I want my environment to be inherited.

I started out by trying to use setfenv on the Lua fn from C++ before I call it, pointing to my environment.  For example, let's call this lua function LuaTest.  I quickly discovered that the setfenv is not inherited through the call stack!  I had to call setfenv in *every* Lua function called in the entire tree that LuaTest might call, which worked, but was not practical at all.

My second atttempt, and the one I have limping along now, was to define a metatable with __index and __newindex entries pointing to my desired environment for LuaTest, and setting this as the metatable for the module that contains LuaTest (*).  This *almost* works.  Every Lua fn called by LuaTest sees my environment table exactly as I wish but *only* within the module defining LuaTest.   Once any function is called outside that module, my environment is lost, replaced with the one defined by the setfenv of the called Lua function. 

That's where I'm at now: I have the behavior I want but only within a single module.  What I really wish is something that seems so very simple: I want the environment i set to be inherited by anything called by LuaTest.  But I can't seem to get that behavior!  I've tried some of the "pseudo-index" things too, but whatever I try, it seems some "function environment" always clobbers the real environment I'm trying to have inherited down through the Lua call stack.

Is there a nice simple way to do this?

For a while I was just setting a global "E" with my desired environment, and everybody had to say "E.myvar ...".  That also works, even across modules, but it's a pain, and fragile: if anyone forgets to say "E.myvar", even just once, there's a bug.  With the metatable approach, at least I *know* variables are going into the environment I want them to.

Thanks!


(*) Actually I'm not using the Lua module system, but my own, but that's not related to my query above: I see the same troubles when using Lua's module system.  Mine provides more private module behavior than the Lua module system, so you can say:

local my_module = Game.require('some/file.lua')

Your handle to it is private, and you can say my_module.some_function().  But I want some_function() to see the same Lua environment that its caller does!

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Steve Litt
On Sat, 24 Mar 2012 14:34:46 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

>
>
> I"m using LUA to extend a game engine.  It's generally working OK
> (could wish for static type checking though...), but I've run into
> something I can't work out, and searching the list archives hasn't
> turned up an answer for any search I've tried so far.
>
> The behavior I'm after seems like it should be very simple, maybe the
> most common thing to want, but I can't find a combination of
> metatable/setfenv behaviors to do it.  I have a bunch of C++ internal
> game objects.  Each of these C++ objects has a Lua table associated
> with it which defines a desired Lua global environment.  All I want
> is to call a Lua fn from C++, and have this environment table be used
> for the Lua fn I call, and anything that *it* calls, recursively for
> the entire call stack.  In other words, I want my environment to be
> inherited.
>
>
> I started out by trying to use setfenv on the Lua fn from C++ before
> I call it, pointing to my environment.  For example, let's call this
> lua function LuaTest.  I quickly discovered that the setfenv is not
> inherited through the call stack!  I had to call setfenv in *every*
> Lua function called in the entire tree that LuaTest might call, which
> worked, but was not practical at all.
>
> My second atttempt, and the one I have limping along now, was to
> define a metatable with __index and __newindex entries pointing to my
> desired environment for LuaTest, and setting this as the metatable
> for the module that contains LuaTest (*).  This *almost* works.
> Every Lua fn called by LuaTest sees my environment table exactly as I
> wish but *only* within the module defining LuaTest.   Once any
> function is called outside that module, my environment is lost,
> replaced with the one defined by the setfenv of the called Lua
> function. 
>
>
> That's where I'm at now: I have the behavior I want but only within a
> single module.  What I really wish is something that seems so very
> simple: I want the environment i set to be inherited by anything
> called by LuaTest.  But I can't seem to get that behavior!  I've
> tried some of the "pseudo-index" things too, but whatever I try, it
> seems some "function environment" always clobbers the real
> environment I'm trying to have inherited down through the Lua call
> stack.
>
>
> Is there a nice simple way to do this?
>
> For a while I was just setting a global "E" with my desired
> environment, and everybody had to say "E.myvar ...".  That also
> works, even across modules, but it's a pain, and fragile: if anyone
> forgets to say "E.myvar", even just once, there's a bug.  With the
> metatable approach, at least I *know* variables are going into the
> environment I want them to.
>
>
> Thanks!

Hi Codemonkey,

It sounds to me like what you really want is the Lua equivalent of the
(Linux/Unix) bash export statement, which makes changes made to
environment variables in this program available in all programs
forked/execed from this program. And it sounds like according to your
research, there's no such thing in Lua.

If there's no such thing, what's wrong with every subroutine making a
call to a quick program to retrieve the intended environment? You say
above that you don't want to do that, but it doesn't sound that bad to
me. In the following discussion, I'll use generalized OOP terms, as
there are lots of ways to do OOP in Lua...

You could make a class that implements a stack of environments, with
each environment being a table of envvar/value pairs. You'd also have a
persistent object variable that keeps track of your current level in
the stack. Such a class would need the following methods:

* getenvtable() gets the environment table at the lowest level, and then
  sets the level counter one deeper. This is run at the beginning of
  every function.
* setenv(envvar, value) sets the current level's envvar to whatever
  value, and just for fun returns that value.
* return() tells the stack object you're done with this function, and
  to decrement the stack level. This is called at the end of every
  function that has called getenvtable(), just before the function's
  return statement.

So a function might look something like this:

function whatever(stackobj)
        envtable = stackobj:getenvtable()
        -- Do some stuff
        envtable[minutes] = stackobj:setenv(minutes+1)
        -- Do more stuff, then call another function
        son_of_whatever(stackobj)
        envtable[color] = stackobj:setenv(green)
        -- Do more stuff
        stackobj:return()
end

I'm not sure why simply passing envtable every time wouldn't work, but
if it didn't, I imagine this stackobj technique would.

HTH

SteveT
       

I haven't implemented this exact stack, but it seems to me like it
wouldn't be difficult, and I doubt it would be especially slow.

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

codemonkey Monkey
Hi Steve!  Thanks for your reply.

> If there's no such thing, what's wrong with every subroutine making a
call to a quick program to retrieve the intended environment?

Well, a few things.  The major one is that you have to do it each and every
time, or it's a bug.  I had essentially that when I was calling "setfenv"
in every Lua function.  First it litters up every Lua fn with extra
lines of code, but more importantly it's fragile against programmer errors.
I was constantly forgetting to do it, and then I had subtle defects in
my Lua code where some variable that should have gone to the
environment-of-interest really went elsewhere.  Since there's no
compile-time checking in Lua (already dangerous itself), that's a pretty
dangerous situation.

Since I'm intending for the game community to be able to write these
Lua scripts, I don't want everybody to have to understand they must keep
calling such functions.  The real solution should be robust and automatic,
leaving no chance to screw it up :).  I screwed it up enough myself, and
I understood what was happening!  I'd rather have the C++ calling functions
set something up that "just works".

Also, performance matters here, second after robustness; I don't want extra
function calls in every Lua function.  (In fact, I can optionally switch in
LuaJIT to get more performance, and that's what I intend to use for real).

I'm not trying to claim there's no way to do what I want, only that
I don't know what the way is :).  Seems like there must be one...

CM


From: Steve Litt <[hidden email]>
To: [hidden email]
Cc: codemonkey Monkey <[hidden email]>
Sent: Saturday, March 24, 2012 4:32 PM
Subject: Re: callstack recursive environment

On Sat, 24 Mar 2012 14:34:46 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

>
>
> I"m using LUA to extend a game engine.  It's generally working OK
> (could wish for static type checking though...), but I've run into
> something I can't work out, and searching the list archives hasn't
> turned up an answer for any search I've tried so far.
>
> The behavior I'm after seems like it should be very simple, maybe the
> most common thing to want, but I can't find a combination of
> metatable/setfenv behaviors to do it.  I have a bunch of C++ internal
> game objects.  Each of these C++ objects has a Lua table associated
> with it which defines a desired Lua global environment.  All I want
> is to call a Lua fn from C++, and have this environment table be used
> for the Lua fn I call, and anything that *it* calls, recursively for
> the entire call stack.  In other words, I want my environment to be
> inherited.
>
>
> I started out by trying to use setfenv on the Lua fn from C++ before
> I call it, pointing to my environment.  For example, let's call this
> lua function LuaTest.  I quickly discovered that the setfenv is not
> inherited through the call stack!  I had to call setfenv in *every*
> Lua function called in the entire tree that LuaTest might call, which
> worked, but was not practical at all.
>
> My second atttempt, and the one I have limping along now, was to
> define a metatable with __index and __newindex entries pointing to my
> desired environment for LuaTest, and setting this as the metatable
> for the module that contains LuaTest (*).  This *almost* works.
> Every Lua fn called by LuaTest sees my environment table exactly as I
> wish but *only* within the module defining LuaTest.   Once any
> function is called outside that module, my environment is lost,
> replaced with the one defined by the setfenv of the called Lua
> function. 
>
>
> That's where I'm at now: I have the behavior I want but only within a
> single module.  What I really wish is something that seems so very
> simple: I want the environment i set to be inherited by anything
> called by LuaTest.  But I can't seem to get that behavior!  I've
> tried some of the "pseudo-index" things too, but whatever I try, it
> seems some "function environment" always clobbers the real
> environment I'm trying to have inherited down through the Lua call
> stack.
>
>
> Is there a nice simple way to do this?
>
> For a while I was just setting a global "E" with my desired
> environment, and everybody had to say "E.myvar ...".  That also
> works, even across modules, but it's a pain, and fragile: if anyone
> forgets to say "E.myvar", even just once, there's a bug.  With the
> metatable approach, at least I *know* variables are going into the
> environment I want them to.
>
>
> Thanks!

Hi Codemonkey,

It sounds to me like what you really want is the Lua equivalent of the
(Linux/Unix) bash export statement, which makes changes made to
environment variables in this program available in all programs
forked/execed from this program. And it sounds like according to your
research, there's no such thing in Lua.

If there's no such thing, what's wrong with every subroutine making a
call to a quick program to retrieve the intended environment? You say
above that you don't want to do that, but it doesn't sound that bad to
me. In the following discussion, I'll use generalized OOP terms, as
there are lots of ways to do OOP in Lua...

You could make a class that implements a stack of environments, with
each environment being a table of envvar/value pairs. You'd also have a
persistent object variable that keeps track of your current level in
the stack. Such a class would need the following methods:

* getenvtable() gets the environment table at the lowest level, and then
  sets the level counter one deeper. This is run at the beginning of
  every function.
* setenv(envvar, value) sets the current level's envvar to whatever
  value, and just for fun returns that value.
* return() tells the stack object you're done with this function, and
  to decrement the stack level. This is called at the end of every
  function that has called getenvtable(), just before the function's
  return statement.

So a function might look something like this:

function whatever(stackobj)
    envtable = stackobj:getenvtable()
    -- Do some stuff
    envtable[minutes] = stackobj:setenv(minutes+1)
    -- Do more stuff, then call another function
    son_of_whatever(stackobj)
    envtable[color] = stackobj:setenv(green)
    -- Do more stuff
    stackobj:return()
end

I'm not sure why simply passing envtable every time wouldn't work, but
if it didn't, I imagine this stackobj technique would.

HTH

SteveT
   

I haven't implemented this exact stack, but it seems to me like it
wouldn't be difficult, and I doubt it would be especially slow.


Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

David Given
On 24/03/12 22:56, codemonkey Monkey wrote:
[...]
> Also, performance matters here, second after robustness; I don't want extra
> function calls in every Lua function.  (In fact, I can optionally switch in
> LuaJIT to get more performance, and that's what I intend to use for real).
>
> I'm not trying to claim there's no way to do what I want, only that
> I don't know what the way is :).  Seems like there must be one...

I suspect the simplest and most robust thing here is to simply compile
the Lua script multiple times, so each game object maintains a reference
to the compiled chunk that it runs. That way you get total control over
the environment by passing in whatever object you like as the fourth
parameter to load().

This has a number of advantages: it lends itself very nicely to an
architecture where each type of game object gets its own script, which
can then use local variables etc for speed in a completely natural way;
it allows you to easily extend things so you run your scripts in
multiple Lua states, if you want multithreading or better
compartmentalisation; and it may well simplify your APIs.

The big disadvantage, of course, is that you need to store multiple
compiled chunks in memory for the same piece of Lua code, but these
aren't large.

--
┌─── dg@cowlark.com ───── http://www.cowlark.com ─────

│ "Never attribute to malice what can be adequately explained by
│ stupidity." --- Nick Diamos (Hanlon's Razor)


signature.asc (262 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Kevin Martin
In reply to this post by codemonkey Monkey
I'm only just getting my head around this at the moment, but I think you automatically get this behaviour in lua 5.2.

As I understand it, if you load the function, then change the first upvalue to be the table associated with your c++ object. That table will the become the global environment for the function, and anything it calls.

Kev
Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Miles Bader-2
Kevin Martin <[hidden email]> writes:
> I'm only just getting my head around this at the moment, but I think
> you automatically get this behaviour in lua 5.2.
>
> As I understand it, if you load the function, then change the first
> upvalue to be the table associated with your c++ object. That table
> will the become the global environment for the function, and anything
> it calls.

No.

[Many people seemed to have _thought_ that's how things worked in 5.2,
but they were wrong.]

The global environment reference in 5.2 is still "static", as it was
in 5.1, but now it uses an upvalue instead of a special environment
slot in the function.

-miles

--
The trouble with most people is that they think with their hopes or
fears or wishes rather than with their minds.  -- Will Durant

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Roberto Ierusalimschy
> > As I understand it, if you load the function, then change the first
> > upvalue to be the table associated with your c++ object. That table
> > will the become the global environment for the function, and anything
> > it calls.
>
> No.
>
> [Many people seemed to have _thought_ that's how things worked in 5.2,
> but they were wrong.]
>
> The global environment reference in 5.2 is still "static", as it was
> in 5.1, but now it uses an upvalue instead of a special environment
> slot in the function.

In particular, if you load and run a chunk that defines several
functions, changing its upvalue changes the environment for all
functions it created (as they are statically inside the chunk).

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Steve Litt
In reply to this post by codemonkey Monkey
On Sat, 24 Mar 2012 15:56:17 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

> Hi Steve!  Thanks for your reply.
>
> > If there's no such thing, what's wrong with every subroutine making
> > a
> call to a quick program to retrieve the intended environment?
>
> Well, a few things.  The major one is that you have to do it each and
> every time, or it's a bug. 

For practical purposes that's true.

> I had essentially that when I was calling
> "setfenv" in every Lua function.  First it litters up every Lua fn
> with extra lines of code, but more importantly it's fragile against
> programmer errors. I was constantly forgetting to do it, and then I
> had subtle defects in my Lua code where some variable that should
> have gone to the environment-of-interest really went elsewhere.

True enough.

> Since there's no compile-time checking in Lua (already dangerous
> itself), that's a pretty dangerous situation.

Hmmmm...

>
>
> Since I'm intending for the game community to be able to write these
> Lua scripts, I don't want everybody to have to understand they must
> keep calling such functions.  The real solution should be robust and
> automatic, leaving no chance to screw it up :).  I screwed it up
> enough myself, and I understood what was happening!  I'd rather have
> the C++ calling functions set something up that "just works".

Obviously I agree. The problem I see is that in Lua, tables are
passed/assigned by reference, not by value. So if a called function
modifies a table value, it remains modified after returning to the
caller, which I don't think is what you wanted. So you can't use any
data structure created from a table.

>
>
> Also, performance matters here, second after robustness; I don't want
> extra function calls in every Lua function.  (In fact, I can
> optionally switch in LuaJIT to get more performance, and that's what
> I intend to use for real).

At this point I don't think anyone can tell how much of a performance
hit these extra calls would make. After all, if there were such a thing
as a passed/stacked environment, you'd still need to query it.

>
>
> I'm not trying to claim there's no way to do what I want, only that
> I don't know what the way is :).  Seems like there must be one...

I think probably somebody will come forth with a Lua-provided way to do
it. But if not, I'd kinda like to help with the thinking on doing the
best possible workaround. I might learn something.

In another email I'll throw out some off-the-cuff rough ideas.

Thanks

SteveT

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Steve Litt
In reply to this post by codemonkey Monkey
On Sat, 24 Mar 2012 15:56:17 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

> Hi Steve!  Thanks for your reply.
>
> > If there's no such thing, what's wrong with every subroutine making
> > a
> call to a quick program to retrieve the intended environment?
>
> Well, a few things.  The major one is that you have to do it each and
> every time, or it's a bug.  I had essentially that when I was calling
> "setfenv" in every Lua function.  First it litters up every Lua fn
> with extra lines of code, but more importantly it's fragile against
> programmer errors. I was constantly forgetting to do it, and then I
> had subtle defects in my Lua code where some variable that should
> have gone to the environment-of-interest really went elsewhere.
> Since there's no compile-time checking in Lua (already dangerous
> itself), that's a pretty dangerous situation.

OK, now I understand your situation, so forget my stack of environments
object idea. And I don't think my idea would have worked anyway,
because in Lua tables are passed/assigned by reference, so changes at
level 7 would affect level 6, which is not what you want.

So if I understand you correctly, you want to be able to look at
variables and have them be the value they were in the caller, but
change them without affecting the caller. Do I have it right so far?

Thanks

SteveT

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

codemonkey Monkey
> So if I understand you correctly, you want to be able to look at
variables and have them be the value they were in the caller, but
change them without affecting the caller. Do I have it right so far?

Almost: changes should impact the caller too (hence, I set both
__index and __newindex right now).

Basically, my C++ code maintains a bunch of separate Lua tables,
one per game object.  When it's about to call a Lua fn for one of
these, it sets up the module's metatable.  Then, any variables either
referenced or created by the Lua functions being called go into
the right table.  This is what i want, except that I also want it to
follow the entire Lua call stack.

function a()
  some_var = true
  b()
  print(some_var)
end

function b()
  print(some_var)
  some_var = 5
end

Should print true and 5, even if b is in another module somewhere.

CM

PS - I've had pretty good results using boost::serialization to
serialize these Lua tables out to disk for game saving.  I do it
recursively, so you can put almost arbitrary stuff in these
environments, and they spring back to life on reloading.

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

codemonkey Monkey
In reply to this post by David Given

> compile the Lua script multiple times...

Hmm.  Interesting idea, but it's a little bit outside my architecture,
which is this:  my engine is already multi-threaded, and I maintain N+1 Lua
contexts, one per game AI thread plus one for the game's rendering thread,
since you can do lots of drawing stuff through the Lua extensions as well.
Each game object might run under any of those Lua environments at
any time, depending on which Posix thread happened to pop it off
the work queue (or deque, more properly).

But I pile a lot of stuff into these enivronments - many thousands of lines
of Lua code arleady, and I'm just getting warmed up :).  It's OK for me
to multiply that by 5 or 9 or whatever # of threads there are, but I'd
rather not multiply it by hundreds or thousands - seems like a lot of stuff,
and a lot of time to re-parse it all from the .lua files.

Hey, thanks for the comments everybody.  You guys are awesome :).

CM


From: David Given <[hidden email]>
To: [hidden email]
Sent: Saturday, March 24, 2012 5:30 PM
Subject: Re: callstack recursive environment

On 24/03/12 22:56, codemonkey Monkey wrote:
[...]
> Also, performance matters here, second after robustness; I don't want extra
> function calls in every Lua function.  (In fact, I can optionally switch in
> LuaJIT to get more performance, and that's what I intend to use for real).
>
> I'm not trying to claim there's no way to do what I want, only that
> I don't know what the way is :).  Seems like there must be one...

I suspect the simplest and most robust thing here is to simply compile
the Lua script multiple times, so each game object maintains a reference
to the compiled chunk that it runs. That way you get total control over
the environment by passing in whatever object you like as the fourth
parameter to load().

This has a number of advantages: it lends itself very nicely to an
architecture where each type of game object gets its own script, which
can then use local variables etc for speed in a completely natural way;
it allows you to easily extend things so you run your scripts in
multiple Lua states, if you want multithreading or better
compartmentalisation; and it may well simplify your APIs.

The big disadvantage, of course, is that you need to store multiple
compiled chunks in memory for the same piece of Lua code, but these
aren't large.

--
┌─── dg@cowlark.com ───── http://www.cowlark.com ─────

│ "Never attribute to malice what can be adequately explained by
│ stupidity." --- Nick Diamos (Hanlon's Razor)



Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Steve Litt
In reply to this post by codemonkey Monkey
On Sat, 24 Mar 2012 17:38:11 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

> > So if I understand you correctly, you want to be able to look at
> variables and have them be the value they were in the caller, but
> change them without affecting the caller. Do I have it right so far?
> Almost: changes should impact the caller too (hence, I set both
> __index and __newindex right now).
>
> Basically, my C++ code maintains a bunch of separate Lua tables,
> one per game object.  When it's about to call a Lua fn for one of
> these, it sets up the module's metatable.  Then, any variables either
> referenced or created by the Lua functions being called go into
> the right table.  This is what i want, except that I also want it to
> follow the entire Lua call stack.
>
> function a()
>   some_var = true
>   b()
>   print(some_var)
> end
>
> function b()
>   print(some_var)
>   some_var = 5
> end
>
> Should print true and 5, even if b is in another module somewhere.

CM, let me play devil's advocate here...

From my reading of what you wrote above, you basically want every
variable in every function to be, for want of a better word, a "public
variable." Is that REALLY what you want? That sounds a little dangerous to me.

SteveT


Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

codemonkey Monkey
> From my reading of what you wrote above, you basically want every
variable in every function to be, for want of a better word, a "public
variable."

Well, I have a lot of local vars.  The Lua fns have "local foo=..." all
over the place, so in that sense it's just like the out of the box Lua
behavior.  Local vars are local, and once you don't declare local
to go _G, or in my case, the particular environment I'm trying to use
at the time.  I'm not trying to change the basic way Lua works
in that sense, just trying to steer things over to a different table, but
being able to chose that table on a callstack basis.

CM



From: Steve Litt <[hidden email]>
To: [hidden email]
Sent: Saturday, March 24, 2012 7:11 PM
Subject: Re: callstack recursive environment

On Sat, 24 Mar 2012 17:38:11 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

> > So if I understand you correctly, you want to be able to look at
> variables and have them be the value they were in the caller, but
> change them without affecting the caller. Do I have it right so far?
> Almost: changes should impact the caller too (hence, I set both
> __index and __newindex right now).
>
> Basically, my C++ code maintains a bunch of separate Lua tables,
> one per game object.  When it's about to call a Lua fn for one of
> these, it sets up the module's metatable.  Then, any variables either
> referenced or created by the Lua functions being called go into
> the right table.  This is what i want, except that I also want it to
> follow the entire Lua call stack.
>
> function a()
>   some_var = true
>   b()
>   print(some_var)
> end
>
> function b()
>   print(some_var)
>   some_var = 5
> end
>
> Should print true and 5, even if b is in another module somewhere.

CM, let me play devil's advocate here...

From my reading of what you wrote above, you basically want every
variable in every function to be, for want of a better word, a "public
variable." Is that REALLY what you want? That sounds a little dangerous to me.

SteveT




Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Tony Finch
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy <[hidden email]> wrote:
>
> In particular, if you load and run a chunk that defines several
> functions, changing its upvalue changes the environment for all
> functions it created (as they are statically inside the chunk).

I suppose codemonkey could use lua_upvaluejoin on all the loaded chunks,
so setting that one joined upvalue changes the global environment of all
the code.

Tony.
--
f.anthony.n.finch  <[hidden email]>  http://dotat.at/
Tyne, Dogger: Variable 3 or 4. Smooth or slight. Mainly fair. Moderate or
good, occasionally poor.

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Peng Zhicheng
In reply to this post by codemonkey Monkey
于 2012-3-25 5:34, codemonkey Monkey 写道:

I"m using LUA to extend a game engine.  It's generally working OK (could wish for static type checking though...), but I've run into something I can't work out, and searching the list archives hasn't turned up an answer for any search I've tried so far.

The behavior I'm after seems like it should be very simple, maybe the most common thing to want, but I can't find a combination of metatable/setfenv behaviors to do it.  I have a bunch of C++ internal game objects.  Each of these C++ objects has a Lua table associated with it which defines a desired Lua global environment.  All I want is to call a Lua fn from C++, and have this environment table be used for the Lua fn I call, and anything that *it* calls, recursively for the entire call stack.  In other words, I want my environment to be inherited.

I started out by trying to use setfenv on the Lua fn from C++ before I call it, pointing to my environment.  For example, let's call this lua function LuaTest.  I quickly discovered that the setfenv is not inherited through the call stack!  I had to call setfenv in *every* Lua function called in the entire tree that LuaTest might call, which worked, but was not practical at all.

My second atttempt, and the one I have limping along now, was to define a metatable with __index and __newindex entries pointing to my desired environment for LuaTest, and setting this as the metatable for the module that contains LuaTest (*).  This *almost* works.  Every Lua fn called by LuaTest sees my environment table exactly as I wish but *only* within the module defining LuaTest.   Once any function is called outside that module, my environment is lost, replaced with the one defined by the setfenv of the called Lua function. 

That's where I'm at now: I have the behavior I want but only within a single module.  What I really wish is something that seems so very simple: I want the environment i set to be inherited by anything called by LuaTest.  But I can't seem to get that behavior!  I've tried some of the "pseudo-index" things too, but whatever I try, it seems some "function environment" always clobbers the real environment I'm trying to have inherited down through the Lua call stack.

Is there a nice simple way to do this?

For a while I was just setting a global "E" with my desired environment, and everybody had to say "E.myvar ...".  That also works, even across modules, but it's a pain, and fragile: if anyone forgets to say "E.myvar", even just once, there's a bug.  With the metatable approach, at least I *know* variables are going into the environment I want them to.

Thanks!


(*) Actually I'm not using the Lua module system, but my own, but that's not related to my query above: I see the same troubles when using Lua's module system.  Mine provides more private module behavior than the Lua module system, so you can say:

local my_module = Game.require('some/file.lua')

Your handle to it is private, and you can say my_module.some_function().  But I want some_function() to see the same Lua environment that its caller does!


IMHO, Lua is **lexical** scoped, not dynamic scoped, and what are callable are actually closures, not function prototypes.
a closure is an object(or instance) of one prototype, with some special variables, i.e. the upvalues, linked with it.
and the upvalues get linked to the closures when the closure is created, not when the closure is called.
what's more, I see the **environment** of a closure as a special upvalue(and in Lua 5.2, it is a upvalue), which is also attached to a closure when it is created. (a function prototype has no environments)

so you may not easily change the environment of closures when they got called.

but I do have a means to simulate this behavior, but with huge performance degradation.
if you don't care about the performance degradation, you may try this:

1.create a empty table _EMPTY(with its metatable _MT) as environment to load/compile all the scripts.
       you may need to set the _MT.__index and _MT.__newindex to ensure the compilation would suceed.

2. before calling some closure/function, if you want use _ENV as the environment,
       just set the _MT.__index and __MT.__newindex to point to _ENV.



BTW:
although this might work, I strongly recommend you rethink you design in Lua.
USE LUA IN THE WAY LUA IS.

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

codemonkey Monkey
> although this might work, I strongly recommend you rethink you design in Lua.  USE LUA IN THE WAY LUA IS.

Well, Lua the "the way it is", with no changes, means every variable written by any script will go into the same namespace, _G, and that's a pretty error prone thing.  There will be conflicts between names used by different scripts and objects, and it'll just be unworkable.

Using a namespace per script, lexically scoped, doesn't work well either, because each script can be used by many game objects, and there will be conficts between game objects using the same script.  Similar situation: it'll be a mess of defects.  I don't think reusing the same lua script for different game objects is so bad, really: there will be too many game objects to dupliate every script for each one, and anyway, the code doesn't change per object, so why should I duplicate it thusly?

So the only thing that made much sense to me was to use a dynamically scoped global environment.  I could turn off the global environment entirely, but that just makes script writers prefix every non-local they use with "something.myvar", which doesn't seem desirable either.

I couldn't think of any real alternatives to the approach I'm taking that didn't have drawbacks.  I do care about performance, but robustiness of scripts is an even higher priority than that, so I want to design for good encapsulation (and if I can figure out how to, for as much static checking as possible.  I found a web page talking about static checking tools for Lua, but haven't dug into it yet).


Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Coda Highland
> So the only thing that made much sense to me was to use a dynamically scoped
> global environment.  I could turn off the global environment entirely, but
> that just makes script writers prefix every non-local they use with
> "something.myvar", which doesn't seem desirable either.

Why not? Especially if that something is "self" it's a pretty standard idiom.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Steve Litt
On Wed, 28 Mar 2012 00:30:02 -0500
Coda Highland <[hidden email]> wrote:

> > So the only thing that made much sense to me was to use a
> > dynamically scoped global environment.  I could turn off the global
> > environment entirely, but that just makes script writers prefix
> > every non-local they use with "something.myvar", which doesn't seem
> > desirable either.
>
> Why not? Especially if that something is "self" it's a pretty
> standard idiom.
>
> /s/ Adam
>

Hey CodeMonkey,

I think Adam has a point. Honestly, I think global variables suck. You
once mentioned that making the programmer prefix them with something is
a bug. I understand your point, but I've found in every language I've
dealt with that global variables resulted in far more esoteric and hard
to find bugs.

I'll keep trying to think of something that works for your situation,
but I have a feeling in the end you'll do what most framewords do and
call object.method() (or object:method(), depending on how you code it).

SteveT

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Eike Decker
In reply to this post by codemonkey Monkey


On Mar 28, 2012 5:53 AM, "codemonkey Monkey" <[hidden email]> wrote:
>
> > although this might work, I strongly recommend you rethink you design in Lua.  USE LUA IN THE WAY LUA IS.
>
> Well, Lua the "the way it is", with no changes, means every variable written by any script will go into the same namespace, _G, and that's a pretty error prone thing.  There will be conflicts between names used by different scripts and objects, and it'll just be unworkable.

"Prefixing" works pretty nicely, same as using the "local" keyword to create variables. This is much like in C. I have maintained quite a bit of Lua code and globals weren't a problem. If some code is doing something wrong regarding globals, you can use metatables to find the cause without much of a problem.
Don't try to make your engine bulletproof for people who don't understand what they do. They'll just break it differently and the more you try to prevent them from doing it, the more it'll hurt in the end. The earlier they are confronted with their own code that doesn't work, the better. That's my opinion at least.

Cheers,
Eike

Reply | Threaded
Open this post in threaded view
|

Re: callstack recursive environment

Steve Litt
In reply to this post by codemonkey Monkey
On Sat, 24 Mar 2012 18:55:44 -0700 (PDT)
codemonkey Monkey <[hidden email]> wrote:

> > From my reading of what you wrote above, you basically want every
> variable in every function to be, for want of a better word, a "public
> variable."
>
> Well, I have a lot of local vars.  The Lua fns have "local foo=..."
> all over the place, so in that sense it's just like the out of the
> box Lua behavior.  Local vars are local, and once you don't declare
> local to go _G, or in my case, the particular environment I'm trying
> to use at the time.  I'm not trying to change the basic way Lua works
> in that sense, just trying to steer things over to a different table,
> but being able to chose that table on a callstack basis.
>
>
> CM

Hey Codemonkey, I just had a thought...

First, a disclaimer. I don't know how global variables work in any
language because I never use them, so I'm just reading your words and
trying to understand everything.

It sounds to me, from your words above, that all globals go in _G. Now
watch this...

Give _G a metatable that, when you're in diagnostic mode, errors out on
every global not specifically declared in _G's metatable (I have no
idea how to do this, but...). Now, when users of your object.method()
or object.property framework accidentally use method() or property,
when it's in diagnostic mode it errors out on every unintended global,
which is a defense against your very valid worry that leaving out the
object in object.property will silently fail in a different part of the
program.

Actually, in my personal opinion, global variables are so dangerous
that it makes sense to ask the programmer to explicitly declare them as
a global. As a matter of fact, I feel much more comfortable with
languages where local is the default, and you have to explicitly
declare all globals. I sometimes forget to put the word "local" (or
"my" in Perl), setting myself up for a later subtle bug.

I think this metatable for _G might go a long way to giving your
framework the robustness you're looking for.

SteveT

12