Lua strictlessness

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lua strictlessness

Rafis Ganeyev
Why Lua 5.1 allows to define local variables more than once and not throws error?
local a = 1
local a = a or 2
print(a) -- will print "1"

function f(a)
  -- we don't see it, but variable `a` is already defined in function's scope
  -- and now we still can define local variable with same name `a`
  local a = a or 4
  print(a)
end
f(3) -- will print "3"

--
rafis
Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Leo Razoumov
On Fri, Aug 23, 2013 at 4:32 AM, Rafis DoctorInfo
<[hidden email]> wrote:

> Why Lua 5.1 allows to define local variables more than once and not throws
> error?
> local a = 1
> local a = a or 2
> print(a) -- will print "1"
>
> function f(a)
>   -- we don't see it, but variable `a` is already defined in function's
> scope
>   -- and now we still can define local variable with same name `a`
>   local a = a or 4
>   print(a)
> end
> f(3) -- will print "3"
>
> --
> rafis

Lua defined two local variables for you with the same name 'a'. The
second one shadows the first.

--Leo--

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Rafis Ganeyev
Omg, shadows? It is even worse: does what means if I shadow redefine table, it will be never garbage collected?
local a = {} -- this table will be never collected?
local a = nil -- shadow the previous definition


2013/8/23 Leo Razoumov <[hidden email]>
On Fri, Aug 23, 2013 at 4:32 AM, Rafis DoctorInfo
<[hidden email]> wrote:
> Why Lua 5.1 allows to define local variables more than once and not throws
> error?
> local a = 1
> local a = a or 2
> print(a) -- will print "1"
>
> function f(a)
>   -- we don't see it, but variable `a` is already defined in function's
> scope
>   -- and now we still can define local variable with same name `a`
>   local a = a or 4
>   print(a)
> end
> f(3) -- will print "3"
>
> --
> rafis

Lua defined two local variables for you with the same name 'a'. The
second one shadows the first.

--Leo--




--
С уважением,
инженер-программист ООО "СТ-ПРЕМИУМ"
Рафис Ганеев
+79257371586
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Leo Razoumov
On Fri, Aug 23, 2013 at 5:33 AM, Rafis DoctorInfo
<[hidden email]> wrote:
> Omg, shadows? It is even worse: does what means if I shadow redefine table,
> it will be never garbage collected?
> local a = {} -- this table will be never collected?
> local a = nil -- shadow the previous definition
>
>

No, garbage collector will collect the table when the first 'local a'
goes out of scope. No surprises here.

--Leo--

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Tangent 128
On 08/23/2013 05:40 AM, Leo Razoumov wrote:
> No, garbage collector will collect the t
Actually, a quick experiment:

> w = setmetatable({}, {__mode="kv"})
> function cg()
>     collectgarbage()
>     for k,v in pairs(w) do
>         print(k,v)
>     end
>     print"---"
> end
> function t()
>     local a = {}
>     w[1] = a
>     cg() cg()
>
>     local a = 2
>     cg() cg()
> end
>
> cg()
> t()
> cg()

shows that the local continues to prevent garbage collection as long as
it's on the stack (that is, until t() returns), even if it's lexically
shadowed.

~Joseph Wallace

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Tim Hill

On Aug 23, 2013, at 5:59 PM, Tangent 128 <[hidden email]> wrote:

function t()
   local a = {}
   w[1] = a
   cg() cg()

   local a = 2
   cg() cg()
end

This is really a side-effect of the non-optimizing Lua compiler; a quick look at luac output shows that each "a" gets a different stack slot, since the compiler does not perform scope analysis (hardly surprising).

However, this does bring up an interesting issue for locals declared at the top-level scope in a long-lived script; as the earlier poster noted, these shadowed items will indeed live on until the script terminates so far as I can see.

local a = { a_very_big_table }
local a = 10
-- The table is now inaccessible but reachable by the GC

Of course, the quick answer is "don't do this", so I guess it might be worth noting somewhere in the Lua docs or wiki?

--Tim

Reply | Threaded
Open this post in threaded view
|

RE: Lua strictlessness

Thijs Schreijer


> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Tim Hill
> Sent: zaterdag 24 augustus 2013 3:39
> To: Lua mailing list
> Cc: [hidden email]
> Subject: Re: Lua strictlessness
>
>
> On Aug 23, 2013, at 5:59 PM, Tangent 128 <[hidden email]> wrote:
>
>
> function t()
>    local a = {}
>    w[1] = a
>    cg() cg()
>
>    local a = 2
>    cg() cg()
> end
>
> This is really a side-effect of the non-optimizing Lua compiler; a quick
> look at luac output shows that each "a" gets a different stack slot, since
> the compiler does not perform scope analysis (hardly surprising).
>
> However, this does bring up an interesting issue for locals declared at
> the top-level scope in a long-lived script; as the earlier poster noted,
> these shadowed items will indeed live on until the script terminates so
> far as I can see.
>
> local a = { a_very_big_table }
> local a = 10
> -- The table is now inaccessible but reachable by the GC
>
> Of course, the quick answer is "don't do this", so I guess it might be
> worth noting somewhere in the Lua docs or wiki?
>
> --Tim

Theoretical correct, but what code would do this? Define a variable, and then make it inaccessible by shadowing it. Common cases are to do this in inner blocks etc, different scoping, so after the shadowing scope ends the original is accessible again. If you don't use the '{ a_very_big_table }' any more, remove it.

It's not a practical case. Though it might be a source of errors, I doubt it to be worth the extra compiler checks and balances for each local that is created.

Thijs

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Tim Hill

On Aug 24, 2013, at 12:37 AM, Thijs Schreijer <[hidden email]> wrote:

It's not a practical case. Though it might be a source of errors, I doubt it to be worth the extra compiler checks and balances for each local that is created.

I just suggested it be mentioned as an anti-pattern .. it would not be a good use of compiler time to check for such things imho.

--Tim

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Dirk Laurie-2
2013/8/24 Tim Hill <[hidden email]>:
> It's not a practical case. Though it might be a source of errors, I doubt it
> to be worth the extra compiler checks and balances for each local that is
> created.
>
>
> I just suggested it be mentioned as an anti-pattern .. it would not be a
> good use of compiler time to check for such things imho.

That's what tools like lua-inspect are for.

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Rafis Ganeyev
In reply to this post by Thijs Schreijer
2013/8/24 Thijs Schreijer <[hidden email]>


> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Tim Hill
> Sent: zaterdag 24 augustus 2013 3:39
> To: Lua mailing list
> Cc: [hidden email]
> Subject: Re: Lua strictlessness
>
>
> On Aug 23, 2013, at 5:59 PM, Tangent 128 <[hidden email]> wrote:
>
>
> function t()
>    local a = {}
>    w[1] = a
>    cg() cg()
>
>    local a = 2
>    cg() cg()
> end
>
> This is really a side-effect of the non-optimizing Lua compiler; a quick
> look at luac output shows that each "a" gets a different stack slot, since
> the compiler does not perform scope analysis (hardly surprising).
>
> However, this does bring up an interesting issue for locals declared at
> the top-level scope in a long-lived script; as the earlier poster noted,
> these shadowed items will indeed live on until the script terminates so
> far as I can see.
>
> local a = { a_very_big_table }
> local a = 10
> -- The table is now inaccessible but reachable by the GC
>
> Of course, the quick answer is "don't do this", so I guess it might be
> worth noting somewhere in the Lua docs or wiki?
>
> --Tim

Theoretical correct, but what code would do this? Define a variable, and then make it inaccessible by shadowing it. Common cases are to do this in inner blocks etc, different scoping, so after the shadowing scope ends the original is accessible again. If you don't use the '{ a_very_big_table }' any more, remove it.

It's not a practical case. Though it might be a source of errors, I doubt it to be worth the extra compiler checks and balances for each local that is created.

Thijs

I thought about run-time checks only, compiler time checks for such task looks odd to me (it can't handle loadstring/loadfile). In case of global scope I can check it through strict.lua (http://lua-users.org/wiki/DetectingUndefinedVariables). But in case of local scope I can't manage it. It would be nice to have some function to enable such check on run-time debug.strict_locals(true). Is it possible to write such function in C using Lua API ABI or do I need to modify Lua core?
Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Coda Highland
On Mon, Aug 26, 2013 at 8:00 AM, Rafis DoctorInfo
<[hidden email]> wrote:
> 2013/8/24 Thijs Schreijer <[hidden email]>
> I thought about run-time checks only, compiler time checks for such task
> looks odd to me (it can't handle loadstring/loadfile). In case of global
> scope I can check it through strict.lua
> (http://lua-users.org/wiki/DetectingUndefinedVariables). But in case of
> local scope I can't manage it. It would be nice to have some function to
> enable such check on run-time debug.strict_locals(true). Is it possible to
> write such function in C using Lua API ABI or do I need to modify Lua core?

I would argue that loadstring/loadfile wouldn't apply ANYWAY, as they
create new scopes -- if a variable declared inside loadstring/loadfile
shadows something in the parent scope, it's actually DESIRABLE to
permit it.

I think it makes perfect sense to handle such a check at compile-time,
since the compiler can easily identify when the same name is used a
second time in the same scope -- it has to map names to stack slots,
after all.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

steve donovan
On Mon, Aug 26, 2013 at 7:03 PM, Coda Highland <[hidden email]> wrote:
> I think it makes perfect sense to handle such a check at compile-time,
> since the compiler can easily identify when the same name is used a
> second time in the same scope -- it has to map names to stack slots,
> after all.

Indeed! consider shadow.lua

local a = 1
do
    local a = 2
end
print(a)

$ luac -l -l shadow.lua
main <shadow.lua:0,0> (6 instructions, 24 bytes at 0000000000496FD0)
0+ params, 3 slots, 0 upvalues, 2 locals, 3 constants, 0 functions
        1       [1]     LOADK           0 -1    ; 1
        2       [3]     LOADK           1 -2    ; 2
        3       [5]     GETGLOBAL       1 -3    ; print
        4       [5]     MOVE            2 0
        5       [5]     CALL            1 2 1
        6       [5]     RETURN          0 1
constants (3) for 0000000000496FD0:
,,,
locals (2) for 0000000000496FD0:
        0       a       2       6
        1       a       3       3
upvalues (0) for 0000000000496FD0:

When you get both locals called 'a'  with the same end instruction,
then you have the 'shadow in same scope' issue.

It should not be too difficult to teach something like lglob to do
this, since it's already parsing the local information for each
function.

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Sven Olsen
Well, there is a bit of a hiccup, because Lua creates implicit "shadowed" locals anytime you write nested for loops.

But, I suppose such cases are easily identified -- they'll be overlaps between locals with names that aren't legal ids -- things like "(for generator)".  

If the behavior you want is to emit an error in response to something like Steve's shadow.lua, hooking into the parser to turn such situations into compile time errors would be fairly straightforward.  I'd suggest performing a check after each call to adjustlocalvars()  -- iterating through the local variable list, and checking that the new locals have startpc's higher than the endpc's of any locals that share the same name.

I'm still not convinced it's a great idea though :)  As the nested for loop case illustrates; there are perfectly valid reasons for Lua programmers to use shadowed locals.  So if you do write a patch for it, I'd certainly make it something that can be turned on and off via a debug function.

-Sven
Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Sven Olsen
I'd suggest performing a check after each call to adjustlocalvars()  -- iterating through the local variable list, and checking that the new locals have startpc's higher than the endpc's of any locals that share the same name.

(actually, now that I look at it, this suggestion may not be quite right.  unless you edit registerlocalvar as well, active locals may have endpc values that just contained freshly allocated memory.  but performing a post-compile check; basically by making a hacked copy of ldebug.c:findlocal, should be perfectly doable.)
Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Tim Hill
In reply to this post by Sven Olsen

On Aug 26, 2013, at 12:15 PM, Sven Olsen <[hidden email]> wrote:

> Well, there is a bit of a hiccup, because Lua creates implicit "shadowed" locals anytime you write nested for loops.
>
> But, I suppose such cases are easily identified -- they'll be overlaps between locals with names that aren't legal ids -- things like "(for generator)".  
>
> If the behavior you want is to emit an error in response to something like Steve's shadow.lua, hooking into the parser to turn such situations into compile time errors would be fairly straightforward.  I'd suggest performing a check after each call to adjustlocalvars()  -- iterating through the local variable list, and checking that the new locals have startpc's higher than the endpc's of any locals that share the same name.
>
> I'm still not convinced it's a great idea though :)  As the nested for loop case illustrates; there are perfectly valid reasons for Lua programmers to use shadowed locals.  So if you do write a patch for it, I'd certainly make it something that can be turned on and off via a debug function.
>
> -Sven

I don't think this is quite as simple as all that. Consider this:

local a = 1
function foo()
        return a
end
local a = 2
print(foo())

Clearly, foo() prints 2, as it should, since a is an upvalue. Now, it's been a while since I looked at Roberto's paper on how the VM handles upvalues, but clearly deciding on when a "new" local of the same name can reuse a stack slot assigned to a previous one is not just as a simple as checking the name and scope of the local. I suspect that is why the compiler takes a conservative approach (remember, the compiler is supposed to be very fast, and these smart checks take time), since the downside is pretty minor.

My original parasitic case seemed sufficiently odd that I felt it only warranted a "don't do this" note in the docs.

--Tim


Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Javier Guerra Giraldez
On Mon, Aug 26, 2013 at 4:00 PM, Tim Hill <[hidden email]> wrote:
> Clearly, foo() prints 2

really?

--
Javier

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Andrew Starks
In reply to this post by Tim Hill



On Mon, Aug 26, 2013 at 4:00 PM, Tim Hill <[hidden email]> wrote:

I don't think this is quite as simple as all that. Consider this:

local a = 1
function foo()
        return a
end
local a = 2
print(foo())

Clearly, foo() prints 2, as it should...
--Tim


Hmm... not in my world. It returns 1 as I think it should.

However:

local a = 1
function foo()
        return a
end
 a = 2
print(foo())
--> 2

When you declare a variable with local local, you make a _new_ variable. You do not overwrite the old one. The second local declaration in your code is not lexically in scope of foo, so why would it overwrite it?

When you set a variable without local, you assign a new value to the existing variable, if it exists, or to the _G/_ENV table, if it does not. 


-Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Tim Hill
In reply to this post by Javier Guerra Giraldez

On Aug 26, 2013, at 2:37 PM, Javier Guerra Giraldez <[hidden email]> wrote:

On Mon, Aug 26, 2013 at 4:00 PM, Tim Hill <[hidden email]> wrote:
Clearly, foo() prints 2

really?

--
Javier


Sigh .. if only I could type …

Clearly, foo() prints 1

--Tim

Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Tim Hill
In reply to this post by Andrew Starks

On Aug 26, 2013, at 4:16 PM, Andrew Starks <[hidden email]> wrote:

On Mon, Aug 26, 2013 at 4:00 PM, Tim Hill <[hidden email]> wrote:


When you declare a variable with local local, you make a _new_ variable. You do not overwrite the old one. The second local declaration in your code is not lexically in scope of foo, so why would it overwrite it?

When you set a variable without local, you assign a new value to the existing variable, if it exists, or to the _G/_ENV table, if it does not. 

Of course, as I said it was just a silly typo on my part. My point, however, was that the reuse of stack slots by the compiler involves more than just a quick check to see if the new local name matches any existing one at the same scope.

If memory serves me right, the upvalue in foo() that references the first "a" still references the stack slot (hence it changing to 2 as per your example). it's only when the containing function goes out of scope that Lua has to migrate the upvalue off the stack.

--Tim


Reply | Threaded
Open this post in threaded view
|

Re: Lua strictlessness

Sven Olsen
My point, however, was that the reuse of stack slots by the compiler involves more than just a quick check to see if the new local name matches any existing one at the same scope.

Yes, I think we're actually in agreement here.  While it's probably doable, a "strict locals" patch won't help programmers avoid most of the suboptimal code that involves memory needlessly trapped by local variables.  For example, consider:

function chunk()
  local a=big_table()
  do_something(a)
  do_something_else()
end

The problem here is that, if do_something_else() takes a while, and uses a lot of resources, a will still be hogging up memory while it runs -- Lua's only confident that it can release a after chunk() returns.

The fix is to wrap local a inside it's own block.  i.e.:

function chunk()
  do
    local a=big_table()
    do_something(a)
  end
  do_something_else()
end

(Now a can be freed before do_something_else() is called.)

So, while you could certainly hack together a "strict locals" patch, one that would do for locals about what strict.lua does for globals -- I'm not convinced it's a particularly good idea.  If the problem we're trying to fix is unnecessary memory use by locals, the solution is to use more do-blocks.  Enforcing naming discipline on locals is a mostly unrelated topic -- and most examples of local shadowing will happen inside nested blocks, a situation where they're not, actually, likely to be a source of memory inefficiencies.  

-Sven
12