Varargs efficiency

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Varargs efficiency

Soni "They/Them" L.
Hello,

How are varargs implemented? Are they array-backed, or linkedlist-backed? I'd think a linkedlist-backed approach would be better/faster:

With the array approach, you have to copy it every time you want to pass it around.

With the linked list approach, (assuming a singly linked list) you just pass the first node around. You don't have to worry about copying, moving, etc. To push an item you just make a new first node that links to the previous first node. To pop an item you just replace the first node with the next from the first node. To replace an item you don't have to copy the whole thing.

With arrays:
- Creation of varargs is fast.
- Reading varargs is fast.
- Passing varargs around is SLOW!
- Replacing an item is SLOW!
- Pushing and popping is SLOW!
- select(n, ...) is SLOW, as it requires copying the varargs, not once, but twice. First it does a full copy of ... to pass it in, then a copy of len(...)-n to return the new one.
- Memory cost can be HUGE in some cases. (see below)

With linked lists:
- Creation of varargs is so-so. (requires allocating nodes, altho the allocation could be batched so you only allocate once for, say, 100 nodes)
- Reading varargs is SLOW!
- Passing varargs around is fast.
- Replacing an item is so-so! (it's slow, but doesn't require a full copy, unlike with arrays)
- Pushing and popping is fast.
- select(n, ...) is fast, as it does no copying, and instead only returns the n-th node.
- Memory cost can be TINY in some cases. (see below)

Memory cost example:

Say you have a function f(...) that calls coroutines g(...), h(...), i(...), j(v, ...), with arguments g(1, ...), h(select(2, ...)), i(2, select(2, ...)), j(...).

With arrays:

The call to f(...) makes a copy.
The call to g(1, ...) makes a copy with space for the extra arg.
The call to h(select(2, ...)) makes a copy when calling into select(), followed by making another copy without the first argument, followed by making another copy when calling h.
The call to i(2, select(2, ...))
makes a copy when calling into select(), followed by making another copy without the first argument, followed by making another copy when calling i, with space for the extra arg.
The call to j(...) makes a copy without the first argument, and puts that first argument into a register.

With linked lists:

The call to f(...) passes a pointer to f.
The call to g(1, ...)
creates a node and passes a pointer to g.
The call to h(select(2, ...)) passes a pointer to select(), which returns a pointer to the next node, which's then passed to h.
The call to i(2, select(2, ...))
passes a pointer to select(), which returns a pointer to the next node, which's linked to by the node that's passed to i.
The call to j(...) puts the first node into a register and passes a pointer to the next node.

I don't know, but linked lists seem much better. (Did I miss any pros/cons?)
-- 
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.
Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Sean Conner
It was thus said that the Great Soni L. once stated:
> Hello,
>
> How are varargs implemented? Are they array-backed, or
> linkedlist-backed? I'd think a linkedlist-backed approach would be
> better/faster:

  Have you tried measuring both approaches?

> With arrays:
> - Passing varargs around is SLOW!

  How do you know this?  For instance:

        function a(...)
          -- code
        end

        function b(...)
          a(...)
        end

        function c(...)
          b(...)
        end

When c() calls b(), a pointer to the array-based stack can be passed to b
(nothing copied, very vast).  Same with b() calling a().  Remember, Lua is
written in C, and in C, when you have a pointer to X, you also have an array
of X (arrays references decay into pointers).

> - Replacing an item is SLOW!

  How so?

> - Pushing and popping is SLOW!

  For arrays, pushing and popping at the front of the array is slow; doing
so at the end is fast.  With a single linked list (say, like Lisp [1]),
pushing and popping from the front is fast, but pushing popping from the end
is slower than any array based version.

> With linked lists:
> - Creation of varargs is so-so. (requires allocating nodes, altho the
> allocation could be batched so you only allocate once for, say, 100 nodes)

  You still have to create the links in each node.  

> With arrays:
>
> The call to f(...) makes a copy.
> The call to g(1, ...) makes a copy with space for the extra arg.
> The call to h(select(2, ...)) makes a copy when calling into select(),
> followed by making another copy without the first argument, followed by
> making another copy when calling h.
> The call to i(2, select(2, ...)) makes a copy when calling into
> select(), followed by making another copy without the first argument,
> followed by making another copy when calling i, with space for the extra
> arg.
> The call to j(...) makes a copy without the first argument, and puts
> that first argument into a register.
>
> With linked lists:
>
> The call to f(...) passes a pointer to f.
> The call to g(1, ...) creates a node and passes a pointer to g.
> The call to h(select(2, ...)) passes a pointer to select(), which
> returns a pointer to the next node, which's then passed to h.
> The call to i(2, select(2, ...)) passes a pointer to select(), which
> returns a pointer to the next node, which's linked to by the node that's
> passed to i.
> The call to j(...) puts the first node into a register and passes a
> pointer to the next node.

  It sounds like you work a lot with varargs [2].  

> I don't know, but linked lists seem much better. (Did I miss any pros/cons?)

  It would probably complicate the implementation of Lua for potentially
little used programming paridigm?  I don't know how often varargs are used
the way you want to use it?  [3]  

  -spc (Have you tried programming in Lisp? [1])

[1] From all the posts you make here, I get the impression you would do
        far better programming in Lisp than in Lua (or any other non-Lisp
        language).  Every crazy thing you want to do is possible in Lisp.

        But that's just my impression.

[2] What are you trying to do anyway?  Create some form of virtual Lua
        environment?

[3] I don't use it often.  I just checked my own codebase at work, and
        out of 18,204 lines of Lua, only 17 uses of varargs, and only 4 of
        those do not deal with logging of some sort (calling either
        print(...) or f:write(...) or some variation on that).


Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Dirk Laurie-2
In reply to this post by Soni "They/Them" L.
2015-04-18 4:16 GMT+02:00 Soni L. <[hidden email]>:

> How are varargs implemented? Are they array-backed,
> or linkedlist-backed? I'd think a linkedlist-backed approach
> would be better/faster:

Array, i.e. a consecutive block on the main stack, in the frame
belonging to the calling function. One of the advantages of so
doing is that select(n,...) is O(1), whereas in a list-based approach
it would be O(n).

Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Soni "They/Them" L.
In reply to this post by Sean Conner


On 18/04/15 12:18 AM, Sean Conner wrote:
> It was thus said that the Great Soni L. once stated:
>> Hello,
>>
>> How are varargs implemented? Are they array-backed, or
>> linkedlist-backed? I'd think a linkedlist-backed approach would be
>> better/faster:
>    Have you tried measuring both approaches?
No, not really

>
>    It sounds like you work a lot with varargs [2].
>
> -spc (Have you tried programming in Lisp? [1])
> [1] From all the posts you make here, I get the impression you would
> do far better programming in Lisp than in Lua (or any other non-Lisp
> language). Every crazy thing you want to do is possible in Lisp. But
> that's just my impression.
> [2] What are you trying to do anyway? Create some form of virtual Lua
> environment?
I'm trying to make a Forth VM, using varargs as the stack.
> [3] I don't use it often. I just checked my own codebase at work, and
> out of 18,204 lines of Lua, only 17 uses of varargs, and only 4 of
> those do not deal with logging of some sort (calling either print(...)
> or f:write(...) or some variation on that).

--
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Tim Hill

>> [1] From all the posts you make here, I get the impression you would do far better programming in Lisp than in Lua (or any other non-Lisp language). Every crazy thing you want to do is possible in Lisp. But that's just my impression.
>> [2] What are you trying to do anyway? Create some form of virtual Lua environment?
> I'm trying to make a Forth VM, using varargs as the stack.
>> [3] I don't use it often. I just checked my own codebase at work, and out of 18,204 lines of Lua, only 17 uses of varargs, and only 4 of those do not deal with logging of some sort (calling either print(...) or f:write(...) or some variation on that).
>

You should realize that the caller of a vararg function has no idea that the target function is vararg; it just pushes its arguments onto the Lua stack normally. Thus the resulting data structure is just a continuous block of Lua stack slots which, as Dirk noted, can be accessed in an array-like manner (since the Lua stack is indexable).

I would doubt that using varargs as a Forth stack is the best approach. What’s wrong with a Lua array (aka table)?

—Tim



Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Soni "They/Them" L.


On 18/04/15 01:50 PM, Tim Hill wrote:
>>> [1] From all the posts you make here, I get the impression you would do far better programming in Lisp than in Lua (or any other non-Lisp language). Every crazy thing you want to do is possible in Lisp. But that's just my impression.
>>> [2] What are you trying to do anyway? Create some form of virtual Lua environment?
>> I'm trying to make a Forth VM, using varargs as the stack.
>>> [3] I don't use it often. I just checked my own codebase at work, and out of 18,204 lines of Lua, only 17 uses of varargs, and only 4 of those do not deal with logging of some sort (calling either print(...) or f:write(...) or some variation on that).
> You should realize that the caller of a vararg function has no idea that the target function is vararg; it just pushes its arguments onto the Lua stack normally. Thus the resulting data structure is just a continuous block of Lua stack slots which, as Dirk noted, can be accessed in an array-like manner (since the Lua stack is indexable).
>
> I would doubt that using varargs as a Forth stack is the best approach. What’s wrong with a Lua array (aka table)?
Lua arrays can't do nil, Lua stacks can.
> —Tim
>
>
>

--
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Hisham
On 18 April 2015 at 14:02, Soni L. <[hidden email]> wrote:

>
>
> On 18/04/15 01:50 PM, Tim Hill wrote:
>>>>
>>>> [1] From all the posts you make here, I get the impression you would do
>>>> far better programming in Lisp than in Lua (or any other non-Lisp language).
>>>> Every crazy thing you want to do is possible in Lisp. But that's just my
>>>> impression.
>>>> [2] What are you trying to do anyway? Create some form of virtual Lua
>>>> environment?
>>>
>>> I'm trying to make a Forth VM, using varargs as the stack.
>>>>
>>>> [3] I don't use it often. I just checked my own codebase at work, and
>>>> out of 18,204 lines of Lua, only 17 uses of varargs, and only 4 of those do
>>>> not deal with logging of some sort (calling either print(...) or
>>>> f:write(...) or some variation on that).
>>
>> You should realize that the caller of a vararg function has no idea that
>> the target function is vararg; it just pushes its arguments onto the Lua
>> stack normally. Thus the resulting data structure is just a continuous block
>> of Lua stack slots which, as Dirk noted, can be accessed in an array-like
>> manner (since the Lua stack is indexable).
>>
>> I would doubt that using varargs as a Forth stack is the best approach.
>> What’s wrong with a Lua array (aka table)?
>
> Lua arrays can't do nil, Lua stacks can.

If you're already writing a VM to a different language, you can just
map your Forth's nil to a non-nil Lua singleton.

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Why is implicit and explicit 'nil' treated differently?

Tony Papadimitriou
As you can see in the sample code below, print does not show the implicit nil (when it is last in the list of things to print), yet it shows the explicit one.  However, in every other regard, nil seems to behave the same in both cases.
 
Can someone explain why there is a difference in 'nil' treatment between these two?
(I can see the compiler produces slightly different code for each case but if they are equivalent, shouldn’t it produce the exact same code for both cases?)
 
And, is the difference evident only in print or elsewhere also?
 
I’m mostly interested if I should expect the exact same execution path for code receiving either nil, or not?  Because obviously print has a different execution for each case.
 
Thank you.
 
function implicit()
  return                      --implicit nil
end
 
function explicit()
  return nil                  --explicit nil
end
 
print(implicit()==nil,implicit() or 'hidden',implicit())
print(explicit()==nil,explicit() or 'shown',explicit())
print(implicit() == explicit())
 
--- And here’s what the compiler output looks like for the two functions ---
 
function <implicit> (2 instructions at 004CC030)
0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
        1       [2]     RETURN          0 1
        2       [3]     RETURN          0 1
 
function <explicit> (3 instructions at 004CE790)
0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
        1       [6]     LOADNIL         0 0
        2       [6]     RETURN          0 2
        3       [7]     RETURN          0 1
 
Reply | Threaded
Open this post in threaded view
|

Re: Why is implicit and explicit 'nil' treated differently?

Coda Highland
On Sat, Apr 18, 2015 at 11:45 AM,  <[hidden email]> wrote:

> As you can see in the sample code below, print does not show the implicit
> nil (when it is last in the list of things to print), yet it shows the
> explicit one.  However, in every other regard, nil seems to behave the same
> in both cases.
>
> Can someone explain why there is a difference in 'nil' treatment between
> these two?
> (I can see the compiler produces slightly different code for each case but
> if they are equivalent, shouldn’t it produce the exact same code for both
> cases?)
>
> And, is the difference evident only in print or elsewhere also?
>
> I’m mostly interested if I should expect the exact same execution path for
> code receiving either nil, or not?  Because obviously print has a different
> execution for each case.
>
> Thank you.
>
> function implicit()
>   return                      --implicit nil
> end
>
> function explicit()
>   return nil                  --explicit nil
> end
>
> print(implicit()==nil,implicit() or 'hidden',implicit())
> print(explicit()==nil,explicit() or 'shown',explicit())
> print(implicit() == explicit())
>
> --- And here’s what the compiler output looks like for the two functions ---
>
> function <implicit> (2 instructions at 004CC030)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [2]     RETURN          0 1
>         2       [3]     RETURN          0 1
>
> function <explicit> (3 instructions at 004CE790)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [6]     LOADNIL         0 0
>         2       [6]     RETURN          0 2
>         3       [7]     RETURN          0 1
>

The main difference is in the number of return values, which the
caller has to care about to know how many stack slots are being used
(though this fact is invisible to Lua code, but if you're writing code
with the C API you need to care).

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Why is implicit and explicit 'nil' treated differently?

Jan Behrens-4
In reply to this post by Tony Papadimitriou
On Sat, 18 Apr 2015 21:45:01 +0300
<[hidden email]> wrote:

> As you can see in the sample code below, print does not show the
> implicit nil (when it is last in the list of things to print), yet it
> shows the explicit one.  However, in every other regard, nil seems to
> behave the same in both cases.
>
> Can someone explain why there is a difference in 'nil' treatment
> between these two? (I can see the compiler produces slightly
> different code for each case but if they are equivalent, shouldn?t it
> produce the exact same code for both cases?)

It will be shown if you write:

print(implicit()==nil, implicit(), implicit() or 'hidden')
-->  true    nil     hidden

or

print(implicit()==nil, implicit() or 'hidden', (implicit()))
-->  true    hidden  nil

Generally "nil" and "none" (none = no value) are distinct when you call
a function or return from a function. However, "none" gets implicitly
converted to "nil" if you store it in a variable (or table) or if you
put parenthesis around it (or use it in an expression where a single
value is expected).

If you use a function call as last(!) argument to another function
call, e.g. f(arg1, arg2, g()), then multiple arguments returned from
the last function will be passed to the outer function:

function g()
  return "Hello", "World"
end

print("Note:", g())
-->  Note:   Hello   World

That's why

print(implicit()==nil, implicit() or 'hidden', implicit())

won't print "nil", because implicit() is last in the argument list,
which means that any variable number of return values (possibly zero
return values) will be passed to the print function in addition to the
two first arguments.

>
> And, is the difference evident only in print or elsewhere also?

I think "print" and "select" are the two notable cases. You can use
"select" to write your own functions to distinguish between "nil" and
"none":

function test(...)
  if select("#", ...) == 0 then
    return "no value given"
  elseif ... == nil then
    return "nil value given"
  else
    return "non-nil value given"
  end
end

test()
-->  no value given
test(nil)
--> nil value given
test(false)
--> non-nil value given

>
> I?m mostly interested if I should expect the exact same execution
> path for code receiving either nil, or not?  Because obviously print
> has a different execution for each case.

If you store a return value in a local variable or table, then "none"
always gets automatically converted to "nil". If you want to perform
this conversion explicitly, use parenthesis. This is also helpful to
truncate a variable number of return values. Compare:

print("Pos:", string.find("Hello World", "World"))
-->  Pos:    7       11

print("Pos:", (string.find("Hello World", "World")))
-->  Pos:    7


>
> Thank you.

Kind Regards,
Jan Behrens


>
> function implicit()
>   return                      --implicit nil
> end
>
> function explicit()
>   return nil                  --explicit nil
> end
>
> print(implicit()==nil,implicit() or 'hidden',implicit())
> print(explicit()==nil,explicit() or 'shown',explicit())
> print(implicit() == explicit())
>
> --- And here?s what the compiler output looks like for the two
> functions ---
>
> function <implicit> (2 instructions at 004CC030)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [2]     RETURN          0 1
>         2       [3]     RETURN          0 1
>
> function <explicit> (3 instructions at 004CE790)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [6]     LOADNIL         0 0
>         2       [6]     RETURN          0 2
>         3       [7]     RETURN          0 1

Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Tim Hill
In reply to this post by Soni "They/Them" L.


I would doubt that using varargs as a Forth stack is the best approach. What’s wrong with a Lua array (aka table)?
Lua arrays can't do nil, Lua stacks can.

Well, it’s true that a “true” Lua array (a sequence) cannot contain a nil value, but if you maintain your own top-of-stack (aka array size), then there is no reason why you cannot have nil as the value of some elements…

t = { “a”, “b”, “c” }
t.len = 3
t[2] = nil

—Tim


Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Sean Conner
It was thus said that the Great Tim Hill once stated:

>
> >>
> >> I would doubt that using varargs as a Forth stack is the best approach. What’s wrong with a Lua array (aka table)?
> > Lua arrays can't do nil, Lua stacks can.
>
> Well, it’s true that a “true” Lua array (a sequence) cannot contain a nil value, but if you maintain your own top-of-stack (aka array size), then there is no reason why you cannot have nil as the value of some elements…
>
> t = { “a”, “b”, “c” }
> t.len = 3
> t[2] = nil

  And in Lua 5.2:

t = setmetatable(
        { _len = 3 },
        { __len = function(self) return self._len end }
)

print(#t)

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Andrew Starks


On Saturday, April 18, 2015, Sean Conner <[hidden email]> wrote:
It was thus said that the Great Tim Hill once stated:
>
> >>
> >> I would doubt that using varargs as a Forth stack is the best approach. What’s wrong with a Lua array (aka table)?
> > Lua arrays can't do nil, Lua stacks can.
>
> Well, it’s true that a “true” Lua array (a sequence) cannot contain a nil value, but if you maintain your own top-of-stack (aka array size), then there is no reason why you cannot have nil as the value of some elements…
>
> t = { “a”, “b”, “c” }
> t.len = 3
> t[2] = nil

  And in Lua 5.2:

t = setmetatable(
        { _len = 3 },
        { __len = function(self) return self._len end }
)

print(#t)

  -spc


Lua doesn't have nil, inasmuch as nil in a sequence does not work. However, you would almost certainly get better performance by overloading __new/index and storing length in `.n`, like old Lua did, and then adding __len to read it. The other approach is to use a sentinel, but that's not false-y, if that matters. 

I'm not only poking my nose in to state the obvious. I have a similar issue in my system, where I need to "delete" keyframes and nil can't be used and false is a legal value. So, I use a delete method, instead. I may have used NUL sentinel, but it seemed unLua like, to me. 

varargs are the way they are because they were simple to implement in Lua, and people didn't like throwing a table away on every function call that had varargs. My sense is that their existence eats away at Roberto ( not being overly serious, but I say this based on the 2014 talk that he gave).

The way I see it: I use varargs all of the time, don't want to see them deprecated, and so I keep my fool mouth shut about them, and just accect them as they are. ;)

Rock on though. In the beginning of this thread, I was going to guess "cannabis use" but didn't know how to write it without sounding like a jerk or a pot head. 

-Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Why is implicit and explicit 'nil' treated differently?

Andrew Starks
In reply to this post by Jan Behrens-4


On Saturday, April 18, 2015, Jan Behrens <[hidden email]> wrote:
On Sat, 18 Apr 2015 21:45:01 +0300
<<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;tonyp@acm.org&#39;)">tonyp@...> wrote:

> As you can see in the sample code below, print does not show the
> implicit nil (when it is last in the list of things to print), yet it
> shows the explicit one.  However, in every other regard, nil seems to
> behave the same in both cases.
>
> Can someone explain why there is a difference in 'nil' treatment
> between these two? (I can see the compiler produces slightly
> different code for each case but if they are equivalent, shouldn?t it
> produce the exact same code for both cases?)

It will be shown if you write:

print(implicit()==nil, implicit(), implicit() or 'hidden')
-->  true    nil     hidden

or

print(implicit()==nil, implicit() or 'hidden', (implicit()))
-->  true    hidden  nil

Generally "nil" and "none" (none = no value) are distinct when you call
a function or return from a function. However, "none" gets implicitly
converted to "nil" if you store it in a variable (or table) or if you
put parenthesis around it (or use it in an expression where a single
value is expected).

If you use a function call as last(!) argument to another function
call, e.g. f(arg1, arg2, g()), then multiple arguments returned from
the last function will be passed to the outer function:

function g()
  return "Hello", "World"
end

print("Note:", g())
-->  Note:   Hello   World

That's why

print(implicit()==nil, implicit() or 'hidden', implicit())

won't print "nil", because implicit() is last in the argument list,
which means that any variable number of return values (possibly zero
return values) will be passed to the print function in addition to the
two first arguments.

>
> And, is the difference evident only in print or elsewhere also?

I think "print" and "select" are the two notable cases. You can use
"select" to write your own functions to distinguish between "nil" and
"none":

function test(...)
  if select("#", ...) == 0 then
    return "no value given"
  elseif ... == nil then
    return "nil value given"
  else
    return "non-nil value given"
  end
end

test()
-->  no value given
test(nil)
--> nil value given
test(false)
--> non-nil value given

>
> I?m mostly interested if I should expect the exact same execution
> path for code receiving either nil, or not?  Because obviously print
> has a different execution for each case.

If you store a return value in a local variable or table, then "none"
always gets automatically converted to "nil". If you want to perform
this conversion explicitly, use parenthesis. This is also helpful to
truncate a variable number of return values. Compare:

print("Pos:", string.find("Hello World", "World"))
-->  Pos:    7       11

print("Pos:", (string.find("Hello World", "World")))
-->  Pos:    7


>
> Thank you.

Kind Regards,
Jan Behrens


>
> function implicit()
>   return                      --implicit nil
> end
>
> function explicit()
>   return nil                  --explicit nil
> end
>
> print(implicit()==nil,implicit() or 'hidden',implicit())
> print(explicit()==nil,explicit() or 'shown',explicit())
> print(implicit() == explicit())
>
> --- And here?s what the compiler output looks like for the two
> functions ---
>
> function <implicit> (2 instructions at 004CC030)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [2]     RETURN          0 1
>         2       [3]     RETURN          0 1
>
> function <explicit> (3 instructions at 004CE790)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [6]     LOADNIL         0 0
>         2       [6]     RETURN          0 2
>         3       [7]     RETURN          0 1


There is one other case where nil/none appear. It almost never matters, but it thwarted me from being too creative with tables: __index is always adjust to one return value. So:

__index = function(t, i, x)
    --x is always absent (empty/nil), btw. This would almost certainly never matter, but it's true. I'm not sure how I know this, actually.it wouldn't happen from trying `t[1, 2]`, because that's an error. Anyway...
 end

f = function()
end

print (select("#", __index()), select("#", f()))
-->1   0

So you can never signal the difference between nothing and `nil` from your index metamethod. Again, not complaining, I'm just sayin'.

Reply | Threaded
Open this post in threaded view
|

Re: Why is implicit and explicit 'nil' treated differently?

Andrew Starks


On Saturday, April 18, 2015, Andrew Starks <[hidden email]> wrote:


On Saturday, April 18, 2015, Jan Behrens <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;jbe-lua-l@public-software-group.org&#39;);" target="_blank">jbe-lua-l@...> wrote:
On Sat, 18 Apr 2015 21:45:01 +0300
<[hidden email]> wrote:

> As you can see in the sample code below, print does not show the
> implicit nil (when it is last in the list of things to print), yet it
> shows the explicit one.  However, in every other regard, nil seems to
> behave the same in both cases.
>
> Can someone explain why there is a difference in 'nil' treatment
> between these two? (I can see the compiler produces slightly
> different code for each case but if they are equivalent, shouldn?t it
> produce the exact same code for both cases?)

It will be shown if you write:

print(implicit()==nil, implicit(), implicit() or 'hidden')
-->  true    nil     hidden

or

print(implicit()==nil, implicit() or 'hidden', (implicit()))
-->  true    hidden  nil

Generally "nil" and "none" (none = no value) are distinct when you call
a function or return from a function. However, "none" gets implicitly
converted to "nil" if you store it in a variable (or table) or if you
put parenthesis around it (or use it in an expression where a single
value is expected).

If you use a function call as last(!) argument to another function
call, e.g. f(arg1, arg2, g()), then multiple arguments returned from
the last function will be passed to the outer function:

function g()
  return "Hello", "World"
end

print("Note:", g())
-->  Note:   Hello   World

That's why

print(implicit()==nil, implicit() or 'hidden', implicit())

won't print "nil", because implicit() is last in the argument list,
which means that any variable number of return values (possibly zero
return values) will be passed to the print function in addition to the
two first arguments.

>
> And, is the difference evident only in print or elsewhere also?

I think "print" and "select" are the two notable cases. You can use
"select" to write your own functions to distinguish between "nil" and
"none":

function test(...)
  if select("#", ...) == 0 then
    return "no value given"
  elseif ... == nil then
    return "nil value given"
  else
    return "non-nil value given"
  end
end

test()
-->  no value given
test(nil)
--> nil value given
test(false)
--> non-nil value given

>
> I?m mostly interested if I should expect the exact same execution
> path for code receiving either nil, or not?  Because obviously print
> has a different execution for each case.

If you store a return value in a local variable or table, then "none"
always gets automatically converted to "nil". If you want to perform
this conversion explicitly, use parenthesis. This is also helpful to
truncate a variable number of return values. Compare:

print("Pos:", string.find("Hello World", "World"))
-->  Pos:    7       11

print("Pos:", (string.find("Hello World", "World")))
-->  Pos:    7


>
> Thank you.

Kind Regards,
Jan Behrens


>
> function implicit()
>   return                      --implicit nil
> end
>
> function explicit()
>   return nil                  --explicit nil
> end
>
> print(implicit()==nil,implicit() or 'hidden',implicit())
> print(explicit()==nil,explicit() or 'shown',explicit())
> print(implicit() == explicit())
>
> --- And here?s what the compiler output looks like for the two
> functions ---
>
> function <implicit> (2 instructions at 004CC030)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [2]     RETURN          0 1
>         2       [3]     RETURN          0 1
>
> function <explicit> (3 instructions at 004CE790)
> 0 params, 2 slots, 0 upvalues, 0 locals, 0 constants, 0 functions
>         1       [6]     LOADNIL         0 0
>         2       [6]     RETURN          0 2
>         3       [7]     RETURN          0 1


There is one other case where nil/none appear. It almost never matters, but it thwarted me from being too creative with tables: __index is always adjust to one return value. So:

__index = function(t, i, x)
    --x is always absent (empty/nil), btw. This would almost certainly never matter, but it's true. I'm not sure how I know this, actually.it wouldn't happen from trying `t[1, 2]`, because that's an error. Anyway...
 end

f = function()
end

print (select("#", __index()), select("#", f()))
-->1   0

So you can never signal the difference between nothing and `nil` from your index metamethod. Again, not complaining, I'm just sayin'.


Blech the above is wrong, unless you take it to mean that __index is being invoked as a metamethod on a table, not just a function hanging out in space, as the variable named __index. Sorry for the noise.  
Reply | Threaded
Open this post in threaded view
|

Re: Why is implicit and explicit 'nil' treated differently?

Jan Behrens-4
On Sat, 18 Apr 2015 23:21:47 -0500
Andrew Starks <[hidden email]> wrote:

> On Saturday, April 18, 2015, Andrew Starks <[hidden email]>
> wrote:
>
> >
> > > [...]
> >
> > There is one other case where nil/none appear. It almost never
> > matters, but it thwarted me from being too creative with tables:
> > __index is always adjust to one return value. So:
> >
> > __index = function(t, i, x)
> >     --x is always absent (empty/nil), btw. This would almost
> > certainly never matter, but it's true. I'm not sure how I know
> > this, actually.it wouldn't happen from trying `t[1, 2]`, because
> > that's an error. Anyway... end
> >
> > f = function()
> > end
> >
> > print (select("#", __index()), select("#", f()))
> > -->1   0
> >
> > So you can never signal the difference between nothing and `nil`
> > from your index metamethod. Again, not complaining, I'm just sayin'.
> >
> >
> Blech the above is wrong, unless you take it to mean that __index is
> being invoked as a metamethod on a table, not just a function hanging
> out in space, as the variable named __index. Sorry for the noise.



Indeed:

function __index(self, key)
  return "a", "b"
end

t = setmetatable({}, {__index=__index})

print(select("#", __index()), select("#", t.x))
-->  2       1

print(__index())
-->  a       b
-- multiple values passed to the print function, because
-- __index() is last argument

print(t.x)
-->  a
-- because t.x is expected to be a _single_ value, the list
-- of arguments is truncated



Similarly, "none" would be converted to "nil" if the __index function
returns no value *and* is invoked as a metamethod:

function __index(self, key)
  return
end

t = setmetatable({}, {__index=__index})

print(select("#", __index()), select("#", t.x))
-->  0       1

print(__index())
-->  
-- nil is not shown, because __index() may pass a variable
-- number of arguments (here: zero) to the print function
-- (__index() is the last entry in the argument list for
-- print)

print(t.x)
-->  nil
-- because t.x is expected to be a _single_ value, nil is
-- printed

print((__index()))
-->  nil
-- also here (__index()) in parenthesis must be a single
-- value, hence nil is printed


Regards,
Jan

Reply | Threaded
Open this post in threaded view
|

Re: Why is implicit and explicit 'nil' treated differently?

Lorenzo Donati-3
In reply to this post by Tony Papadimitriou


[...]

Please, don't highjack threads like this (it's not the first time you do
it).

If you want to begin a new thread send a new mail message to the mailing
list with a new subject, don't reply to another thread changing its subject.

Thanks!

-- Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Roberto Ierusalimschy
In reply to this post by Andrew Starks
> varargs are the way they are because they were simple to implement in Lua,
> and people didn't like throwing a table away on every function call that
> had varargs. My sense is that their existence eats away at Roberto ( not
> being overly serious, but I say this based on the 2014 talk that he gave).

That is true... :-(

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Nagaev Boris
In reply to this post by Andrew Starks
On Sat, Apr 18, 2015 at 10:07 PM, Andrew Starks <[hidden email]> wrote:
> varargs are the way they are because they were simple to implement in Lua,
> and people didn't like throwing a table away on every function call that had
> varargs. My sense is that their existence eats away at Roberto ( not being
> overly serious, but I say this based on the 2014 talk that he gave).

Could you provide a link to this talk, please?


--


Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

Re: Varargs efficiency

Andrew Starks


On Monday, April 20, 2015, Nagaev Boris <[hidden email]> wrote:
On Sat, Apr 18, 2015 at 10:07 PM, Andrew Starks <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;andrew.starks@trms.com&#39;)">andrew.starks@...> wrote:
> varargs are the way they are because they were simple to implement in Lua,
> and people didn't like throwing a table away on every function call that had
> varargs. My sense is that their existence eats away at Roberto ( not being
> overly serious, but I say this based on the 2014 talk that he gave).

Could you provide a link to this talk, please?


--


Best regards,
Boris Nagaev



It was at the 2013 Lua Workshop in Virginia. Not sure where, inside this hour long video. 
12