Next Version of Lua?

classic Classic list List threaded Threaded
143 messages Options
123456 ... 8
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

steve donovan
On Wed, Jun 10, 2009 at 1:59 PM, Luiz Henrique de
Figueiredo<[hidden email]> >   > setmetatable(t,{
>            __call = function (x,y,z) return next(x,z) end
>    })
>
>    for k,v in t do print(k,v) end
>
> No parentheses needed.

That is actually rather cute, but how would you do that for an
ipairs-style iterator?

steve d.
Reply | Threaded
Open this post in threaded view
|

Re[2]: Next Version of Lua?

Bulat Ziganshin
In reply to this post by Duncan Cross
Hello Duncan,

Wednesday, June 10, 2009, 3:32:52 PM, you wrote:

> I had an idea for an explicit syntax for this, that overloads the ... operator:

> local tbl = {SixValues()..., 7, 8, 9}

better

table = table1 .. table2

but the next question will be - does we concat on pairs or ipairs?
pitting both arrays and hashes to the same datatype strikes us here

--
Best regards,
 Bulat                            mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Peter Cawley
In reply to this post by steve donovan
On Wed, Jun 10, 2009 at 1:38 PM, steve donovan<[hidden email]> wrote:

> On Wed, Jun 10, 2009 at 1:59 PM, Luiz Henrique de
> Figueiredo<[hidden email]> >   > setmetatable(t,{
>>            __call = function (x,y,z) return next(x,z) end
>>    })
>>
>>    for k,v in t do print(k,v) end
>>
>> No parentheses needed.
>
> That is actually rather cute, but how would you do that for an
> ipairs-style iterator?

t = {"hello", "world", "from", "Lua"}

setmetatable(t, {__call = function(t,v,i)
  i = (i or 0) + 1
  v = t[i]
  if v then return i, v end
end})

for k, v in t do print(k, v) end
--> 1  hello
--> 2  world
--> 3  from
--> 4  Lua
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Peter Cawley
In reply to this post by steve donovan
On Wed, Jun 10, 2009 at 1:38 PM, steve donovan<[hidden email]> wrote:

> On Wed, Jun 10, 2009 at 1:59 PM, Luiz Henrique de
> Figueiredo<[hidden email]> >   > setmetatable(t,{
>>            __call = function (x,y,z) return next(x,z) end
>>    })
>>
>>    for k,v in t do print(k,v) end
>>
>> No parentheses needed.
>
> That is actually rather cute, but how would you do that for an
> ipairs-style iterator?

Or, alternatively, using the undocumented behaviour of ipairs:

ipairs_next = ipairs{}
t = {"hello", "world", "from", "Lua"}
setmetatable(t, {__call = function(t,_,i) return ipairs_next(t, i or 0) end})
for k, v in t do print(k, v) end
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

steve donovan
On Wed, Jun 10, 2009 at 2:49 PM, Peter Cawley<[hidden email]> wrote:
> Or, alternatively, using the undocumented behaviour of ipairs:
>
> ipairs_next = ipairs{}
> t = {"hello", "world", "from", "Lua"}
> setmetatable(t, {__call = function(t,_,i) return ipairs_next(t, i or 0) end})
> for k, v in t do print(k, v) end

Very elegant - actually, it is documented that the ipairs() function
would return 'an iterator function' as its first result.
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Luiz Henrique de Figueiredo
> Very elegant - actually, it is documented that the ipairs() function
> would return 'an iterator function' as its first result.

But not that the iterator function does not depend on the table...
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: Next Version of Lua?

steve donovan
In reply to this post by Bulat Ziganshin
On Wed, Jun 10, 2009 at 2:40 PM, Bulat
Ziganshin<[hidden email]> wrote:
> table = table1 .. table2
>
> but the next question will be - does we concat on pairs or ipairs?
> pitting both arrays and hashes to the same datatype strikes us here

Well, seen as sequences, it would be on ipairs. But it would be
confusing if this happened for all tables, and unnecessary, since it's
so simple to overload __concat appropriately.
Reply | Threaded
Open this post in threaded view
|

RE: Next Version of Lua? (__iter metamethod)

John Hind
In reply to this post by Luiz Henrique de Figueiredo
Ugg! Not a good choice, in my opinion! There is a "right choice" of iterator
for different types of object and it should be contextually selected not
having to remember whether to use pairs or ipairs. If you go down this way,
we'll end up defining __pairs or __ipairs to give error messages if the
wrong one is used! This is like saying the user has to use a different
function for adding real and complex numbers and if they pick the wrong one
they'll get a result, but a meaningless one.

Apart from this consideration of principle, it is using two metamethods to
do a half-assed job that could be properly done with just one!

I cannot see why this is syntactically impossible. Just check first for a
__iter when parsing the for statement, if not present, fall back to
processing as currently.

> -----Original Message-----
> From: [hidden email] [mailto:lua-
> [hidden email]] On Behalf Of Luiz Henrique de
> Figueiredo
> Sent: 10 June 2009 13:34
> To: Lua list
> Subject: Re: Next Version of Lua? (__iter metamethod)
>
> > Could I push my luck by asking if there is a position on '__iter'?
>
> There'll be metamethods __pairs and __ipairs. See
> http://lua-users.org/wiki/LuaFiveTwo

Reply | Threaded
Open this post in threaded view
|

Re[4]: Next Version of Lua?

Bulat Ziganshin
In reply to this post by steve donovan
Hello steve,

Wednesday, June 10, 2009, 4:59:14 PM, you wrote:

>> table = table1 .. table2
>>
>> but the next question will be - does we concat on pairs or ipairs?
>> pitting both arrays and hashes to the same datatype strikes us here

> Well, seen as sequences, it would be on ipairs.

the problem is that concatenation of hash tables is also useful


--
Best regards,
 Bulat                            mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua? (__iter metamethod)

Peter Cawley
In reply to this post by Luiz Henrique de Figueiredo
On Wed, Jun 10, 2009 at 2:16 PM, John Hind<[hidden email]> wrote:
> Ugg! Not a good choice, in my opinion! There is a "right choice" of iterator
> for different types of object and it should be contextually selected not
> having to remember whether to use pairs or ipairs. If you go down this way,
> we'll end up defining __pairs or __ipairs to give error messages if the
> wrong one is used! This is like saying the user has to use a different
> function for adding real and complex numbers and if they pick the wrong one
> they'll get a result, but a meaningless one.
pairs and ipairs are "intended" for iterating over table-like objects.
__pairs and __ipairs allow non-tables to behave like tables do in more
situations. If you want to attach them to a more complex object, then
simply set __pairs and __ipairs to the same function if you want to
make pairs() and ipairs() both do the default iteration. For complex
objects, it may not be obvious (to people other than the library
author) what the default iteration is. In this case, it can be clearer
if the complex object has members on it for iteration, like "for k,v
in obj:children() do" or "for callback in obj:callbacks() do", thus
making it explicit what part of the object is being iterated.

On Wed, Jun 10, 2009 at 2:16 PM, John Hind<[hidden email]> wrote:
> I cannot see why this is syntactically impossible. Just check first for a
> __iter when parsing the for statement, if not present, fall back to
> processing as currently.

The parser doesn't know what type of value "t" is in "for k, v in t
do", let alone if it has an __iter metamethod or not.
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Niklas Frykholm
In reply to this post by Olivier Hamel
2009/6/10 Olivier Hamel <[hidden email]>:
> Ah, I thank you for point this page out to me. I'll take this opportunity to
> dump my suggestions/wishes/hopes for Lua 5.2. Basically Lua 5.1 is very
> close to what I would consider the penultimate scripting language

I'm using lua as a game scripting language and the only thing I'm
missing is support for fast value types, i. e. user-created objects
with the same semantics as lua_Numbers.

Currently, if you want to create Complex or Vector3 type you have to
do it as a table or a userdata. And if you do a serious ammount of
processing on them, the ammount of garbage generated by simple
arithmetic operations will soon put a significant strain on the
system. (You could use a pool of such objects, but that would mean
resorting to manual memory management with all its pains - especially
when you are using it for something as simple as numbers.)

A generational garbage collector could be an alternative to value
types, I guess. Provided it is fast and smart enough to keep up with
the arithmetic garbage.

// Niklas
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Duncan Cross
On Wed, Jun 10, 2009 at 2:55 PM, Niklas Frykholm<[hidden email]> wrote:
> 2009/6/10 Olivier Hamel <[hidden email]>:
> Currently, if you want to create Complex or Vector3 type you have to
> do it as a table or a userdata. And if you do a serious ammount of
> processing on them, the ammount of garbage generated by simple
> arithmetic operations will soon put a significant strain on the
> system. (You could use a pool of such objects, but that would mean
> resorting to manual memory management with all its pains - especially
> when you are using it for something as simple as numbers.)

Perhaps I am misunderstanding, but if the pool is a table that has
been set to have weak values (i.e. its metatable's __mode field is set
to 'v'), you should not have to do any manual memory management -
values that only exist in the pool will be eligible for garbage
collection.

-Duncan
Reply | Threaded
Open this post in threaded view
|

RE: Next Version of Lua? (__iter metamethod)

John Hind
In reply to this post by Peter Cawley

> pairs and ipairs are "intended" for iterating over table-like objects.
> __pairs and __ipairs allow non-tables to behave like tables do in more
> situations. If you want to attach them to a more complex object, then
> simply set __pairs and __ipairs to the same function if you want to
> make pairs() and ipairs() both do the default iteration. For complex
> objects, it may not be obvious (to people other than the library
> author) what the default iteration is. In this case, it can be clearer
> if the complex object has members on it for iteration, like "for k,v
> in obj:children() do" or "for callback in obj:callbacks() do", thus
> making it explicit what part of the object is being iterated.
 
Looking at it fundamentally, any object that implements collection semantics
benefits from iterator functionality. "pairs" is appropriate for
dictionary-like collections (whether implemented using tables or otherwise)
while "ipairs" is appropriate for "list" or "array" like collections
(whether implemented using tables or otherwise). With basic Lua whether a
table is "dictionary-like" or "list-like" is about how you assign the keys,
but with true object orientation (using metatables) you will usually decide
the structure of the table in the class design. OK sometimes an object
requires more than one type of iterator, but rarely exactly "pairs" and
"ipairs" - so it would be better to cope with this by allowing "__iter" to
be parameterised (or as you suggest, using methods for non-default
iterators).

The Lua "String" type got an object oriented library in the last version,
but think about how the same would have to be done for "Table". You would
need to fork the Table type into two Classes- many of the functions in the
library only work on "List-Like" tables so you'd want one metatable for
those and another for "Generic" "Dictionary-like" tables. "pairs" would be
the right iterator in the generic class and "ipairs" in the "List-Like"
case.

> The parser doesn't know what type of value "t" is in "for k, v in t
> do", let alone if it has an __iter metamethod or not.

OK, but the case is no different from any other "core" metamethod i.e.
"__add". I admit I've never looked into the details, but if this metamethod
has to be resolved at runtime so could the "__iter" metamethod be. Maybe
that involves a new bytecode, but that is what new versions are for surely?
I get that "__pairs" and "__ipairs" are "library metamethods" in the same
sense as "__tostring" is, but this means they are rather trivial and could
be implemented easily on the current language platform just by shelling the
pairs and ipairs functions.


Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Niklas Frykholm
In reply to this post by Duncan Cross
2009/6/10 Duncan Cross <[hidden email]>:

> On Wed, Jun 10, 2009 at 2:55 PM, Niklas Frykholm<[hidden email]> wrote:
>> 2009/6/10 Olivier Hamel <[hidden email]>:
>> Currently, if you want to create Complex or Vector3 type you have to
>> do it as a table or a userdata. And if you do a serious ammount of
>> processing on them, the ammount of garbage generated by simple
>> arithmetic operations will soon put a significant strain on the
>> system. (You could use a pool of such objects, but that would mean
>> resorting to manual memory management with all its pains - especially
>> when you are using it for something as simple as numbers.)
>
> Perhaps I am misunderstanding, but if the pool is a table that has
> been set to have weak values (i.e. its metatable's __mode field is set
> to 'v'), you should not have to do any manual memory management -
> values that only exist in the pool will be eligible for garbage
> collection.

My description was a bit short, but the whole point of having a pool
was to avoid generating garbage. (By using a free object from the
pool, rather than creating a new one, whenever a new object is
needed.) But this requires us to manually track which objects in the
pool are free or not (i.e., manual memory management).

If the objects in the pool are eligible for garbage collection, then
the pool doesn't really buy us anything. We will generate the same
ammount of garbage with the pool as without it.

// Niklas
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Mark Hamburg
In reply to this post by Olivier Hamel
On Jun 10, 2009, at 3:02 AM, Olivier Hamel wrote:

> * (VM?) + Lexer: Have an explicitly defined handling of cases like  
> {[3] = 3, 1, 2, 3} and do not adjust the return codes to one unless  
> an operation is directly performed on it. Example:
> local tbl = {ReturnSixIncrementalValuesFromZeroToSix(), 7, 8, 9} --  
> this would equate to doing {1, 2, 3, 4, 5, 6, 7, 8, 9}
> local tbl = {ReturnSixIncrementalValuesFromZeroToSix() + 10, 7, 8,  
> 9} -- this would equate to doing {11, 7, 8, 9}
> The alternate suggestion of using ';' to elaborate when to use  
> multiple returns might be better (feedback on this please?)?

I'll let others discuss the pros and cons -- one of the big cons being  
compatibility -- but I will concur that it would be nice to have some  
sort of splicing construction for tables if only to allow for  
functions that returned nothing and didn't then insert a nil into the  
table which then tells ipairs that it's the end of the table:

ViewList = {
        Button( "I'm always here" ),
        Optional( TestCondition(), Button( "Condition Was True" ) )
        Button( "I'm always here too" )
}

The current work around for this is to have Optional return a "should  
not be here value" and have a post-process remove such values. One can  
also address splicing that way. But that doesn't feel quite as natural  
when building structures.

On the other hand, I don't know what syntax I would propose for this.  
It probably needs to stand out, but the more it stands out the more it  
risks not being used properly.

Mark

Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Mark Hamburg
In reply to this post by Olivier Hamel
On Jun 10, 2009, at 3:02 AM, Olivier Hamel wrote:

> * (VM?) + Lexer: Have an explicitly defined handling of cases like  
> {[3] = 3, 1, 2, 3} and do not adjust the return codes to one unless  
> an operation is directly performed on it. Example:
> local tbl = {ReturnSixIncrementalValuesFromZeroToSix(), 7, 8, 9} --  
> this would equate to doing {1, 2, 3, 4, 5, 6, 7, 8, 9}
> local tbl = {ReturnSixIncrementalValuesFromZeroToSix() + 10, 7, 8,  
> 9} -- this would equate to doing {11, 7, 8, 9}
> The alternate suggestion of using ';' to elaborate when to use  
> multiple returns might be better (feedback on this please?)?

I'll let others discuss the pros and cons -- one of the big cons being  
compatibility -- but I will concur that it would be nice to have some  
sort of splicing construction for tables if only to allow for  
functions that returned nothing and didn't then insert a nil into the  
table which then tells ipairs that it's the end of the table:

ViewList = {
        Button( "I'm always here" ),
        Optional( TestCondition(), Button( "Condition Was True" ) )
        Button( "I'm always here too" )
}

The current work around for this is to have Optional return a "should  
not be here value" and have a post-process remove such values. One can  
also address splicing that way. But that doesn't feel quite as natural  
when building structures.

On the other hand, I don't know what syntax I would propose for this.  
It probably needs to stand out, but the more it stands out the more it  
risks not being used properly.

Mark

Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Duncan Cross
In reply to this post by Niklas Frykholm
On Wed, Jun 10, 2009 at 3:42 PM, Niklas Frykholm<[hidden email]> wrote:

> 2009/6/10 Duncan Cross <[hidden email]>:
>> On Wed, Jun 10, 2009 at 2:55 PM, Niklas Frykholm<[hidden email]> wrote:
>>> 2009/6/10 Olivier Hamel <[hidden email]>:
>>> Currently, if you want to create Complex or Vector3 type you have to
>>> do it as a table or a userdata. And if you do a serious ammount of
>>> processing on them, the ammount of garbage generated by simple
>>> arithmetic operations will soon put a significant strain on the
>>> system. (You could use a pool of such objects, but that would mean
>>> resorting to manual memory management with all its pains - especially
>>> when you are using it for something as simple as numbers.)
>>
>> Perhaps I am misunderstanding, but if the pool is a table that has
>> been set to have weak values (i.e. its metatable's __mode field is set
>> to 'v'), you should not have to do any manual memory management -
>> values that only exist in the pool will be eligible for garbage
>> collection.
>
> My description was a bit short, but the whole point of having a pool
> was to avoid generating garbage. (By using a free object from the
> pool, rather than creating a new one, whenever a new object is
> needed.) But this requires us to manually track which objects in the
> pool are free or not (i.e., manual memory management).
>
> If the objects in the pool are eligible for garbage collection, then
> the pool doesn't really buy us anything. We will generate the same
> ammount of garbage with the pool as without it.
>
> // Niklas
>

Well, it depends - are you ending up with many many separate copies of
exactly the same vector? My assumption of how the pool would work was
that the key used to insert values into the pool would be a serialized
version of the object (for example using 'luaL_pushlstring(L, (const
char*)pvector, sizeof(VECTOR));' to generate the key, assuming your
vector is a simple struct stored in a full userdata) and the pool
would be queried by value to see if an existing vector object could be
reused.

But if that isn't the case, and the vectors you are dealing with are
unique in the majority of cases, then sure, it doesn't help. (It also
doesn't help if you need the vectors to be mutable rather than
immutable objects.)

-Duncan
Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua? (__iter metamethod)

Adrian Sietsma
In reply to this post by John Hind
John Hind wrote:
> Ugg! Not a good choice, in my opinion! There is a "right choice" of iterator
> for different types of object and it should be contextually selected not
> having to remember whether to use pairs or ipairs.
Not true.
I often have multi-use tables, such that t[id] = t[n] = object.
I can then lookup objects by id, or iterate them with ipairs.
In this case, how would your code 'contextually select' the iterator ?

> Apart from this consideration of principle, it is using two metamethods to
> do a half-assed job that could be properly done with just one!
Is the solution half-assed, or your understanding of the problem ?



Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua?

Mark Hamburg
In reply to this post by Niklas Frykholm
The argument for having support for __iter is that if one starts to  
build iterations via sequences of filters, maps, and so forth then  
__iter allows one to avoid having to tack something on at the end to  
actually convert to iteration form while at the same time avoiding the  
performance costs of going through the __call metamethod lookup on  
every loop iteration.

The expense of doing this is the extra opcode at the beginning of a  
for loop to see whether the first argument to the for loop has an  
__iter metamethod and if so calling it.

This arguably comes down to a question of how much Lua should embrace  
an object-oriented approach to operations and how much it should stick  
to a more procedural/functional approach. __iter suggests that we  
think about for as taking iterable objects.

While on the subject of iteration, at the expense of probably breaking  
a number of iterator implementations, it seems like it would be nice  
if one could signal done by having the iterator return nothing instead  
of nil. That way one could build iterators that did in fact return  
nil. I don't have a good example of usage handy, but imagine wanting  
to return just the values in an array over a range. The work around  
right now is to return an extra initial parameter and simply ignore it.

In particular, here is a semi-desirable use case:

        for v in ... do
                -- process varargs
        end

This doesn't iterate the varargs because it actually means use the ...  
parameters as the arguments. Hence, we probably need some other syntax  
anyway such as:

        for v in iter( ... ) do
                -- process varargs
        end

It isn't a huge burden to have this also return an index which can  
then be ignored, but the index might be seen as a distraction:

        for _, v in iter( ... ) do
                -- process varargs
        end

With or without the change to support nil values from iterations, some  
way to iterate (and for that matter index) ... would be really nice  
since use of select tends to lead to quadratic time complexity.

Mark

Reply | Threaded
Open this post in threaded view
|

Re: Next Version of Lua? (__iter metamethod)

Mark Hamburg
In reply to this post by Adrian Sietsma
With regard to __setindex, I was just thinking this morning about the  
costs associated with creating read-only tables and thinking that it  
would be worth benchmarking the approach of using a stub userdata:

        ud.mt = { __index = read-only-table }

And using a proxy table:

        {}.mt = { __index = read-only-table, __newindex = error-function }

__setindex would be the simplest option but would then need to be  
benchmarked for its effect on all table assignments.

Mark

123456 ... 8