Adding direct enumeration of collections to Lua generic for loops

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding direct enumeration of collections to Lua generic for loops

Jean-Luc Jumpertz-2
Hi,

When enumerating a collection in Lua (i.e. a table or some kind of userdata), a generic for loop is probably the simplest and most straightforward solution.

So most of the time, you write it like:

for key, value in pairs(myCollection) do
  — some stuff
end

Frankly, when you know other languages, the call to ’pairs()’ here looks rather strange and inelegant. What you mean is « iterate in every entry of myCollection and do some stuff », which would probably better be written as:

for key, value in myCollection do
  — some stuff
end

Of course, because you are an experienced Lua programmer, you know what is really going on here. For a table, direct use of the for-in syntax could read something like:

for key, value in next, myCollection, nil do
  — some stuff
end

which, although it carries the full power of iterator functions, is not very readable and exposes to the reader the unneeded complexity of the collection's underlying implementation.

So we are back to calling `pairs()`, which is made compatible with custom types thanks to the "__pairs" metamethod. But again, having to call pairs() is odd, especially if your custom collection type has a single value per entry, or a triplet. :-)

Fore the reasons above, and because my application provides a bridge allowing to transparently mix Lua tables and native iOS collection classes, I started yesterday to investigate if providing direct enumeration of collections in Lua would be complex to implement, and the short answer is that it is finally an easy change.

If you are interested by the code, you can find the diff based on the latest available Lua 5.4 source code here  https://github.com/jlj/lua/commit/d4b680b10cffc048aa85dbd9d4338fa804f79666 (actually I implemented it first in a Lua 5.2 branch, but there is not that much difference between 5,2, 5.3 and 5.4 regarding this part).

In this implementation, if  `collection` is a Lua table or a userdata with the appropriate metamethod, you can now write:

for var1, var2, var3 in collection do
    -- do some stuff
end

Direct for..in enumeration can be customized by defining a "__forgen" (for generator) metamethod that shall return the 3 values needed by the generic for loop: an iterator function, a state, and the initial value of the iterator variable.

If no "__forgen" metamethod is defined and the collection is a Lua table, direct for..in enumeration is equivalent to calling the standard Lua library function pair().

Internally, a new Lua op-code 'OP_TFORPREP' has been added, that replaces the initial JMP instruction when initializing a generic for loop. When executed, this opcode checks for the presence of a "__forgen" metamethod in the loop base object and calls it if present.

I think this is a nice (and fully compatible) addition to Lua, so I decided to share it here.

Lua team, any opinion about this?

Jean-Luc


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Adding direct enumeration of collections to Lua generic for loops

Jean-Luc Jumpertz-2
Oops, sorry, wrong link to the code, the actual commit is https://github.com/jlj/lua/commit/688bebeb611c3d317acddb47e280c32e59280614

> Le 3 avr. 2018 à 20:15, Jean-Luc Jumpertz <[hidden email]> a écrit :
>
> Hi,
>
> When enumerating a collection in Lua (i.e. a table or some kind of userdata), a generic for loop is probably the simplest and most straightforward solution.
>
> So most of the time, you write it like:
>
> for key, value in pairs(myCollection) do
>  — some stuff
> end
>
> Frankly, when you know other languages, the call to ’pairs()’ here looks rather strange and inelegant. What you mean is « iterate in every entry of myCollection and do some stuff », which would probably better be written as:
>
> for key, value in myCollection do
>  — some stuff
> end
>
> Of course, because you are an experienced Lua programmer, you know what is really going on here. For a table, direct use of the for-in syntax could read something like:
>
> for key, value in next, myCollection, nil do
>  — some stuff
> end
>
> which, although it carries the full power of iterator functions, is not very readable and exposes to the reader the unneeded complexity of the collection's underlying implementation.
>
> So we are back to calling `pairs()`, which is made compatible with custom types thanks to the "__pairs" metamethod. But again, having to call pairs() is odd, especially if your custom collection type has a single value per entry, or a triplet. :-)
>
> Fore the reasons above, and because my application provides a bridge allowing to transparently mix Lua tables and native iOS collection classes, I started yesterday to investigate if providing direct enumeration of collections in Lua would be complex to implement, and the short answer is that it is finally an easy change.
>
> If you are interested by the code, you can find the diff based on the latest available Lua 5.4 source code here  https://github.com/jlj/lua/commit/d4b680b10cffc048aa85dbd9d4338fa804f79666 (actually I implemented it first in a Lua 5.2 branch, but there is not that much difference between 5,2, 5.3 and 5.4 regarding this part).
>
> In this implementation, if  `collection` is a Lua table or a userdata with the appropriate metamethod, you can now write:
>
> for var1, var2, var3 in collection do
>    -- do some stuff
> end
>
> Direct for..in enumeration can be customized by defining a "__forgen" (for generator) metamethod that shall return the 3 values needed by the generic for loop: an iterator function, a state, and the initial value of the iterator variable.
>
> If no "__forgen" metamethod is defined and the collection is a Lua table, direct for..in enumeration is equivalent to calling the standard Lua library function pair().
>
> Internally, a new Lua op-code 'OP_TFORPREP' has been added, that replaces the initial JMP instruction when initializing a generic for loop. When executed, this opcode checks for the presence of a "__forgen" metamethod in the loop base object and calls it if present.
>
> I think this is a nice (and fully compatible) addition to Lua, so I decided to share it here.
>
> Lua team, any opinion about this?
>
> Jean-Luc
>


Reply | Threaded
Open this post in threaded view
|

Re: Adding direct enumeration of collections to Lua generic for loops

Michal Kottman
In reply to this post by Jean-Luc Jumpertz-2
On Tue, Apr 3, 2018, 8:15 PM Jean-Luc Jumpertz <[hidden email]> wrote:
What you mean is « iterate in every entry of myCollection and do some stuff », which would probably better be written as:

for key, value in myCollection do
  — some stuff
end

What are the differences between your approach and "Self-iterating Objects" patch from http://lua-users.org/wiki/LuaPowerPatches ?
Reply | Threaded
Open this post in threaded view
|

Re: Adding direct enumeration of collections to Lua generic for loops

Jean-Luc Jumpertz-2

> Le 3 avr. 2018 à 20:57, Michal Kottman <[hidden email]> a écrit :
>
> What are the differences between your approach and "Self-iterating Objects" patch from http://lua-users.org/wiki/LuaPowerPatches ?

Good point, I was not of the existence of this specific power patch. After reading it, I would say that the need and the idea is essentially the same.
However it differs by the way it is implemented:
- the power patch modifies the behavior of the OP_TFORCALL instruction, executed at every iteration of the loop, and adds code for detecting if this is the first iteration, and in this case to modify the registers used by this instruction to force « self-iteration » iteration;
- my implementation adds a new instruction OP_TFORPREP that is executed only once when the loop is initialized, and is symmetrical with the OP_FORPREP used to initialize the numeric for. In this regard, I would consider it as less hack-ish than the power patch…

Although these differences are mainly implementation details, this thread could have the interest of bringing back the feasibility and benefits of this feature to the top of the stack. :-)



Reply | Threaded
Open this post in threaded view
|

Re: Adding direct enumeration of collections to Lua generic for loops

Daurnimator
In reply to this post by Jean-Luc Jumpertz-2
On 4 April 2018 at 04:15, Jean-Luc Jumpertz <[hidden email]> wrote:
> Direct for..in enumeration can be customized by defining a "__forgen" (for generator) metamethod that shall return the 3 values needed by the generic for loop: an iterator function, a state, and the initial value of the iterator variable.
>
> If no "__forgen" metamethod is defined and the collection is a Lua table, direct for..in enumeration is equivalent to calling the standard Lua library function pair().

Must be that time of the year for the __iter proposal :P
See https://duckduckgo.com/?q=inurl%3Alua-users.org%2Flists%2Flua-l%2F+"__iter"

Generally the outcome is:
  - Lua used to do bare iteration but removed it
  - There are often multiple ways to iterate an object (see ipairs vs
pairs). It's limiting to pick just one.
  - You can already do it with  __call