[ANN] Lua 5.4.0 (work1) now available

classic Classic list List threaded Threaded
164 messages Options
1234567 ... 9
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Roberto Ierusalimschy
> and speaking of readline, it would still be good for lua.c to set
> rl_readline_name and (maybe) rl_inhibit_completion as discussed
> recently.

Yes, we will do it.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Coda Highland
In reply to this post by Rena
On Wed, Mar 14, 2018 at 11:00 AM, Rena <[hidden email]> wrote:

> On Wed, Mar 14, 2018, 10:27 Roberto Ierusalimschy, <[hidden email]>
> wrote:
>>
>> > In the presence of holes, #t seems to still have the same
>> > easy-to-define/hard-to-understand semantics from Lua 5.1-5.3.
>>
>> I said this in my first post:
>>
>>   You still can create tables with holes, but you must get out of your
>>   way to do it.
>>
>> Once you get out of your way and create holes, all the "standard
>> behavior" are back. But you asked for it...
>>
>> -- Roberto
>
>
> Well, it seems like now that tables can contain nil, it makes little sense
> that sequences can't.

You misunderstand.

The example given had a sequence that contained nil. It was only when
the table key was explicitly removed with undef that a hole was
introduced.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Dirk Laurie-2
In reply to this post by Roberto Ierusalimschy
2018-03-14 16:18 GMT+02:00 Roberto Ierusalimschy <[hidden email]>:

>> If you enable LUA_NILINTABLES (as explain in detail in a new thread),
>> there is another value of type nil, call it 'empty'. You can't tell
>> the difference between the values 'nil' and 'empty' at the level of
>> the API, let alone in Lua.
>
> I don't think these implementation details help this discussion, on the
> contrary.
>
> The implementation could be completely different, for instance an extra
> field in each table entry telling whether it is empty or not. Actually,
> this second implementation gives a much better mental model of the
> semantics then the real implementation, but it would be more expensive.

They are confusing, yes. Particularly because nil behaves like empty
and undef like nil. But bear with me while we make a thought
experiment.

Lua 5.3: There is a type 'nil' and a value also called 'nil'.

Experiment: There is a type 'nil' and two predefined values 'empty'
and 'undef'   both visible at the Lua level. If LUA_NILINTABLES is not
defined, they are the same and behave just like the value 'nil' does.
Otherwise there is a difference. 'undef' is actually quite close to
the present value 'nil': it is what too-short tuples are filled with;
it defines where a boundary in a table is. 'empty' behaves like the
empty value in the current implementation.

If we do 'nil=empty', the work1 implementation could also serve as an
implementation of this model.

The question is: is this model really conceptually worse? Would it not
in fact be quite useful to have access to both these values at the Lua
level?

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Pierre Chapuis
In reply to this post by Luiz Henrique de Figueiredo
On Wed, Mar 14, 2018, at 15:55, Roberto Ierusalimschy wrote:

> Every year or so the list has its recurrent discussion about the length
> operator and holes in sequences. Quite often, this discussion leads some
> people to propose that Lua needs an array type, different from tables.

In practice, this is not the reason why people (at least me, but not only)
wish Lua had an array type.

The main reason is that Lua is a dynamic language where you can
inspect the type of values (with type() mostly), but there is no
standard way to know if a table is supposed to be a sequence
or not when you get it. But in the outside world, things are not
tables, they tend to be sequences or map. This causes issues
when we try to interact with it using Lua.

I said it's not only me because I had feedback on that just yesterday
on Twitter [1] in a thread I started asking why Lua ended up in the list
of "most dreaded languages" in the 2018 Stack Overflow study...

In my own case, here is the kind of trick I have to resort to to deal
with this when using, for instance, François Perrad's MessagePack
library [2]:

    local mp = require "MessagePack"

    rawset(mp.packers, "function", function(buf, f) f(buf) end)

    function mp.MAP(t)
        return function(buf)
            local n = 0
            for _ in pairs(t) do n = n + 1 end
            mp.packers["map"](buf, t, n)
        end
    end

    function mp.ARRAY(t)
        return function(buf)
            mp.packers["array"](buf, t, #t)
        end
    end

    local my_binary = mp.encode {
        an_empty_array = mp.ARRAY {},
        an_empty_map = mp.MAP {},
        -- ... etc
    }

In another dynamic language (Ruby, Python, JavaScript, etc) this code
would look like this instead:

    local mp = require "MessagePack"

    local my_binary = mp.encode {
        an_empty_array = [],
        an_empty_map = {},
    }

Sure, the library itself could provide a helper and it could look like this:

    local mp = require "MessagePack"

    local my_binary = mp.encode {
        an_empty_array = mp.sequence(),
        an_empty_map = {},
    }

but even if it did, that does not solve the issue of structures created
by code that doesn't even know about the library.

For instance, this is dangerous:

    local mp = require "MessagePack"
    local json = require "json"

    local my_json = get_json()
    local my_binary = mp.encode(json.decode(my_json))

whereas in another language it would be fine (disregarding error
handling etc, this is an example).

[1] https://twitter.com/karlseguin/status/973582788950573058
[2] https://github.com/fperrad/lua-MessagePack

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Gavin Wraith
In reply to this post by Roberto Ierusalimschy
In message <[hidden email]>
          Roberto Ierusalimschy <[hidden email]> wrote:

>> I mean: if I can pass something to a function, and I can get it back from a
>> call... well it is a value!
>
> That is true. But you cannot pass 'undef' to a function or return
> an 'undef' from a function (anymore than you can pass 'while' to a
> function).

It seems to me that the proposed syntax "x = undef" looks like
an assignment of a value. Why not "x undef" or "undef x"? Or have
I missed that discussion? Sorry for the noise if I have.
--
Gavin Wraith ([hidden email])
Home page: http://www.wra1th.plus.com/

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Oliver Kroth
In reply to this post by Dirk Laurie-2


Am 14.03.2018 um 18:16 schrieb Dirk Laurie:

> 2018-03-14 16:18 GMT+02:00 Roberto Ierusalimschy <[hidden email]>:
>>> If you enable LUA_NILINTABLES (as explain in detail in a new thread),
>>> there is another value of type nil, call it 'empty'. You can't tell
>>> the difference between the values 'nil' and 'empty' at the level of
>>> the API, let alone in Lua.
>> I don't think these implementation details help this discussion, on the
>> contrary.
>>
>> The implementation could be completely different, for instance an extra
>> field in each table entry telling whether it is empty or not. Actually,
>> this second implementation gives a much better mental model of the
>> semantics then the real implementation, but it would be more expensive.
> They are confusing, yes. Particularly because nil behaves like empty
> and undef like nil. But bear with me while we make a thought
> experiment.
>
> Lua 5.3: There is a type 'nil' and a value also called 'nil'.
>
> Experiment: There is a type 'nil' and two predefined values 'empty'
> and 'undef'   both visible at the Lua level. If LUA_NILINTABLES is not
> defined, they are the same and behave just like the value 'nil' does.
> Otherwise there is a difference. 'undef' is actually quite close to
> the present value 'nil': it is what too-short tuples are filled with;
> it defines where a boundary in a table is. 'empty' behaves like the
> empty value in the current implementation.
>
> If we do 'nil=empty', the work1 implementation could also serve as an
> implementation of this model.
>
> The question is: is this model really conceptually worse? Would it not
> in fact be quite useful to have access to both these values at the Lua
> level?
>
I like the idea of having two values for the type nil: one for "there is
no value", and the other "table entry does not exist"

The question is which one we name nil

Like Lua 5.3 and before: nil = 'table entry does not exist',
or as has been implemented in Lua 5.4, nil = 'no value'

--
Oliver




Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Viacheslav Usov
In reply to this post by Roberto Ierusalimschy
On Wed, Mar 14, 2018 at 3:01 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> How do you store a 'while' in a table?

If you mean that 'undef' is a keyword on par with 'while', then perhaps it could be used with a syntax that is not a value-like, for example:

undef table[index]

Such a syntax does not suggest that 'undef' can be stored, retrieved, etc.

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Albert Chan
In reply to this post by Gavin Wraith
On Mar 14, 2018, at 5:33 PM, Gavin Wraith <[hidden email]> wrote:

>
> It seems to me that the proposed syntax "x = undef" looks like
> an assignment of a value. Why not "x undef" or "undef x"? Or have
> I missed that discussion? Sorry for the noise if I have.
> --
> Gavin Wraith ([hidden email])
> Home page: http://www.wra1th.plus.com/
>

what is so bad about the old syntax x = nil ?

false were added in lua to "store" nil in table, there is no need for undef.

I am curious, is there an example where undef is truly needed ?


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Sean Conner
It was thus said that the Great albertmcchan once stated:

> On Mar 14, 2018, at 5:33 PM, Gavin Wraith <[hidden email]> wrote:
>
> >
> > It seems to me that the proposed syntax "x = undef" looks like
> > an assignment of a value. Why not "x undef" or "undef x"? Or have
> > I missed that discussion? Sorry for the noise if I have.
> > --
> > Gavin Wraith ([hidden email])
> > Home page: http://www.wra1th.plus.com/
> >
>
> what is so bad about the old syntax x = nil ?
>
> false were added in lua to "store" nil in table, there is no need for undef.
>
> I am curious, is there an example where undef is truly needed ?

  Assume you have the following JSON:

        [ 1 , 2 , true , false , nil , "alpha" , "beta" ]

  Under the current (5.3) behavior, you can convert this to the following
Lua table:

        { 1 , 2 , true , false , nil , "alpha" , "beta" }

but the length is undefined, because the nil creates a hole.  If you use
false instead:

        { 1 , 2 , true , false , false , "alpha" , "beta" }

the length *is* defined, but it may not have the same semantic meaning.
Now, with the proposed undef (which seems to be confusing a lot of people),
the Lua table:

        { 1 , 2 , true , false , nil , "alpha" , "beta" }

would have a defined length and semantically matches the original JSON.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Roberto Ierusalimschy
In reply to this post by Dirk Laurie-2
> Experiment: There is a type 'nil' and two predefined values 'empty'
> and 'undef'   both visible at the Lua level. If LUA_NILINTABLES is not
> defined, they are the same and behave just like the value 'nil' does.
> Otherwise there is a difference. 'undef' is actually quite close to
> the present value 'nil': it is what too-short tuples are filled with;
> it defines where a boundary in a table is. 'empty' behaves like the
> empty value in the current implementation.
>
> If we do 'nil=empty', the work1 implementation could also serve as an
> implementation of this model.
>
> The question is: is this model really conceptually worse? Would it not
> in fact be quite useful to have access to both these values at the Lua
> level?

The problems I am trying to solve are these:

1) A constructor like {x, y, z} should always create a sequence with
three elements. A constructor like {...} should always get all arguments
passed to a function. A constructor like {f(x)} should always get
all results returned by the function. '#' should always work on these
tables correctly.

2) A statement like 't[#t + 1] = x' should always add one more element
at the end of a sequence.

These are what confuse people all the time, these are what start new
rounds of discussions around '#'.

Once 'empty' and 'undef' are values, you break all those properties.

I think we cannot use a value to represent the absence of a value, no
matter how many no-value values we add. It is a contradiction. Several
languages followed this path, always with bad results.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Russell Haley


On Wed, Mar 14, 2018 at 11:24 AM, Roberto Ierusalimschy <[hidden email]> wrote:
> Experiment: There is a type 'nil' and two predefined values 'empty'
> and 'undef'   both visible at the Lua level. If LUA_NILINTABLES is not
> defined, they are the same and behave just like the value 'nil' does.
> Otherwise there is a difference. 'undef' is actually quite close to
> the present value 'nil': it is what too-short tuples are filled with;
> it defines where a boundary in a table is. 'empty' behaves like the
> empty value in the current implementation.
>
> If we do 'nil=empty', the work1 implementation could also serve as an
> implementation of this model.
>
> The question is: is this model really conceptually worse? Would it not
> in fact be quite useful to have access to both these values at the Lua
> level?

The problems I am trying to solve are these:

1) A constructor like {x, y, z} should always create a sequence with
three elements. A constructor like {...} should always get all arguments
passed to a function. A constructor like {f(x)} should always get
all results returned by the function. '#' should always work on these
tables correctly.

2) A statement like 't[#t + 1] = x' should always add one more element
at the end of a sequence.

These are what confuse people all the time, these are what start new
rounds of discussions around '#'.

Once 'empty' and 'undef' are values, you break all those properties.

I think we cannot use a value to represent the absence of a value, no
matter how many no-value values we add. It is a contradiction. Several
languages followed this path, always with bad results.

-- Roberto


So in the end, one would turn this feature on and only use undef if the desire was to *create* holes in the table 'sequence'? Other than that, we would have the sequence like behavior and can stop having to add (useless) values of True to have a table of named elements?

Russ
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Russell Haley
In reply to this post by Andrew Gierth


On Wed, Mar 14, 2018 at 12:38 AM, Andrew Gierth <[hidden email]> wrote:
>>>>> "Luiz" == Luiz Henrique de Figueiredo <[hidden email]> writes:

 Luiz> All feedback welcome. Thanks.

Building on freebsd doesn't work without adding CC=cc (since gcc is not
installed by default and if you install it from ports, it's usually
under a version-specific name such as gcc6), and also enabling readline
requires adding -I/usr/local/include and -L/usr/local/lib, or tweaking
things to use libedit instead.

I've tried these, which all seem to work on my system (though obviously
the presence of a port build would make this somewhat moot):

# with base system libedit:
freebsd:
        $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"

# ... with readline from ports:
freebsd-readline:
        $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE -I/usr/local/include" SYSLIBS="-Wl,-E -L/usr/local/lib -lreadline"

# ... with editing disabled
freebsd-noedit:
        $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E"

and speaking of readline, it would still be good for lua.c to set
rl_readline_name and (maybe) rl_inhibit_completion as discussed
recently.

--
Andrew.

https://reviews.freebsd.org/D13690

The above is a link to my suggested patches to the Lua 5.3 port for FreeBSD to build on FreeBSD 10-12 (9 should work, I haven't tested below 10. HEAD is 12). This also contains the suggested libedit patch that I came to independently. If desired, I would like to confirm with some NetBSD and OpenBSD fellows to see if we can have a standard target for BSD?

I'll be putting together a port file for 5.4 and posting it for review in the next week or so. Andrew, any testing and reporting you can do would push my patches forward. If your interested in pushing the default Lua version up (currently sits at 5.2), I could use some help with that. Ping me if you're interested. 

Russ
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Roberto Ierusalimschy
In reply to this post by Viacheslav Usov
> On Wed, Mar 14, 2018 at 3:01 PM, Roberto Ierusalimschy <
> [hidden email]> wrote:
>
> > How do you store a 'while' in a table?
>
> If you mean that 'undef' is a keyword on par with 'while', then perhaps it
> could be used with a syntax that is not a value-like, for example:
>
> undef table[index]
>

Answering that again:

Yes, I mean that 'undef' is a keyword on par with 'while'. And
't[k]=undef' is a new construction, with the same semantics of
'undef t[k]', that looks like an assignment but it is not an assignment.

Once again: 't[k]=undef' looks like an assignment, but it is not an
assignment. It means "remove the key 'k' from table 't'".

Yes, we could use a syntax that is not a value-like. But that would make
all programs that use that syntax unable to run (or even compile) in older
versions of Lua.

The proposed syntax would allow programs written for 5.4 to run on 5.3
without changes (pending metamethods), and would also allow programs
written for Lua 5.3 to be written with undef (only as an idiom), so
that they would more easily migrate to a future version of Lua where
nil-in-table would be the default semantics (if ever there will be such
version).

So, 't[k]=undef' looks like an assignment on purpose, because in older
versions of Lua the semantics of removing an item from a list was done
through an assignment. But only because it looks like an assignment
it does not mean it is an an assignment.

Just to be sure: 't[k]=undef' looks like an assignment, but
it is not an assignment. It means "remove the key 'k' from table 't'".

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Albert Chan
In reply to this post by Sean Conner


On Mar 14, 2018, at 2:12 PM, Sean Conner <[hidden email]> wrote:

> It was thus said that the Great albertmcchan once stated:
>>
>> what is so bad about the old syntax x = nil ?
>>
>> false were added in lua to "store" nil in table, there is no need for undef.
>>
>> I am curious, is there an example where undef is truly needed ?
>
>  Assume you have the following JSON:
>
>    [ 1 , 2 , true , false , nil , "alpha" , "beta" ]
>
>  Under the current (5.3) behavior, you can convert this to the following
> Lua table:
>
>    { 1 , 2 , true , false , nil , "alpha" , "beta" }
>
> but the length is undefined, because the nil creates a hole.  If you use
> false instead:
>
>    { 1 , 2 , true , false , false , "alpha" , "beta" }
>
> the length *is* defined, but it may not have the same semantic meaning.
> Now, with the proposed undef (which seems to be confusing a lot of people),
> the Lua table:
>
>    { 1 , 2 , true , false , nil , "alpha" , "beta" }
>
> would have a defined length and semantically matches the original JSON.
>
>  -spc
>
>

If you really want a table without holes, just use false for nil in table.
Your example have BOTH false and nil, that meant you wanted the hole.

Yes, "new" syntax undef can fill holes, but it can also make holes.

All it does is shift the name for nil. Now, undef does the work of nil.

Once, this shifting names of nil started, it will never end.
Say, add undef2 so that undef can store in table, undef3 for undef2 ...

You can argue we will never add keyword undef2 to allow undef in table.
But, is that what the "new" undef does to allow nil in table ?





Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Sean Conner
In reply to this post by Pierre Chapuis
It was thus said that the Great Pierre Chapuis once stated:

> On Wed, Mar 14, 2018, at 15:55, Roberto Ierusalimschy wrote:
>
> > Every year or so the list has its recurrent discussion about the length
> > operator and holes in sequences. Quite often, this discussion leads some
> > people to propose that Lua needs an array type, different from tables.
>
> In practice, this is not the reason why people (at least me, but not only)
> wish Lua had an array type.
>
> The main reason is that Lua is a dynamic language where you can
> inspect the type of values (with type() mostly), but there is no
> standard way to know if a table is supposed to be a sequence
> or not when you get it. But in the outside world, things are not
> tables, they tend to be sequences or map. This causes issues
> when we try to interact with it using Lua.

  I had the same issue when writing my CBOR interface [1].  By default, an
empty table becomes an empty map.  I can get around this a few ways:

        --[[ explicitely call on empty sequence ]]

        a = {}
        x = cbor.TYPE.ARRAY(a)

        --[[ use metatable on sequences ]]

        a = setmetatable(
                {},
                {
                  __tocbor = function(...)
                    return cbor.TYPE.ARRAY(...)
                 end
                }
        )

        x = cbor.encode(a)

        --[[ change default behavior of mapping tables ]]

        cbor.__ENCODE_MAP.table = function(value,...)
          if next(value) == nil then
            return cbor.TYPE.ARRAY(value,...)
          elseif #value > 0 then
            return cbor.TYPE.ARRAY(value,...)
          else
            return cbor.TYPE.MAP(value,...)
          end
        end

        a = {}
        x = cbor.encode(a)

  Each has their tradeoffs.

  An issue with having a separate array type is how do you know when one is
created?  Sure, this:

        { 1 , 2 , 3 , 4 , 5 }

is an array.  This:

        { [100] = 1 , [200] = 2 , [300] = 3 , [400] = 4 , [500] = 5 }

results in a map (or associative array).  It's obviously NOT an array.  But
then this:

        { [1] = 1 , [2] = 2 , [3] = 3 , [4] = 4 , [5] = 5 }

Should that be an array?  Or a map?  And there are times when I've done
this:

        {
          alpha = 1 ,
          beta = 2 ,
          gamma = 3 ,
          delta = 4 ,
          [1] = alpha,
          [2] = beta,
          [3] = gamma ,
          [4] = delta
        }

(usually via programming, but not always).  It is useful, and I used such a
construction in my CBOR encoder/decoder:

        TAG = setmetatable(
          {
            --[[ encode a tag of __datetime ]]

            _datetime = function(value)
              return cbor_c.encode(0xC0,0) .. TYPE.TEXT(value)
            end,

            --[[ decode a tag of 0 ]]
       
            [0] = function(packet,pos,conv,ref)
              local value,npos,ctype = decode(packet,pos,conv,ref)
              if ctype == 'TEXT' then
                return value,npos,'_datetime'
              else
                throw(pos,"_datetime: wanted TEXT, got %s",ctype)
              end
            end,
        ...

so I *like* the current state of affairs.  

  -spc

[1] https://github.com/spc476/CBOR

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Andrew Gierth
In reply to this post by Russell Haley
>>>>> "Russell" == Russell Haley <[hidden email]> writes:

 >> https://reviews.freebsd.org/D13690

 Russell> The above is a link to my suggested patches to the Lua 5.3
 Russell> port for FreeBSD to build on FreeBSD 10-12 (9 should work, I
 Russell> haven't tested below 10. HEAD is 12). This also contains the
 Russell> suggested libedit patch that I came to independently.

Is there some strong reason to patch the source to reference
edit/readline/readline.h rather than just doing -I/usr/include/edit as I
did?

Also there are more bugfix patches available than you have included,
notably the one I reported for lua_pushcfunction.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Enrico Colombini
In reply to this post by Roberto Ierusalimschy
On 14-Mar-18 19:59, Roberto Ierusalimschy wrote:
> The proposed syntax would allow programs written for 5.4 to run on 5.3
> without changes (pending metamethods), and would also allow programs
> written for Lua 5.3 to be written with undef (only as an idiom), so
> that they would more easily migrate to a future version of Lua where
> nil-in-table would be the default semantics (if ever there will be such
> version).

Assuming I understood it right (which is definitely not given):

1) Wouldn't many existing programs (starting with the 'rolling table'
buffer in PiL) develop a memory leak if used in this mode, because
entries set to 'nil' will still exist?

2) Would this risk a split in Lua, with some people using the new
'undef' and others not, and with people unsure if a given program will
run correctly (see point 1) in the variant they are using? There could
be 'some' confusion.

It would a bit sad to see one of the simple, creative and useful ideas
in Lua being removed (or half-removed, or split, if you like) just
because many people try to bend programming languages to fit their
pre-existing mind scheme, rather than the other way around.

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Sean Conner
In reply to this post by Albert Chan
It was thus said that the Great albertmcchan once stated:

>
>
> On Mar 14, 2018, at 2:12 PM, Sean Conner <[hidden email]> wrote:
>
> > It was thus said that the Great albertmcchan once stated:
> >>
> >> what is so bad about the old syntax x = nil ?
> >>
> >> false were added in lua to "store" nil in table, there is no need for undef.
> >>
> >> I am curious, is there an example where undef is truly needed ?
> >
> >  Assume you have the following JSON:
> >
> >    [ 1 , 2 , true , false , nil , "alpha" , "beta" ]
> >
> >  Under the current (5.3) behavior, you can convert this to the following
> > Lua table:
> >
> >    { 1 , 2 , true , false , nil , "alpha" , "beta" }
> >
> > but the length is undefined, because the nil creates a hole.  If you use
> > false instead:
> >
> >    { 1 , 2 , true , false , false , "alpha" , "beta" }
> >
> > the length *is* defined, but it may not have the same semantic meaning.
> > Now, with the proposed undef (which seems to be confusing a lot of people),
> > the Lua table:
> >
> >    { 1 , 2 , true , false , nil , "alpha" , "beta" }
> >
> > would have a defined length and semantically matches the original JSON.
> >
> >  -spc
> >
> >
>
> If you really want a table without holes, just use false for nil in table.
> Your example have BOTH false and nil, that meant you wanted the hole.

  As I said, that might create a semantic difference when converting from
JSON to Lua.  

  CBOR defines both "Null" and "Undefined".  "Null" is not defined, but is
presumed to have the same meaning as JSON's null, or Lua's nil, or NULL from
C or similar notations.  Here is what the spec for CBOR says about
"Undefined":

        3.8.  Undefined Values
       
           In some CBOR-based protocols, the simple value (Section 2.3) of
           Undefined might be used by an encoder as a substitute for a data item
           with an encoding problem, in order to allow the rest of the enclosing
           data items to be encoded without harm.

but otherwise, it leaves "Undefined" undefined.  Now, switching gears.

  A major complaint about Lua is that nil cannot be used in a sequence.

        x = { 1 , 2 , nil , 4 , 5 }

is not a sequence (and what #x returns is undefined).  And it has
implications:

        function foo(...)
          local x = { ... }
          print(select("#",...)
          dump("x",x)
        end

        foo(1,2,nil,4,5)
        5
        x =
        {
          [1] = 1.000000,
          [2] = 2.000000,
          [4] = 4.000000,
          [5] = 5.000000,
        }

There's also:

        function bar(...)
          local x = table.pack(...)
          print(select('#",...)
          dump("x",x)
        end

        bar(1,2,nil,4,5)
        5
        x =
        {
          [1] = 1.000000,
          [2] = 2.000000,
          [4] = 4.000000,
          [5] = 5.000000,
          n = 5.000000,
        }

  Not very elegant.  

> Yes, "new" syntax undef can fill holes, but it can also make holes.
>
> All it does is shift the name for nil. Now, undef does the work of nil.

  Not really.  nil still means "no value", but it becomes possible (with
"undef") to store one in a sequence, such that:

        x = { nil , nil , nil , nil , nil }

is of length 5.  If you do:

        x[3] = undef

*THEN* you get a hole in the sequence.  undef is NOT a value, even though
the syntax makes it look like a value, because it's backwards compatible
with Lua 5.3. [2]

  -spc

[1] http://cbor.io/

[2] I don't necessarily agree with Roberto's reasoning here to prevent
        code breakage.  Any code using the bit operators of Lua 5.3 can't be
        used under Lua 5.2 and he seemed fine with that.  I'm not sure why
        he didn't do

                undef x[3]

        Why is 5.4 compatibility with 5.3 important, but not 5.3 to 5.2?

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Petri Häkkinen
In reply to this post by Roberto Ierusalimschy

On 14 Mar 2018, at 20.59, Roberto Ierusalimschy <[hidden email]> wrote:

> The proposed syntax would allow programs written for 5.4 to run on 5.3
> without changes

Is this really so important? For example, you can’t run programs with ”goto” on earlier versions of Lua that didn’t have it.

Since you’re already adding a new keyword (disallowing its use as a global var name which can break some programs), adding a new syntax for it shouldn’t hurt much more.

The fact that ”undef” looks like a value when it’s not seems to be very confusing to many people...

Petri
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Lua 5.4.0 (work1) now available

Egor Skriptunoff-2
In reply to this post by Roberto Ierusalimschy
On Wed, Mar 14, 2018 at 9:24 PM, Roberto Ierusalimschy wrote:

The problems I am trying to solve are these:

1) A constructor like {x, y, z} should always create a sequence with
three elements. A constructor like {...} should always get all arguments
passed to a function. A constructor like {f(x)} should always get
all results returned by the function. '#' should always work on these
tables correctly.

2) A statement like 't[#t + 1] = x' should always add one more element
at the end of a sequence.

...

t[i]=undef is a special syntax form that means "remove the key 'i' from table 't'"


Remark #1
At first sight, I like this idea with undef.
Syntax is a bit confusing, but it is a step towards more convenient
work with lists/tuples.
nil is not merely a substitution for missed values anymore.
nil now is a full-fledged value!
 
Question:
Once we could have nil as a valid value in tables,
could we use nil as a valid key?
t[nil]=42



 
Remark #2
At second sight, this "undef" idea has big problems with compatibility.
If I understood correctly, in Lua with NILINTABLES table keys
with nil values will NOT be GC-ed.
To release a key in a table, undef must be pseudo-assigned instead of nil.
Unfortunately, this breaks most of old Lua code.
Almost all Lua projects will need to change its internal logic
to avoid memory leakage. (Memory leakage here is about having a lot of
unneeded keys with nil values which are not available for GC)
Yes, searching for literal nil assignment will find most of places
we should change.
But there would be a lot of hard-to-find places such as
   -- func may return nil sometimes
   t[k] = func(k)
or
   -- intentionally skips nils and stores only non-nils
   table.insert(t, get_next_val())  
Rewriting old projects would be hard and error-prone.
We all have a lot of old code we don't remember in details how it works.
The price of rewriting Lua 5.3 code to Lua 5.4 will be too high.


 
Remark #3
Probably, metatable-based approach would be convenient?
We could define special field "__undef" in table's metatable for storing
a value which is used to delete table entry from this table.
If there is no "__undef" metafield, then the value nil is used to delete
table key (this behavior is compatible with previous Lua versions).
But in Lua 5.4 we could store nils by assigning non-nil value to "__undef":
   t = setmetatable({}, {__undef = 'A'})
   t[1] = nil   -- adding key 1 with value nil
   assert(#t == 1 and t[1] == nil)
   t[1] = 'A'   -- deleting key 1 from the table
   assert(#t == 0 and t[1] == nil)
But what if user needs string 'A' in this table?
To avoid accidental collision, we could generate unique __undef value:
   t = setmetatable({}, {__undef = {}})
To delete a key:
   function table.delete(t, key)
      t[key] = (getmetatable(t) or {}).__undef
   end
__undef metafield also affects #/next/pairs/ipairs:
iterating must skip all pairs (key, value) with value equals to __undef.
 
The interesting question is about table constructors:
   t1 = {1, 2, nil, 4, a=nil, b=nil}
   t2 = { func() }
Table constructors are applied at the time when a table doesn't have
a metatable yet, so table constructor must store all values it has
(including nils).

   t = {a=nil}
   setmetatable(t, {__undef = t}) 
   -- the following line will print "a", nil
   for k, v in pairs(t) do print(k, v) end  

   t = {nil}      -- table actually has key 1 with nil value
   setmetatable(t, {__undef = t}) 
   print(#t)  -- prints 1

The backward compatibility is guaranteed due to next/pairs/ipairs
respect __undef metafield.

   t = {a=nil}    -- table actually has key "a" with nil value
   -- the following line prints nothing due to next() skips
   -- values equals to __undef (__undef==nil)
   for k, v in pairs(t) do print(k, v) end  

   t = {nil}      -- table actually has key 1 with nil value
   print(#t)  -- prints 0
   -- the following line prints nothing due to ipairs() returns
   -- an iterator which skips values equals to __undef (__undef==nil)
   for k, v in ipairs(t) do print(k, v) end 
 
As you can see, this approach allows storing nils in tables
without syntax compatibility problems and without new keywords.
Both Roberto's problems (see quoted text) are solved
(for problem #2 user must prepare metafield prior to assigning nil).

You can prepare simple global metatable:
   _NILINSIDE = { __undef = {} }
and use it for every table which you want to insert nils into:
   local function f() return nil, nil end
   local t = setmetatable( { f() }, _NILINSIDE )
   t[#t+1] = nil
   assert(#t == 3)

Assigning a metatable is verbose, but only some tables need this feature.
 
More details:
1) __undef metafield is compared by rawequal().
2) You can change __undef metafield on-the-fly, it may (or may not,
depending on the implementation) delete existing keys which have
value equals to __undef.
3) While iterating with ipairs(), depending on implementation,
__undef may be read once at the moment of ipairs invocation
or may be reread on every iteration.
4) next() may (or may not, depending on the implementation)
delete skipped table items from the table
(that is, keys having value equals to the current __undef metafield)
5) Iterator returned by ipairs() may (or may not, depending
on the implementation) delete skipped table items from the table
 

1234567 ... 9