new "empty" value/type in Lua?

classic Classic list List threaded Threaded
102 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

new "empty" value/type in Lua?

Tim Hill
I'm going to get shot for suggesting this… (hides under bed)

I've been writing a lot of Lua code recently, some of it quite generic in nature. One thing I frequently come up against is the need to handle data polymorphically in tables. And an effect of this is to hit against the overloaded use of "nil" in Lua.

It seems to me that "nil" is overloaded; it means both a noun: "no value" and a verb: "delete this table entry". Now, most of the time, the delete is hidden since reading a non-existent table entry returns nil. So having an explicitly stored nil versus having no value stored at all is equivalent (all this I love, btw … 100 points to Lua).

But it all rather breaks down with arrays (sequences). I cannot have sparse arrays, which in Lua terms means I cannot have "nil" in an array element. So, when writing polymorphic code, how do I handle cases (which, trust me, i DO have to handle, with "no value" in an array element? There are, of course, a whole bunch of work-arounds: Use a sentinel value; use an empty table (assuming empty tables aren't value data, and ignoring that each one is unique); use a magic userdata (assuming I have access to C code to make one); etc etc.

But, let's face it; they ARE all workarounds. I think there IS a case to be made for being able to store a value in a Lua array that is uniquely NOT any other Lua value. Essentially, I think that value is "empty"; a valid Lua value that is simply not the same as any other Lua value (including nil).

Basically, empty would act like nil in most regards, except:
-- Storing empty in a table element actually stores the value, rather than deletes it.
-- When used in a boolean context, the empty value evaluates to true (unlike nil).
-- Has a type() of "empty"

Other arguments in favor of this:
-- Judging by the mail list chatter, a lot of Lua beginners struggle with the problem of nil in sequences.

-- Better impedance matching to SQL NULL values; since empty can bridge to a SQL NULL when a Lua array is used to hold a SQL row.

Thoughts anyone?
--Tim


Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Rena

On 2013-06-27 8:57 PM, "Tim Hill" <[hidden email]> wrote:
>
> I'm going to get shot for suggesting this… (hides under bed)
>
> I've been writing a lot of Lua code recently, some of it quite generic in nature. One thing I frequently come up against is the need to handle data polymorphically in tables. And an effect of this is to hit against the overloaded use of "nil" in Lua.
>
> It seems to me that "nil" is overloaded; it means both a noun: "no value" and a verb: "delete this table entry". Now, most of the time, the delete is hidden since reading a non-existent table entry returns nil. So having an explicitly stored nil versus having no value stored at all is equivalent (all this I love, btw … 100 points to Lua).
>
> But it all rather breaks down with arrays (sequences). I cannot have sparse arrays, which in Lua terms means I cannot have "nil" in an array element. So, when writing polymorphic code, how do I handle cases (which, trust me, i DO have to handle, with "no value" in an array element? There are, of course, a whole bunch of work-arounds: Use a sentinel value; use an empty table (assuming empty tables aren't value data, and ignoring that each one is unique); use a magic userdata (assuming I have access to C code to make one); etc etc.
>
> But, let's face it; they ARE all workarounds. I think there IS a case to be made for being able to store a value in a Lua array that is uniquely NOT any other Lua value. Essentially, I think that value is "empty"; a valid Lua value that is simply not the same as any other Lua value (including nil).
>
> Basically, empty would act like nil in most regards, except:
> -- Storing empty in a table element actually stores the value, rather than deletes it.
> -- When used in a boolean context, the empty value evaluates to true (unlike nil).
> -- Has a type() of "empty"
>
> Other arguments in favor of this:
> -- Judging by the mail list chatter, a lot of Lua beginners struggle with the problem of nil in sequences.
>
> -- Better impedance matching to SQL NULL values; since empty can bridge to a SQL NULL when a Lua array is used to hold a SQL row.
>
> Thoughts anyone?
> --Tim
>
>

I'm not really sure what the difference would be between storing a sentinel table reference and storing "empty". It actually occupies the table entry and it evaluates to true, so it meets the first two differences you mentioned. With some metatable magic you could give it a type of "empty" too if you really wanted.

The fact that empty tables are unique helps with this method, because you can define an "empty" variable and use it just as you would if it were a special, built-in value (like nil), while not affecting the ability to store other, unrelated empty tables (e.g. a list that just happens to be empty at the moment). You just need to take care not to modify it or shadow it.

The one significant advantage I can see is that a database wrapper can automatically convert this magic value to and from null. Of course this can still be done if "empty" is an ordinary variable rather than a keyword; it's just a little more difficult because you need to make sure the database module is using the same "empty" as the rest of the code.

I can see how it'd be helpful if the standard Lua libraries defined a global "empty" variable to help ensure every module is using the same value, and if you're paranoid (or debugging, perhaps) it could even use a metatable on _G to prevent that variable being modified (although that wouldn't prevent a "local empty = {}" from causing trouble), but I don't think a whole new type is needed.

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Andrew Starks
On Thu, Jun 27, 2013 at 8:28 PM, Rena <[hidden email]> wrote:

> On 2013-06-27 8:57 PM, "Tim Hill" <[hidden email]> wrote:
>>
>> I'm going to get shot for suggesting this… (hides under bed)
>>
>> I've been writing a lot of Lua code recently, some of it quite generic in
>> nature. One thing I frequently come up against is the need to handle data
>> polymorphically in tables. And an effect of this is to hit against the
>> overloaded use of "nil" in Lua.
>>
>> It seems to me that "nil" is overloaded; it means both a noun: "no value"
>> and a verb: "delete this table entry". Now, most of the time, the delete is
>> hidden since reading a non-existent table entry returns nil. So having an
>> explicitly stored nil versus having no value stored at all is equivalent
>> (all this I love, btw … 100 points to Lua).
>>
>> But it all rather breaks down with arrays (sequences). I cannot have
>> sparse arrays, which in Lua terms means I cannot have "nil" in an array
>> element. So, when writing polymorphic code, how do I handle cases (which,
>> trust me, i DO have to handle, with "no value" in an array element? There
>> are, of course, a whole bunch of work-arounds: Use a sentinel value; use an
>> empty table (assuming empty tables aren't value data, and ignoring that each
>> one is unique); use a magic userdata (assuming I have access to C code to
>> make one); etc etc.
>>
>> But, let's face it; they ARE all workarounds. I think there IS a case to
>> be made for being able to store a value in a Lua array that is uniquely NOT
>> any other Lua value. Essentially, I think that value is "empty"; a valid Lua
>> value that is simply not the same as any other Lua value (including nil).
>>
>> Basically, empty would act like nil in most regards, except:
>> -- Storing empty in a table element actually stores the value, rather than
>> deletes it.
>> -- When used in a boolean context, the empty value evaluates to true
>> (unlike nil).
>> -- Has a type() of "empty"
>>
>> Other arguments in favor of this:
>> -- Judging by the mail list chatter, a lot of Lua beginners struggle with
>> the problem of nil in sequences.
>>
>> -- Better impedance matching to SQL NULL values; since empty can bridge to
>> a SQL NULL when a Lua array is used to hold a SQL row.
>>
>> Thoughts anyone?
>> --Tim
>>
>>

I think that you can have sparse arrays.

As I'm sure you're aware, you can modify __ipairs  and __len in any
way that you wish. So a metatable that marks the holes or provides the
length can be used in your version and then BAM! Sparse arrays. Heck,
you don't even need any storage. Just use make an __ipairs function
where the factory uses pairs to find the max int, picks the biggest
index and then poops out an iterator function that keeps going until
math.max is reached.


The implementation doesn't matter. I've posted emails just like yours
and like you, lately I've been doing more writing in Lua than writing
about it. The more that I use it, the more that I've come to
appreciate that it just isn't made to be the most convenient language
in the world. You end up doing quite a bit of typing, or you can use
its mechanisms to get what you want, ala how Penlight does it, which
probably has a sparse array module....

So, methods, not policies. Those work arounds that you've mentioned
are _how_ you do it, in Lua. That is the way. No need for yet another
"thing."

As an example, in my little library that I've built up for my project:
type looks at __type, error and assert use LuaLogger as well as
string.format, dogs hang out with cats, math has is_int, table is
overloaded with Penlight's tablex, coroutines can be made with tables
that have a __coroutine metatable field, and all manner of other
fluffery abounds. It's all good because I put it there and that's how
you're supposed to do it, if that's how you choose to roll... at least
that's what I've come to understand.

-Andrew

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Tim Hill
Good points from all, but I don't think I can agree.

I think the point here is that there are different degrees/scopes to standardization:
1. At the bottom, there is the "per project" conventions. The work-arounds I mentioned, as well as Andrews's, fall into this bucket. Yes, a single empty table could act as a flag WITHIN a single Lua project or program, but...

2. Next, you have informal conventions that are often embodied in "best practices" or accepted ways to do things. Much of the Lua module system works like this.

3. Next you have the standard runtime/library.

4. Finally, you have the enforced conventions of the language itself.

To my mind you try to stay as far "up" that stack as you can. So why am I arguing for something all the way down at #4? Because at every other level any solution is scoped wrongly. Yes, for a given project I can use an empty table as a special placeholder, or (as per the other suggestion) I can use metatables to "hack" sparse arrays (though this is harder than it might at first appear, at least to do efficiently).

But what if (as I am currently doing), I am coding a general purpose C library that has to handle polymorphic arrays? How do I project "empty" behavior "up" into Lua? I *could* do it with a variety of more or less elegant hacks. Then someone else comes along with the same problem, and they solve it in a slightly different way. Then a third … and so on. Pretty soon you have a big mess on your hands. This is the classic balancing act: too many standards and everyone is handcuffed, too few and you create chaos.

You see, when you move down that stack above, you are going from a single local project to a broader and broader scope, until at #4 you reach everyone. At the top, hacks are acceptable (as was stated in a reply). But the further down you go, the cleaner you need to be. Some of the stuff I'm working on is hovering between #2 and #3, and I for one am uneasy of level #1 hacks drifting down that stack too far. Hence my desire for a much less hack-y way of doing things.

To take a canonical example, what is a clean way to surface a database result set that might include NULLs? In a standard library that many people are likely to use? Yes, there are a dozen ways, but I don't regard any of them as clean, and when I analyzed the problem, it really came down to this lack of a storable empty value and the overloading of nil.

And, to be honest, I *do* think Lua lacks the concept of empty; the current Lua documentation has to kind of wriggle a bit when explaining arrays precisely because you cannot have an empty array element. Even after 3 years of working with Lua I still find this one of the "oh really?" bits.

--Tim




On Jun 27, 2013, at 8:40 PM, Andrew Starks <[hidden email]> wrote:

> On Thu, Jun 27, 2013 at 8:28 PM, Rena <[hidden email]> wrote:
>> On 2013-06-27 8:57 PM, "Tim Hill" <[hidden email]> wrote:
>>>
>>> I'm going to get shot for suggesting this… (hides under bed)
>>>
>>> I've been writing a lot of Lua code recently, some of it quite generic in
>>> nature. One thing I frequently come up against is the need to handle data
>>> polymorphically in tables. And an effect of this is to hit against the
>>> overloaded use of "nil" in Lua.
>>>
>>> It seems to me that "nil" is overloaded; it means both a noun: "no value"
>>> and a verb: "delete this table entry". Now, most of the time, the delete is
>>> hidden since reading a non-existent table entry returns nil. So having an
>>> explicitly stored nil versus having no value stored at all is equivalent
>>> (all this I love, btw … 100 points to Lua).
>>>
>>> But it all rather breaks down with arrays (sequences). I cannot have
>>> sparse arrays, which in Lua terms means I cannot have "nil" in an array
>>> element. So, when writing polymorphic code, how do I handle cases (which,
>>> trust me, i DO have to handle, with "no value" in an array element? There
>>> are, of course, a whole bunch of work-arounds: Use a sentinel value; use an
>>> empty table (assuming empty tables aren't value data, and ignoring that each
>>> one is unique); use a magic userdata (assuming I have access to C code to
>>> make one); etc etc.
>>>
>>> But, let's face it; they ARE all workarounds. I think there IS a case to
>>> be made for being able to store a value in a Lua array that is uniquely NOT
>>> any other Lua value. Essentially, I think that value is "empty"; a valid Lua
>>> value that is simply not the same as any other Lua value (including nil).
>>>
>>> Basically, empty would act like nil in most regards, except:
>>> -- Storing empty in a table element actually stores the value, rather than
>>> deletes it.
>>> -- When used in a boolean context, the empty value evaluates to true
>>> (unlike nil).
>>> -- Has a type() of "empty"
>>>
>>> Other arguments in favor of this:
>>> -- Judging by the mail list chatter, a lot of Lua beginners struggle with
>>> the problem of nil in sequences.
>>>
>>> -- Better impedance matching to SQL NULL values; since empty can bridge to
>>> a SQL NULL when a Lua array is used to hold a SQL row.
>>>
>>> Thoughts anyone?
>>> --Tim
>>>
>>>
>
> I think that you can have sparse arrays.
>
> As I'm sure you're aware, you can modify __ipairs  and __len in any
> way that you wish. So a metatable that marks the holes or provides the
> length can be used in your version and then BAM! Sparse arrays. Heck,
> you don't even need any storage. Just use make an __ipairs function
> where the factory uses pairs to find the max int, picks the biggest
> index and then poops out an iterator function that keeps going until
> math.max is reached.
>
>
> The implementation doesn't matter. I've posted emails just like yours
> and like you, lately I've been doing more writing in Lua than writing
> about it. The more that I use it, the more that I've come to
> appreciate that it just isn't made to be the most convenient language
> in the world. You end up doing quite a bit of typing, or you can use
> its mechanisms to get what you want, ala how Penlight does it, which
> probably has a sparse array module....
>
> So, methods, not policies. Those work arounds that you've mentioned
> are _how_ you do it, in Lua. That is the way. No need for yet another
> "thing."
>
> As an example, in my little library that I've built up for my project:
> type looks at __type, error and assert use LuaLogger as well as
> string.format, dogs hang out with cats, math has is_int, table is
> overloaded with Penlight's tablex, coroutines can be made with tables
> that have a __coroutine metatable field, and all manner of other
> fluffery abounds. It's all good because I put it there and that's how
> you're supposed to do it, if that's how you choose to roll... at least
> that's what I've come to understand.
>
> -Andrew
>


Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Tim Mensch
In reply to this post by Andrew Starks
On 6/27/2013 9:40 PM, Andrew Starks wrote:
> As I'm sure you're aware, you can modify __ipairs and __len in any way
> that you wish.

Unless you're using Lua 5.1 or LuaJIT (without the LJ_52 compatibility
preprocessor flag).

Tim

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Jorge Visca
On 28/06/13 01:42, Tim Mensch wrote:
> On 6/27/2013 9:40 PM, Andrew Starks wrote:
>> As I'm sure you're aware, you can modify __ipairs and __len in any
>> way that you wish.
>
> Unless you're using Lua 5.1 or LuaJIT (without the LJ_52 compatibility
> preprocessor flag).

As the original post proposes a change, it's a moot point  :)

Jorge

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

steve donovan
In reply to this post by Tim Hill
On Fri, Jun 28, 2013 at 2:57 AM, Tim Hill <[hidden email]> wrote:
I'm going to get shot for suggesting this… (hides under bed)

Nah, it's a perenial issue.  But it always ends up complicating the language, which we don't tend to like...

I've been writing a lot of Lua code recently, some of it quite generic in nature. One thing I frequently come up against is the need to handle data polymorphically in tables. And an effect of this is to hit against the overloaded use of "nil" in Lua.

OK, now if you're storing objects, why not the old standby `false`?  It strikes me as unusual to have containers of genuinely unrelated values - generally I like to put Cats and Dogs in separate tables, unless they do share some common behaviour. For tables, strings and userdata, `false` has the right semantics - because it's false.

for _,o in ipairs(list) do
  if o then
    -- do something!
  end
end

There's an issue of course if you're found of standard map and filter operations.

steve d.

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Dirk Laurie-2
In reply to this post by Tim Hill
2013/6/28 Tim Hill <[hidden email]>:

> I'm going to get shot for suggesting this… (hides under bed)

No, you won't. We've all been through this phase and survived it.

> I think there IS a case to be made for being able to store
> a value in a Lua array that is uniquely NOT any other Lua value.
> Essentially, I think that value is "empty"; a valid Lua value
> that is simply not the same as any other Lua value (including nil).
>
> Basically, empty would act like nil in most regards, except:
> -- Storing empty in a table element actually stores the value,
>    rather than deletes it.
> -- When used in a boolean context, the empty value evaluates
>    to true (unlike nil).
> -- Has a type() of "empty"

Would `empty` be allowed as a table key?

> -- Judging by the mail list chatter, a lot of Lua beginners
>    struggle with the problem of nil in sequences.

It would be so much nicer to have them struggle with `empty`
instead.

> But, let's face it; they ARE all workarounds.

Those workarounds are not workarounds, they are special-purpose
solutions to special-purpose problems.

Here's another "workaround":

   NaN = 0/0

It's different from all Lua values, including `nil` _and itself_.
It occupies a TValue, but is not a valid table key.

The proposed `empty` is likewise a mere workaround. Another
special-purpose solution to another special-purpose problem.

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Rena
In reply to this post by Tim Hill

On Jun 28, 2013 12:35 AM, "Tim Hill" <[hidden email]> wrote:
> To take a canonical example, what is a clean way to surface a database result set that might include NULLs? In a standard library that many people are likely to use? Yes, there are a dozen ways, but I don't regard any of them as clean, and when I analyzed the problem, it really came down to this lack of a storable empty value and the overloading of nil.

Personally I haven't yet encountered a case where I needed a null value in a database to correspond to anything other than nil in a table. I suppose that's because I generally abstract each table (or sometimes sets of tables) to functions like:

function add_foo(foo)
  db:query("INSERT INTO foos (name, type, value) VALUES (?, ?, ?)",
    foo.name, foo.type, foo.value)
end

in which db:query() uses its vararg params to form a parameterized query; a nil argument gets translated to NULL automatically. Similarly get_foo() would look up a row by some criteria and return it (or perhaps a list of rows) with fields like {name='a', type='t', value=7}; if one of those fields were null in the database, it would just be missing from the returned row, and when the application reads that field from the table, it gets nil just as you'd expect.

There are probably cases where this won't do, but I usually feel like if the best solution to a problem involves modifying Lua itself, what I really need to do is rethink the design so that the problem no longer exists, as it's usually reflecting a fundamental weakness in the design itself and not in the language.

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Michal Kolodziejczyk-3
In reply to this post by Tim Hill
On 28.06.2013 02:57, Tim Hill wrote:

> But, let's face it; they ARE all workarounds. I think there IS a case to be made for being able to store a value in a Lua array that is uniquely NOT any other Lua value. Essentially, I think that value is "empty"; a valid Lua value that is simply not the same as any other Lua value (including nil).

So what is wrong with it? Lua is small and flexible enough that you can
- if you need -  add a few lines of code and then forget that there is
any workaround. But others, who don't need that (because they learned to
live with current implementation), can continue using original (read:
small and fast, so portable, not bloated with every feature added at the
language level) implementation.

> Basically, empty would act like nil in most regards, except:
> -- Storing empty in a table element actually stores the value, rather than deletes it.
> -- When used in a boolean context, the empty value evaluates to true (unlike nil).
> -- Has a type() of "empty"

That code should do it:

_EMPTYMT={__tostring=function() return '<empty>' end}
empty=setmetatable({}, _EMPTYMT)
type=function(o)
  if getmetatable(o)==_EMPTYMT then
    return 'empty'
  else
    return type(o)
  end
end

And then some tests:

print(empty)
assert(1~=empty)
assert(nil~=empty)
assert(true~=empty)
assert(false~=empty)
if empty then print('empty is true') end
t={1,empty,3}
for k,v in ipairs(t) do print(k, v) end
t2={[1]=1, [empty]=empty, [3]=3}
for k,v in pairs(t2) do print(k, v) end
print('Type:', type(empty))

Regards,
        miko


Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Michal Kottman-2
On 28 June 2013 10:03, Michal Kolodziejczyk <[hidden email]> wrote:
type=function(o)
  if getmetatable(o)==_EMPTYMT then
    return 'empty'
  else
    return type(o)
  end
end

This of course would have to be:

 do
  local oldtype=type
  type=function(o)
    if getmetatable(o)==_EMPTYMT then
      return 'empty'
    else
      return oldtype(o)
    end
  end
end

Otherwise you end up in an infinite tail-call loop :)
Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Tim Hill
Thanks for all the suggestions, but again I think you are missing the point. It's not HOW to work around the issue for an application, it's if it's GOOD or BAD that I SHOULD have to work around it. I can think of (easily) a dozen ways to handle this. But I stand by my point; it's an oddity of the language that I cannot use nil to mark an array entry as empty, and my suggestion was to fix that in a clean simple manner. Nothing more, nothing less.

As for the "Lua is clean and tight, don't extend it" .. yes I agree, one of the best aspects of the language is the tight, clean, elegant design. But if EVERY suggestion is met with "we can't add that because Lua is small" then NOTHING will ever change and Lua is complete NOW. Which is clearly not the case.

No-one has answered my original question: How can I write a CLEAN C library that emits a SQL rowset that might include NULL values? Without "empty", I cannot see a clean way to do this. And, to be honest, I don't accept the "everything is a workaround" argument, since if thats the case JavaScript or APL or FORTRAN for that matter (shudder) are all as good as Lua.

--Tim


On Jun 28, 2013, at 1:36 AM, Michal Kottman <[hidden email]> wrote:

On 28 June 2013 10:03, Michal Kolodziejczyk <[hidden email]> wrote:
type=function(o)
  if getmetatable(o)==_EMPTYMT then
    return 'empty'
  else
    return type(o)
  end
end

This of course would have to be:

 do
  local oldtype=type
  type=function(o)
    if getmetatable(o)==_EMPTYMT then
      return 'empty'
    else
      return oldtype(o)
    end
  end
end

Otherwise you end up in an infinite tail-call loop :)

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

steve donovan
On Fri, Jun 28, 2013 at 11:17 AM, Tim Hill <[hidden email]> wrote:
No-one has answered my original question: How can I write a CLEAN C library that emits a SQL rowset that might include NULL values? Without "empty", I cannot see a clean way to do this.

But then your library would define a null constant, and you would use that. That's how the same problem is sorted with JSON encoding.

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Philipp Janda
In reply to this post by Tim Hill
Hi!

Am 28.06.2013 06:35 schröbte Tim Hill:
> Good points from all, but I don't think I can agree.
>
> I think the point here is that there are different degrees/scopes to standardization:
> 1. At the bottom, there is the "per project" conventions. The work-arounds I mentioned, as well as Andrews's, fall into this bucket. Yes, a single empty table could act as a flag WITHIN a single Lua project or program, but...
>
> 2. Next, you have informal conventions that are often embodied in "best practices" or accepted ways to do things. Much of the Lua module system works like this.
>
> 3. Next you have the standard runtime/library.

There already is a level-3 solution to your problem: Have a look at
`table.pack` which supports arrays containing nils.

>
> 4. Finally, you have the enforced conventions of the language itself.
>
> To my mind you try to stay as far "up" that stack as you can. So why
> am I arguing for something all the way down at #4? Because at every
> other level any solution is scoped wrongly. Yes, for a given project I
> can use an empty table as a special placeholder, or (as per the other
> suggestion) I can use metatables to "hack" sparse arrays (though this is
> harder than it might at first appear, at least to do efficiently).

sparse.lua:
-----------8<-------------

     local setmetatable = setmetatable

     local function ipairs_iter( state, var )
       var = var + 1
       if var <= state.n then
         return var, state[ var ]
       end
     end

     local sparse_meta = {
       __len = function( self )
         return self.n
       end,
       __ipairs = function( self )
         return ipairs_iter, self, 0
       end,
     }

     return function( n )
       return setmetatable( { n = n }, sparse_meta )
     end

-----------8<-------------

Did I miss something?

In your case this probably is unnecessary anyway, because the user
usually knows how many values to expect from a database operation --
he/she has written the SQL statement after all ...

>
> You see, when you move down that stack above, you are going from a
> single local project to a broader and broader scope, until at #4 you
> reach everyone.

Why do you want to reach everyone? So far the "empty" type is only
useful under very special circumstances (to map the C/SQL NULL value to
Lua), and you don't need a new type for that.

Then there are other implications to consider:
Should `local x` contain nil or empty? Should `table.pack( nil, nil )`
translate the nils into empties or not? What about `{ ... }`? Should
`table.unpack( t )` translate empties into nils? Should accessing an
empty field invoke __(new)index metamethods? `t[ 1 ] or 0` won't work
anymore. You would need two conditions to check for valid table
elements. There will be sparse arrays where you *don't* want to use
empty anyway, e.g.: `{ n=1000, [499]=1, [500]=2, [501]=1 }` ...

>
> To take a canonical example, what is a clean way to surface a
> databaseresult set that might include NULLs? In a standard library that many
> people are likely to use? Yes, there are a dozen ways, but I don't
> regard any of them as clean, and when I analyzed the problem, it really
> came down to this lack of a storable empty value and the overloading of nil.

Use either one of `nil`, lightuserdata NULL, or some other special
sentinel value. The question is: How would your "empty" type help?

empty type:
     local res = sql:getrow()
     local id,name = table.unpack( res )   -- or res[ 1 ], res[ 2 ]
     if name ~= empty then ... end

nil (without sparse array metatable):
     local res = sql:getrow()
     local id,name = table.unpack( res, 1, 2 )
     if name ~= nil then ... end

nil (with sparse array metatable):
     local res = sql:getrow()
     local id,name = table.unpack( res )
     if name ~= nil then ... end

some sentinel value:
     local res = sql:getrow()
     local id,name = table.unpack( res )
     if name ~= sql.NULL then ... end   -- or name ~= sql.empty

generic:
     local res = sql:getrow()
     local id,name = table.unpack( res, 1, 2 )
     if not sql.is_null( name ) then ... end

>
> And, to be honest, I *do* think Lua lacks the concept of empty; the
> current Lua documentation has to kind of wriggle a bit when explaining
> arrays precisely because you cannot have an empty array element.

You cannot have an empty array element in C. In Lua you *can* have empty
array elements, but Lua cannot automatically keep track of the array
size for you if you do.

>
> --Tim

Philipp



Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Jorge Visca
In reply to this post by steve donovan
On 06/28/2013 06:31 AM, steve donovan wrote:

> On Fri, Jun 28, 2013 at 11:17 AM, Tim Hill <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     No-one has answered my original question: How can I write a CLEAN C
>     library that emits a SQL rowset that might include NULL values?
>     Without "empty", I cannot see a clean way to do this.
>
>
> But then your library would define a null constant, and you would use
> that. That's how the same problem is sorted with JSON encoding.
>


I agree: I don't see why a NULL representation has to have a meaning
outside the SQL library, or to "have a well known" semanthics to be
respetcted by every bit of lua codebase. A SQL NULL is just a SQL concept.

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Tim Hill
In reply to this post by Philipp Janda

On Jun 28, 2013, at 2:42 AM, Philipp Janda <[hidden email]> wrote:

There already is a level-3 solution to your problem: Have a look at `table.pack` which supports arrays containing nils.

Table.pack() creates a Lua table that (as the docs state) is *not* an array if the list contains nil values.

In your case this probably is unnecessary anyway, because the user usually knows how many values to expect from a database operation -- he/she has written the SQL statement after all …

No, this is not true. This is a generic library that must pack SQL results .. it has no inherent knowledge of the schema or intent of the user (and nor should it). In such a library it is FAR better than it does NOT make assumptions about the statement or schema.

Then there are other implications to consider:
Should `local x` contain nil or empty? Should `table.pack( nil, nil )` translate the nils into empties or not? What about `{ ... }`? Should `table.unpack( t )` translate empties into nils? Should accessing an empty field invoke __(new)index metamethods? `t[ 1 ] or 0` won't work anymore. You would need two conditions to check for valid table elements. There will be sparse arrays where you *don't* want to use empty anyway, e.g.: `{ n=1000, [499]=1, [500]=2, [501]=1 }` ...

I don't see any of these as very hard to answer:
-- "local x" should initialize to nil as it currently does; I see no reason to change the basic language semantics.
-- table.pack(<anything>) should behave as expected, so table.pack(nil, nil) yields {nil, nil} while table.pack(empty, empty) yields {empty, empty}
-- Same for {…}
-- table.unpacl() should not translate nil to empty

"empty" has no magic (unlike nil), for example to my mind "empty" when used in a boolean context should return true like any other value except false/nil. "empty" is just a value that is totally ordinary EXCEPT it's NOT equal to any other Lua value except itself. So of course it doesn't modify __newindex() in any special way. In fact, what you have highlighted is just how special "nil" is in Lua, and how it is overloaded in odd ways. For example: "x = nil" has different behaviors for globals and locals. Yes, I completely understand why, but the fact is it's different.

And again I'll point out that Lua arrays DO behave oddly when faced with nil. I have no problems with this, I've been developing for so long in so many languages I'm used to god knows how many quirks, but quirks they remain. Stop and think about it:

a = {1,2,3} -- it's an array
a[4] = "hello" -- still an array
a[2] = 1000 -- STILL an array
a[4] = nil -- STILL an array
a[2] = nil -- oops .. not an array any more
a[5] = 99 -- not an array
a[2] = 1 -- Hey! .. I'm an array again

For a junior developer, it's even harder to see when hidden behind locals:
local x, y = 10
a = {1,2,3} -- It's an array
a[2] = y -- Not an array any more
a[2] = x -- Wait .. I'm an array again

Imagine a Java/C++/C# collection behaving like that. I'd be scratching my head a bit at such a thing. (OK, so Java/C++/C# collections have a LOT of other problems, but two wrongs don't make a right! <grin>).

There are plenty of cases where being able to "mark" an array entry as done/dead/empty etc is useful as part of an algorithm. In many cases you can do that using a sentinel value that is outside the normal range of expected array types and/or values, but it's all very domain specific (can I use an empty string? -1? false? auxiliary state?). I'm basically arguing that "empty" allows you to do this in a nice, clean, self-documenting, portable manner. I don't really see anything heretical in that. To be honest, the most heretical thing is that you would introduce a new keyword, and THAT can of course break existing code.

--Tim

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Dirk Laurie-2
In reply to this post by Tim Hill
2013/6/28 Tim Hill <[hidden email]>:

> But I stand by my point; it's an oddity of the language that I cannot
> use nil to mark an array entry as empty, and my suggestion was to fix
> that in a clean simple manner. Nothing more, nothing less.

It's a feature of the language, call it an oddity if you prefer, that
unused array items quite likely do not occupy any actual storage. The
Lua `nil` fits in very nicely with that. It is not intended to mark an
array entry as empty. The ways to do so are domain-specific because the
necessity to so is domain-specific.

I ask again: should `empty` be a valid table key?

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Vadi
In reply to this post by Jorge Visca
It's a concept in JSON as well. Hard fact learnt from serializing t = {[13435] = "hi"}; yajl.to_string(t)
Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Enrico Colombini
In reply to this post by Tim Hill
On 28/06/2013 13.03, Tim Hill wrote:
> There are plenty of cases where being able to "mark" an array entry
> as done/dead/empty etc is useful as part of an algorithm. In many
> cases you can do that using a sentinel value that is outside the
> normal range of expected array types and/or values, but it's all very
> domain specific (can I use an empty string? -1? false? auxiliary
> state?). I'm basically arguing that "empty" allows you to do this in
> a nice, clean, self-documenting, portable manner. I don't really see
> anything heretical in that.

empty = {}

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: new "empty" value/type in Lua?

Philipp Janda
In reply to this post by Tim Hill
Am 28.06.2013 13:03 schröbte Tim Hill:
>
> On Jun 28, 2013, at 2:42 AM, Philipp Janda <[hidden email]> wrote:
>
>> There already is a level-3 solution to your problem: Have a look at `table.pack` which supports arrays containing nils.
>
> Table.pack() creates a Lua table that (as the docs state) is *not* an array if the list contains nil values.

Then my suggestion is: Don't use arrays for your SQL result rows! Use
tables instead. `table.pack()` shows how you can replace an array with a
table and still support array-like operations.

>
>> In your case this probably is unnecessary anyway, because the user usually knows how many values to expect from a database operation -- he/she has written the SQL statement after all …
>
> No, this is not true. This is a generic library that must pack SQL results .. it has no inherent knowledge of the schema or intent of the user (and nor should it). In such a library it is FAR better than it does NOT make assumptions about the statement or schema.
>
>> Then there are other implications to consider:
>> Should `local x` contain nil or empty? Should `table.pack( nil, nil )` translate the nils into empties or not? What about `{ ... }`? Should `table.unpack( t )` translate empties into nils? Should accessing an empty field invoke __(new)index metamethods? `t[ 1 ] or 0` won't work anymore. You would need two conditions to check for valid table elements. There will be sparse arrays where you *don't* want to use empty anyway, e.g.: `{ n=1000, [499]=1, [500]=2, [501]=1 }` ...
>
> I don't see any of these as very hard to answer:
> -- "local x" should initialize to nil as it currently does; I see no reason to change the basic language semantics.
> -- table.pack(<anything>) should behave as expected, so table.pack(nil, nil) yields {nil, nil} while table.pack(empty, empty) yields {empty, empty}
> -- Same for {…}
> -- table.unpack() should not translate nil to empty

In that case you just add another new table type:
   1) general table (possibly with holes)
   2) proper sequences (no holes, integer keys 1...n)
   and additionally
   3) proper sequences with empty values.

The array-like tables created by `table.pack` and `{...}` would still be
there, so you don't reduce complexity, you just add ...

>
> "empty" has no magic (unlike nil), for example to my mind "empty"
> when  used in a boolean context should return true like any other value except
> false/nil. "empty" is just a value that is totally ordinary EXCEPT it's
> NOT equal to any other Lua value except itself.

We already have that: {}.
And the bonus is: You can have as many different "empty" values as you
want (one for NULL, one for NaN, etc.).

> So of course it doesn't
> modify __newindex() in any special way. In fact, what you have
> highlighted is just how special "nil" is in Lua, and how it is
> overloaded in odd ways. For example: "x = nil" has different behaviors
> for globals and locals. Yes, I completely understand why, but the fact
> is it's different.
>
> And again I'll point out that Lua arrays DO behave oddly when faced with nil. I have no problems with this, I've been developing for so long in so many languages I'm used to god knows how many quirks, but quirks they remain. Stop and think about it:
>
> a = {1,2,3} -- it's an array
> a[4] = "hello" -- still an array
> a[2] = 1000 -- STILL an array
> a[4] = nil -- STILL an array
> a[2] = nil -- oops .. not an array any more
> a[5] = 99 -- not an array
> a[2] = 1 -- Hey! .. I'm an array again
>
> For a junior developer, it's even harder to see when hidden behind locals:
> local x, y = 10
> a = {1,2,3} -- It's an array
> a[2] = y -- Not an array any more
> a[2] = x -- Wait .. I'm an array again

Why do you care if it's an "array"? No matter what, you can access the
values for the integer keys from 1 to n. If nil is not allowed as a
value, Lua gives you a fast way to calculate that n. If nil may be in
there somewhere, somebody better tell you where this table is supposed
to end because that information is not obvious!
And if a nil ends up in a table that cannot handle it, that's a bug, and
you have more important things to worry about than the array-status of
the table.

And again, introducing "empty" would only make the situation more
complicated ...

>
> Imagine a Java/C++/C# collection behaving like that. I'd be scratching my head a bit at such a thing. (OK, so Java/C++/C# collections have a LOT of other problems, but two wrongs don't make a right! <grin>).
>
> There are plenty of cases where being able to "mark" an array entry
> as  done/dead/empty etc is useful as part of an algorithm. In many cases you
> can do that using a sentinel value that is outside the normal range of
> expected array types and/or values, but it's all very domain specific

What are the cases where you cannot use a sentinel value? `{}` will only
be equal to itself, so this is a natural candidate for a sentinel ...

> (can I use an empty string? -1? false? auxiliary state?). I'm basically
arguing that "empty" allows you to do this in a nice, clean,
self-documenting, portable manner. I don't really see anything heretical
in that. To be honest, the most heretical thing is that you would
introduce a new keyword, and THAT can of course break existing code.
>
> --Tim
>

Philipp




1234 ... 6