Lua 5.3: wrong coercion?

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lua 5.3: wrong coercion?

Lorenzo Donati-3
Hi all!

I just found something that seems odd in Lua 5.3.

According to sect 3.4.3 in the refman:

"The conversion from strings to numbers goes as follows: First, the
string is converted to an integer or a float, following its syntax and
the rules of the Lua lexer. (The string may have also leading and
trailing spaces and a sign.) Then, the resulting number is converted to
the required type (float or integer) according to the previous rules."

Thus if I have a string which is "42" it should be coerced to an
integer, since 42 is parsed as an integer constant by the lexer, whereas
I get:

print( 43, "42" + 1 ) --> 43   43.0

where it seems that the addition is a float addition, signaling that one
of the operands is float. Is this a bug, an error in the refman or maybe
something on my side (Windows 7-64, using 32bit executables compiled
with TDM-GCC 4.9.2 32 bit)? Am I missing something?


-- Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Dirk Laurie-2
2015-01-16 21:41 GMT+02:00 Lorenzo Donati <[hidden email]>:

> Hi all!
>
> I just found something that seems odd in Lua 5.3.
>
> According to sect 3.4.3 in the refman:
>
> "The conversion from strings to numbers goes as follows: First, the string
> is converted to an integer or a float, following its syntax and the rules of
> the Lua lexer. (The string may have also leading and trailing spaces and a
> sign.) Then, the resulting number is converted to the required type (float
> or integer) according to the previous rules."
>
> Thus if I have a string which is "42" it should be coerced to an integer,
> since 42 is parsed as an integer constant by the lexer, whereas I get:
>
> print( 43, "42" + 1 ) --> 43   43.0
>
> where it seems that the addition is a float addition, signaling that one of
> the operands is float. Is this a bug, an error in the refman or maybe
> something on my side (Windows 7-64, using 32bit executables compiled with
> TDM-GCC 4.9.2 32 bit)? Am I missing something?

It also happens under Ubuntu 14.04 LTS. But:

   print( 43, tonumber"42" + 1 ) --> 43   43

Let's be careful with the word "bug", shall we? Don't want to annoy people.
But certainly this seems to be a documentation/implementation anomaly.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3


On 16/01/2015 20:53, Dirk Laurie wrote:

> 2015-01-16 21:41 GMT+02:00 Lorenzo Donati <[hidden email]>:
>> Hi all!
>>
>> I just found something that seems odd in Lua 5.3.
>>
>> According to sect 3.4.3 in the refman:
>>
>> "The conversion from strings to numbers goes as follows: First, the string
>> is converted to an integer or a float, following its syntax and the rules of
>> the Lua lexer. (The string may have also leading and trailing spaces and a
>> sign.) Then, the resulting number is converted to the required type (float
>> or integer) according to the previous rules."
>>
>> Thus if I have a string which is "42" it should be coerced to an integer,
>> since 42 is parsed as an integer constant by the lexer, whereas I get:
>>
>> print( 43, "42" + 1 ) --> 43   43.0
>>
>> where it seems that the addition is a float addition, signaling that one of
>> the operands is float. Is this a bug, an error in the refman or maybe
>> something on my side (Windows 7-64, using 32bit executables compiled with
>> TDM-GCC 4.9.2 32 bit)? Am I missing something?
>
> It also happens under Ubuntu 14.04 LTS. But:

Good to know it is repeatable on other platforms. Was it reported before
on the list? If it was, did Lua team said something about that?

>
>     print( 43, tonumber"42" + 1 ) --> 43   43
>
> Let's be careful with the word "bug", shall we? Don't want to annoy people.
> But certainly this seems to be a documentation/implementation anomaly.
>
>

Well, if it is by design (so it is not a bug, but an implementation
detail), and it works differently among platforms, the manual
overspecified the thing.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Roberto Ierusalimschy
In reply to this post by Lorenzo Donati-3
> According to sect 3.4.3 in the refman:
>
> "The conversion from strings to numbers goes as follows: First, the
> string is converted to an integer or a float, following its syntax
> and the rules of the Lua lexer. (The string may have also leading
> and trailing spaces and a sign.) Then, the resulting number is
> converted to the required type (float or integer) according to the
> previous rules."
>
> Thus if I have a string which is "42" it should be coerced to an
> integer, since 42 is parsed as an integer constant by the lexer,
> whereas I get:
>
> print( 43, "42" + 1 ) --> 43   43.0
>
> where it seems that the addition is a float addition, signaling that
> one of the operands is float. Is this a bug, an error in the refman
> or maybe something on my side (Windows 7-64, using 32bit executables
> compiled with TDM-GCC 4.9.2 32 bit)? Am I missing something?

You are missing the last sentence that you quoted!  "Then, the resulting
number is converted to the required type (float or integer) according
to the previous rules." The "required type" in this case is a float, as
explained in the rule for addition (Section 3.4.1):

  [...] the arithmetic operators work as follows: If both operands are
  integers, the operation is performed over integers and the result is an
  integer. Otherwise, if both operands are numbers or strings that can be
  converted to numbers (see §3.4.3), then they are converted to floats,
  [...]

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3


On 16/01/2015 21:49, Roberto Ierusalimschy wrote:

>> According to sect 3.4.3 in the refman:
>>
>> "The conversion from strings to numbers goes as follows: First, the
>> string is converted to an integer or a float, following its syntax
>> and the rules of the Lua lexer. (The string may have also leading
>> and trailing spaces and a sign.) Then, the resulting number is
>> converted to the required type (float or integer) according to the
>> previous rules."
>>
>> Thus if I have a string which is "42" it should be coerced to an
>> integer, since 42 is parsed as an integer constant by the lexer,
>> whereas I get:
>>
>> print( 43, "42" + 1 ) --> 43   43.0
>>
>> where it seems that the addition is a float addition, signaling that
>> one of the operands is float. Is this a bug, an error in the refman
>> or maybe something on my side (Windows 7-64, using 32bit executables
>> compiled with TDM-GCC 4.9.2 32 bit)? Am I missing something?
>
> You are missing the last sentence that you quoted!  "Then, the resulting
> number is converted to the required type (float or integer) according
> to the previous rules." The "required type" in this case is a float, as
> explained in the rule for addition (Section 3.4.1):
>
>    [...] the arithmetic operators work as follows: If both operands are
>    integers, the operation is performed over integers and the result is an
>    integer. Otherwise, if both operands are numbers or strings that can be
>    converted to numbers (see §3.4.3), then they are converted to floats,
>    [...]
>
Ah! Thanks! You are right. A bit unexpected, but correct.
So it should work like this on all platforms.


Is there a reason why the coercion first recognizes the string as
representing an int and then "throws away" that info and converts it
unconditionally into a float?




> -- Roberto
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Tim Hill
In reply to this post by Roberto Ierusalimschy

> On Jan 16, 2015, at 12:49 PM, Roberto Ierusalimschy <[hidden email]> wrote:
>
>> According to sect 3.4.3 in the refman:
>>
>> "The conversion from strings to numbers goes as follows: First, the
>> string is converted to an integer or a float, following its syntax
>> and the rules of the Lua lexer. (The string may have also leading
>> and trailing spaces and a sign.) Then, the resulting number is
>> converted to the required type (float or integer) according to the
>> previous rules."
>>
>> Thus if I have a string which is "42" it should be coerced to an
>> integer, since 42 is parsed as an integer constant by the lexer,
>> whereas I get:
>>
>> print( 43, "42" + 1 ) --> 43   43.0
>>
>> where it seems that the addition is a float addition, signaling that
>> one of the operands is float. Is this a bug, an error in the refman
>> or maybe something on my side (Windows 7-64, using 32bit executables
>> compiled with TDM-GCC 4.9.2 32 bit)? Am I missing something?
>
> You are missing the last sentence that you quoted!  "Then, the resulting
> number is converted to the required type (float or integer) according
> to the previous rules." The "required type" in this case is a float, as
> explained in the rule for addition (Section 3.4.1):
>
>  [...] the arithmetic operators work as follows: If both operands are
>  integers, the operation is performed over integers and the result is an
>  integer. Otherwise, if both operands are numbers or strings that can be
>  converted to numbers (see §3.4.3), then they are converted to floats,
>  [...]
>
> -- Roberto
>

I’m scratching my head on this one…

(1) 3.4.3 "First, the string is converted to an integer or a float, following its syntax and the rules of the Lua lexer...”
—> So, “42” should convert to the integer 42. So we now have 42 + 1 as the print() expression.

(2) 3.4.3 “...Then, the resulting number is converted to the required type (float or integer) according to the previous rules.”
—> If by “previous rules” the manual means “...All other arithmetic operations applied to mixed numbers (integers and floats) convert the integer operand to a float; this is called the usual rule.”, then this doesn’t seem to apply here, since both operands are integers. So we still have 42 + 1.

(3) 3.4.1 "If both operands are integers, the operation is performed over integers and the result is an integer. Otherwise, if both operands are numbers or strings that can be converted to numbers (see §3.4.3), then they are converted to floats"
—> Yes, they are both numbers or strings. So we convert them to float and get 42.0 + 1.0

But why? Given that the parser has determined “42” is an integer, why does the arithmetic operator then somehow keep track of the origins of the integer as a string and coerce it to a float? I have to confess I find the following surprising:

print(“42” + 1)
—> 43.0
print(tonumber(“42”) + 1)
—> 43

Is this behavior (the first print()) for backward compatibility? That makes sense to me, but otherwise I’m trying to understand the rationale.

—Tim




Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Roberto Ierusalimschy
In reply to this post by Lorenzo Donati-3
> Is there a reason why the coercion first recognizes the string as
> representing an int and then "throws away" that info and converts it
> unconditionally into a float?

We want a fast track for integers, so the first check is simply whether
both operands are integers. After that check, it is too cumbersome to
recheck whether the strings are "integers" and then also do an integer
operation.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Enrico Tassi
In reply to this post by Tim Hill
On Fri, Jan 16, 2015 at 01:35:35PM -0800, Tim Hill wrote:
> print(“42” + 1)
> —> 43.0
> print(tonumber(“42”) + 1)
> —> 43

I've to admit I was puzzled too... but I guess we all want

  print("42" + 1)

to behave like

  local x = "42"
  print(x + 1)

so the ony coercion triggered here is the one documented for arithmetic
operatos, that clearly turns "42" into "42.0" (x is a string, we want
no coercion mapping "42" into an int/float in my second example).

Best,
--
Enrico Tassi

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3
In reply to this post by Roberto Ierusalimschy


On 16/01/2015 22:38, Roberto Ierusalimschy wrote:

>> Is there a reason why the coercion first recognizes the string as
>> representing an int and then "throws away" that info and converts it
>> unconditionally into a float?
>
> We want a fast track for integers, so the first check is simply whether
> both operands are integers. After that check, it is too cumbersome to
> recheck whether the strings are "integers" and then also do an integer
> operation.
>
> -- Roberto
>
>
Thanks for the clarification!

I'm not really sure, maybe I just must get used to it, but the whole
string->number automatic coercion thing seems to have become a bit
overcomplicated. I only relied a few times on this behavior before (pre
5.3), so I wouldn't miss it should you remove it from the language and
raise an error whenever a string is used where a number is expected. As
it stands now the feature seems to me a bit counterintuitive or at least
too convoluted and perhaps can cause more harm than good. Just a gut
feeling, though. (my 2c)

Cheers!

-- Lorenzo

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Coda Highland
On Fri, Jan 16, 2015 at 2:29 PM, Lorenzo Donati
<[hidden email]> wrote:

>
>
> On 16/01/2015 22:38, Roberto Ierusalimschy wrote:
>>>
>>> Is there a reason why the coercion first recognizes the string as
>>> representing an int and then "throws away" that info and converts it
>>> unconditionally into a float?
>>
>>
>> We want a fast track for integers, so the first check is simply whether
>> both operands are integers. After that check, it is too cumbersome to
>> recheck whether the strings are "integers" and then also do an integer
>> operation.
>>
>> -- Roberto
>>
>>
> Thanks for the clarification!
>
> I'm not really sure, maybe I just must get used to it, but the whole
> string->number automatic coercion thing seems to have become a bit
> overcomplicated. I only relied a few times on this behavior before (pre
> 5.3), so I wouldn't miss it should you remove it from the language and raise
> an error whenever a string is used where a number is expected. As it stands
> now the feature seems to me a bit counterintuitive or at least too
> convoluted and perhaps can cause more harm than good. Just a gut feeling,
> though. (my 2c)
>
> Cheers!
>
> -- Lorenzo
>

Isn't there a #define you can set now to disable string-to-number coercions?

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Dirk Laurie-2
In reply to this post by Lorenzo Donati-3
2015-01-17 0:29 GMT+02:00 Lorenzo Donati <[hidden email]>:

> the whole string->number automatic coercion thing seems to have become a bit
> overcomplicated.

It's the same as it always was, but we've started talking
about it a lot more. At the C code level it just does the
minimum, as Roberto explained.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Alexandre Rion
In reply to this post by Lorenzo Donati-3
I was not expecting that behaviour neither. However, it is consistent with what Roberto told during the conversation about the power (x^y) operator: the result type of the operator should depends only on the operandes types and not the operandes values.
If string where converted to integer or float according their values in the addition operation, you would have:
  String + Integer -> Integer (if the string represents an integer) or Float (if the string represents a float)
The current behaviour give:
  String + Integer -> Float
  String + Float -> Float
in all cases (given that the string represents a valid number) and so there is no surprise about the result type.

  Alexandre

--- Original Message ---

From: "Dirk Laurie" <[hidden email]>
Sent: 17 January 2015 07:20
To: "Lua mailing list" <[hidden email]>
Subject: Re: Lua 5.3: wrong coercion?

2015-01-17 0:29 GMT+02:00 Lorenzo Donati <[hidden email]>:

> the whole string->number automatic coercion thing seems to have become a bit
> overcomplicated.

It's the same as it always was, but we've started talking
about it a lot more. At the C code level it just does the
minimum, as Roberto explained.


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Tim Hill

> On Jan 17, 2015, at 12:29 AM, Alexandre (gaumerie) <[hidden email]> wrote:
>
> I was not expecting that behaviour neither. However, it is consistent with what Roberto told during the conversation about the power (x^y) operator: the result type of the operator should depends only on the operandes types and not the operandes values.
> If string where converted to integer or float according their values in the addition operation, you would have:
>  String + Integer -> Integer (if the string represents an integer) or Float (if the string represents a float)
> The current behaviour give:
>  String + Integer -> Float
>  String + Float -> Float
> in all cases (given that the string represents a valid number) and so there is no surprise about the result type.
>

For me, the “surprise" is that implicit coercion of a string to an integer behaves differently to explicit coercion (via tonumber()). I’m not saying it’s not according to the spec, just that it’s a bit .. well .. surprising.

—Tim



Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3
In reply to this post by Coda Highland


On 17/01/2015 00:04, Coda Highland wrote:

> On Fri, Jan 16, 2015 at 2:29 PM, Lorenzo Donati
> <[hidden email]> wrote:
>>
>>
>> On 16/01/2015 22:38, Roberto Ierusalimschy wrote:
>>>>
>>>> Is there a reason why the coercion first recognizes the string as
>>>> representing an int and then "throws away" that info and converts it
>>>> unconditionally into a float?
>>>
>>>
>>> We want a fast track for integers, so the first check is simply whether
>>> both operands are integers. After that check, it is too cumbersome to
>>> recheck whether the strings are "integers" and then also do an integer
>>> operation.
>>>
>>> -- Roberto
>>>
>>>
>> Thanks for the clarification!
>>
>> I'm not really sure, maybe I just must get used to it, but the whole
>> string->number automatic coercion thing seems to have become a bit
>> overcomplicated. I only relied a few times on this behavior before (pre
>> 5.3), so I wouldn't miss it should you remove it from the language and raise
>> an error whenever a string is used where a number is expected. As it stands
>> now the feature seems to me a bit counterintuitive or at least too
>> convoluted and perhaps can cause more harm than good. Just a gut feeling,
>> though. (my 2c)
>>
>> Cheers!
>>
>> -- Lorenzo
>>
>
> Isn't there a #define you can set now to disable string-to-number coercions?
>
Really? I didn't know that. Thanks for the tip!

I'd like not to use that, since I use Lua essentially as an external
script interpreter and not embedded in C applications, so I prefer to
stick with the defaults settings to lessen "compatibility" problems.

Probably I should grow the habit of using "tonumber" more often, which
BTW is also good programming practice.

> /s/ Adam
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3
In reply to this post by Dirk Laurie-2


On 17/01/2015 07:20, Dirk Laurie wrote:

> 2015-01-17 0:29 GMT+02:00 Lorenzo Donati <[hidden email]>:
>
>> the whole string->number automatic coercion thing seems to have become a bit
>> overcomplicated.
>
> It's the same as it always was, but we've started talking
> about it a lot more. At the C code level it just does the
> minimum, as Roberto explained.
>
>
I wasn't clear, I meant "overcomplicated" for the user, not in the
implementation.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3
In reply to this post by Alexandre Rion


On 17/01/2015 09:29, Alexandre (gaumerie) wrote:
> I was not expecting that behaviour neither. However, it is consistent with what Roberto told during the conversation about the power (x^y) operator: the result type of the operator should depends only on the operandes types and not the operandes values.
> If string where converted to integer or float according their values in the addition operation, you would have:
>    String + Integer -> Integer (if the string represents an integer) or Float (if the string represents a float)
> The current behaviour give:
>    String + Integer -> Float
>    String + Float -> Float
> in all cases (given that the string represents a valid number) and so there is no surprise about the result type.
>

Ah! Yes! This makes sense. I didn't got that point. This is indeed a
very sensible choice. Maybe it would be nice to stress this thing in the
manual. As It was shown to me in another post, the manual is precise
about that, but I think it is not straightforward.

>    Alexandre
>
> --- Original Message ---
>
> From: "Dirk Laurie" <[hidden email]>
> Sent: 17 January 2015 07:20
> To: "Lua mailing list" <[hidden email]>
> Subject: Re: Lua 5.3: wrong coercion?
>
> 2015-01-17 0:29 GMT+02:00 Lorenzo Donati <[hidden email]>:
>
>> the whole string->number automatic coercion thing seems to have become a bit
>> overcomplicated.
>
> It's the same as it always was, but we've started talking
> about it a lot more. At the C code level it just does the
> minimum, as Roberto explained.
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Lorenzo Donati-3
In reply to this post by Tim Hill


On 17/01/2015 11:29, Tim Hill wrote:

>
>> On Jan 17, 2015, at 12:29 AM, Alexandre (gaumerie) <[hidden email]> wrote:
>>
>> I was not expecting that behaviour neither. However, it is consistent with what Roberto told during the conversation about the power (x^y) operator: the result type of the operator should depends only on the operandes types and not the operandes values.
>> If string where converted to integer or float according their values in the addition operation, you would have:
>>   String + Integer -> Integer (if the string represents an integer) or Float (if the string represents a float)
>> The current behaviour give:
>>   String + Integer -> Float
>>   String + Float -> Float
>> in all cases (given that the string represents a valid number) and so there is no surprise about the result type.
>>
>
> For me, the “surprise" is that implicit coercion of a string to an integer behaves differently to explicit coercion (via tonumber()). I’m not saying it’s not according to the spec, just that it’s a bit .. well .. surprising.
>
I agree. That's surprising to me too. As I said, I begin to feel we
could get rid of this automatic string->number coercion thing and live
happily using tonumber explicitly.

I wonder whether there are use cases where that automatic coercion is so
important that not having it could amount to a showstopper.
To me it seems just a little convenience which we could do without in
exchange for a "cleaner" semantics and more explicit and robust code
(not really sure about it, though).

> —Tim
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

David Favro
In reply to this post by Coda Highland

On 07/31/2014 08:24 PM, Roberto Ierusalimschy wrote:
> Just in case, Lua 5.3 will have compile-time options for easily
> disabling these coercions (one for string->number, another for
> number->string), but the "standard" will be with coercions enabled.

On 01/17/2015 06:04 AM, Coda Highland wrote:
> Isn't there a #define you can set now to disable string-to-number
> coercions?

I would love to turn off automatic string-to-number conversion, but from
looking through the final 5.3.0 luaconf.h, I'm not clear on how to do
this -- did it make it into the release?  Am I missing something?


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Roberto Ierusalimschy
> On 01/17/2015 06:04 AM, Coda Highland wrote:
> >Isn't there a #define you can set now to disable string-to-number
> >coercions?
>
> I would love to turn off automatic string-to-number conversion, but
> from looking through the final 5.3.0 luaconf.h, I'm not clear on how
> to do this -- did it make it into the release?  Am I missing
> something?

#define LUA_NOCVTN2S
#define LUA_NOCVTS2N

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: wrong coercion?

Andrew Starks
In reply to this post by Lorenzo Donati-3


On Saturday, January 17, 2015, Lorenzo Donati <[hidden email]> wrote:


On 17/01/2015 11:29, Tim Hill wrote:

On Jan 17, 2015, at 12:29 AM, Alexandre (gaumerie) <[hidden email]> wrote:

I was not expecting that behaviour neither. However, it is consistent with what Roberto told during the conversation about the power (x^y) operator: the result type of the operator should depends only on the operandes types and not the operandes values.
If string where converted to integer or float according their values in the addition operation, you would have:
  String + Integer -> Integer (if the string represents an integer) or Float (if the string represents a float)
The current behaviour give:
  String + Integer -> Float
  String + Float -> Float
in all cases (given that the string represents a valid number) and so there is no surprise about the result type.


For me, the “surprise" is that implicit coercion of a string to an integer behaves differently to explicit coercion (via tonumber()). I’m not saying it’s not according to the spec, just that it’s a bit .. well .. surprising.

I agree. That's surprising to me too. As I said, I begin to feel we could get rid of this automatic string->number coercion thing and live happily using tonumber explicitly.

I wonder whether there are use cases where that automatic coercion is so important that not having it could amount to a showstopper.
To me it seems just a little convenience which we could do without in exchange for a "cleaner" semantics and more explicit and robust code (not really sure about it, though).

—Tim







I go back and forth between caring and suspecting that I just want to be one of the kewl kids that hates coercion. 

However, if I may proffer a guess as to how it would change:

1: A big section of us compiles Lua with the flags turned on. 
2: When modules fail, submit pull requests (it will take 2 seconds to patch them). 
3: Report back that the world didn't end. 

Alternately we can leave N2S off, but if we're going rouge, we may as well do a proper job of it. Making the modules we use explicit would be nice and it will mean that we can report back the level of pain for both flags, which will be useful, for science. 

I'll go first. You're welcome. 

-Andrew
12