Floats and %d

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Floats and %d

Rena
This is something I've been stubbing my toe on a lot lately and it's
rather annoying:

Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
; print(string.format('%d', 1.1))
stdin:1: bad argument #2 to 'format' (number has no integer representation)
stack traceback:
    [C]: in function 'string.format'
    stdin:1: in main chunk
    [C]: in ?
;

In previous versions - and in C - it would just silently truncate the
number (drop the fractional part). Why was this changed?

--
Sent from my Game Boy.

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Daurnimator
On 15 July 2015 at 10:58, Rena <[hidden email]> wrote:

> This is something I've been stubbing my toe on a lot lately and it's
> rather annoying:
>
> Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> ; print(string.format('%d', 1.1))
> stdin:1: bad argument #2 to 'format' (number has no integer representation)
> stack traceback:
>     [C]: in function 'string.format'
>     stdin:1: in main chunk
>     [C]: in ?
> ;
>
> In previous versions - and in C - it would just silently truncate the
> number (drop the fractional part). Why was this changed?

Truncation is undefined and a surprising source of errors.
See the subthread starting at
http://lua-users.org/lists/lua-l/2014-12/msg00259.html

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Parke
In reply to this post by Rena
On Tue, Jul 14, 2015 at 5:58 PM, Rena <[hidden email]> wrote:
> This is something I've been stubbing my toe on a lot lately and it's
> rather annoying:
>
> Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> ; print(string.format('%d', 1.1))

> In previous versions - and in C - it would just silently truncate the
> number (drop the fractional part).

In C?  Really?  What compiler?

$ cat test.c
#include <stdio.h>
int main () {
  printf ( "%d\n", 2.3 );
  return 0;
}

$ gcc test.c
test.c: In function ‘main’:
test.c:3:3: warning: format ‘%d’ expects argument of type ‘int’, but
argument 2 has type ‘double’ [-Wformat=]
   printf ( "%d\n", 2.3 );
   ^

$ gcc -Wformat=0 test.c

$ ./a.out
-782778040

-Parke

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Rena
On Tue, Jul 14, 2015 at 9:20 PM, Parke <[hidden email]> wrote:

> On Tue, Jul 14, 2015 at 5:58 PM, Rena <[hidden email]> wrote:
>> This is something I've been stubbing my toe on a lot lately and it's
>> rather annoying:
>>
>> Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
>> ; print(string.format('%d', 1.1))
>
>> In previous versions - and in C - it would just silently truncate the
>> number (drop the fractional part).
>
> In C?  Really?  What compiler?
>

Maybe it was only the one in my imagination... :)

--
Sent from my Game Boy.

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Roberto Ierusalimschy
> On Tue, Jul 14, 2015 at 9:20 PM, Parke <[hidden email]> wrote:
> > On Tue, Jul 14, 2015 at 5:58 PM, Rena <[hidden email]> wrote:
> >> This is something I've been stubbing my toe on a lot lately and it's
> >> rather annoying:
> >>
> >> Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> >> ; print(string.format('%d', 1.1))
> >
> >> In previous versions - and in C - it would just silently truncate the
> >> number (drop the fractional part).
> >
> > In C?  Really?  What compiler?
> >
>
> Maybe it was only the one in my imagination... :)

Any correct C compiler will truncate as you said (towards zero), when
it converts a float to an integer. 'printf' is an exception, because
it is a vararg function and, therefore, without type checking. (With
no type checking, the compiler does not know it has to convert.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Rena

On Jul 16, 2015 8:06 AM, "Roberto Ierusalimschy" <[hidden email]> wrote:
>
> > On Tue, Jul 14, 2015 at 9:20 PM, Parke <[hidden email]> wrote:
> > > On Tue, Jul 14, 2015 at 5:58 PM, Rena <[hidden email]> wrote:
> > >> This is something I've been stubbing my toe on a lot lately and it's
> > >> rather annoying:
> > >>
> > >> Lua 5.3.1  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> > >> ; print(string.format('%d', 1.1))
> > >
> > >> In previous versions - and in C - it would just silently truncate the
> > >> number (drop the fractional part).
> > >
> > > In C?  Really?  What compiler?
> > >
> >
> > Maybe it was only the one in my imagination... :)
>
> Any correct C compiler will truncate as you said (towards zero), when
> it converts a float to an integer. 'printf' is an exception, because
> it is a vararg function and, therefore, without type checking. (With
> no type checking, the compiler does not know it has to convert.)
>
> -- Roberto
>

Right, I think I mixed up the automatic casting when assigning to an int and the automatic promotion to int when passing through a vararg list.

In any case, I'm still wishing Lua would truncate for %d. I keep ending up with seemingly harmless UI code crashing with a "number has no integer representation" error, and ugly floor() littered everywhere. It reminds me of having to use tostring() with %s in earlier versions (a similar case which annoyed me as well). But perhaps I'm alone here...

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Dirk Laurie-2
2015-07-18 20:29 GMT+02:00 Rena <[hidden email]>:

> On Jul 16, 2015 8:06 AM, "Roberto Ierusalimschy" <[hidden email]>
> wrote:
>> Any correct C compiler will truncate as you said (towards zero), when
>> it converts a float to an integer. 'printf' is an exception, because
>> it is a vararg function and, therefore, without type checking. (With
>> no type checking, the compiler does not know it has to convert.)
>>
>> -- Roberto
>>
>
> Right, I think I mixed up the automatic casting when assigning to an int and
> the automatic promotion to int when passing through a vararg list.
>
> In any case, I'm still wishing Lua would truncate for %d. I keep ending up
> with seemingly harmless UI code crashing with a "number has no integer
> representation" error, and ugly floor() littered everywhere. It reminds me
> of having to use tostring() with %s in earlier versions (a similar case
> which annoyed me as well). But perhaps I'm alone here...

That precedent lends conceptual integrity to your request.

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Parke
In reply to this post by Rena
On Sat, Jul 18, 2015 at 11:29 AM, Rena <[hidden email]> wrote:
> In any case, I'm still wishing Lua would truncate for %d. I keep ending up
> with seemingly harmless UI code crashing with a "number has no integer
> representation" error, and ugly floor() littered everywhere. It reminds me
> of having to use tostring() with %s in earlier versions (a similar case
> which annoyed me as well). But perhaps I'm alone here...

You are not alone.  After reflection, truncation for %d makes sense to me.

In Lua 5.2, %d would truncate.  In Python also.

Is anyone willing to suggest that the error in Lua 5.3 has value?

Regarding:

http://lua-users.org/lists/lua-l/2014-12/msg00259.html

>   string.format("%d", 2.7)

In this case, string.format does not have to convert 2.7 to an integer
(although it could).  string.format in this case is converting a float
to a string, and the request for truncation is not hidden, it is
explicit in %d.

-Parke

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Dirk Laurie-2
2015-07-20 23:14 GMT+02:00 Parke <[hidden email]>:
>>   string.format("%d", 2.7)
>
> In this case, string.format does not have to convert 2.7 to an integer
> (although it could).  string.format in this case is converting a float
> to a string, and the request for truncation is not hidden, it is
> explicit in %d.

This is a persuasive argument. However, the manual says:
    Options c, d, i, o, u, X, and x expect an integer.
Should the argument apply to the others too? Maybe %i?
Maybe not %x?

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Tim Hill

> On Jul 20, 2015, at 6:59 PM, Dirk Laurie <[hidden email]> wrote:
>
> 2015-07-20 23:14 GMT+02:00 Parke <[hidden email]>:
>>>  string.format("%d", 2.7)
>>
>> In this case, string.format does not have to convert 2.7 to an integer
>> (although it could).  string.format in this case is converting a float
>> to a string, and the request for truncation is not hidden, it is
>> explicit in %d.
>
> This is a persuasive argument. However, the manual says:
>    Options c, d, i, o, u, X, and x expect an integer.
> Should the argument apply to the others too? Maybe %i?
> Maybe not %x?
>

If you view string.format() as a thin wrapper for sprintf(), then it’s unlikely to change. Of course, the entire logic behind things like %d and %g in sprintf() really doesn’t apply to Lua — unlike in C the actual types of the values may be determined at runtime. In a more “Lua-ish” formatter, something like %d might well mean “format a number as an integer using xxx truncation rule”, but I doubt a re-write of sprintf() is high on Roberto’s todo list. Though is IS quite high on my list, I wonder if others here would be interested in such a facility?

—Tim


Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Dirk Laurie-2
2015-07-21 6:49 GMT+02:00 Tim Hill <[hidden email]>:

>> On Jul 20, 2015, at 6:59 PM, Dirk Laurie <[hidden email]> wrote:
>If you view string.format() as a thin wrapper for sprintf(), then it’s unlikely to
> change.

It's not that thin. There is a separate case for every specifier. At present
the following are all treated the same.

        case 'd': case 'i':
        case 'o': case 'u': case 'x': case 'X': {
          lua_Integer n = luaL_checkinteger(L, arg);
          addlenmod(form, LUA_INTEGER_FRMLEN);
          nb = sprintf(buff, form, n);
          break;
        }

> I doubt a re-write of sprintf() is high on Roberto’s todo list. Though is
> IS quite high on my list, I wonder if others here would be interested
> in such a facility?

The OP's problem can be solved with something much less ambitious.
Replace the first line above by:

      case 'd':
          lua_Integer n = (lua_Integer)luaL_checknumber(L, arg);
          addlenmod(form, LUA_INTEGER_FRMLEN);
          nb = sprintf(buff, form, (n);
          break;
      case 'i':

My question was: which of the other five cases should also be
treated like case 'd?

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Daurnimator
On 21 July 2015 at 20:48, Dirk Laurie <[hidden email]> wrote:

> 2015-07-21 6:49 GMT+02:00 Tim Hill <[hidden email]>:
>
>>> On Jul 20, 2015, at 6:59 PM, Dirk Laurie <[hidden email]> wrote:
>>If you view string.format() as a thin wrapper for sprintf(), then it’s unlikely to
>> change.
>
> It's not that thin. There is a separate case for every specifier. At present
> the following are all treated the same.
>
>         case 'd': case 'i':
>         case 'o': case 'u': case 'x': case 'X': {
>           lua_Integer n = luaL_checkinteger(L, arg);
>           addlenmod(form, LUA_INTEGER_FRMLEN);
>           nb = sprintf(buff, form, n);
>           break;
>         }
>
>> I doubt a re-write of sprintf() is high on Roberto’s todo list. Though is
>> IS quite high on my list, I wonder if others here would be interested
>> in such a facility?
>
> The OP's problem can be solved with something much less ambitious.
> Replace the first line above by:
>
>       case 'd':
>           lua_Integer n = (lua_Integer)luaL_checknumber(L, arg);
>           addlenmod(form, LUA_INTEGER_FRMLEN);
>           nb = sprintf(buff, form, (n);
>           break;
>       case 'i':
>
> My question was: which of the other five cases should also be
> treated like case 'd?
>

IIUC, this would break it for some numbers between 2^53 and 2^63

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Roberto Ierusalimschy
> > The OP's problem can be solved with something much less ambitious.
> > Replace the first line above by:
> >
> >       case 'd':
> >           lua_Integer n = (lua_Integer)luaL_checknumber(L, arg);
> >           addlenmod(form, LUA_INTEGER_FRMLEN);
> >           nb = sprintf(buff, form, (n);
> >           break;
> >       case 'i':
> >
> > My question was: which of the other five cases should also be
> > treated like case 'd?
> >
>
> IIUC, this would break it for some numbers between 2^53 and 2^63

Most of them, actually. It also has undefined behavior for numbers
larger than 2^63. (On most machines this "undefined behavior" manifests
as plainly wrong results.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

William Ahern
In reply to this post by Parke
On Mon, Jul 20, 2015 at 02:14:36PM -0700, Parke wrote:

> On Sat, Jul 18, 2015 at 11:29 AM, Rena <[hidden email]> wrote:
> > In any case, I'm still wishing Lua would truncate for %d. I keep ending up
> > with seemingly harmless UI code crashing with a "number has no integer
> > representation" error, and ugly floor() littered everywhere. It reminds me
> > of having to use tostring() with %s in earlier versions (a similar case
> > which annoyed me as well). But perhaps I'm alone here...
>
> You are not alone.  After reflection, truncation for %d makes sense to me.
>
> In Lua 5.2, %d would truncate.  In Python also.
>
> Is anyone willing to suggest that the error in Lua 5.3 has value?

Many people might argue that automagic coercion like this hides more bugs
than it fixes. I'm not once of those people, but it's a reasonable argument.

Also, the internal implementation rightly takes care to fail when losing
information in a coercion. luaL_checkinteger calls lua_integerx and errors
when lua_integerx says the value wasn't an integer. Your suggested fix in
this one case would make the implementation's treatment of coercions less
consistent. For example, the bit operators use luaL_checkinteger. Are you
also suggesting that the following should work?

        print(2.7 >> 1)

Is it worth the inconsistency in both the language semantics as well as the
implementation?

> Regarding:
>
> http://lua-users.org/lists/lua-l/2014-12/msg00259.html
>
> >   string.format("%d", 2.7)
>
> In this case, string.format does not have to convert 2.7 to an integer
> (although it could).  string.format in this case is converting a float
> to a string, and the request for truncation is not hidden, it is
> explicit in %d.
>

It's not explicit. Explicit would be

        string.format("%d", (math.modf(2.7)))

I don't do much floating point math, but AFAIU truncation is not always the
_obvious_ choice. It appears a natural choice to me, but I don't do much
numerical analysis.

There's significant tension in Lua regarding strict typing versus automatic
coercions. IIRC, over the years there have been suggestions to make code
like this fail:

        string.format("%d", "2")
        string.format("%s", 2)

Be careful what you ask for. You may end up getting the complete opposite.
The Lua authors might eventually believe the creeping complexity of coercion
rules detracts from the simplicity of the language and it's implementation.

IMO, permitting 2.0 to coerce to 2, but not 2.7, seems like a very sensible
approach. And sticking to this rule also seems very sensible because it's
principled--it emphasizes value preservation, without the burdens of
worrying about context or strict typing. If you start having to worry about
context (whether it should coerced or truncated in one construct but not
another), strict typing might seem like the simpler approach overall.


Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Parke
On Tue, Jul 21, 2015 at 1:14 PM, William Ahern
<[hidden email]> wrote:
> Many people might argue that automagic coercion like this hides more bugs
> than it fixes. I'm not once of those people, but it's a reasonable argument.

Actually, I may be shifting to that position.  Why?  I have realized
that choice of either %s or %.0f may be an adequate alternative for
%d.

> Are you also suggesting that the following should work?
>
>         print(2.7 >> 1)

No, I do not believe I have ever suggested that.  I'd prefer an error.

My suggestion was limited to string.format.

> There's significant tension in Lua regarding strict typing versus automatic
> coercions. IIRC, over the years there have been suggestions to make code
> like this fail:

>         string.format("%d", "2")

Failure would be fine.

>         string.format("%s", 2)

Failure would be inconvenient, IMO.  Unless there is some other easy
way that I am unaware of to have 2.0 format to '2' and 2.7 format to
'2.7' ?

> IMO, permitting 2.0 to coerce to 2, but not 2.7, seems like a very sensible
> approach. And sticking to this rule also seems very sensible because it's
> principled--it emphasizes value preservation, without the burdens of
> worrying about context or strict typing. If you start having to worry about
> context (whether it should coerced or truncated in one construct but not
> another), strict typing might seem like the simpler approach overall.

Yes, I am now mostly inclined to agree.

-Parke

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Rena
In reply to this post by Roberto Ierusalimschy
On Tue, Jul 21, 2015 at 8:28 AM, Roberto Ierusalimschy
<[hidden email]> wrote:

>> > The OP's problem can be solved with something much less ambitious.
>> > Replace the first line above by:
>> >
>> >       case 'd':
>> >           lua_Integer n = (lua_Integer)luaL_checknumber(L, arg);
>> >           addlenmod(form, LUA_INTEGER_FRMLEN);
>> >           nb = sprintf(buff, form, (n);
>> >           break;
>> >       case 'i':
>> >
>> > My question was: which of the other five cases should also be
>> > treated like case 'd?
>> >
>>
>> IIUC, this would break it for some numbers between 2^53 and 2^63
>
> Most of them, actually. It also has undefined behavior for numbers
> larger than 2^63. (On most machines this "undefined behavior" manifests
> as plainly wrong results.)
>
> -- Roberto
>

That can still be addressed:
case 'd':
    lua_Integer n;
    if(lua_isinteger(L, arg)) n = lua_tointeger(L, arg);
    else n = (lua_Integer)luaL_checknumber(L, arg);
    addlenmod(form, LUA_INTEGER_FRMLEN);
    nb = sprintf(buff, form, (n);
    break;
case 'i':

IIRC lua_Number already isn't able to reliably represent values
between 2^53 and 2^63, so the truncation shouldn't be an issue?

I have been using %1.0f instead, but it just feels awkward. %d is
requesting display as an integer, whether the input is an integer,
float, string, etc. I can understand raising an error when information
is being lost, but not in this particular case. If I were concerned
about the precision, I'd use e.g. %1.2f.

Otherwise it seems like the consistent thing would be to raise an
error also for string.format("%1.2f", 0.005), which would be
especially silly.

--
Sent from my Game Boy.

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Dirk Laurie-2
2015-07-22 0:32 GMT+02:00 Rena <[hidden email]>:

> On Tue, Jul 21, 2015 at 8:28 AM, Roberto Ierusalimschy
> <[hidden email]> wrote:
>>> > The OP's problem can be solved with something much less ambitious.
>>> > Replace the first line above by:
>>> >
>>> >       case 'd':
>>> >           lua_Integer n = (lua_Integer)luaL_checknumber(L, arg);
>>> >           addlenmod(form, LUA_INTEGER_FRMLEN);
>>> >           nb = sprintf(buff, form, (n);
>>> >           break;
>>> >       case 'i':
>>> >
>>> > My question was: which of the other five cases should also be
>>> > treated like case 'd?
>>> >
>>>
>>> IIUC, this would break it for some numbers between 2^53 and 2^63
>>
>> Most of them, actually. It also has undefined behavior for numbers
>> larger than 2^63. (On most machines this "undefined behavior" manifests
>> as plainly wrong results.)
>>
>> -- Roberto
>>
>
> That can still be addressed:
> case 'd':
>     lua_Integer n;
>     if(lua_isinteger(L, arg)) n = lua_tointeger(L, arg);
>     else n = (lua_Integer)luaL_checknumber(L, arg);
>     addlenmod(form, LUA_INTEGER_FRMLEN);
>     nb = sprintf(buff, form, (n);
>     break;
> case 'i':

Excellent! We have now demonstrated that a very easy change
allows anybody to build a custom string.format that will silently
throw away the fractional part of a non-integer. Such a person
can be assumed to be a responsible adult who will use this
dangerous tool with caution.

BTW does anybody know if C makes any distinction between
%d and %i? "man 3 printf" in Linux does not show any.

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Parke
On Tue, Jul 21, 2015 at 11:33 PM, Dirk Laurie <[hidden email]> wrote:
> BTW does anybody know if C makes any distinction between
> %d and %i? "man 3 printf" in Linux does not show any.

Posix and C++ also ascribe %d and %i the same behavior.

http://pubs.opengroup.org/onlinepubs/9699919799/functions/printf.html

http://www.cplusplus.com/reference/cstdio/printf/

-Parke

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Coda Highland
In reply to this post by Dirk Laurie-2
On Tue, Jul 21, 2015 at 11:33 PM, Dirk Laurie <[hidden email]> wrote:
> BTW does anybody know if C makes any distinction between
> %d and %i? "man 3 printf" in Linux does not show any.

There's no difference for printf, but for scanf %d always interprets
the number as decimal while %i allows octal and hexadecimal,
distinguished by prefix.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Floats and %d

Philipp Janda
In reply to this post by Dirk Laurie-2
Am 22.07.2015 um 08:33 schröbte Dirk Laurie:

> 2015-07-22 0:32 GMT+02:00 Rena <[hidden email]>:
>> On Tue, Jul 21, 2015 at 8:28 AM, Roberto Ierusalimschy
>> <[hidden email]> wrote:
>>>>> The OP's problem can be solved with something much less ambitious.
>>>>> Replace the first line above by:
>>>>>
>>>>>        case 'd':
>>>>>            lua_Integer n = (lua_Integer)luaL_checknumber(L, arg);
>>>>>            addlenmod(form, LUA_INTEGER_FRMLEN);
>>>>>            nb = sprintf(buff, form, (n);
>>>>>            break;
>>>>>        case 'i':
>>>>>
>>>>> My question was: which of the other five cases should also be
>>>>> treated like case 'd?
>>>>>
>>>>
>>>> IIUC, this would break it for some numbers between 2^53 and 2^63
>>>
>>> Most of them, actually. It also has undefined behavior for numbers
>>> larger than 2^63. (On most machines this "undefined behavior" manifests
>>> as plainly wrong results.)
>>>
>>> -- Roberto
>>>
>>
>> That can still be addressed:
>> case 'd':
>>      lua_Integer n;
>>      if(lua_isinteger(L, arg)) n = lua_tointeger(L, arg);
>>      else n = (lua_Integer)luaL_checknumber(L, arg);
>>      addlenmod(form, LUA_INTEGER_FRMLEN);
>>      nb = sprintf(buff, form, (n);
>>      break;
>> case 'i':
>
> Excellent! We have now demonstrated that a very easy change
> allows anybody to build a custom string.format that will silently
> throw away the fractional part of a non-integer.

Obviously the change isn't that easy, because the code above is still
incorrect: It yields undefined behavior for large floating point numbers
as Roberto already mentioned. ISO C 90:

> When a value of floating type is converted to integral type, the
> fractional part is discarded.  If the value of the integral part
> cannot be represented by the integral type, the behavior is
> undefined.

There are probably a lot of places in C code for Lua 5.2 and before
where this problem might occur, so raising an error by default like in
Lua 5.3 is actually a very good choice, IMHO.

One way to address this issue for `string.format` would be to use `%f`
with a zero precision for the `sprintf` if the number is a floating
point number, and the LUA_INTEGER_FRM* in case the value is already an
integer.


Philipp