bug report: string.format rounding error

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

bug report: string.format rounding error

Thomas Jenni
In Lua 5.3.5 on a 64-bit machine (macOS 10.14.6), the following two lines produce different results

print(string.format("%.2f",1.625000000000001))
1.63

print(string.format("%.2f",1.625000000000000))
1.62

It seems unclear to me, why in the latter case 1.625 is rounded to 1.62.
Thanks for fixing this bug.

best regards

T. Jenni
Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

markjynx
On 2019-09-18 09:47, Thomas Jenni wrote:

> In Lua 5.3.5 on a 64-bit machine (macOS 10.14.6), the following two
> lines produce different results
>
> print(string.format("%.2f",1.625000000000001))
> 1.63
>
> print(string.format("%.2f",1.625000000000000))
> 1.62
>
> It seems unclear to me, why in the latter case 1.625 is rounded to
> 1.62.
> Thanks for fixing this bug.
>
> best regards
>
> T. Jenni

https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Kat Kioo
"Hey why does this happen?"
*Hands 44 page manual to other person without explaining anything*

On Wed, Sep 18, 2019, 3:36 AM <[hidden email]> wrote:
On 2019-09-18 09:47, Thomas Jenni wrote:
> In Lua 5.3.5 on a 64-bit machine (macOS 10.14.6), the following two
> lines produce different results
>
> print(string.format("%.2f",1.625000000000001))
> 1.63
>
> print(string.format("%.2f",1.625000000000000))
> 1.62
>
> It seems unclear to me, why in the latter case 1.625 is rounded to
> 1.62.
> Thanks for fixing this bug.
>
> best regards
>
> T. Jenni

https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Storkman
On September 18, 2019 11:21:00 AM UTC, Meepen <[hidden email]> wrote:
>"Hey why does this happen?"
>*Hands 44 page manual to other person without explaining anything*


Read The Friendly Manual, as they used to say back when.
It's not every day you get to see a programmer complain of too much documentation.

--
Storkman

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Michal Kottman
Some briefer and bit more accessible references (that do link to the referenced document):

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Kat Kioo
In reply to this post by Storkman
Instead of just handing the manual over, you could say a brief reason of what is happening or at least the section to read in there. No sane developer that's just casually using lua is going to read a manual about floating point numbers to report something they thought could be a bug.

On Wed, Sep 18, 2019, 7:37 AM Storkman <[hidden email]> wrote:
On September 18, 2019 11:21:00 AM UTC, Meepen <[hidden email]> wrote:
>"Hey why does this happen?"
>*Hands 44 page manual to other person without explaining anything*


Read The Friendly Manual, as they used to say back when.
It's not every day you get to see a programmer complain of too much documentation.

--
Storkman

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Coda Highland
On Wed, Sep 18, 2019 at 7:10 AM Meepen <[hidden email]> wrote:
Instead of just handing the manual over, you could say a brief reason of what is happening or at least the section to read in there. No sane developer that's just casually using lua is going to read a manual about floating point numbers to report something they thought could be a bug.

Perhaps not, and I'm not defending the practice, but it IS an annoyingly common thing to ask about, especially since it's not only not a bug in Lua, it's not a bug at all and it's not specific to Lua. It's something that every developer needs to know.

(Other practices I don't defend: top-posting in contravention of list etiquette.)

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

RE: bug report: string.format rounding error

Thomas Buergel
>> print(string.format("%.2f",1.625000000000001))
>> 1.63
>>
>> print(string.format("%.2f",1.625000000000000))
>> 1.62

> Perhaps not, and I'm not defending the practice, but it IS an annoyingly common
> thing to ask about, especially since it's not only not a bug in Lua, it's not a bug at
> all and it's not specific to Lua. It's something that every developer needs to know.

Knowledge about binary floating point is certainly something every developer who
handles number should know about.

In this particular case, though...

1.625 is exactly representable in IEEE754 (float and double). There is no loss of
Information from the conversion. I would have expected both inputs to produce
a result of 1.63.

A quick attempt shows the following:

- Lua 5.1 (32-bit Windows)
- Lua 5.2 (32-bit Windows)
- C (Gcc 4.8.1)
- C (Visual Studio 2017)
  1.63 after rounding for both literals

- Lua 5.3 (32-bit Windows)
  1.63/1.62

So, yes, it is a valid question as to why there is a different result for the same input.
I don't have an explanation.

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Coda Highland
On Wed, Sep 18, 2019 at 11:20 AM Thomas Buergel <[hidden email]> wrote:
>> print(string.format("%.2f",1.625000000000001))
>> 1.63
>>
>> print(string.format("%.2f",1.625000000000000))
>> 1.62

> Perhaps not, and I'm not defending the practice, but it IS an annoyingly common
> thing to ask about, especially since it's not only not a bug in Lua, it's not a bug at
> all and it's not specific to Lua. It's something that every developer needs to know.

Knowledge about binary floating point is certainly something every developer who
handles number should know about.

In this particular case, though...

1.625 is exactly representable in IEEE754 (float and double). There is no loss of
Information from the conversion. I would have expected both inputs to produce
a result of 1.63.

A quick attempt shows the following:

- Lua 5.1 (32-bit Windows)
- Lua 5.2 (32-bit Windows)
- C (Gcc 4.8.1)
- C (Visual Studio 2017)
  1.63 after rounding for both literals

- Lua 5.3 (32-bit Windows)
  1.63/1.62

So, yes, it is a valid question as to why there is a different result for the same input.
I don't have an explanation.

string.format offers no particular promises about rounding behavior. Neither does the sprintf() function underlying string.format. Therefore, the particular details of this behavior aren't well-defined.

However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

The fifth one is the behavior typically taught to elementary school students and what appears to be the expected behavior in this discussion. The sixth one is the way some printf implementations have historically chosen to do it, but its behavior on values like .999... have made this rather unpopular.

However, the behavior we're seeing here is actually the fourth one, which is IEEE-754's recommended rounding behavior for binary floating-point numbers: 1.625 is a tie between 1.62 and 1.63, so it rounds to the one that ends with an even digit. On the other hand, 1.625000000000001 isn't a tie, so both round-to-nearest behaviors agree that 1.63 is the correct answer.

There was discussion in the lead-up to Lua 5.3 about whether or not control over the floating-point rounding mode should be exposed to scripts, but in the end it was not.

(As a side note: I don't know if it impacts this specific example, but just saying "32-bit Windows" isn't enough to describe the platform when talking about floating-point rounding. There's also a difference in rounding behavior depending on whether it's compiled using 53-bit-mantissa doubles or 64-bit-mantissa doubles.)

/s/ Adam


Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Philippe Verdy-2
In reply to this post by Thomas Buergel
May be the rounding mode may explain the difference. Note you display only 2 decimals and the remainder .25 is correctly rounded to 0.

Le mer. 18 sept. 2019 à 18:20, Thomas Buergel <[hidden email]> a écrit :
>> print(string.format("%.2f",1.625000000000001))
>> 1.63
>>
>> print(string.format("%.2f",1.625000000000000))
>> 1.62

> Perhaps not, and I'm not defending the practice, but it IS an annoyingly common
> thing to ask about, especially since it's not only not a bug in Lua, it's not a bug at
> all and it's not specific to Lua. It's something that every developer needs to know.

Knowledge about binary floating point is certainly something every developer who
handles number should know about.

In this particular case, though...

1.625 is exactly representable in IEEE754 (float and double). There is no loss of
Information from the conversion. I would have expected both inputs to produce
a result of 1.63.

A quick attempt shows the following:

- Lua 5.1 (32-bit Windows)
- Lua 5.2 (32-bit Windows)
- C (Gcc 4.8.1)
- C (Visual Studio 2017)
  1.63 after rounding for both literals

- Lua 5.3 (32-bit Windows)
  1.63/1.62

So, yes, it is a valid question as to why there is a different result for the same input.
I don't have an explanation.

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Coda Highland
In reply to this post by Coda Highland


On Wed, Sep 18, 2019 at 12:21 PM Coda Highland <[hidden email]> wrote:
However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

It occurs to me that truncating the decimal representation is actually equivalent to rounding towards 0, so that's only five after all and all of them are documented. Wasn't thinking about that.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

szbnwer@gmail.com
hi there! :)

`('%a'):format(yourNumber)` can give some enlightening results. it can
also be nice for serialization, but note that it doesnt work on 5.1
(however luajit supports it)

(((ps: thx, ive learned something new today as well, "read the
fri*ndly manual" X'D )))

all the bests to all of you! :)

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Tim Hill
In reply to this post by Coda Highland


On Sep 18, 2019, at 6:24 PM, Coda Highland <[hidden email]> wrote:



On Wed, Sep 18, 2019 at 12:21 PM Coda Highland <[hidden email]> wrote:
However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

It occurs to me that truncating the decimal representation is actually equivalent to rounding towards 0, so that's only five after all and all of them are documented. Wasn't thinking about that.

/s/ Adam

Not for negative numbers.
Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Coda Highland


On Wed, Sep 18, 2019 at 3:06 PM Tim Hill <[hidden email]> wrote:


On Sep 18, 2019, at 6:24 PM, Coda Highland <[hidden email]> wrote:



On Wed, Sep 18, 2019 at 12:21 PM Coda Highland <[hidden email]> wrote:
However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

It occurs to me that truncating the decimal representation is actually equivalent to rounding towards 0, so that's only five after all and all of them are documented. Wasn't thinking about that.

/s/ Adam

Not for negative numbers.

Yes for negative numbers. If you write -1.23456, and you truncate that representation (e.g. to -1.23), then you have rounded towards zero.

Besides, if it WERE different for negative numbers, then it would be equivalent to rounding towards -inf instead, which would still mean it's not a distinct rounding mode.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Tim Hill


On Sep 18, 2019, at 9:12 PM, Coda Highland <[hidden email]> wrote:



On Wed, Sep 18, 2019 at 3:06 PM Tim Hill <[hidden email]> wrote:


On Sep 18, 2019, at 6:24 PM, Coda Highland <[hidden email]> wrote:



On Wed, Sep 18, 2019 at 12:21 PM Coda Highland <[hidden email]> wrote:
However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

It occurs to me that truncating the decimal representation is actually equivalent to rounding towards 0, so that's only five after all and all of them are documented. Wasn't thinking about that.

/s/ Adam

Not for negative numbers.

Yes for negative numbers. If you write -1.23456, and you truncate that representation (e.g. to -1.23), then you have rounded towards zero.

Besides, if it WERE different for negative numbers, then it would be equivalent to rounding towards -inf instead, which would still mean it's not a distinct rounding mode.

/s/ Adam

Lol you’re right. What WAS I drinking last night? :)

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Thomas Jenni
From a decimal perspective the binary rounding seems strange. 
Especially if the representation is not exact, one has to trust 
the hidden "round to nearest“ function as one can see in the 
following example.

-- test
n={6.05,6.15,6.25,6.35,6.45,6.55,6.65,6.75,6.85,6.95}

for i = 1,#n do
x = string.format("%.1f",n[i])
print(n[i].." "..x.." "..(x-n[i]))
end

-- output
6.05 6.0 -0.05
6.15 6.2 0.05
6.25 6.2 -0.05
6.35 6.3 -0.05
6.45 6.5 0.05
6.55 6.5 -0.05
6.65 6.7 0.05
6.75 6.8 0.05
6.85 6.8 -0.05
6.95 7.0 0.05

Nevertheless it is the right way to round with as less bias as possible.
So i agree with you, that it's a feature and not a bug. Thx for the 
patience and explanations. 

Thomas

Am 18.09.2019 um 23:55 schrieb Tim Hill <[hidden email]>:



On Sep 18, 2019, at 9:12 PM, Coda Highland <[hidden email]> wrote:



On Wed, Sep 18, 2019 at 3:06 PM Tim Hill <[hidden email]> wrote:


On Sep 18, 2019, at 6:24 PM, Coda Highland <[hidden email]> wrote:



On Wed, Sep 18, 2019 at 12:21 PM Coda Highland <[hidden email]> wrote:
However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

It occurs to me that truncating the decimal representation is actually equivalent to rounding towards 0, so that's only five after all and all of them are documented. Wasn't thinking about that.

/s/ Adam

Not for negative numbers.

Yes for negative numbers. If you write -1.23456, and you truncate that representation (e.g. to -1.23), then you have rounded towards zero.

Besides, if it WERE different for negative numbers, then it would be equivalent to rounding towards -inf instead, which would still mean it's not a distinct rounding mode.

/s/ Adam

Lol you’re right. What WAS I drinking last night? :)

—Tim


Reply | Threaded
Open this post in threaded view
|

helping Was: bug report: string.format rounding error

Andrew Starks-2
In reply to this post by Kat Kioo
[off topic for Lua]

This discussion reminded me of a conversation on the Ezra Klein Show with XKCD author Randall Munroe. 


He talks at length about how it’s best to assume that a) people are really smart and not lazy and b) people don’t have the context for the problem their trying to solve. The mode in their head is leading them astray because of some flaw in their thinking early on. 

Anyway, I’d like to be more like Mr. Munroe after listening to this podcast. 

- Andrew Starks

On Sep 18, 2019, at 06:21, Meepen <[hidden email]> wrote:

"Hey why does this happen?"
*Hands 44 page manual to other person without explaining anything*

On Wed, Sep 18, 2019, 3:36 AM <[hidden email]> wrote:
On 2019-09-18 09:47, Thomas Jenni wrote:
> In Lua 5.3.5 on a 64-bit machine (macOS 10.14.6), the following two
> lines produce different results
>
> print(string.format("%.2f",1.625000000000001))
> 1.63
>
> print(string.format("%.2f",1.625000000000000))
> 1.62
>
> It seems unclear to me, why in the latter case 1.625 is rounded to
> 1.62.
> Thanks for fixing this bug.
>
> best regards
>
> T. Jenni

https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf

Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Coda Highland
In reply to this post by Thomas Jenni


On Thu, Sep 19, 2019 at 2:24 AM Thomas Jenni <[hidden email]> wrote:
From a decimal perspective the binary rounding seems strange. 
Especially if the representation is not exact, one has to trust 
the hidden "round to nearest“ function as one can see in the 
following example.

-- test
n={6.05,6.15,6.25,6.35,6.45,6.55,6.65,6.75,6.85,6.95}

for i = 1,#n do
x = string.format("%.1f",n[i])
print(n[i].." "..x.." "..(x-n[i]))
end

-- output
6.05 6.0 -0.05
6.15 6.2 0.05
6.25 6.2 -0.05
6.35 6.3 -0.05
6.45 6.5 0.05
6.55 6.5 -0.05
6.65 6.7 0.05
6.75 6.8 0.05
6.85 6.8 -0.05
6.95 7.0 0.05

Nevertheless it is the right way to round with as less bias as possible.
So i agree with you, that it's a feature and not a bug. Thx for the 
patience and explanations. 

Thomas

I'll point out that we haven't actually discussed the solution to this:

If you don't like the rounding mode, then round it yourself -- leave string.formatting for string formatting, and do the math using whatever other tools you like. The standard way to do this would be something like math.floor(x * 100 + 0.5) / 100 if you want to round to two decimal places with ties rounding towards positive infinity. You'll have to fiddle a bit more than that if you specifically want to round away from zero, but it's not egregiously bad.

There are also plenty of third-party libraries that offer rounding functions, and it's not uncommon at all for Lua embedders to explicitly set the rounding mode in C code before starting the interpreter.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: bug report: string.format rounding error

Thomas Jenni
I wrote a lua library that can handle physical units and uncertainty propagation. 


I use lualatex for generating the solutions for textbook physics problems. I have to think about what the best solution is.

In python there exists a decimal module, which does the decimal rounding correctly. The tradeoff is, that its slower than native computations. 

Thomas 


Am 19.09.2019 um 15:48 schrieb Coda Highland <[hidden email]>:



On Thu, Sep 19, 2019 at 2:24 AM Thomas Jenni <[hidden email]> wrote:
From a decimal perspective the binary rounding seems strange. 
Especially if the representation is not exact, one has to trust 
the hidden "round to nearest“ function as one can see in the 
following example.

-- test
n={6.05,6.15,6.25,6.35,6.45,6.55,6.65,6.75,6.85,6.95}

for i = 1,#n do
x = string.format("%.1f",n[i])
print(n[i].." "..x.." "..(x-n[i]))
end

-- output
6.05 6.0 -0.05
6.15 6.2 0.05
6.25 6.2 -0.05
6.35 6.3 -0.05
6.45 6.5 0.05
6.55 6.5 -0.05
6.65 6.7 0.05
6.75 6.8 0.05
6.85 6.8 -0.05
6.95 7.0 0.05

Nevertheless it is the right way to round with as less bias as possible.
So i agree with you, that it's a feature and not a bug. Thx for the 
patience and explanations. 

Thomas

I'll point out that we haven't actually discussed the solution to this:

If you don't like the rounding mode, then round it yourself -- leave string.formatting for string formatting, and do the math using whatever other tools you like. The standard way to do this would be something like math.floor(x * 100 + 0.5) / 100 if you want to round to two decimal places with ties rounding towards positive infinity. You'll have to fiddle a bit more than that if you specifically want to round away from zero, but it's not egregiously bad.

There are also plenty of third-party libraries that offer rounding functions, and it's not uncommon at all for Lua embedders to explicitly set the rounding mode in C code before starting the interpreter.

/s/ Adam