On Fri, Feb 12, 2021 at 10:08 PM Lux Lang <

[hidden email]> wrote:

>

> Hello, everyone.

>

> There's an issue that bothers me with regards to float-to-string formatting in Lua.

>

> It appears to be imprecise, unless you're doing hexadecimal formatting.

>

> I'm getting these results in Lua 5.4.2

>

> > tostring(0x1.a8e64000018dp-1)

> 0.82988166809153

> > 0x1.a8e64000018dp-1 == tonumber(tostring(0x1.a8e64000018dp-1))

> false

> > 0x1.a8e64000018dp-1 == tonumber(string.format("%f", 0x1.a8e64000018dp-1))

> false

> > 0x1.a8e64000018dp-1 == tonumber(string.format("%e", 0x1.a8e64000018dp-1))

> false

> > 0x1.a8e64000018dp-1 == tonumber(string.format("%a", 0x1.a8e64000018dp-1))

> true

You're not specifying the precision. Without an explicit precision,

Lua (or more probably, the underlying C library) guesses a sensible

default based on the assumption that the resulting string is going to

be printed. That assumption on my system is around 6 decimal places,

which is well less than what is required to maintain accuracy on an

IEEE double-precision floating point number (Lua's default float

type). You generally need 15-17 significant digits in decimal to

correctly round-trip between IEEE doubles and the decimal

representation of that double.

In your case:

> 0x1.a8e64000018dp-1 == tonumber(string.format("%.16f", 0x1.a8e64000018dp-1))

true

> 0x1.a8e64000018dp-1 == tonumber(string.format("%.16e", 0x1.a8e64000018dp-1))

true

In this case, 16 digits of precision does the trick. Depending on your

value, more or fewer digits may be required.

Why do you need so much precision? Because IEEE floating-point numbers

(regardless of precision) are stored in binary (base-2), not decimal

(base-10). Binary cannot exactly represent many (most, in fact)

fractional decimal numbers, so a seemingly simple decimal number like

0.1 has a lengthy and inexact representation in IEEE floating-point

numbers, leading to this classic illustration of the pitfalls of IEEE

floats:

> 0.1 + 0.1 + 0.1 == 0.3

false

Yes, this is from the Lua REPL, and no, it is not a bug in Lua. It is,

however, an illustration of why floating point numbers should not be

relied on for critical applications where perfect precision with

fractional numbers is necessary (such as financial applications that

deal with cents and other fractional currency units). In those cases,

either use a library that offers arbitrary-precision decimal math, or

rely solely on integers (e.g. calculations involving US dollars can be

done safely by multiplying the dollars by 100 and doing all

calculations in integer cents, dividing by 100 only when needed for

display).