Hello, everyone. There's an issue that bothers me with regards to float-to-string formatting in Lua. It appears to be imprecise, unless you're doing hexadecimal formatting. I'm getting these results in Lua 5.4.2 > tostring(0x1.a8e64000018dp-1) 0.82988166809153 > 0x1.a8e64000018dp-1 == tonumber(tostring(0x1.a8e64000018dp-1)) false > 0x1.a8e64000018dp-1 == tonumber(string.format("%f", 0x1.a8e64000018dp-1)) false > 0x1.a8e64000018dp-1 == tonumber(string.format("%e", 0x1.a8e64000018dp-1)) false > 0x1.a8e64000018dp-1 == tonumber(string.format("%a", 0x1.a8e64000018dp-1)) true There seems to be some loss of precision in non-hex formatting. Is this something fixable? Is there some way for me (in Lua) to format floats to decimal strings and not lose precision? |
On Fri, Feb 12, 2021 at 10:08 PM Lux Lang <[hidden email]> wrote:
> > Hello, everyone. > > There's an issue that bothers me with regards to float-to-string formatting in Lua. > > It appears to be imprecise, unless you're doing hexadecimal formatting. > > I'm getting these results in Lua 5.4.2 > > > tostring(0x1.a8e64000018dp-1) > 0.82988166809153 > > 0x1.a8e64000018dp-1 == tonumber(tostring(0x1.a8e64000018dp-1)) > false > > 0x1.a8e64000018dp-1 == tonumber(string.format("%f", 0x1.a8e64000018dp-1)) > false > > 0x1.a8e64000018dp-1 == tonumber(string.format("%e", 0x1.a8e64000018dp-1)) > false > > 0x1.a8e64000018dp-1 == tonumber(string.format("%a", 0x1.a8e64000018dp-1)) > true You're not specifying the precision. Without an explicit precision, Lua (or more probably, the underlying C library) guesses a sensible default based on the assumption that the resulting string is going to be printed. That assumption on my system is around 6 decimal places, which is well less than what is required to maintain accuracy on an IEEE double-precision floating point number (Lua's default float type). You generally need 15-17 significant digits in decimal to correctly round-trip between IEEE doubles and the decimal representation of that double. In your case: > 0x1.a8e64000018dp-1 == tonumber(string.format("%.16f", 0x1.a8e64000018dp-1)) true > 0x1.a8e64000018dp-1 == tonumber(string.format("%.16e", 0x1.a8e64000018dp-1)) true In this case, 16 digits of precision does the trick. Depending on your value, more or fewer digits may be required. Why do you need so much precision? Because IEEE floating-point numbers (regardless of precision) are stored in binary (base-2), not decimal (base-10). Binary cannot exactly represent many (most, in fact) fractional decimal numbers, so a seemingly simple decimal number like 0.1 has a lengthy and inexact representation in IEEE floating-point numbers, leading to this classic illustration of the pitfalls of IEEE floats: > 0.1 + 0.1 + 0.1 == 0.3 false Yes, this is from the Lua REPL, and no, it is not a bug in Lua. It is, however, an illustration of why floating point numbers should not be relied on for critical applications where perfect precision with fractional numbers is necessary (such as financial applications that deal with cents and other fractional currency units). In those cases, either use a library that offers arbitrary-precision decimal math, or rely solely on integers (e.g. calculations involving US dollars can be done safely by multiplying the dollars by 100 and doing all calculations in integer cents, dividing by 100 only when needed for display). |
Free forum by Nabble | Edit this page |