Have I or Plua gone mad (math?)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Have I or Plua gone mad (math?)

Morten Agerlin Petersen (MAP)
I am really mystified - when running the test program below in
Plua2.0 both on my Palm Z22 and in Palmsim on my Windows PC I get the
results:

low1: 22.752188171 (it should be 18.457220875)
low2: 18.409012

I have no use for all the decimals but I find it worrying if Plua can
go that wrong?

It's generally a very nice program!

Regards
Morten

-- Calculation error.lua
low1=256*0.000777193+16*1.623011077-7.709917765
low2=256*0.000777   +16*1.62       -7.7099

print(low1)
print(low2)
gui.event()


Reply | Threaded
Open this post in threaded view
|

Re: Have I or Plua gone mad (math?)

ygfz4v902
Plua appears to use some form of fixed point for calculations (with
32-bit integers), and that amount of decimal places causes an overflow.

print(17.179869184) => 0
print(17.179869183) => 4.294967295

Without the decimal point, 4294967295 == 2^32-1. Also, 17.179869184/4
== 4.294967296, FWIW.

Checking with the palm's calculator, this appears to be Plua's fault,
not PalmOS's.

Plua 2.0b10, Palm Z31, PalmOS v5.2.8

--- In [hidden email], "m_agerlin" <map@...> wrote:

>
> I am really mystified - when running the test program below in
> Plua2.0 both on my Palm Z22 and in Palmsim on my Windows PC I get the
> results:
>
> low1: 22.752188171 (it should be 18.457220875)
> low2: 18.409012
>
> I have no use for all the decimals but I find it worrying if Plua can
> go that wrong?
>
> It's generally a very nice program!
>
> Regards
> Morten
>
> -- Calculation error.lua
> low1=256*0.000777193+16*1.623011077-7.709917765
> low2=256*0.000777   +16*1.62       -7.7099
>
> print(low1)
> print(low2)
> gui.event()
>