lua_number2int

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

lua_number2int

Brian Weed-2
Today, we converted a project from Lua v5.02 to v5.1.1, and we started having a problem when converting double to int. I have narrowed my problem down to code that has nothing to do with Lua. But I still wanted to get other people's opinions about it.

double d = 0xFFFFFFFF;
int n;
n = (int)d;

// n now contains 0x80000000

n = (int)(unsigned int)d;

// n now contains 0xFFFFFFFF

(this happens on both VS2005, and GCC 4.0.1 (Apple Computer, Inc. build 5363))

Since lua_number2int and lua_number2integer both basically do this: "n = (int)d;"

Wouldn't it be safer to do this: "n = (int)(unsigned int)d;" so as to preserve the values that would otherwise be truncated ?

What sort of problems would this cause?  Is my fix unportable?

The initial problem we got stems from doing this in Lua:

d = 0xFFFFFFFF
Foo(d)

// Assuming Foo() is something like this...

int bind_Foo(lua_State *L)
{
Foo(lua_tointeger(L, 1)); // value passed to Foo() is now truncated to 0x80000000

   return(0);
}

Brian





Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Given
Brian Weed wrote:
> Today, we converted a project from Lua v5.02 to v5.1.1, and we started
> having a problem when converting double to int.
> I have narrowed my problem down to code that has nothing to do with
> Lua.  But I still wanted to get other people's opinions about it.
> 
> double d = 0xFFFFFFFF;
> int n;
> n = (int)d;
> 
> // n now contains 0x80000000

On my system (Ubuntu Edgy Linux, gcc 4.1.2) n now containts 0x7FFFFFFF, which
is what I would consider to be correct.

> n = (int)(unsigned int)d;
> 
> // n now contains 0xFFFFFFFF

I get this too. I also consider this to be correct, because casting d to an
unsigned int will fit, and will therefore not truncate the value; and casting
an unsigned int to an int doesn't truncate (which I've never really understood).

-- 
âââDavid GivenâââMcQââ "A line dancer near a graduated cylinder, the
ââ [hidden email]âââââ blithe spirit inside some wheelbarrow, and a tomato
ââ([hidden email])ââ are what made America great!" --- received via spam
âââwww.cowlark.comââââ

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

Asko Kauppi
In reply to this post by Brian Weed-2

There may be several abnormal things regarding the normal Lua integer handling.

What I find most weird (met in making the "integer patches") is that floating point values are silently converted into integers by 'lua_tointeger()'. I would consider this a BUG, since if a C routine specifically requests an integer I think it should be provided one. Anyways, that behaviour remains in the "integer patch" so that outside behaviour remains unaltered.

Could we have this changed, in say 5.2?

Also "lua_tounsigned()" would be handy.

-asko


On Mon, 13 Nov 2006 21:52:56 -0500
 "Brian Weed" <[hidden email]> wrote:
Today, we converted a project from Lua v5.02 to v5.1.1, and we started having a problem when converting double to int. I have narrowed my problem down to code that has nothing to do with Lua. But I still wanted to get other people's opinions about it.

double d = 0xFFFFFFFF;
int n;
n = (int)d;

// n now contains 0x80000000

n = (int)(unsigned int)d;

// n now contains 0xFFFFFFFF

(this happens on both VS2005, and GCC 4.0.1 (Apple Computer, Inc. build 5363))

Since lua_number2int and lua_number2integer both basically do this: "n = (int)d;"

Wouldn't it be safer to do this: "n = (int)(unsigned int)d;" so as to preserve the values that would otherwise be truncated ?

What sort of problems would this cause? Is my fix unportable?

The initial problem we got stems from doing this in Lua:

d = 0xFFFFFFFF
Foo(d)

// Assuming Foo() is something like this...

int bind_Foo(lua_State *L)
{
Foo(lua_tointeger(L, 1)); // value passed to Foo() is now truncated to 0x80000000

   return(0);
}

Brian






Reply | Threaded
Open this post in threaded view
|

RE: lua_number2int

Jerome Vuarand-2
Here I get this:

% gcc --version
gcc (GCC) 3.4.4 (mingw special)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

% cat test.c
#include <stdio.h>

int main()
{
	printf("0x%08x\n", (int)(double)0xffffffff);
	printf("0x%08x\n", (int)(unsigned int)(double)0xffffffff);
	printf("%d\n", (int)(double)-4.0);
	printf("%d\n", (int)(unsigned int)(double)-4.0);
	return 0;
}
% make test
% test
0x7fffffff
0xffffffff
-4
0

As you can see with your additionnal unsigned int cast you lose all negative values. That's because (unsigned int)0xffffffff and (int)0xffffffff have two different representations in double precision floating point format. 

> -----Message d'origine-----
> De : [hidden email] 
> [[hidden email]] De la part de 
> [hidden email]
> Envoyé : 14 novembre 2006 07:21
> À : Lua list
> Objet : Re: lua_number2int
> 
> 
> There may be several abnormal things regarding the normal Lua 
> integer handling.
> 
> What I find most weird (met in making the "integer
> patches") is that floating point values are silently 
> converted into integers by 'lua_tointeger()'.  I would 
> consider this a BUG, since if a C routine specifically 
> requests an integer I think it should be provided one. 
> Anyways, that behaviour remains in the "integer patch" so 
> that outside behaviour remains unaltered.
> 
> Could we have this changed, in say 5.2?
> 
> Also "lua_tounsigned()" would be handy.
> 
> -asko
> 
> 
> On Mon, 13 Nov 2006 21:52:56 -0500
>   "Brian Weed" <[hidden email]> wrote:
> > Today, we converted a project from Lua v5.02 to v5.1.1, and 
> we started 
> >having a problem when converting double to int.
> > I have narrowed my problem down to code that has nothing to do with 
> >Lua.  But I still wanted to get other people's opinions about it.
> > 
> > double d = 0xFFFFFFFF;
> > int n;
> > n = (int)d;
> > 
> > // n now contains 0x80000000
> > 
> > n = (int)(unsigned int)d;
> > 
> > // n now contains 0xFFFFFFFF
> > 
> > (this happens on both VS2005, and GCC 4.0.1 (Apple Computer, Inc. 
> >build 5363))
> > 
> > Since lua_number2int and lua_number2integer both basically 
> do this:  
> >"n = (int)d;"
> > 
> > Wouldn't it be safer to do this: "n = (int)(unsigned 
> int)d;" so as to 
> >preserve the values that would otherwise be truncated ?
> > 
> > What sort of problems would this cause?  Is my fix unportable?
> > 
> > The initial problem we got stems from doing this in Lua:
> > 
> > d = 0xFFFFFFFF
> >Foo(d)
> > 
> > // Assuming Foo() is something like this...
> > 
> > int bind_Foo(lua_State *L)
> > {
> >    Foo(lua_tointeger(L, 1));   // value passed to Foo() 
> >is now
> > truncated to 0x80000000
> > 
> >    return(0);
> > }
> > 
> > Brian
> > 
> > 
> > 
> > 
> 
> 


Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

Brian Weed-2
Jerome Vuarand wrote:
Here I get this:

% gcc --version
gcc (GCC) 3.4.4 (mingw special)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

% cat test.c
#include <stdio.h>

int main()
{
	printf("0x%08x\n", (int)(double)0xffffffff);
	printf("0x%08x\n", (int)(unsigned int)(double)0xffffffff);
	printf("%d\n", (int)(double)-4.0);
	printf("%d\n", (int)(unsigned int)(double)-4.0);
	return 0;
}
% make test
% test
0x7fffffff
0xffffffff
-4
0

As you can see with your additionnal unsigned int cast you lose all negative values. That's because (unsigned int)0xffffffff and (int)0xffffffff have two different representations in double precision floating point format.

I get different results in VS2005...

0xffffffff
0xffffffff
-4
-4

Perhaps there is a compiler setting for choosing the float rounding technique?

Brian

Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Given
Brian Weed wrote:
> Jerome Vuarand wrote:
[...]
>> % make test
>> % test
>> 0x7fffffff
>> 0xffffffff
[...]
> I get different results in VS2005...
> 
> 0xffffffff
> 0xffffffff

You know, I've just had a horrible thought.

Does the C standard even *allow* you to cast an unsigned int to an int if the
value won't fit? Doesn't the result depend on your sign representation?
(0xFFFFFFFF == -1 only on two's-complement machines.) Are we seeing the
dreaded 'undefined behaviour'?

[rummages through the 'net]

Yes, I was right. This is from WG14 N1124, which is a reasonably current C99.

http://www.open-std.org/jtc1/sc22/wg14/www/standards

âââââ

6.3.1.3 Signed and unsigned integers

Â1 When a value with integer type is converted to another integer type other
than _Bool, if the value can be represented by the new type, it is unchanged.
Â2 Otherwise, if the new type is unsigned, the value is converted by
repeatedly adding or subtracting one more than the maximum value that can be
represented in the new type until the value is in the range of the new type.
Â3 Otherwise, the new type is signed and the value cannot be represented in
it; either the result is implementation-deïned or an implementation-deïned
signal is raised.

6.3.1.4 Real ïoating and integer

Â1 When a ïnite value of real ïoating type is converted to an integer type
other than _Bool, the fractional part is discarded (i.e., the value is
truncated toward zero). If the value of the integral part cannot be
represented by the integer type, the behavior is undeïned.)
Â2 When a value of integer type is converted to a real ïoating type, if the
value being converted can be represented exactly in the new type, it is
unchanged. If the value being converted is in the range of values that can be
represented but cannot be represented exactly, the result is either the
nearest higher or nearest lower representable value, chosen in an
implementation-deïned manner. If the value being converted is outside the
range of values that can be represented, the behavior is undeïned.

âââââ

Therefore, casting (double)0xFFFFFFFF to an unsigned int is valid, but it's
*not* valid to cast it to an int. Casting it to an unsigned int and *then* to
an int is valid, and will result in 0xFFFFFFF.

Anything else invoces undefined behaviour, and results in daemons flying out
of your nose.

-- 
âââDavid GivenâââMcQââ "...electrons, nuclei and other particles are good
ââ [hidden email]âââââ approximations to perfectly elastic spherical
ââ([hidden email])ââ cows." --- David M. Palmer on r.a.sf.c
âââwww.cowlark.comââââ

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

RE: lua_number2int

Jerome Vuarand-2
David Given wrote:
> Therefore, casting (double)0xFFFFFFFF to an unsigned int is 
> valid, but it's
> *not* valid to cast it to an int. Casting it to an unsigned 
> int and *then* to an int is valid, and will result in 0xFFFFFFF.
> 
> Anything else invoces undefined behaviour, and results in 
> daemons flying out of your nose.

lua_Integer is a typedef for ptrdiff_t, which must be signed. So for
safe extraction of unsigned integers in stock Lua there may be a need
for a lua_UnsignedInteger typedef, and a lua_tounsignedinteger function.
Maybe lua_Unsigned and lua_tounsigned would be clear enough.


Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

Asko Kauppi

Me, yesterday, 14:21

Also "lua_tounsigned()" would be handy.

:)  imho, the whole integer support should be revised.


Jerome Vuarand kirjoitti 15.11.2006 kello 0.13:

David Given wrote:
Therefore, casting (double)0xFFFFFFFF to an unsigned int is
valid, but it's
*not* valid to cast it to an int. Casting it to an unsigned
int and *then* to an int is valid, and will result in 0xFFFFFFF.

Anything else invoces undefined behaviour, and results in
daemons flying out of your nose.

lua_Integer is a typedef for ptrdiff_t, which must be signed. So for
safe extraction of unsigned integers in stock Lua there may be a need
for a lua_UnsignedInteger typedef, and a lua_tounsignedinteger function.
Maybe lua_Unsigned and lua_tounsigned would be clear enough.


Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

Roberto Ierusalimschy
In reply to this post by David Given
> Â1 When a ïnite value of real ïoating type is converted to an integer type
> other than _Bool, the fractional part is discarded (i.e., the value is
> truncated toward zero). If the value of the integral part cannot be
> represented by the integer type, the behavior is undeïned.)
> [...]
> Therefore, casting (double)0xFFFFFFFF to an unsigned int is valid, [...]

But it seems that casting (double)-1 to an unsigned int will result in
daemons flying out of our noses.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

RE: lua_number2int

Jerome Vuarand-2
In reply to this post by Asko Kauppi
Asko Kauppi wrote:
> Me, yesterday, 14:21
> 
> > Also "lua_tounsigned()" would be handy.
> 
> :)  imho, the whole integer support should be revised.

Maybe the best solution is to remove completely integer support. Keep
one type to contain them all. No more lua_Integer or lua_tointeger. Keep
Lua high level and let the user convert the doubles to whatever fits the
C code. tointeger/tounsigned are easy to write above tonumber. As the
thread as shown, default ones are error prone, may cross C definition
boundaries and lead to undefined behaviour (and daemons invasions :-) ).



Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Given
In reply to this post by Roberto Ierusalimschy
Roberto Ierusalimschy wrote:
[...]
> But it seems that casting (double)-1 to an unsigned int will result in
> daemons flying out of our noses.

>From my understanding of the specification, that's correct. It will.

It just so happens that on all platforms and all compilers that I know of it
produces the same result as (int)(unsigned int)(double)-1, but that's
coincidental.

(You know, it would be rather informative some time to put together a C
compiler which actually *checked* for all these undefined behaviour cases. I
suspect we might learn something...)

-- 
âââDavid GivenâââMcQââ "...electrons, nuclei and other particles are good
ââ [hidden email]âââââ approximations to perfectly elastic spherical
ââ([hidden email])ââ cows." --- David M. Palmer on r.a.sf.c
âââwww.cowlark.comââââ

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

Brian Weed-2
David Given wrote:
Roberto Ierusalimschy wrote:
[...]
But it seems that casting (double)-1 to an unsigned int will result in
daemons flying out of our noses.

>From my understanding of the specification, that's correct. It will.

It just so happens that on all platforms and all compilers that I know of it
produces the same result as (int)(unsigned int)(double)-1, but that's
coincidental.

(You know, it would be rather informative some time to put together a C
compiler which actually *checked* for all these undefined behaviour cases. I
suspect we might learn something...)

So, to be safe, maybe I should do this?

#define lua_number2integer(i, d) ((d) < 0 ? (i) = (int)(d) : (i) = (int)(unsigned int)(d))

Brian

Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Given
Brian Weed wrote:
[...]
> So, to be safe, maybe I should do this?
> 
> #define lua_number2integer(i, d) ((d) < 0 ? (i) = (int)(d) : (i) =
> (int)(unsigned int)(d))

Actually, I think the only way to be completely safe would be to do:


int lua_number2integer_f(double d)
{
	if (d < INT_MIN)
		return INT_MIN;
	if (d > INT_MAX)
		return INT_MAX;
	return (int)d;
}

#define lua_number2integer(i, d) i=lua_number2integer_f(d)


...but that's rather a mouthful, and of course you'd need a completely
separate version for unsigned ints.

-- 
âââDavid GivenâââMcQââ "...electrons, nuclei and other particles are good
ââ [hidden email]âââââ approximations to perfectly elastic spherical
ââ([hidden email])ââ cows." --- David M. Palmer on r.a.sf.c
âââwww.cowlark.comââââ

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

RE: lua_number2int

Grellier, Thierry
In reply to this post by Jerome Vuarand-2
Well, another solution may be to introduce an unsigned type with bit
wise operators besides arithmetic operators aiming at representing
addresses and accepting hexadecimal literals only... 
I would be really happy with that.
This would also allow simplifying literal support for current number,
and this address type could be also optional in luaconf.h for those not
willing to pay the overhead of a new type.

-----Original Message-----
From: [hidden email]
[[hidden email]] On Behalf Of Jerome Vuarand
Sent: Wednesday, November 15, 2006 12:37 AM
To: Lua list
Subject: RE: lua_number2int

Asko Kauppi wrote:
> Me, yesterday, 14:21
> 
> > Also "lua_tounsigned()" would be handy.
> 
> :)  imho, the whole integer support should be revised.

Maybe the best solution is to remove completely integer support. Keep
one type to contain them all. No more lua_Integer or lua_tointeger. Keep
Lua high level and let the user convert the doubles to whatever fits the
C code. tointeger/tounsigned are easy to write above tonumber. As the
thread as shown, default ones are error prone, may cross C definition
boundaries and lead to undefined behaviour (and daemons invasions :-) ).



Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Jones-2
In reply to this post by David Given

On 15 Nov 2006, at 00:39, David Given wrote:

Brian Weed wrote:
[...]
So, to be safe, maybe I should do this?

#define lua_number2integer(i, d) ((d) < 0 ? (i) = (int)(d) : (i) =
(int)(unsigned int)(d))

Actually, I think the only way to be completely safe would be to do:


int lua_number2integer_f(double d)
{
	if (d < INT_MIN)
		return INT_MIN;
	if (d > INT_MAX)
		return INT_MAX;
	return (int)d;
}

#define lua_number2integer(i, d) i=lua_number2integer_f(d)


...but that's rather a mouthful, and of course you'd need a completely
separate version for unsigned ints.

I think you're right; this is the only safe way. I'm amazed that this has gone unnoticed. Of course, even with this solution it's _possible_ to construct a perverse C implementation where it would be undefined behavious (if INT_MAX was huge enough to be outside the range of double).

drj

Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Given
David Jones wrote:
[...]
> I think you're right; this is the only safe way.  I'm amazed that this
> has gone unnoticed.  Of course, even with this solution it's _possible_
> to construct a perverse C implementation where it would be undefined
> behavious (if INT_MAX was huge enough to be outside the range of double).

Actually, talking to someone who knows more about floating-point
representations than I do: the Evil Hack to convert a double to an int, viz.:

#define lua_number2int(i,d) \
  { volatile union luai_Cast u; u.l_d = (d) + 6755399441055744.0; (i) = u.l_l; }

...should produce the 'right' results --- i.e. (double)-1 == (int)-1 ==
(unsigned int)0xFFFFFFFF --- everywhere where the hack does work, because the
hack relies on a particular binary representation. It's only when you do
things correctly that it will start to fail. You're on a Mac, right, Brian? If
so, you won't be using the hack.

Actually, I think that the better way to do things is to allow lua_tointeger()
to fail if the conversion won't work; that is, if the value is outside
INT_MIN..INT_MAX or not a valid number (such as NaN or +Inf or -Inf). It's a
pity that it's too late to change this now, and it'd probably be too slow
anyway...

-- 
âââDavid GivenâââMcQââ "A line dancer near a graduated cylinder, the
ââ [hidden email]âââââ blithe spirit inside some wheelbarrow, and a tomato
ââ([hidden email])ââ are what made America great!" --- received via spam
âââwww.cowlark.comââââ

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

Brian Weed-2
David Given wrote:
David Jones wrote:
[...]
I think you're right; this is the only safe way.  I'm amazed that this
has gone unnoticed.  Of course, even with this solution it's _possible_
to construct a perverse C implementation where it would be undefined
behavious (if INT_MAX was huge enough to be outside the range of double).

Actually, talking to someone who knows more about floating-point
representations than I do: the Evil Hack to convert a double to an int, viz.:

#define lua_number2int(i,d) \
  { volatile union luai_Cast u; u.l_d = (d) + 6755399441055744.0; (i) = u.l_l; }

...should produce the 'right' results --- i.e. (double)-1 == (int)-1 ==
(unsigned int)0xFFFFFFFF --- everywhere where the hack does work, because the
hack relies on a particular binary representation. It's only when you do
things correctly that it will start to fail. You're on a Mac, right, Brian? If
so, you won't be using the hack.

The project in question is both PC, and Mac.

"n = (int)(unsigned int)d;" worked on the PC, but not the Mac (Intel) with negative values.

It seems as though redefining lua_Integer to be 64bit would solve this for all platforms, but Lua has too many implicit casts from lua_Integer to int, so I abandoned that change (the more mods I make to Lua, the less clean it feels).

I'm also thinking that I may have to re-think our entire use of flags, and bitfields, since we have other projects on the PS2 where lua_Number is a float, and therefore can't hold all 32bit integer values.

Brian


Reply | Threaded
Open this post in threaded view
|

Re[2]: lua_number2int

Gunnar Zötl
In reply to this post by David Given
Hi,

David Given wrote:
DG> [interesting stuff]

DG> Anything else invoces undefined behaviour, and results in daemons flying out
DG> of your nose.

boy, I gotta try that one day... ;-)

Gunnar




Reply | Threaded
Open this post in threaded view
|

Re: lua_number2int

David Given
Gunnar ZÃtl wrote:
[...]
> DG> Anything else invoces undefined behaviour, and results in daemons flying out
> DG> of your nose.
> 
> boy, I gotta try that one day... ;-)

It's a vitally important phrase when discussing anything related to C.

http://www.catb.org/jargon/html/N/nasal-demons.html

(Lua doesn't summon daemons, but it still has its own share of 'undefined
behaviour' issues as well. I think it would be amazingly handy to be able to
test for these in a debug situation and trap them as errors. For example, that
hoary old array-with-holes-in business --- I'm quite certain there's
production code out there that suffers from that.)

-- 
âââDavid GivenâââMcQââ "...electrons, nuclei and other particles are good
ââ [hidden email]âââââ approximations to perfectly elastic spherical
ââ([hidden email])ââ cows." --- David M. Palmer on r.a.sf.c
âââwww.cowlark.comââââ

Attachment: signature.asc
Description: OpenPGP digital signature

Reply | Threaded
Open this post in threaded view
|

Is there a C API equivalent to the new # operator for tables?

Ram Firestone
In reply to this post by Gunnar Zötl
I think the title says it all, but just to make sure I would like to know if there is a C API function (or another easy way) to get
the length of a table that is being used as an array. Indexing will start at one and there will be no holes. I didn't see anything
in the documentation but I wanted to make sure I didn't miss anything. I am looking for a method that is reasonably fast. 




12