datatype conversion

Community Support
1

Hi there,

I happened to find a interesting problem when dealing with type conversion. Please see my code below (result marked in red)

(2 xexp 60)- (2 xexp 60) + 16f

0f

        (2 xexp 10)- (2 xexp 10) + 16f

-16f

(7h$(2 xexp 60))- (7h$(2 xexp 60)) + 16f

0f

(7h$(2 xexp 60))- (7h$(2 xexp 60)) + 16

-16j

        

        (7h$(2 xexp 10))- (7h$(2 xexp 10)) + 16f

-16f


From my understanding, all the code above should return a value of -16 regardless of datatypes. However the first and third line of code obviously return a different result.


It seems that type conversion on big numbers will somehow result in loss of accuracy?

Could someone explain this to me? Hope it is not a stupid mistake made by me.




Thanks,


Xinyu


2

q)-8!(2 xexp 60)

0x0100000011000000f7efffffffffffaf43

q)-8!(2 xexp 60)+16f

0x0100000011000000f7efffffffffffaf43

ieee754 float > ~2^53 are not represented precisely.

http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

3

ahh that’s pretty clear, thanks! 

Xinyu