Hi, I am using KDB+ 3.5, and I have a question about the accuracy of “sum”.
For example, I have a list of float number, and the actual sum value is 1699950.92. But when I use KDB+ to calculate the sum value, it shows 1699951f.
The code shows below:
test:(145570 621250 229750 191143.2 461635.4 -145570 -325572 149744.5 69962.85 115413.9 -1050000 353084.2 61710 -175000 675667.5 321161.3);
sum test
How could I solve this problem?
Hi Tongwei,
The number is being calculated correctly; increasing the display precision will show the whole number
// Default precision
q)\P
7i
q)sum test
1699951f
q)\P 10
q)sum test
1699950.85
‘-P x’ on the command line will set this at startup, where x is required precision.
Thanks
Hi,
Its display problem and not the calculation one. Change your display precision using command \P. By default it is set to 7.
Ex:
\P 14
sum test
/ output 1699950.85
More details on wiki: http://code.kx.com/q/ref/cmdline/#-p-display-precision
Thanks for the answers, I do appreciate your time.
? 2017?11?29??? UTC+8??7:12:30?Tongwei Liu???
Hi, I am using KDB+ 3.5, and I have a question about the accuracy of “sum”.
For example, I have a list of float number, and the actual sum value is 1699950.92. But when I use KDB+ to calculate the sum value, it shows 1699951f.
The code shows below:
test:(145570 621250 229750 191143.2 461635.4 -145570 -325572 149744.5 69962.85 115413.9 -1050000 353084.2 61710 -175000 675667.5 321161.3);
sum test
How could I solve this problem?