Author Topic: Precision of the operation with integer and float  (Read 908 times)

Offline Qedo

  • Dr. Type
  • ****
  • Posts: 352
  • to program what I have todo how should I program?
    • View Profile
I write this because nobody ever said it (or at least I never read it anywhere).
When performing a number operation having integer variables and float together, pay attention to the order of the variables.
Example with the calculator the operation "911/5/15*2*1.1" returns 26.72266667
in GLbasic:
Aint% = 5, Bint% = 911, Cint% = 15, Dint% = 2, Mflt # = 1.1
1) Nflt# = Bint / Aint / Cint * Dint * Mflt    // returns 26.4    // WRONG
2) Nflt# = Mflt * (Bint / Aint / Cint * Dint)    // returns 26.4    // WRONG
3) Nflt# = Mflt * Bint / Aint / Cint * Dint    // returns 26.72266667    // OK

I think this depends on the order in which operations are performed from left to right. If the first variable is float the precision will remain with the comma and there will be no rounding to the integers.
I hope it is useful
Ciao

Offline dreamerman

  • Global Moderator
  • Dr. Type
  • *******
  • Posts: 401
    • View Profile
    • my personal website
Yeah this can cause some unexected behaviour (I overlooked such things so many times :D), and this applies in general to any programming language, it's always advised to use proper type casting in calculations.
In GLBasic it's easy to avoid, as You mentioned type order is important so all what you need to do to get proper ## result is add "1.0 *" before rest of calculation.
In C++ you will get same problems without proper type casting.
Code: (glbasic) [Select]
FUNCTION test_int_float%:
LOCAL ga% = 5, gb% = 911, gc% = 15, gd% = 2, gm# = 1.1
LOCAL result1#, result2#
INLINE
int a = 5, b = 911, c = 15, d = 2;
double m = 1.1, resultf;
result1 = b / a / c * d * m;
result2 = double(b) / double(a) / double(c) * double(d) * m;
ENDINLINE
DEBUG "results, 1: " + result1 + ", 2: " + result2 + ", pure glb: " + (1.0 * gb / ga / gc * gd * gm) + "\n"
ENDFUNCTION
Not sure if that isn't specified in one of tutorials, but it could be.
Check my source code editor for GLBasic - link Update: 20.04.2020

Offline Kitty Hello

  • code monkey
  • Administrator
  • Prof. Inline
  • *******
  • Posts: 10815
  • here on my island the sea says 'hello'
    • View Profile
    • http://www.glbasic.com
or explicitly cast the left integer that causes the problem with NUMBER(i).
Expressions are evaluated left-to-right. So,
Code: (glbasic) [Select]
LOCAL i=5, k=2, j#=7
i/k*j // = INTEGER(i/k) * j = 2*7.0=14.0
NUMBER(i)/k*j // = 5.0 / 2 * j = 2.5*7.0
If any of the two operands is a floating point, so will be the result. If both are integer, the result is, too.
The problem usually only apears for divisions.

Offline Qedo

  • Dr. Type
  • ****
  • Posts: 352
  • to program what I have todo how should I program?
    • View Profile
I have discovered the existence of NUMBER () but it is not in the help.
What does this command do?

Offline dreamerman

  • Global Moderator
  • Dr. Type
  • *******
  • Posts: 401
    • View Profile
    • my personal website
Yeah, quick looks it seems that standard manual / help file doesn't have that info, but it's in online manual.
Quote
flt# = NUMBER(str$)

Explicitly convert a string or any other number into a floating point number.

LOCAL n# = NUMBER("1.23") * 100.0
Check my source code editor for GLBasic - link Update: 20.04.2020

Offline Qedo

  • Dr. Type
  • ****
  • Posts: 352
  • to program what I have todo how should I program?
    • View Profile
Great  :good: