easy calculation goes wrong

This forum is meant for questions and discussions about the X# language and tools
Post Reply
User avatar
Horst
Posts: 331
Joined: Tue Oct 13, 2015 3:27 pm

easy calculation goes wrong

Post by Horst »

Hello
This line is calculating 806.38+24.54 and the result is 830.93 and thats wrong , look at screenshot.

Code: Select all

aMwstCodes [nCnt][04] := aMwstCodes [nCnt][04]  + fSteuer  
i changed all to decimal variables

Code: Select all

fSteuerTmp  := fBetrag - fBetrag / (100+fProzent) * 100 
fSteuer	   += fSteuerTmp           
fSteuerTotal := aMwstCodes [nCnt][04]                   
fSteuerTotal := fSteuerTotal +fSteuer      
aMwstCodes [nCnt][04] := fSteuerTotal  
but the same result, all other calculations are ok.
any idea ?

Horst
Attachments
WhatsApp Image 2024-04-28 at 11.16.58.jpeg
User avatar
Horst
Posts: 331
Joined: Tue Oct 13, 2015 3:27 pm

Re: easy calculation goes wrong

Post by Horst »

Hello
After my post i sleep a houre, think has to do something with the setdecimal or something else in teh settings. Will read the help now.
g.bunzel@domonet.de
Posts: 97
Joined: Tue Mar 01, 2016 11:50 am
Location: Germany

Re: easy calculation goes wrong

Post by g.bunzel@domonet.de »

Horst,

ich verwende diese Einstellung und hatte noch nie ein Problem mit Rundungsfehlern:
SetFloatDelta (0.0001)

Um bei solchen Berechnung mit möglichst wenig Nachkommastellen zu arbeiten, würde ich erst die Multiplikation und dann die Division machen - damit die Werte der Variablen nicht so klein sind und viele Nachkommastellen haben, die dann beim Runden abgeschnitten werden.
Damit alle Werte der Zeile als FLOAT behandelt werden, sollte diese Zeile so aussehen:

fSteuerTmp := fBetrag - fBetrag * 100.0 / (100.0 + fProzent)


HTH

Gerhard
User avatar
robert
Posts: 4483
Joined: Fri Aug 21, 2015 10:57 am
Location: Netherlands

Re: easy calculation goes wrong

Post by robert »

Guys,
This problem is caused by the fact that certain floating point numbers cannot exactly be represented.
If you want 100% accuracy you should either calculate with integers (in cents) or with the Decimal type.

Robert
XSharp Development Team
The Netherlands
robert@xsharp.eu
FFF
Posts: 1573
Joined: Fri Sep 25, 2015 4:52 pm
Location: Germany

Re: easy calculation goes wrong

Post by FFF »

But Horst wrote, that he changed to decimal with no avail?
Regards
Karl
(on Win8.1/64, Xide32 2.20, X#2.20.0.3)
User avatar
Horst
Posts: 331
Joined: Tue Oct 13, 2015 3:27 pm

Re: easy calculation goes wrong

Post by Horst »

Hello

It was my fault

i changed to SetFloatDelta (0.0001) then i saw more decimal numbers 806.389 + 24.544 is 830.93
Wired, it was the only one in more then 300 calculations.
Murphys law :-)

Horst
Post Reply