How comes my computer calculate into this?

It calculate 0.9 - (0.5 * (i-1)) into -0.9999

Where did I go wrong?



i = 3 btw

Ew, basing a "My First RPG" on MapleStory.

Looks like a Double rounding error. It's doing it right - 0.9 - (0.5 * (3-1)) = -0.1, but it's storing it as -0.09999999 because of the way computers handle floating point numbers. Make sure you are using a Double instead of a Float.

You mean -0.0999... - which is a pretty significant difference.

Nowhere! The computer calculates 0.9 - (0.5 * (i - 1)) by first calculating i - 1 = 2, then 0.5 * 2 = 1.0. Finally, it computes 0.9 - 1.0 to either -0.0999999978 or -0.100000024 because of floating point accuracy problem: As the computer only sees 1s and 0s, it cannot represent -0.10 in any way without using fractions. Fractions are usually slow, and are thus not used unless specified.

It is probably easier to understand if you compare it with some fraction we can't express properly: 1/3. If we say 1/3 * 3 to a computer, it can actually compute it correctly, but if we say it to a "human computer", it would first calculate 1/3 = 0.3333333... and then multiply it by 3, which is 0.9999999....

Noah

Wow, I never know this (or forgot about it already lol).

Thanks for quick replies.

If you care about avoiding the floating point accuracy problem, C# has a "decimal" type that does math like a human doing it by hand.