How comes my computer calculate into this? - Printable Version +- Southperry.net (https://www.southperry.net) +-- Forum: Social (https://www.southperry.net/forumdisplay.php?fid=14) +--- Forum: Rubik's Cube (https://www.southperry.net/forumdisplay.php?fid=58) +--- Thread: How comes my computer calculate into this? (/showthread.php?tid=43240)

How comes my computer calculate into this? - Unauthorized Intruder - 2011-06-19 It calculate 0.9 - (0.5 * (i-1)) into -0.9999 Where did I go wrong? i = 3 btw How comes my computer calculate into this? - Dusk - 2011-06-19 Ew, basing a "My First RPG" on MapleStory. Looks like a Double rounding error. It's doing it right - 0.9 - (0.5 * (3-1)) = -0.1, but it's storing it as -0.09999999 because of the way computers handle floating point numbers. Make sure you are using a Double instead of a Float. How comes my computer calculate into this? - Noah - 2011-06-19 You mean -0.0999... - which is a pretty significant difference. Nowhere! The computer calculates 0.9 - (0.5 * (i - 1)) by first calculating i - 1 = 2, then 0.5 * 2 = 1.0. Finally, it computes 0.9 - 1.0 to either -0.0999999978 or -0.100000024 because of floating point accuracy problem: As the computer only sees 1s and 0s, it cannot represent -0.10 in any way without using fractions. Fractions are usually slow, and are thus not used unless specified. It is probably easier to understand if you compare it with some fraction we can't express properly: 1/3. If we say 1/3 * 3 to a computer, it can actually compute it correctly, but if we say it to a "human computer", it would first calculate 1/3 = 0.3333333... and then multiply it by 3, which is 0.9999999.... Noah How comes my computer calculate into this? - Unauthorized Intruder - 2011-06-19 Wow, I never know this (or forgot about it already lol). Thanks for quick replies. How comes my computer calculate into this? - Spaz - 2011-06-19 If you care about avoiding the floating point accuracy problem, C# has a "decimal" type that does math like a human doing it by hand.