This post has been republished via RSS; it originally appeared at: Small Basic Blog articles.
First published on MSDN on Aug 21, 2017Authored by Nonki Takahashi
Computer calculation always has some errors. I wrote following program to investigate how Math.Round operation works. Guess what type of errors occur in following case.
TextWindow.WriteLine(Math.Round("2.50000000000000017763568394")) ' 2
TextWindow.WriteLine(Math.Round( 2.5000000000000001776356839401 )) ' 2
TextWindow.WriteLine(Math.Round("2.500000000000000200000")) ' 2
TextWindow.WriteLine(Math.Round("2.5000000000000002000")) ' 2
TextWindow.WriteLine(Math.Round("2.500000000000000200")) ' 2
TextWindow.WriteLine(Math.Round("2.5000000000000002")) ' 2
TextWindow.WriteLine(Math.Round( 2.5000000000000051 )) ' 2
TextWindow.WriteLine(Math.Round( 2.5000000000000052 )) ' 3
TextWindow.WriteLine(Math.Round("2.5000000000000001776356839401")) ' 3
TextWindow.WriteLine(Math.Round("2.50000000000000020")) ' 3
Actuary, I can't understand yet what happens in the program above...