Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!ames!rex!cleeland From: cleeland@rex.cs.tulane.edu (Chris Cleeland) Newsgroups: comp.sys.mac Subject: Re: Bug in EXCEL v. 2.2, and a flame Message-ID: <959@rex.cs.tulane.edu> Date: 1 Aug 89 00:16:21 GMT References: Reply-To: cleeland@rex.UUCP (Chris Cleeland) Distribution: usa Organization: Computer Science Dept., Tulane Univ., New Orleans, LA Lines: 23 In article jk3t+@andrew.cmu.edu (Jonathan King) writes: > >Interesting. My calculator (a Sharp EL-512 II) gives virtually the same >erroneous square root as Excel! That is, the square root of .999975 is >given as .999987499. It looks like this problem is not unique to Excel; >to make its way into the entirely silicon brain of my calculator this error >must be the result of an ancient and honorable (although badly conditioned) >numerics routine... > Funny, your Sharp and my HP must hail from a common genetic origin. My HP-28C give SQRT(.999975) as 0.999987499922; upon hitting the "x^2" key, my trusty calc gives the original number -- .999975. The 28C calculates things to 12 digits of accuracy, and only displays 10 (presumably so that all 10 are usable). I hope that my calculator isn't wrong, but then again, maybe I could blame some bad test grades on it ;-> Anybody else? -- Thanks Chris Cleeland, Tulane University ADDRESS: cleeland@rex.tulane.cs.edu Disclaimer: "I'm a student -- I can't afford to buy one!"