Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!yale!cs.yale.edu!doctor.chem.yale.edu From: jim@doctor.chem.yale.edu (James F. Blake) Newsgroups: comp.lang.c Subject: Re: RMS deviation between two sets of cartesian coords Message-ID: <27637@cs.yale.edu> Date: 6 Dec 90 20:46:50 GMT References: <14666@smoke.brl.mil> Sender: news@cs.yale.edu Lines: 15 Nntp-Posting-Host: doctor.chem.yale.edu From article <14666@smoke.brl.mil>, by gwyn@smoke.brl.mil (Doug Gwyn): > > It's hard to see how you could get 5% fuzz in a variance computation > if you use the direct definition (sums of squares of deviations). > The textbook rewrite as mean-of-squares minus square-of-mean is well > known to produce bogus answers, however, particularly when the variance > is relatively small. You're correct. I believe the problem was a combination of the equations and a "feature." The original code allowed the axis to be transformed such that they were nonorthogonal, if this would produce a better fit. This probably produced most of the fuzz. Thanks to all who have responded, your insight is very much appreciated. Jim Brought to you by Super Global Mega Corp .com