Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!yale!cs.yale.edu!doctor.chem.yale.edu
From: jim@doctor.chem.yale.edu (James F. Blake)
Newsgroups: comp.lang.c
Subject: Re: RMS deviation between two sets of cartesian coords
Message-ID: <27637@cs.yale.edu>
Date: 6 Dec 90 20:46:50 GMT
References: <14666@smoke.brl.mil>
Sender: news@cs.yale.edu
Lines: 15
Nntp-Posting-Host: doctor.chem.yale.edu

From article <14666@smoke.brl.mil>, by gwyn@smoke.brl.mil (Doug Gwyn):
> 
> It's hard to see how you could get 5% fuzz in a variance computation
> if you use the direct definition (sums of squares of deviations).
> The textbook rewrite as mean-of-squares minus square-of-mean is well
> known to produce bogus answers, however, particularly when the variance
> is relatively small.

     You're correct.  I believe the problem was a combination of the
equations and a "feature."  The original code allowed the axis to be 
transformed such that they were nonorthogonal, if this would produce a
better fit.  This probably produced most of the fuzz.  Thanks to all who
have responded, your insight is very much appreciated.

  Jim