Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!lll-crg!nike!sri-spam!parcvax!hplabs!tektronix!uw-beaver!ubc-vision!alberta!calgary!radford
From: radford@calgary.UUCP (Radford Neal)
Newsgroups: net.arch
Subject: Re: Incorrect Benchmark summary.
Message-ID: <405@vaxb.calgary.UUCP>
Date: Mon, 29-Sep-86 13:18:59 EDT
Article-I.D.: vaxb.405
Posted: Mon Sep 29 13:18:59 1986
Date-Received: Thu, 2-Oct-86 20:52:56 EDT
References: <322@oblio.UUCP> <20954@rochester.ARPA> <393@vaxb.calgary.UUCP> <1675@ames.UUCP>
Organization: U. of Calgary, Calgary, Ab.
Lines: 38
Summary: What's wrong with Flemming & Wallace's paper?

In article <1675@ames.UUCP>, eugene@ames.UUCP (Eugene Miya) writes:

> > The averaging in this table is done incorrectly. As noted in a recent CACM
> > article, normalized benchmark results should be averaged with a geometric
> > mean, not an arithmetic mean.
> 
> There is no clear cut evidence that the geometric mean is any more correct
> than any other [Re: the Flemming and Wallace paper].  Jack Wolton of
> Los Alamost published a paper in 1984 (IEEE Compcon) [touting]
> HARMONIC mean as the "correct" mean...
> The proof offered by W&F is not a sufficiently
> rigorous proof.  And I think a poor proof is worse than no proof.

Could you elaborate on what's wrong with the proof offered by W&F? I 
re-read the article last night, and they seem to me to prove what they
set out to prove. The harmonic mean certainly doesn't work for normalized
numbers.

W&F do not claim that the geometric mean is the only good way to average
benchmarks, just the only way to average *normalized* benchmarks. If you
have a good idea of your job mix, a weighted arithmetic mean of the raw data
is the way to go. Then you can normalize this mean to one machine if you feel
like it.

Regardless, the case *against* arithmetic means of normalized numbers 
seems completely incontrovertable, regardless of what one considers the
"best" replacement. It's not acceptable for the results to depend on 
which machine one arbitrarily decides to normalize to. 

The paper discussed here is:

  Flemming, Philip J. and Wallace, John J. "How not to lie with
    statistics: The correct way to summarize benchmark resuls",
    Communications of the ACM, Vol. 29, No. 3. (March 1986).


Radford Neal
The University of Calgary