Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!caip!sri-spam!nike!aurora!ames!eugene
From: eugene@ames.UUCP (Eugene Miya)
Newsgroups: net.arch
Subject: Re: Incorrect Benchmark summary.
Message-ID: <1675@ames.UUCP>
Date: Thu, 25-Sep-86 14:38:33 EDT
Article-I.D.: ames.1675
Posted: Thu Sep 25 14:38:33 1986
Date-Received: Fri, 26-Sep-86 18:50:02 EDT
References: <322@oblio.UUCP> <20954@rochester.ARPA> <393@vaxb.calgary.UUCP>
Organization: NASA-Ames Research Center, Mtn. View, CA
Lines: 66

> 
> The averaging in this table is done incorrectly. As noted in a recent CACM
> article, normalized benchmark results should be averaged with a geometric
> mean, not an arithmetic mean.
>     Radford Neal
>     The University of Calgary

There is no clear cut evidence that the geometric mean is any more correct
than any other [Re: the Flemming and Wallace paper].  Jack Wolton of
Los Alamost published a paper in 1984 (IEEE Compcon, with the title:
Bottleneckology: something, my proceedings are lent out at the moment).
This paper touted the HARMONIC mean as the "correct" mean.  This draws
both suspect.  The proof offered by W&F is not a sufficiently
rigorous proof.  And I think a poor proof is worse than no proof.

The science and art of measurement are independent of statistics (I
realize an over simplification).  We resort to statistics because our
measurement tool on the computer are so poor.  What sense is it to
average numbers intended to show some sort of peak performance?
(Don't say average peak performance.)

I suggest reading two texts:

%A Darrell Huff
%T How to Lie with Statistics
%I Norton
%C NY
%D 1954

%A Edward R. Tufte
%T The Visual Display of Quatitative Information
%I Graphics Press
%C Cheshire, CO
%D 1983

Separate note on performance measurement:
I have been talking to colleagues at Berkeley, LBL, LLNL,
and other locations.  We will be having an informal dinner meeting,
probably to be held at UC Berkeley (to the shagrin of the Stanford people)
sometime during the end of October.  I am trying to think of a name to
characterize this group: New Generation Performance Measurement Group
(naw, make it a Ring), Bay Area Performance Measurement Ring,
Those People Interested in Improving Performance Measurement People.
Anyway, if you are in the Bay Area and are seriously interested,
contact me.  We have several mail correspondents like Jack Dongarra
and Ken Dymond, but we want to have this meeting, too.

Added reference:

%A Philip J. Flemming
%A John J. Wallace
%T How Not to Lie with Statistics: The Correct Way to Summarize
Benchmark Results
%J CACM
%V 29
%N 3
%D March 1986
%P 218-221
%X Waste of a good title.

From the Rock of Ages Home for Retired Hackers:
--eugene miya
  NASA Ames Research Center
  com'on do you trust Reply commands with all these different mailers?
  {hplabs,ihnp4,dual,hao,decwrl,tektronix,allegra}!ames!aurora!eugene
  eugene@ames-aurora.ARPA