Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!samsung!olivea!uunet!decwrl!sgi!rpw3@rigden.wpd.sgi.com
From: rpw3@rigden.wpd.sgi.com (Rob Warnock)
Newsgroups: comp.arch
Subject: Re: Suggestions for SPEC 3.0 CPU Performance Evaluation Suite
Message-ID: <112406@sgi.sgi.com>
Date: 22 Jun 91 00:55:57 GMT
References: <403@validgh.com>
Sender: guest@sgi.sgi.com
Reply-To: rpw3@sgi.com (Rob Warnock)
Organization: Silicon Graphics, Inc., Mountain View, CA
Lines: 34

In article <403@validgh.com> dgh@validgh.com (David G. Hough on validgh)
writes:
+---------------
| Why Geometric Mean is Best for SPEC
|      The progress of some end users is limited by the time it takes a fixed
| series of computational tasks to complete.  They then think about the results
| and decide what to do next.  The appropriate metric for them is the total
| elapsed time for the applications to complete, so the arithmetic mean of times
| is the appropriate summary statistic.  If rates, the inverse of times, happen
| to be available instead, the appropriate statistic is the harmonic mean of
| rates.  If application A runs ten times as long as application B, then a 2X
| improvement in application A is ten times as important as a 2X improvement in
| application B.
+---------------

In Mike Johnson's new book "Superscalar Micorprocessor Design", he argues
(pp.37-40) that the harmonic mean is the right thing to use when comparing
"speedups", because it tends to assign larger weighting to the program with
the smallest speedup. I agree. In particular, a single isolated "gigantic"
speedup is almost completely ignored, which is what you want for a "systems"
benchmark.

So it seems that the harmonic mean of the individual test ratios would be
useful as "the SPECratio" when comparing several systems to a reference
base system.


-ROb

-----
Rob Warnock, MS-1L/515		rpw3@sgi.com		rpw3@pei.com
Silicon Graphics, Inc.		(415)335-1673		Protocol Engines, Inc.
2011 N. Shoreline Blvd.
Mountain View, CA  94039-7311