Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!lll-crg!nike!think!mit-eddie!genrad!decvax!tektronix!uw-beaver!ubc-vision!alberta!calgary!radford
From: radford@calgary.UUCP (Radford Neal)
Newsgroups: net.arch
Subject: Incorrect Benchmark summary.
Message-ID: <393@vaxb.calgary.UUCP>
Date: Mon, 22-Sep-86 14:39:21 EDT
Article-I.D.: vaxb.393
Posted: Mon Sep 22 14:39:21 1986
Date-Received: Thu, 25-Sep-86 07:29:10 EDT
References: <322@oblio.UUCP> <20954@rochester.ARPA>
Organization: U. of Calgary, Calgary, Ab.
Lines: 58
Keywords: 386, 286, 68020, 32032, 32100, CACHE
Summary: Use geometric means to average normalized figures.

In article <20954@rochester.ARPA>, crowl@rochester.ARPA (Lawrence Crowl) writes:
> The table below is a reorganization of the following table. [I ommitted
> this table in this posting. It contained the times for the benchmarks in
> seconds. RN]
> 				Relative Performance
> 
> processor	80286	80386	68000	68020	68020	32032	32100	32100
> cache (MHz)	(10)	(16)	(8)	N (16)	C (16)	(10)	N (18)	C (18)
> 
> string search	1.37	1.00	3.85	2.25	1.08	3.51	4.71	1.89
> bit manipulate5.52	2.09	5.91	2.25	1.00	5.29	3.58	1.74
> linked list	3.08	1.70	4.11	1.79	1.00	2.90	2.36	1.28
> quicksort	4.07	2.49	4.39	2.05	1.00	3.12	3.12	1.32
> matrix trans	6.42	2.05	5.49	1.58	1.00	4.33	3.04	1.47
> 
> average	4.09	1.87	4.75	1.98	1.02	3.83	3.36	1.54

The averaging in this table is done incorrectly. As noted in a recent CACM
article, normalized benchmark results should be averaged with a geometric
mean, not an arithmetic mean. The geometic mean of N numbers is the Nth 
root of their product. This method gives the correct results:

  RIGHT average	3.60	1.79	4.68	1.97	1.02	3.74	3.28	1.52

In this case, it doesn't seem to make all that much difference in the
conclusions. In can though, consider the following example:

		Machine A	Machine B

  Benchmark 1:	10 seconds	5 seconds
  Benchmark 2:	10 seconds	20 seconds

Look at the results of normalizing these figures to Machine A and then
taking the arithmetic mean of the results:

  		Machine A	Machine B
  Benchmark 1:	  1.0		  0.5
  Benchmark 2:	  1.0		  2.0

  arith. mean:	  1.0		  1.25

Machine B is thus 25% slower than machine A, right? Wrong. Look at what
happens when you take the *same* benchmark results, normalize to machine
B, and take the arithmetic mean:

  		Machine A	Machine B
  Benchmark 1:	  2.0		  1.0
  Benchmark 2:	  0.5		  1.0

  arith. mean:	  1.25		  1.0

Now machine B comes out looking faster! 

If you take the geometric mean, however, machine A and machine B look
equally fast regardless of how you normalize the results. 

    Radford Neal
    The University of Calgary