Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!lll-crg!nike!ucbcad!ucbvax!decvax!decwrl!glacier!mips!hansen
From: hansen@mips.UUCP
Newsgroups: net.arch
Subject: Re: Incorrect Benchmark summary.
Message-ID: <711@mips.UUCP>
Date: Fri, 3-Oct-86 14:14:43 EDT
Article-I.D.: mips.711
Posted: Fri Oct  3 14:14:43 1986
Date-Received: Mon, 6-Oct-86 18:23:24 EDT
References: <322@oblio.UUCP> <20954@rochester.ARPA> <393@vaxb.calgary.UUCP> <1675@ames.UUCP> <405@vaxb.calgary.UUCP>
Organization: MIPS Computer Systems, Sunnyvale, CA
Lines: 42

The geometric and harmonic means are MEANINGLESS for benchmarking machines.
They are only useful for fudging the results to try to make an
unbalanced, ill-designed, machine look good.

Look at it this way: if you've got two jobs to get done and
two machines take the following times:

	Machine A	Machine B
Job 1	10 sec		20 sec
Job 2	10 sec		5 sec

Does anyone disagree that Machine A takes 20 seconds,
and Machine B takes 25 seconds? It should be obvious
that machine A is the faster machine FOR THE GIVEN WORKLOAD.

Even if the times were more extreme:

	Machine A	Machine B
Job 1	10 sec		20 sec
Job 2	10 sec		1 sec

Still, Machine B is slower. It doesn't matter a damn
that Job 2 executes ten times faster on Machine B,
because it was so slow on Job 1 that it already lost
the race before even starting on Job 2.

Now, if you run Job 2 more often than Job 1, or Job 2
is more closely representative of the workload you
intend for the machine, then sure, go ahead and
adjust the weighting. However, the geometric
and harmonic means take neither of these factors into account,
and effectively use inconsistent weightings between machines.

Doesn't anyone remember the parable of the Tortise and the Hare?
I suppose someone will now try and convince us that
the Tortise should have been the winner via the geometric mean!

-- 

Craig Hansen			|	 "Evahthun' tastes
MIPS Computer Systems		|	 bettah when it
...decwrl!mips!hansen		|	 sits on a RISC"