Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!mips!winchester!mash
From: mash@mips.COM (John Mashey)
Newsgroups: comp.benchmarks
Subject: Re: bc benchmark [really: One Number]
Message-ID: <44353@mips.mips.COM>
Date: 27 Dec 90 05:43:31 GMT
References: <44342@mips.mips.COM> <15379@ogicse.ogi.edu>
Sender: news@mips.COM
Reply-To: mash@mips.COM (John Mashey)
Organization: MIPS Computer Systems, Inc.
Lines: 46

In article <15379@ogicse.ogi.edu> borasky@ogicse.ogi.edu (M. Edward Borasky) writes:
>Thank you for at least driving another stake in "bc benchmark"'s heart.
>However, as you and I know, there is a tremendous need out there for
>[sigh] [gasp] A SINGLE NUMBER to characterize JUST EXACLTY HOW FAST
>ANY GIVEN COMPUTER IS.  I have my own personal favorite which I will
>not belabor because everyone has his own personal favorite.  My question
>is this: just as you and I believe that vampires don't exist, do you
>believe that a single-number that measures a computer's speed doesn't
>exist?  I won't state MY belief to avoid bias in the discussion.  My
>use of the word "bias" in the preceding sentence is a HINT on my belief!

There is no One Number that predicts performance.

Let me restate the hypothesis more precisely:
	Let ON(a) be the One Number for machine a.
	We'd then expect that ON(a) / ON(b) would predict the
	relative performance of machines a and b on all benchmarks.
Well, that number doesn't exist, and is easily proved not,
even by looking just at the published SPEC benchmarks.
	How about saying that ON(a) / ON(b) predicts the relative performance
	of any benchmark within 10%?
Well that doesn't exist either, from the same data.
	How about saying that ON(a) / ON(b) predicts the relative performance
	within a factor of 10, i.e., suppose ON(a) / ON(b) == 1.0,
	then it would be OK as long as "a" was no more than 10X faster
	than "b" on any benchmark, or vice-versa.
This might exist, and might even cover the SPEC data, although one may have
to go even higher, like allowing a factor of 20X off.
(I'm just unpacking, and don't have the numbers handy.)
For example, try comparing a CISC micro (like a 486), which has good
integer performance, but whose VAX-relative floating-point is pretty
low, with a vector machine (like the Stardent), or with the IBM RS/6000,
both of whose floating point performance tends to be much stronger
than their VAX-relative integer performance. 

Of course, something that predicts only within a factor of 10X to 20X is pretty
useless..... But even if it were within 20-40%, it's still pretty bad.
(Note, for example, that published Dhrystone results easily mis-predict
SPEC integer benchmarks pretty badly, i.e., it is quite easy for machine
"a" to be 25% faster on Dhrystone than "b", and end up 25% SLOWER on more
realistic integer benchmarks.)
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	 mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash 
DDD:  	408-524-7015, 524-8253 or (main number) 408-720-1700
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086