Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!mcnc!ece-csc!ncrcae!ncr-sd!bigbang!celerity!ps
From: ps@celerity.UUCP (Pat Shanahan)
Newsgroups: comp.arch,comp.org.usenix
Subject: Re: Benchmarking the 532, 68030, MIPS, 386...at a Usenix!
Message-ID: <202@celerity.UUCP>
Date: Thu, 28-May-87 16:22:05 EDT
Article-I.D.: celerity.202
Posted: Thu May 28 16:22:05 1987
Date-Received: Sat, 30-May-87 09:32:09 EDT
References: <324@dumbo.UUCP> <809@killer.UUCP> <2417@homxa.UUCP> <4294@nsc.nsc.com> <2128@hoptoad.uucp>
Reply-To: ps@celerity.UUCP (Pat Shanahan)
Organization: Celerity Computing, San Diego, Ca.
Lines: 48
Xref: mnetor comp.arch:1407 comp.org.usenix:216

In article <2128@hoptoad.uucp> gnu@hoptoad.uucp (John Gilmore) writes:
>In article <4294@nsc.nsc.com>, grenley@nsc.nsc.com (George Grenley) writes:
>> So, here's the deal.  I invite Mot, Intel, and other interested parties
>> to work with me in defining some sort of realistic benchmark, which we'll
>> run (in public).  I expect to have system level hardware late this year,
>> so if we get started now, we'll have very interesting Xmas presents...
>> May the best CPU win!  

Defining a realistic benchmark set seems a good objective. I do think that
it is important to realize that performance is a multi-dimensional quantity.
The only situation in which it is realistic to say "machine X is twice as
fast as machine Y" is if X and Y have identical architectures. Any benchmark
that produces a single number is likely to be misleading, unless it is
understood to be very specialized.

>...
>I'd like to join in the hoopla, rahrah, etc that has followed this
>suggestion, and make a further one:
>
>	Let's have the bake-off in the trade show at, say, next Winter
>	Usenix.  Probably the actual setup and running of the benchmarks
>	can be done a day or two before the show, so the results can be
>	printed for distribution, and to give the losers time to think
>	up (and print up) good explanations before we descend on them :-).
>...
>-- 
>Copyright 1987 John Gilmore; you may redistribute only if your recipients may.
>(This is an effort to bend Stargate to work with Usenet, not against it.)
>{sun,ptsfa,lll-crg,ihnp4,ucbvax}!hoptoad!gnu	       gnu@ingres.berkeley.edu


I doubt whether I could get anything unusual run on a system that is going
to a trade show during the last few days before the show. The system would
normally be being loaded with known demo programs and crated up for
transport. It is also a very busy time for the people involved in the show.

I think it would be better to define the benchmark in advance and encourage
manufacturers to run it on a variety of configurations, rather than just on
what happens to be at one show.

I suggest using existing benchmarks where possible. For example, measure
Fortran loop performance with Livermore loops, rather than writing a new
benchmark.
-- 
	ps
	(Pat Shanahan)
	uucp : {decvax!ucbvax || ihnp4 || philabs}!sdcsvax!celerity!ps
	arpa : sdcsvax!celerity!ps@nosc