Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!mcnc!gatech!hao!husc6!endor!reiter
From: reiter@endor.harvard.edu (Ehud Reiter)
Newsgroups: comp.arch
Subject: Re: Benchmarking
Message-ID: <2100@husc6.UUCP>
Date: Wed, 27-May-87 11:23:37 EDT
Article-I.D.: husc6.2100
Posted: Wed May 27 11:23:37 1987
Date-Received: Fri, 29-May-87 06:29:43 EDT
References: <415@winchester.UUCP> <642@percival.UUCP> <426@winchester.UUCP>
Sender: news@husc6.UUCP
Reply-To: reiter@endor.UUCP (Ehud Reiter)
Organization: Aiken Computation Lab Harvard, Cambridge, MA
Lines: 31

I think some people are perhaps missing the point.  Of course, we would all like
system benchmarks which accurately predict the performance for our workloads.
But such benchmarks are usually impossible, because performance varies quite
a bit depending on workload, and most users just don't have a very good idea
of what their workload is and will evolve into.  Even when the workload is
accurately known, this kind of benchmarking is expensive and time-consuming.

The point is, there is a great demand out there for simple, single figure
performance numbers which are in the public domain.  No matter how much we
complain that single figures are meaningless, people out there in the real
world are going to continue using them.  There's a reason why MIPS and
Dhrystones are so often quoted.

And, we can do better than Dhrystone!  We all know what the problems with
Dhrystone are - can't be globally optimized, too much string handling,
too small, etc.  We can certainly write a benchmark which, although still
"bad", will be much better than Dhrystone.

I think we can even get away with replacing single-number benchmarks by
two number benchmarks, which would give a high and low performance figure
instead of just a single performance figure (that is, the benchmark would
consist of lots of programs.  The performance numbers would be normalized
against some standard (good old 4.2BSD VAX-11/780?), and the summary
statistics would be the highest and lowest of the normalized numbers).

In summary, we can't write a perfect benchmark, but we can write a better
benchmark.

					Ehud Reiter
					reiter@harvard	(ARPA,BITNET,UUCP)
					reiter@harvard.harvard.EDU  (new ARPA)