Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!ut-sally!husc6!endor!reiter
From: reiter@endor.harvard.edu (Ehud Reiter)
Newsgroups: comp.arch
Subject: Re: Benchmarking
Message-ID: <2036@husc6.UUCP>
Date: Wed, 20-May-87 10:24:20 EDT
Article-I.D.: husc6.2036
Posted: Wed May 20 10:24:20 1987
Date-Received: Sat, 23-May-87 06:07:20 EDT
References: <1589@ames.UUCP>
Sender: news@husc6.UUCP
Reply-To: reiter@harvard.UUCP (Ehud Reiter)
Organization: Aiken Computation Lab Harvard, Cambridge, MA
Lines: 22
Keywords: Whiz

In article <1589@ames.UUCP> eugene@pioneer.arpa (Eugene Miya N.) writes:
>150 architecture articles with lots on benchmarking all bad ...
>You guys should go out and get books on
>experiment design: Campbell and Stanley, Cochrane and Cox, or find
>one of Phil Heidelberger's (IBM, IEEE) articles.

I've seen Eugene Miya say good scientific experimental design should be used for
benchmarking many times, but I'm still a bit puzzled as to how this should be
done.  Ideally, if we had good data on exactly what programs were typically
run by each class of user, then we could measure the performance of a machine
for a carefully chosen set of "benchmark" programs, and then use the above data
to extrapolate the machine's performance for each user class.  I assume this
is what Eugene means by a good experiment.

However, I've never seen good data on what programs typical users run, and
without this data we can not perform the above "experiment".  Perhaps this
just means we should try to gather good data on what programs users run -
I think this is a great idea, as long as someone else does it!

					Ehud Reiter
					reiter@harvard	(ARPA,BITNET,UUCP)
					reiter@harvard.harvard.EDU  (new ARPA)