Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!samsung!sol.ctr.columbia.edu!emory!utkcs2!de5
From: de5@ornl.gov (Dave Sill)
Newsgroups: comp.benchmarks
Subject: Re: benchmark evaluations
Message-ID: <1990Dec14.132824.8089@cs.utk.edu>
Date: 14 Dec 90 13:28:24 GMT
References: <12220@hubcap.clemson.edu> <1990Dec12.135910.27667@cs.utk.edu> <7694@eos.arc.nasa.gov>
Sender: news@cs.utk.edu (USENET News System)
Reply-To: Dave Sill <de5@ornl.gov>
Organization: Oak Ridge National Laboratory
Lines: 40

In article <7694@eos.arc.nasa.gov>, eugene@eos.arc.nasa.gov (Eugene Miya) writes:
>In article <1990Dec12.135910.27667@cs.utk.edu> Dave Sill <de5@ornl.gov> writes:
>>In article <12220@hubcap.clemson.edu>, mark@hubcap.clemson.edu
>(Mark Smotherman) writes:
>>>1) Representative
>>Only important if the results are going to be used to predict the
>>performance of the system on other code.
>
>Wrong!  Representativeness is need for any descriptive or diagnostic
>system.  Prediction is icing on a cake.

Okay, I'll backpedal a bit here.  Yes, representativeness is a
requirement to the extent that without it you're not testing what you
think you're testing.  But representativeness can be shown
empirically.

>>>2) Reproducible
>>Not necessary in all cases, e.g., informal testing or repeated tests
>>of the same configuration.
>
>Reproducibility is a hallmark of all good sciences.
>See, The Journal of Irreproducible Results (maybe all benchmarks
>deserve to be there?).

Yes, it's A Good Thing, but how much effort is required to show that
repeated runs of the same test on the same system are reproducible?
Are there not situations where it can be assumed?

>>full, rigorous suites such as SPEC.
>
>If one benchmark is not adequate, and 2 aren't enough,
>when is enough, enough?  42? 700?  I don't think the answer lies solely
>in fixed benchmarks.

Agreed.

-- 
Dave Sill (de5@ornl.gov)
Martin Marietta Energy Systems
Workstation Support