Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sun-barr!ames!pasteur!agate!shelby!eos!eugene
From: eugene@eos.arc.nasa.gov (Eugene Miya)
Newsgroups: comp.benchmarks
Subject: Re: benchmark evaluations
Message-ID: <7693@eos.arc.nasa.gov>
Date: 14 Dec 90 08:09:10 GMT
References: <12220@hubcap.clemson.edu>
Reply-To: eugene@eos.UUCP (Eugene Miya)
Organization: NASA Ames Research Center, Calif.
Lines: 46

In article <12220@hubcap.clemson.edu> mark@hubcap.clemson.edu
(Mark Smotherman) writes:
>1) Representative
>   A) accurate characterization of workload
>   B) exploit system structure (including compiler optimization) only
>      as much as workload will be able to do so
>2) Reproducible
>   A) full system configuration (including OS and compiler versions)
>      specified (e.g., SPEC reporting)
>   B) system unloaded, or load specified and reproducible
>   C) operational rules (e.g., compiler options, program inputs and files)
>3) Compact
>   A) portable, with little or no need of conversion
>   B) inexpensive
>   C) test files (to avoid size and privacy problems with actual data)

Ah! From the home of the fine moderated group: comp.parallel.
Unfortunately Dave mailed me that comp.benchmark isn't archived.

This is a good start, but the list is a bit linear.
Comments:
2) There are two kinds of reproducibility:
	You want to be able to replicate the result yourself (not always
	possible on a computer)
	You want colleagues to be able do it without you.
I too had thought about these issues (you got "representative" as better
than "real" or "realistic."  Did you see my first issues post?

I want to also suggest a short reading, the very last chapter of
Danny Hillis's thesis "The Connection Machine" published by MIT
Press entitled "Why Computer Science is No Good."  Despite Danny's (and
my own) flaming, he points out three things CS needs (benchmarking
as well as any other science):
	1) scale
	2) symmetry
	3) locality of effect
We can learn something from these.  Danny is right.  THESE ARE IMPORTANT
THINGS LEFT OUT OF LEARNING ABOUT BENCHMARKS.

But I just got back from vacation, and I have had to jump back into the
benchmarking fire.  More later.


--e.n. miya, NASA Ames Research Center, eugene@eos.arc.nasa.gov
  {uunet,mailrus,most gateways}!ames!eugene
  AMERICA: CHANGE IT OR LOSE IT.