Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!samsung!usc!elroy.jpl.nasa.gov!ames!ames.arc.nasa.gov!lamaster
From: lamaster@ames.arc.nasa.gov (Hugh LaMaster)
Newsgroups: comp.arch
Subject: Re: benchmarking
Message-ID: <43905@ames.arc.nasa.gov>
Date: 28 Feb 90 17:02:12 GMT
References: <7393@pdn.paradyne.com> <3300102@m.cs.uiuc.edu> <36438@mips.mips.COM> <132232@sun.Eng.Sun.COM> <6336@eos.UUCP>
Sender: usenet@ames.arc.nasa.gov
Organization: NASA - Ames Research Center
Lines: 59

In article <6336@eos.UUCP> eugene@eos.UUCP (Eugene Miya) writes:
>In article <132232@sun.Eng.Sun.COM> lm@sun.UUCP (Larry McVoy) writes:
>>>In article <3300102@m.cs.uiuc.edu> gillies@m.cs.uiuc.edu writes:
>>In article <36438@mips.mips.COM> mash@mips.COM (John Mashey) writes:
:
>>I, for one, think SPEC is great.
>Oh well.  Too bad.
:
>Sorry, John, I tend to suspect SPEC spent a lot of money.

I, for another, like SPEC reasonably well.  It does a good job of balancing
integer and floating point requirements.  It matches reasonably well what
some vendors use as a definition of "MIPS".  Overall, I think it is a good
job.

>You want number?  Try 42.  Douglas Adams published that.
:
>The fundamental idea which separates people is whether or not you believe
>the whole a of benchmark equals or exceeds the sum of its parts.

Of course, every techie realizes that benchmark numbers are just numbers.
I can give you a list of 100 things that no existing benchmark measures well.
But, as a defense against marketing droids, it is a reasonable first line
of defense.

>Users simply concern with pure speed will inevitably be disappointed.

Those looking for a single number to characterize speed will inevitably be
disappointed.  Those looking for a number to demonstrate that certain kinds
of programs will not experience bottlenecks may be much better served.  The
purpose to rating systems with numbers like SPECMARK is not to say,
"My system is better than yours, because it is 17.6 SPECMARKS and yours is
only 16.9"  The purpose is eliminate systems from the solution space because
they won't be fast enough.  Marketing types will misuse it all the same, but
so what?  They have freedom of speech too.

>I don't get any warm fuzzy feeling from the Nelson, the Loops, Dongarra, etc.

Most people don't get warm fuzzies from benchmark programs.  But, they can
narrow your solution space if used correctly.  You don't have to consider
systems which are too slow for your job.  You still need to apply
other measures to make sure that the system meets all your requirements.

The main problem that I see with using benchmark programs is that some
*marketING* driven companies have a tendency to neglect things which aren't
being measured.  For example, context switching speed.  I see some evidence
that after initial euphoria over faster CPU speeds on micros, people are 
beginning to go back to fundamentals, and building more balanced systems.
Of course, *MARKET* driven companies have been doing it all along. 

>  [If it ain't source, it ain't software -- D. Tweten]

Agreed.


  Hugh LaMaster, m/s 233-9,  UUCP ames!lamaster
  NASA Ames Research Center  ARPA lamaster@ames.arc.nasa.gov
  Moffett Field, CA 94035     
  Phone:  (415)604-6117