Path: utzoo!attcan!uunet!zaphod.mps.ohio-state.edu!usc!apple!agate!shelby!eos!eugene
From: eugene@eos.arc.nasa.gov (Eugene Miya)
Newsgroups: comp.benchmarks
Subject: Re: benchmarks (SPECmarks)
Message-ID: <7581@eos.arc.nasa.gov>
Date: 13 Nov 90 01:05:16 GMT
Reply-To: eugene@eos.UUCP (Eugene Miya)
Distribution: comp
Organization: NASA Ames Research Center, Calif.
Lines: 35

In article <9135@ncar.ucar.edu> pack@acd.UCAR.EDU (Dan Packman) writes:
>Excellent point.  The best benchmark is clearly ones own application
>or set of applications including multi-process loads.

It might seem like this is true.  But it is not.

It depends on what you want benchmarks to do.

Buying a machine to print checks is one thing.  The above is fine.
Buying a machine to solve changing fluid dynamics problems is another
(moving target).  Testing (diagnosing performance) is a third.  

We know the answer.  It's 42.  At least those of us who read THAT book.
It's how you ask the question.  Not knowing how to ask the question
results in supercomputers being one generation behind the problems
we would really like to solve.  So the problem is not quite as clear as
one would like.  I agree, there is some truth to using specific existing
applications, but to ignore flaws invites trouble.

This is why SPEC's decision to normalize performance on the DEC VAX-11/780
or say the IBM PC, simply because they are or were common is bad.
This is akin to to taking the wind up clock I used to wake up as a youngster
and designating it as a time standard.  If you think this analogy is
bogus, consider taking a common ruler, 12 inches or a meter stick,
designating that as THE Meter or THE Foot, those wooden things,
then try to make sub-micron lines for your memories or CPUs.  You won't
produce consistent chips for long.  This is called "gold-plating"
a metric.  Ref: "Foundations of Metrology."

We must start at a few more basic building blocks and proceed in a
progression.  Trying to rush the process will only confuse the issue.

--e.n. miya, NASA Ames Research Center, eugene@eos.arc.nasa.gov
  {uunet,mailrus,most gateways}!ames!eugene
  AMERICA: CHANGE IT OR LOSE IT.