Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!rutgers!apple!bionet!agate!eos!eugene
From: eugene@eos.UUCP (Eugene Miya)
Newsgroups: comp.arch
Subject: Re: Benchmarking
Message-ID: <1755@eos.UUCP>
Date: 18 Oct 88 05:03:27 GMT
References: <2220003@hpausla.HP.COM> <46500026@uxe.cso.uiuc.edu> <1988Oct9.011633.13259@utzoo.uucp> <6001@june.cs.washington.edu> <5356@winchester.mips.COM>
Reply-To: eugene@eos.UUCP (Eugene Miya)
Organization: NASA Ames Research Center, Calif.
Lines: 42

In article <5356@winchester.mips.COM> mash@winchester.UUCP (John Mashey) writes:
>In article <6001@june.cs.washington.edu> rik@june.cs.washington.edu (Rik Littlefield) writes:
>>Many postings in this stream seem to assume that "large, real" programs are
>>somehow the most fair to use for benchmarking.  That's not necessarily true.
>As we've said numerous times, the best benchmark for anybody is for them
>to run their own real applications, because such applications obviously
>have the highest correlation with what they'll see in real use.

>When I keep saying "use large, real programs", it's because I usually
>have in front of me numerous statistics about the behavior of programs
>that show that most of the toy benchmarks aren't very good predictors of
>the real applications, especially when applied to the higher-performance
>designs.  Why is this?
>	a) Toys don't stress cache designs,
>	b) Toys don't stress limits.
>	c) Toys don't stress software.

REAL programs have certain biases and short comings, but I am unable to
come up with an elioquent example.  The problem comes with "large, real."
Does large mean memory requirements? (Crank the arrays bigger,
ever see an arrays proposed with 1 Tera word of memory? Read Cray Channels).
Does large mean computationally complex?  Each of these is true to a degree.
Then there is the question of what constitutes "real," and I don't mean
in the metaphysical sense.  I call this "the tension of simplicity."
It affects all we do with measurement: portability, interpretability,
and how we run.

I believe we have to play with some toys before jumping into "real"
programs.  We have to find out what makes them "real." [Many have ideas,
but few are good.]  I think if it weren't for toys, we wouldn't have
things like the NeXT, the Mac, the Apple II.  Computers would be big boxes
behind glass windows, and we would be hung up on who's card deck would be
submitted next.

Another gross generalization from

--eugene miya, NASA Ames Research Center, eugene@aurora.arc.nasa.gov
  resident cynic at the Rock of Ages Home for Retired Hackers:
  "Mailers?! HA!", "If my mail does not reach you, please accept my apology."
  {uunet,hplabs,ncar,decwrl,allegra,tektronix}!ames!aurora!eugene
  "Send mail, avoid follow-ups.  If enough, I'll summarize."
  Actually, I can think about one example at LLNL, but it's classified.