Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!ll-xn!ames!pioneer!eugene
From: eugene@pioneer.arpa (Eugene Miya N.)
Newsgroups: comp.arch,comp.org.usenix
Subject: Re: Benchmarking
Message-ID: <1606@ames.UUCP>
Date: Thu, 21-May-87 12:51:22 EDT
Article-I.D.: ames.1606
Posted: Thu May 21 12:51:22 1987
Date-Received: Sat, 23-May-87 11:44:00 EDT
References: <324@dumbo.UUCP> <809@killer.UUCP> <2417@homxa.UUCP>
Sender: usenet@ames.UUCP
Reply-To: eugene@pioneer.UUCP (Eugene Miya N.)
Organization: NASA Ames Research Center, Moffett Field, Calif.
Lines: 41
Keywords: Okay let do something
Xref: mnetor comp.arch:1378 comp.org.usenix:201

In article <6024@steinmetz.steinmetz.UUCP> William E. Davidsen Jr writes:
>
>After doing benchmarks for about 15 years now, I will assure everyone
>that the hard part is not getting reproducable results, but in (a)
>deciding how these relate to the problem you want to solve, and (b) getting
>people to believe that there is no "one number" which can be used to
>characterize performance. If pressed I use the reciprocal of the total
>real time to run the suite. It's as good as any other voodoo number...

Yes I agree, and I have not had to do it that long.  Let's take a moment
to study ways to relate or characterize end-users applications:
1) without gross generalizations, but real quantitative data,
and 2) using common ideas and tools?  Okay?  Static as well as dynamic
tools.  What can we tell independent of machines and languages?

Second:
There's lots of disciplines which abuse and use single figures of merit
and get away from them.  Consider: earlier in the season, (end of ski
season really): base of NS was a sea of mud, 2/3 way up the mountain
in a sheltered area was the snow gauge reading 5.5 feet.  You think we
have problems with measurement?  Is an average ($ {int int from {all area of
ski resort} depth function dx dy} over area} a reasonable way to
characterize resort coverage?

Do we buy cars on single figures of merit?  If not, then now many?

Consider cardiology: heart function.  Single figures are used: heart
rates, but EKGs are much better they portrary more.  Picture worth a
thousand words?  Try embedding one on the net with any good resolution.

Yes, we can get away, but we have to take others with us.  I better stop
before Alan Smith totally loses respect (probably has already).

From the Rock of Ages Home for Retired Hackers:

--eugene miya
  NASA Ames Research Center
  eugene@ames-aurora.ARPA
  "You trust the `reply' command with all those different mailers out there?"
  "Send mail, avoid follow-ups.  If enough, I'll summarize."
  {hplabs,hao,ihnp4,decwrl,allegra,tektronix,menlo70}!ames!aurora!eugene