Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!samsung!usc!ucsd!pacbell.com!ames!amelia!wilbur.nas.nasa.gov!eugene
From: eugene@wilbur.nas.nasa.gov (Eugene N. Miya)
Newsgroups: comp.graphics
Subject: Re: bogus benchmarks
Message-ID: <7654@amelia.nas.nasa.gov>
Date: 3 Aug 90 17:10:25 GMT
References: <850@earth.cs.utexas.edu> <7009@helios.TAMU.EDU> <4299@tahoe.unr.edu> <140112@sun.Eng.Sun.COM>
Sender: news@amelia.nas.nasa.gov
Reply-To: eugene@wilbur.nas.nasa.gov (Eugene N. Miya)
Organization: NASA Ames Research Center, Moffett Field, CA
Lines: 40

In article <140112@sun.Eng.Sun.COM> herzog@Eng.Sun.COM (Brian Herzog) writes:
>I think benchmarks are important, valuable, complicated, and yes, dangerous.
>
>I don't wish to engage in benchmark wars, and even if I did, this would
>be an inappropriate forum.

Er, which one is?  For graphics systems maybe, maybe not.
See below.

>compare one company's latest and greatest RISC architecture against
>another's aging CISC architecture.
....  Taken to the logic extreme you can compare ENIACs.
>What's the point?  Surely one could select configurations that would better 
>approximate an apples-to-apples comparison.

The problem with seeking "apples to apples" comparisons that that after a
while you are debating "MacIntosh versus Gold Delicious."  You end up
seeking more detail to form an opinion.  I think this is an appropriate
forum, however, we'd have to come to some consensus on methodology.
I think there are a lot of people interested in performance, but few of
them willing to do any work (much of which will be error prone and there
is lots of it).

The basic issues are	"what constitutes 'equivalence?'"
			"how does something compose/decompose?"

When some group of people can reach a consensus on these for graphics
systems (considering their diversity from the Apple II to the top of the
line E&S flight simulator image generation systems).  Then we can compare
systems.  At least two attempts and one SIGGRAPH session have tried to
address some of this without success.  Bogged down.
There is an economic advantage for some parties to keep people
confused and in the dark.

Consensus is how international systems of measurement work.

>Opinions expressed are my own, etc., etc.
Ditto.
--e. nobuo miya, NASA Ames Research Center, eugene@orville.nas.nasa.gov
  {uunet,mailrus,other gateways}!ames!eugene