Path: utzoo!utgpu!jarvis.csri.toronto.edu!rutgers!cs.utexas.edu!uunet!dg!mpogue
From: mpogue@dg.dg.com (Mike Pogue)
Newsgroups: comp.arch
Subject: Re: Criteria ... [really: are N designs better than 1?]
Message-ID: <173@dg.dg.com>
Date: 16 May 89 14:25:46 GMT
References: <2368@ogccse.ogc.edu> <1464@cfa.cfa.harvard.EDU> <141@dg.dg.com> <156@dg.dg.com> <658@pitstop.West.Sun.COM> <19088@winchester.mips.COM> <169@dg.dg.com> <104996@sun.Eng.Sun.COM>
Reply-To: uunet!dg!mpogue (Mike Pogue)
Organization: Data General, Westboro, MA.
Lines: 42

In article <104996@sun.Eng.Sun.COM> khb@sun.UUCP (Keith Bierman - SPD Languages Marketing -- MTS) writes:
>>
>>  My preference is to have a third party do the numbers (no fudging allowed!).
>
>Typically no understanding of the machine either. In former
>incarnations I was on the user side. It was (and still is) that most
>third parties don't know how to run the machine right. So as a
>customer I got each vendor to run my code, and explain what they did.
>directives could increase performance fourfold .... 

  Yes, I agree that it is important to get somebody knowledgable to run
the benchmarks.  Just any third party will not do (sorry I didn't make this
clearer).

>
>It has been my experience that _most_ vendors tell the truth in these
>tests; though they may shade it a bit :>  

  Actually, some vendors shade the data quite a bit.  e.g. Intel quotes
simulated, unrolled BLAS, no-wait-state memory, and bug-free chips in
their i860 benchmarks.  Also, Sun made quite a mistake in claiming that their 
new Sparcstation 1GX could create 3D polygons twice as fast as Silicon
Graphics Personal Iris (Sun's new machine doesn't even handle 3D polygons).
(NOTE:  To be fair, Sun has claimed that their error was an honest mistake, although
it seems to me to be amazingly to their benefit to make such a mistake.)           

  And this is exactly my point.  Vendors who do stuff like this cast doubt on 
ALL the vendor numbers, making a third party solution important (and, as you
mention, the ultimate benchmark is the CUSTOMER's application).

>
> [They] chose the name of the mag w/o realizing that it was in use by one of
>the companies they would be covering .... :> *)

  Actually, as I understand it, MIPS magazine had the name before MIPS (the
company) was created.  :)

Mike Pogue
Data General Co.
Westboro, MA.

My opinions are my own....