Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!ucbvax!bloom-beacon!apple!oliveb!sun!chiba!khb
From: khb%chiba@Sun.COM (Keith Bierman - SPD Languages Marketing -- MTS)
Newsgroups: comp.arch
Subject: Re: Criteria ... [really: are N designs better than 1?]
Message-ID: <104996@sun.Eng.Sun.COM>
Date: 16 May 89 03:59:19 GMT
References: <2368@ogccse.ogc.edu> <1464@cfa.cfa.harvard.EDU> <141@dg.dg.com> <156@dg.dg.com> <658@pitstop.West.Sun.COM> <19088@winchester.mips.COM> <169@dg.dg.com>
Sender: news@sun.Eng.Sun.COM
Reply-To: khb@sun.UUCP (Keith Bierman - SPD Languages Marketing -- MTS)
Organization: Sun Microsystems, Mountain View
Lines: 56

In article <169@dg.dg.com> uunet!dg!mpogue (Mike Pogue) writes:
>
>  MIPS is a LOT better at providing real numbers, and Sun is better than Intel
>(although I understand that their claims for the Sparcstation are still more
>like PEAK numbers than sustained numbers.  Anybody have any REAL data?).

The widely published Sun figures are derived from taking popular (but
generally silly) benchmarks and computing the geometric mean vs. the
mvII. I am sorry, but I must refuse to debate the merits of this
approach publically.

There are groups at Sun who run customer benchmarks (up to a million
lines of code or so) and turn the results over to the customers.
Personally I believe those results much more than I do contrived
benchmarks. Of course the result is typically only something the
owners of the program fully understand ... or have any right to see
(it is, after all, their code ..  not ours :>).  

Overall the SS-1 is slower than a DECstation 3100 which is slower than
a SS 330. Mileage varies from code to code; and often from dataset to
dataset. It should also be noted that the compilers are under constant
improvement.... so the performance changes from time to time ....
(this is true for all other vendors also)

I hope to have a white paper of sorts in a month or so; with some case
studies. 

>
>  My preference is to have a third party do the numbers (no fudging allowed!).

Typically no understanding of the machine either. In former
incarnations I was on the user side. It was (and still is) that most
third parties don't know how to run the machine right. So as a
customer I got each vendor to run my code, and explain what they did.
The more complex the machine (say a vector machine, or a VLIW or
hypercube) the more imperative it was (is) to get a very savvy (yet
honest) benchmarker to do the actual work. On the Cydra 5, for
example, knowing how to run the compiler and employ compiler
directives could increase performance fourfold .... 

It has been my experience that _most_ vendors tell the truth in these
tests; though they may shade it a bit :>  

>Look for an upcoming issue of MIPS magazine for a relatively unbiased set of
>numbers on our AViiON ($7995) workstation.

The folks at MIPS mag seem very well intentioned; and I wish them the
best of luck. But the two issues I've studied do not reflect very much
depth. The AIM benchmark they rely on does not, IMHO, do a very good
job of characterising a system. (* it does seem kinda odd that they
chose the name of the mag w/o realizing that it was in use by one of
the companies they would be covering .... :> *)
Keith H. Bierman      |*My thoughts are my own. Only my work belongs to Sun*
It's Not My Fault     |	Marketing Technical Specialist    ! kbierman@sun.com
I Voted for Bill &    |   Languages and Performance Tools. 
Opus  (* strange as it may seem, I do more engineering now     *)