Path: utzoo!attcan!uunet!lll-winken!ames!vsi1!wyse!mips!mash
From: mash@mips.COM (John Mashey)
Newsgroups: comp.arch
Subject: Re: Criteria ... [really: are N designs better than 1?]
Message-ID: <19483@winchester.mips.COM>
Date: 12 May 89 03:43:36 GMT
References: <2368@ogccse.ogc.edu> <1464@cfa.cfa.harvard.EDU> <141@dg.dg.com> <156@dg.dg.com> <658@pitstop.West.Sun.COM> <19088@winchester.mips.COM> <169@dg.dg.com>
Reply-To: mash@mips.COM (John Mashey)
Organization: MIPS Computer Systems, Sunnyvale, CA
Lines: 56

In article <169@dg.dg.com> uunet!dg!mpogue (Mike Pogue) writes:

>  MIPS is a LOT better at providing real numbers, and Sun is better than Intel
>(although I understand that their claims for the Sparcstation are still more
>like PEAK numbers than sustained numbers.  Anybody have any REAL data?).
OPINION: this is a typical difference between a systems company and a chip
company; not surprising, given the differing natures of business.

>  My preference is to have a third party do the numbers (no fudging allowed!).
>Look for an upcoming issue of MIPS magazine for a relatively unbiased set of
>numbers on our AViiON ($7995) workstation.
This will be good to see.  The only caveat to be careful of is that nobody
is better than the vendor at optimizing code for their own machine, i.e.,
they really know the options and how to use them.  This leads to a good
set of rules of thumb:
	The best benchmarks are run by a 3rd-party, with a vendor looking
	over their should saying "Use -O3!".
	Even with the most honest attitude in teh world, a vendor will
	always use the best options on their own machine, and may miss some
	when running the same benchmark on somebody else's.

>  Anybody know how the SPEC group is doing (John Mashey?)?   
Yep.  We're closing in the the contents of the first tape, which basically
represent mostly technical [CASE, system commands, ECAD, MCAD, scientific]
benchmarks for the first round.  We're busy trading benchmarks around
and checking them out, with the following sorts of rules:
	1) code has to be reasonably portable
	2) code must be public-domain, or copyleft, or something whose
	source can be given out for benchmarking purposes.
	3) The program has to have a nontrivial I-cache or D-cache use,
	or both.
	4) The program should run, at least, about 60 seconds on an
	8-VUPs machine; preferably more like 5-10 minutes [but you'd
	be surprised how HARD it is to get reasoanble benchmarks that
	do this.....]
	5) The benchmarks must be real applications, or at worst
	large hunks of code from real applications.
	6) There must be some added value in us working on this,
	i.e., there is little value for us to include Livermore Loops,
	as it already does what it does, is widely used, etc.
I'll publish some more details a bit later; right now I'm in frenzy
state getting the details together for a "bench-a-thon: or "SPECtacle",
where we hook up a bunch of machines and do the last crunch of
trying to get exactly the same sources, and well-paramaterized makefiles,
and consistent documentation, for this.  This will happen in June,
followed by a 2-month review by all of the rest of the SPEC members,
and then we'll issue the first round tape as we continue evaluating
benchmarks for the next round.  In second round we hope to get more
systems-y and/or commercial benchmarks.  [In fact, if by some miracle,
somebody out there has some really good COBOL benchmarks....]
That's all for now; more later.
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	{ames,decwrl,prls,pyramid}!mips!mash  OR  mash@mips.com
DDD:  	408-991-0253 or 408-720-1700, x253
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086