Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!mips!mash
From: mash@mips.com (John Mashey)
Newsgroups: comp.arch
Subject: Re: More Snake bytes.
Message-ID: <1762@spim.mips.COM>
Date: 3 Apr 91 00:01:52 GMT
References: <2004@kuling.UUCP> <8840021@hpfcso.FC.HP.COM>
Sender: news@mips.COM
Organization: MIPS Computer Systems, Inc.
Lines: 79
Nntp-Posting-Host: winchester.mips.com

In article <8840021@hpfcso.FC.HP.COM> maf@hpfcso.FC.HP.COM (Mark Forsyth) writes:
>>From: mash@mips.com (John Mashey)
>>
>>PLEASE: do we have to keep seeing cost/MIPS, where everybody
>>computes mips differently.  I'll express a little irritation at this:
>>people MIGHT have computed Price/SPECmark and Price/SPECint,
>>where the latter is an intelligent approximation to Price/VAX-mips-integer.
>
>And why not Price/SPECfp also ?  Don't workstation customers still care
>highly about performance on FP applications ?  Why did SPEC choose six
>FP intensive benchmarks if these are not important ?  Is the importance
>of a benchmark suite proportional to how competitive MIPs is on it ?
>Why not also include I/O, graphics, X11, etc. ?
To short-circuit this before it gets out of hand (maf and I have had
side-conversations already, calming things done, I think),
where all of this came from, I think is:
a) It was the understanding of various members of SPEC that if you
published SPECmarks, that the full-disclosure form be available at the
same time, in particular all 10 numbers.  Most people thought that the
license says exactly that, but it says something slightly vaguer.

b) In general, SPEC members, if they provided a SPECmark,
also provided the 10 numbers.  In particular, various members of SPEC,
from day 1, have been adamant in NOT signing up to anything that
didn't have full disclosures of whatever sort the full disclosure
consensus agreed on. (me, among others :-)  For instance, some companies
regularly send out the SPEC form along with the initial press release
on a product.

c) Unfortunately, not everybody at HP was involved in all of the
discussions to this effect, and the license wording is not as explicit
as people thought it had been. In addition, in large companies,
the SPEC members do not have as much direct influence on marketing as
they might in smaller ones. As a result, although CLEARLY not intended
by the HP SPEC folks, and DEFINITELY not signed up to by many of the
people who've helped SPEC exist, was an important period of time
during which:
	a) The analysts and press had the SPECmark number and a MIPS-rating,
	because that is what they'd been given.
	b) They beat up unmercifully on various HP competitors.
	Since all they had was MIPS and SPECmark, that's what they used,
	and whether intended or not, those 2 numbers alone are
	misleading. 
	c) When a press person or analyst called you up, there was no
	rational reply of any sort possible, because you couldn't
	get the 10 individual numbers.  I.e., you were handed a situation
	in which what you'd thought you'd agreed to was a certain kind
	of disclosure (to avoid the problems of single-number things),
	and you'd supported that, but you were now getting hammered
	with exactly the thing you thought you hadn't agreed to....
	(It was bad enough that the numbers are quite good;
	it was worse not knowingthe shape of the curves.)

d) Now, the numbers are now available, and one can do whatever analysis
makes any sense on them, and the 10 numbers are MUCH more meaningful
than 1 number, especially when there is a high variance.
Both integer and FP are important; it is important to distinguish,
not mix them together; it is important to see all 10 numbers to see
the variation pattern.  $/SPECmark is fine, so is $/SPECfp.  However,
using either of those two, and then using $/dhry-mips instead of
$/SPECint is uncool.... (And I know it doesn't originate with the SPEC
folks, HP or otherwise.  Almost EVERYBODY involved in SPEC has
"interesting" times "educating" their marketing groups....)

e) Anyway, SPEC is working to make sure the rules are clarified in
everybody's mind, and it's no big deal in the long run.  HP usually
does an excellent, credible job documenting its performance - the current
performance document is fine.  The main issue here was the gap in time
between when SPECmark numbers were given out and when the full
10-number disclosure was available, and we just have to be clearer about the
rules of the game....  This is especially important as we move from
the (relatively) easy CPU benchmarks towards more complex systems benchmarks.

...well, back to work...
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	 mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash 
DDD:  	408-524-7015, 524-8253 or (main number) 408-720-1700
USPS: 	MIPS Computer Systems MS 1/05, 930 E. Arques, Sunnyvale, CA 94086