Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!mips!mash
From: mash@mips.com (John Mashey)
Newsgroups: comp.benchmarks
Subject: Re: Price/Performance figures for Number-Crunching
Message-ID: <1670@spim.mips.COM>
Date: 1 Apr 91 18:20:42 GMT
References: <MCCALPIN.91Mar18165912@pereland.cms.udel.edu> <21720002@hpfcso.FC.HP.COM>
Sender: news@mips.COM
Organization: MIPS Computer Systems, Inc.
Lines: 68
Nntp-Posting-Host: winchester.mips.com

In article <21720002@hpfcso.FC.HP.COM> maf@hpfcso.FC.HP.COM (Mark Forsyth) writes:
>
><From: mash@mips.com (John Mashey)
><It turns out that the SPECint
><metric (the 4 integer C benchmarks) is relatively stable, i.e.,
><it has a fairly narrow confidence interval, small variance, etc,
><on every machine ever measured.  SPECfp tends to vary a lot more,
><i.e., it is alwasy good to run your own benchmarks, but you'd be especially
><tempted to do so when comparing machines on FP performance, because it
><varies all over the map, especially on the more vectorizable ones.
>
>You seem to be confusing "performance relative to a VAX 11/780" with
>"performance".  Since the units chosen for the SPEC numbers are relative
No, don't think so.
>to a VAX 11/780, a flat curve on the individual components means only
>that the system faithfully reproduces all of the relative strengths, and
>weaknesses, of the VAX, NOT that it is necessarily a "balanced" design. 
>A small variance is not necessarily a good goal to design for.  The 780 
Of course not.  I didn't say that the point of all this was to
build something that mimicked a VAX.  let me try again:
	a) On every machine for which SPEC has published benchmarks
	(with the sole exception of early DN10000s, for which some
	compiler bug was fixed, removing the exception), the bottom-to-
	top ratio amongst the integer benchmarks is at worst 1.5X to 1,
	but more usually, 1.2-1.3X to 1.  Many companies have internal
	data that correlates well with the SPEC integer benchmarks,
	across machines (not just on VAXen, i.e., the ratios have a similar
	property regardless of which machine you pick.)
	Thus, every bit of data that I have says that if I knew I had
	a new integer benchmark to be run on two machines, I'd expect that
	the performance ratio would quite often be close (+/-15%) of
	the ratio of the SPECint metrics.  (Not always, but quite often.)
	b) The point is that the SPECfloat does NOT have this property,
	because the variance is much higher - I've seen at least as much
	as 8X from bottom to top, and as I recall, the new HP's have
	something like 5-6X from bottom to top.  All of this says:
	1) IF I had to predict the performance of a new FP benchmark
	on two machines, and I knew nothing but their SPECfloat numbers,
	and knew nothing about the benchmark, I'd guess the SPECfloat
	ratio, for lack of anything else.
	2) However, I'd expect a MUCH wider range of measured performance
	ratios, i.e., much more often, I'd expect to find inversions in
	the measured performance compared to predicted, or at least
	a wider range of ratios.
	3) All of this behavior is PERFECTLY in line with the observed behavior
	seen in the super-computer/minisupercomputer world forever.
	In particular, it continues to show that there are things to do
	to drastically accelerate vector-ish code, and hard to do much
	about integer and scalar FP.
>was very good at integer intensive workloads, but, one would expect that 
Actually, compared to many machines, the VAX was better on FP than integer...
Or, put another way, for many systems, SPECfp numbers are lower than SPECint.
>modern RISCs with 64 bit floating point units and an additional 10 years 
>of evolution in compiler technology to perform relatively better on FP 
>intensive applications.....
Yes; I think we all agree on that.  The issue is not the performance
of FP relative to integer, but of the variance. An amusing thing to do
is to plot the SPEC numbers for IBM RS6000/540, Stardent 3010,
and MIPS RC6280.  The SPECfp numbers are fairly similar, but the
relative ORDER changes from benchmark to benchmark.
Certainly, given the likely changes going on, we'll probably have to
change the SPEC scale from linear to log, to be able to show both
integer and FP numbers on same chart. :-)
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	 mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash 
DDD:  	408-524-7015, 524-8253 or (main number) 408-720-1700
USPS: 	MIPS Computer Systems MS 1/05, 930 E. Arques, Sunnyvale, CA 94086