Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!uunet!jarthur!bridge2!mips!winchester!mash
From: mash@mips.COM (John Mashey)
Newsgroups: comp.arch
Subject: Re: SPECmarks
Message-ID: <36438@mips.mips.COM>
Date: 25 Feb 90 02:34:34 GMT
References: <7393@pdn.paradyne.com> <3300102@m.cs.uiuc.edu>
Sender: news@mips.COM
Reply-To: mash@mips.COM (John Mashey)
Organization: MIPS Computer Systems, Inc.
Lines: 51

In article <3300102@m.cs.uiuc.edu> gillies@m.cs.uiuc.edu writes:

>Several people have made the point that SPEC is supposed to be a
>"real-world" program benchmark, consisting of "today's" programs.  
>My understanding is that a benchmark is a standard, just as a 1-meter
>bar of platinum stored in France is a standard.  Computers are
>measured by the SPEC standard.  Standards should be timeless.  The
>only time we change them is when we can simplify them and perhaps
>simultaneously make them more precise (e.g. measuring meters using
>atomic emissions).
>
>I question the utility of a benchmark that is not a standard, a
>benchmark that is an ever-moving target composed of "today's"
>programs, a piece of software incapable of predicting the performance
>of "tomorrow's" programs on "today's" computers.
>
>Originally, Mr. Henry M. Spencer complained that a small benchmark
>would have a difficult time testing cacheing affects.  Well, SPEC is
>no better than Dhrystone in this respect.  SPEC is just as frozen in
>time as dhrystone was, and 5 years from now SPEC will be just as
>useless at testing cacheing, since caches will certainly be over a
>megabyte by then, even on PC's.


It has been publicly stated many times that:
	a) Once we put a benchmark in the set, the reference time is frozejn,
	to avoid the "VAX-mips-time-warp" problem of comp;aring against
	different things.
	b) We are actively working to add benchmarks to the set over time.
	c) If we decide a benchmark has become useless, or was a bad idea,
	or turns out not to do what we thought it did, or whatever, we
	have no inhibition from:
		1) Simply deleting it.
		2) Modfiying it to create a new benchmark, under a new number,
		and deleting the old one.
	What we DON'T do is modify one of the components, avoiding having
	multiple variants floating around that get easily confused and
	mis-stated.  It won't bother us at all to supercede Suite 1.0
	with something better.....

I'm sad to hear that what we've done so far is "no better than Dhrystone",
because if that's true, a whole bunch of us have wasted, in toto, at
least several million $ to try to do something better....

In any case, constructive criticism is always welcomed; these things are
hardly perfect.....
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	{ames,decwrl,prls,pyramid}!mips!mash  OR  mash@mips.com
DDD:  	408-991-0253 or 408-720-1700, x253
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086