Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!brutus.cs.uiuc.edu!ux1.cso.uiuc.edu!ux1.cso.uiuc.edu!m.cs.uiuc.edu!gillies
From: gillies@m.cs.uiuc.edu
Newsgroups: comp.arch
Subject: Re: SPECmarks
Message-ID: <3300102@m.cs.uiuc.edu>
Date: 24 Feb 90 10:39:34 GMT
References: <7393@pdn.paradyne.com>
Lines: 21
Nf-ID: #R:pdn.paradyne.com:7393:m.cs.uiuc.edu:3300102:000:1101
Nf-From: m.cs.uiuc.edu!gillies    Feb 23 17:54:00 1990


Several people have made the point that SPEC is supposed to be a
"real-world" program benchmark, consisting of "today's" programs.  
My understanding is that a benchmark is a standard, just as a 1-meter
bar of platinum stored in France is a standard.  Computers are
measured by the SPEC standard.  Standards should be timeless.  The
only time we change them is when we can simplify them and perhaps
simultaneously make them more precise (e.g. measuring meters using
atomic emissions).

I question the utility of a benchmark that is not a standard, a
benchmark that is an ever-moving target composed of "today's"
programs, a piece of software incapable of predicting the performance
of "tomorrow's" programs on "today's" computers.

Originally, Mr. Henry M. Spencer complained that a small benchmark
would have a difficult time testing cacheing affects.  Well, SPEC is
no better than Dhrystone in this respect.  SPEC is just as frozen in
time as dhrystone was, and 5 years from now SPEC will be just as
useless at testing cacheing, since caches will certainly be over a
megabyte by then, even on PC's.