Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!ucsd!ogicse!borasky
From: borasky@ogicse.ogi.edu (M. Edward Borasky)
Newsgroups: comp.benchmarks
Subject: SPEC vs. Dhrystone
Message-ID: <15546@ogicse.ogi.edu>
Date: 3 Jan 91 01:38:43 GMT
References: <44342@mips.mips.COM> <15379@ogicse.ogi.edu> <44353@mips.mips.COM> <1685@marlin.NOSC.MIL>
Distribution: comp.benchmarks
Organization: Oregon Graduate Institute (formerly OGC), Beaverton, OR
Lines: 38

In article <1685@marlin.NOSC.MIL> aburto@marlin.nosc.mil.UUCP (Alfred A. Aburto) writes:
>In article <44353@mips.mips.COM> mash@mips.COM (John Mashey) writes:
>>(Note, for example, that published Dhrystone results easily mis-predict
>>SPEC integer benchmarks pretty badly, i.e., it is quite easy for machine
>>"a" to be 25% faster on Dhrystone than "b", and end up 25% SLOWER on more
>>realistic integer benchmarks.)
>This is an interesting observation (result).
>Dhrystone was intended to be REPRESENTATIVE of TYPICAL integer
>programs. That is, hundreds (I believe) of programs were 
>analyzed to come up with the (ahem) 'typical' high level
>language instructions and their frequency of usage. In view of this 
>I would, at first sight, suspect the Dhrystone to be more accurate
>than SPEC as SPEC is based upon only a few integer programs. 
I suspect there are two factors at work here.  First, Dhrystone is a
fairly small benchmark, and would not exercise the memory hierarchy 
as hard as the real programs in SPEC.  The second factor is that it is
easier to tune a compiler to a small benchmark like Dhrystone (or for
that matter Whetstone and the Livermore Loops) than it is to tune for
a variety of different real programs.  By the way, I believe Dhrystone
was originally written in ADA and was translated to "C", the form in
which it is usually run.
>Real programs also show a great variation in performance.  I noticed
>this recently in a Scientific American article (Jan 1991) which
>showed the comparison of 13 different real programs on a wide
>variety of supercomputers.  
This is the PERFECT CLUB benchmark suite.  I just picked up the article;
when I glanced at it in the supermarket checkout line (some people
look at Oprah's diets, some don't) I suspected that there might be a
howler or two lurking within the article.  If I see anything that looks
suspicious I will post something to comp.sys.super or here or both.
>The program 'megflop' variation in 
>perfromance was truly tremendous especially for the fastest systems
>(Cray and a NEC computer I think).
This variation is a fact of life in supercomputing.  If you take any
suite of benchmarks that is fairly comprehensive, you will see much
more variation in performance for a supercomputer than you will for a 
scalar machine.  This is true for both absolute Megaflops and for speed
relative to a fixed SCALAR machine.