Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!lll-winken!lll-lcc!pyramid!prls!mips!hansen
From: hansen@mips.UUCP (Craig Hansen)
Newsgroups: comp.arch
Subject: Re: Towards A Meaningful Performance Measure
Message-ID: <881@mips.UUCP>
Date: Thu, 5-Nov-87 00:56:57 EST
Article-I.D.: mips.881
Posted: Thu Nov  5 00:56:57 1987
Date-Received: Sat, 7-Nov-87 17:49:37 EST
References: <861@winchester.UUCP> <2993@phri.UUCP> <864@tut.cis.ohio-state.edu> <3907@sol.ARPA>
Lines: 43
Keywords: benchmarks
Summary: Understand intent - but it doesn't work that way

In article <3907@sol.ARPA>, crowl@cs.rochester.edu (Lawrence Crowl) writes:
> My intent was that the base architecture/operating system/compiler be constant.
> This means that some readily available compiler be part of the base system.  I
> wanted to avoid cases where people benchmark relative to two different
> compilers.  I want people to use the compiler appropriate to their machine.

Unfortunately, the base architecture/operating system/compiler, from a
competitive point of view, isn't constant. Each prospective customer has a
set of machines and applications that are relevant to them. VAX (TM DEC) VMS
(TM DEC) users generally stick up their noses at VAX BSD 4.2-relative
measurements, because they know that the BSD compilers are inferior. The VAX
780 is fast becoming an irrelevant machine, because faster/cheaper VAXes are
available. UNIX V (TM ATT) comes with a file-system that's only good for
coffee breaks and lunch hours, so file system performance relative to an
off-the shelf sys-V machine isn't meaningful....you get the idea.

When you're talking about (decimal) orders of magnitude of performance above
a VAX 780, the two machines being compared aren't in the same regime
anymore. (Remember that the VAX 780 is approaching its tenth anniversary for
goshsakes.) The real problem is that the characteristics of the architecture
are going to be different for 10-20-50 MIPS machine than they are for a VAX
780, in terms of cache and memory system design, pipelining, compiler
optimizations, register windows, etc., so that the selection of programs to
run on the machine heavily influences the performance ratio. We've all seen
how much Dhrystone overestimates the performance of "10 MIPS" RISC machines,
compared to larger, more realistic workloads. The only good thing I'll say
about the VAX 780 is that its scalar floating point performance (about a
MegaDoubleWhetstone) is fairly well-balanced against its scalar integer
performance (about 1 MIPS).  Thus, if you build a machine in which FP
applications and integer applications both perform about the same V.R.P.,
you'll have a reasonably well-balanced machine.

As to standarizing on a single compiler/OS, remember that the company
producing the base architecture has an interest in making their machines
look competitive. When competitors say their machine is 10X a VAX 780, when
using trussed up benchmarks and a markedly inferior compiler/OS on the VAX,
DEC should by all rights be screaming bloody murder. Should DEC claim that
their VAX 780 is 1.5 times faster than their VAX 780?

-- 
Craig Hansen
Manager, Architecture Development
MIPS Computer Systems, Inc.
...decwrl!mips!hansen