Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!utgpu!water!watmath!clyde!rutgers!rochester!crowl
From: crowl@rochester.UUCP
Newsgroups: comp.arch
Subject: Towards A Meaningful Performance Measure
Message-ID: <3806@sol.ARPA>
Date: Mon, 2-Nov-87 09:49:44 EST
Article-I.D.: sol.3806
Posted: Mon Nov  2 09:49:44 1987
Date-Received: Wed, 4-Nov-87 20:03:05 EST
References: <861@winchester.UUCP> <2993@phri.UUCP> <864@tut.cis.ohio-state.edu>
Reply-To: crowl@cs.rochester.edu (Lawrence Crowl)
Organization: U of Rochester, CS Dept, Rochester, NY
Lines: 37
Keywords: benchmarks

In article <864@tut.cis.ohio-state.edu> manson@tut.cis.ohio-state.edu
(Bob Manson) writes:
>Alright, I'm a little pissed over the recent posting of these supposed
>"performance figures". Exactly what are these things supposed to mean?  Well,
>they are compiled programs on different systems that are run and supposed to
>represent the speed of the various processors, MIPS etc.  We all know that
>MIPS is mostly a meaningless figure (Okay, so there's some debate there, but I
>see no meaning to how many instructions per second a processor runs-"The
>COPYMEM instruction blockmoves the entire memory to disk space, but takes
>1,000,000 usec to execute, with an effective MIPS of 1/1000000").

This is in part a terminology problem.  John Mashey's performance brief states
mips as relative to a VAX 11/780.  (Note that the 780 executes about 500,000
instructions per second.  Its original one mips figure came from early
comparisons with IBM machines claiming one mips.)  I suggest we stop calling
performance "mips", and start being more specific about what we really mean.
I suggest the term "Vax Relative Performance".

Unfortunately, that is not enough.  We must define what configuration of Vax
we use as the baseline.  I suggest an 11/780 with full memory and a floating
point accellerator.  CPU oriented benchmarks should run completely in physical
memory.

The compiler and operating system also affect performance.  To make the base
machine highly available, both should be common.  I suggest Unix BSD 4.2 as
the base operating system and the portable C compiler as the base C language
compiler.  This allows realistic Unix/C benchmarks like grep, nroff, etc.  Note
that such benchmarks must have the same source.  Putting a better compiler on
the Vax will increase its relative performance, so DEC can honestly sell a 780
as having a Vax Relative Performance greater than one.

Of course, 780's are becoming scarce.  We may have to pick another machine just
to keep the base machine readily available.  Suggestions?
-- 
  Lawrence Crowl		716-275-9499	University of Rochester
		      crowl@cs.rochester.edu	Computer Science Department
...!{allegra,decvax,rutgers}!rochester!crowl	Rochester, New York,  14627