Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!seismo!sundc!pitstop!sun!decwrl!labrea!rutgers!clyde!burl!codas!usfvax2!pdn!alan
From: alan@pdn.UUCP (Alan Lovejoy)
Newsgroups: comp.arch
Subject: Re: Towards A Meaningful Performance Measure
Message-ID: <1714@pdn.UUCP>
Date: Tue, 3-Nov-87 15:00:11 EST
Article-I.D.: pdn.1714
Posted: Tue Nov  3 15:00:11 1987
Date-Received: Sat, 7-Nov-87 10:40:15 EST
References: <861@winchester.UUCP> <2993@phri.UUCP> <864@tut.cis.ohio-state.edu> <3806@sol.ARPA>
Reply-To: alan@pdn.UUCP (0000-Alan Lovejoy)
Organization: Paradyne Corporation, Largo, Florida
Lines: 44
Keywords: benchmarks

In article <3806@sol.ARPA> crowl@cs.rochester.edu (Lawrence Crowl) writes:
>[Re: "mips", let's] start being more specific about what we really mean.
>I suggest the term "Vax Relative Performance".
>
>Unfortunately, that is not enough.  We must define what configuration of Vax
>we use as the baseline.  I suggest an 11/780 with full memory and a floating
>point accellerator.  CPU oriented benchmarks should run completely in physical
>memory.
>
>The compiler and operating system also affect performance.  To make the base
>machine highly available, both should be common.  I suggest Unix BSD 4.2 as
>the base operating system and the portable C compiler as the base C language
>compiler.  This allows realistic Unix/C benchmarks like grep, nroff, etc.  Note
>
>Of course, 780's are becoming scarce.  We may have to pick another machine just
>to keep the base machine readily available.  Suggestions?

The installed base of 11/780's is puny by comparison to the popular
micros.  Why not choose one of them?  I suggest the IBM PS/2 Model 60 as
the baseline.  Most systems will be faster, so that the relative
performance ratios will mostly be above 1.  Most benchmarkers will be
able to get access to such a machine.  Most readers will have some idea
of the real performance of the baseline machine.  And the machine is
likely to be around for a long time, so that familiarity with it will
not be short-lived.  And even after its demise, people will remember it.

I strongly vote against PCC as the compiler.  It is known to vary
considerably in code-generation quality on different architectures.
Not good.  It's better if people use the best compiler avaiable for 
the system, whatever that is.

As for operating system, I suggest an average based on standalone, 
UNIX V and the most widely used OS on that system, if that is not
UNIX V.  UNIX V is clearly headed for standard status.  Leave UNIX out 
if it's not available.  I also strongly urge against UNIX utilities as 
benchmarks:  not everyone uses or can use UNIX (a sacrilege, I know).

--alan@pdn

P.S. I'm a Mac enthusiast and despise the Intel architecture, but that
only makes the idea of having the PS/2 m60 as a baseline even *more*
attractive since it would create an atmosphere where everyone would
market their systems as being x times faster than Big Blue's pride and
joy.