Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!seismo!sundc!pitstop!sun!decwrl!labrea!bloom-beacon!husc6!endor!reiter
From: reiter@endor.harvard.edu (Ehud Reiter)
Newsgroups: comp.arch
Subject: Re: Towards A Meaningful Performance Measure
Message-ID: <3113@husc6.UUCP>
Date: Thu, 5-Nov-87 09:16:24 EST
Article-I.D.: husc6.3113
Posted: Thu Nov  5 09:16:24 1987
Date-Received: Sun, 8-Nov-87 02:45:59 EST
References: <861@winchester.UUCP> <2993@phri.UUCP> <864@tut.cis.ohio-state.edu> <3907@sol.ARPA> <881@mips.UUCP>
Sender: news@husc6.UUCP
Reply-To: reiter@harvard.UUCP (Ehud Reiter)
Organization: Aiken Computation Lab Harvard, Cambridge, MA
Lines: 42
Keywords: benchmarks

In article <881@mips.UUCP> hansen@mips.UUCP (Craig Hansen) writes:
>As to standarizing on a single compiler/OS, remember that the company
>producing the base architecture has an interest in making their machines
>look competitive. When competitors say their machine is 10X a VAX 780, when
>using trussed up benchmarks and a markedly inferior compiler/OS on the VAX,
>DEC should by all rights be screaming bloody murder. Should DEC claim that
>their VAX 780 is 1.5 times faster than their VAX 780?

At one time, I was convinced that there was a well-defined "inflation"
pattern in MIPS as you went from big computer company to little computer
company.  So, for example, an IBM "1 MIPS" machine would have the same
"performance" as a DEC "2 MIPS" machine, which had the same performance as
a SUN "4 MIPS" machine, which had the same performance as an "8 MIPS" machine
from brand X start-up computer company.  Each company in the hierarchy would
ignore its smaller competitors (as being beneath its dignity to comment on),
and proudly claim that it had a large price/performance advantage over its
larger competitors.

Today, as the computer industry finally starts being somewhat competitive
(as opposed to being a monopoly by you know who), I think the big companies
(IBM, DEC) are being forced to be a bit more sophisticated in their
marketing, and stress things customers really care about, like software,
reliability, peripherals, and performance in specific applications (usually
floating-point or I/O intensive applications, not "integer crunching" ones).
It's mainly the little companies (and university people) who still go around
bragging about magic numbers which they pull from thin air and call "MIPS".

The problem with MIPS is that attempts to measure "integer crunching"
performance, and

	(a)   It is impossible to summarize "integer crunching" performance
in one number.
	(b)   In any case, not many customers care about integer crunching
performance.


So, I think that attempts to give good definitions for "MIPS" will remain
fairly academic exercises, of little relevance to the "real world" of computing.

					Ehud Reiter
					reiter@harvard	(ARPA,BITNET,UUCP)
					reiter@harvard.harvard.EDU  (new ARPA)