Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.2 9/18/84; site nmtvax.UUCP
Path: utzoo!linus!philabs!cmcl2!lanl!unmvax!unm-cvax!nmtvax!blaine
From: blaine@nmtvax.UUCP
Newsgroups: net.arch
Subject: Re: MIPs, But really IDC numbers (long)
Message-ID: <408@nmtvax.UUCP>
Date: Thu, 11-Apr-85 14:24:49 EST
Article-I.D.: nmtvax.408
Posted: Thu Apr 11 14:24:49 1985
Date-Received: Mon, 15-Apr-85 04:01:26 EST
References: <>
Reply-To: blaine@nmtvax.UUCP ( Gaither)
Distribution: na
Organization: New Mexico Tech, Socorro
Lines: 90
Summary: 

in article <> wfmans@ihuxb.UUCP (w. mansfield)

MIPs are loosely related to IDC numbers, and in many respects IDC numbers
are more interesting.  MIPs tend to be linear with IDC#s.  
I know of no benchmark for MIPs which is
standardized between vendors.  I have worked for several computer vendors
in the past(non IBM compatible) in competitive performance.  All that we could
find out was that each vendor had his own way of measuring MIPs. (i.e. load
the mix up with "typical jobs" and use a hardware monitor to count
instructions).  

Among non IBM compatible vendors the problems get worse because not only is
there no common workload, but no common operating system, and vastly different
instruction  sets.

IDC numbers are loosely related to MIPs.  Most people get their MIPs and IDC#
directly or indirectly from Computerworld.  Computerworld is a weekly
publication with a very large subscription base.

What used to happen was when a new computer was announced, Computerworld(CW)
would call up vendors and ask for a MIPs rating.  The vendor would complain
that it is not really a meaningfull metric, but in the end would end up
providing a MIPs rating (the last thing you would want is for them to guess
or ask your competitor for a number). 

Eventually CW (as I remember) started publishing "relative performance" 
numbers.  These were cited as having come from the IDC corporation, the sister
company to CW.

A friend of mine, Steve Howard, decided to find out what IDC numbers mean.
Initially, all we could find out was that everyone agrees that the starting
point is that an IBM 370/158-3 =45. Steve went right to the source and 
eventually talked with Will Zachman, V.P. of R&D for International Data
Corporation.  He told Steve that the IDC number is an approximate measure
of performance; in fact, Will said that the numbers do not attempt to predict
throughput, CPU power or any other specific measureable quantity.  He did claim
that the numbers can be used to rank systems by "performance" and indicate
approximately (again no definition) the ratios between them.

IDC usually does not rate non-IBM architectures, but is often asked to do
so by CW(they are sister companies).  When you see "CW estimate" in 
in Computerworld, it means that IDC has refused to give them numbers for that
system. God knows how CW comes up with their estimates when IDC refuses to
give them an estimate, but its probably by the same process that IDC uses.

IDC does no measurements.  The main ingredients of the numbers are 
vendor-supplied statements comparing 2 systems (i.e. The 3033 is n.n times
as powerful as the 4341-2).  If  the number produced by this method seems
unreasonable to IDC or is objected to by users or vendors, "adjustments" 
may be made.  IDC is very willing to accept informantion which is claimed 
to increase the accuracy of its numbers.

The result of this is that almost no work or thought (opinion) is put into
establishing the numbers except to react to complaints.

WHEN THE SQUEAKING STOPS, THE WHEEL IS CONSIDERED OILED.

Given this sorry state of affairs concerning their creation,  we decided to
find out what we could by looking at the numbers themselves.  Why? because
management asks, Why? because marketing asks, Why? because customers ask, 
Why? because the consumer is always right (if not correct).

We decided to look at how IDC#s relate to CPU power.  To find out, we turned
to CPU power informantion published by the Institute for Software Engineering
(ISE).  ISE used instruction kernals to measure CPU power for three workload 
classes:  Moves and compares, Engineering Calculations, and Decimal Arithmetic.

We plotted the IDC number vs. ISE power for each workload type.  The systems
considered were IBM 4331-1, CDC 480-II, IBM 4341-2, IBM 3032, NAS/7000, 
IBM 3033N, IBM 3033U, and AHMDAL V8.  We observed a very linear relationship
with all three Power measures.

This showed us that the published IDC numbers could be used (at some risk) as
a measure of raw CPU power (not throughput) in spite of the haphazard method
used in making them.  One can obtain "accurate" IDC numbers for non-IBM arch
systems by using architecture-independent processor power metrics on your 
systems and a couple IBM systems with known IDC#s, and after assuring yourself
that there are the proper linear relationships, develop and apply the linear 
translation.

Note that IDC#s and MIPs imply a certain combination of power delivered 
to workloads (three cups COBOL, two cups FORTRAN).  So it is really most 
meaningful (if I dare use that word) on business type systems.  

I only use IDC#s or MIPs  when it is the only data available on a competitive
system.  Loose MIPs sink ships!

-- 
Blaine Gaither                    ucbvax!unmvax!nmtvax!blaine
Computer Science Department       blaine@nmt