Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!usc!wuarchive!uunet!telesoft!choll
From: choll@telesoft.com (Chris Holl @adonna)
Newsgroups: comp.benchmarks
Subject: Re: bc benchmark [sigh]
Summary: One number?
Message-ID: <1142@telesoft.com>
Date: 28 Dec 90 01:52:01 GMT
References: <44342@mips.mips.COM> <15379@ogicse.ogi.edu>
Organization: TeleSoft, San Diego, CA.
Lines: 51

In article <15379@ogicse.ogi.edu>, borasky@ogicse.ogi.edu
(M. Edward Borasky) writes:

>   ...do you believe that a single-number that measures a computer's
>   speed doesn't exist?

When I worked at Boeing Computer Services we typically looked for one
number to compare two machines.  The comparison was very narrow in
scope however.  We compared one vendor's computer to their next box.
As long as the architecture stays the same, such comparisons are
valid.  This was needed because as a computer service, there had to be
a way to consistently charge customers independent of which box their
job actually ran on.  The CPU times needed to be normalized so it
wouldn't matter if a job ran on a Cyber 175 or 760.  Cray 1S or X-MP.
In fact, we had to guarantee this for our government customers who
insisted that their bill for the same job should always be within some
percentage (5%, I think).

The CPU ratios was determined by running 10 to 14 CPU kernels such as
linear code, loops, subroutine calls, memory fetches (in and out of
stride), matrix reductions, etc.  Again, as long as the architecture
was the same (or close enough) the ratios stayed pretty constant (and
typically close to the ratio of the clocks, which was usually the
biggest difference).  Where the architecture changed we had trouble 
justifying one number.

For example, we compared a Cray 1 to a Cray X-MP.  The clocks were
12.5 to 9.5 nanoseconds.  All the ratios looked fine (1.32 give or
take a bit) except scatter/gather which was 10 to 14 times faster on
the X!  (Hardware scatter/gather - architecture change.)  A job that
performed a lot of scatter/gather would burn different amounts of CPU
seconds on the different Crays.

The other "one number" we used was for capacity planning.  After
maturing through many yardsticks of throughput, one of my fellows
(Dr. Howard "Doc" Schemising - wonderful guy) developed a capacity 
test that would precisely model the current workload on a machine.
This was used with great accuracy to measure the capacity of different
machine (for that workload).

Anyway, summing up my ramblings, one number is okay for a given
architecture or a given application.  Unfortunately that is not what
most people are looking for.  They want you to tell them how fast
their jobs are going to be on machine X if they are this fast on Y.
My answer has always been "Depends what you're doing." which rarely
satisfies 'em.  :-).

Chris Holl
TeleSoft (formally of BCS)
5959 Cornerstone Ct. W.
San Diego, CA  92121