Path: utzoo!mnetor!uunet!husc6!sri-unix!garth!walter
From: walter@garth.UUCP (Walter Bays)
Newsgroups: comp.arch
Subject: Re: What's a Vax 11/780 MIP really?
Message-ID: <573@garth.UUCP>
Date: 23 Mar 88 19:19:19 GMT
References: <310@uwslh.UUCP> <6304@ames.arpa>
Reply-To: walter@garth.UUCP (Walter Bays)
Organization: INTERGRAPH (APD) -- Palo Alto, CA
Lines: 25
Summary: Simple benchmarks can't measure complex architectures.

In article <6304@ames.arpa> lamaster@ames.arc.nasa.gov.UUCP (Hugh LaMaster) writes:
>In fact, most of the so-called "MIPS" figures are based on benchmarks
>anyway - usually the vendor's idea of the "typical" workload - which
>is begging the question since users often pick machines based on the
>ability to handle particular workloads.  [...]
Yes.  Benchmarks should be at least somewhat representative of the real
workload the user will run.

>Certainly, some simple benchmarks can tell you something- how fast can
>you fork and exec - but how about file system performance, network
>performance, etc. ?
The problem is that computer architectures are becoming too complex for
performance to be accurately measured by simple benchmarks.  These
benchmarks are usually small, with a narrow locality of memory
reference, do no I/O, and are run with a single user and a single
process.  Even if they get the instruction mix right, they can be poor
predictors of performance on real workloads which (MAY) have large
programs, multiple users, multiple processes per user, I/O, network
activity, window management, ...

Benchmark suites that test each of these functions individually are an
improvement, but what is really needed is a way to test them all
concurrently.  (Preferably without porting 100 applications and data
files and running them for a year :-)  (I hope I got this posted right.
I'm still learning my way around the net.)