Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!zaphod.mps.ohio-state.edu!samsung!xylogics!merk!alliant!linus!linus!eachus
From: eachus@linus.mitre.org (Robert I. Eachus)
Newsgroups: comp.benchmarks
Subject: Re: bc benchmark [sigh]
Message-ID: <EACHUS.90Dec31152920@aries.linus.mitre.org>
Date: 31 Dec 90 20:29:20 GMT
References: <44342@mips.mips.COM> <15379@ogicse.ogi.edu>
	<2195@shodha.enet.dec.com>
Sender: usenet@linus.mitre.org
Organization: The Mitre Corporation, Bedford, MA
Lines: 45
Nntp-Posting-Host: aries.mitre.org
In-reply-to: alan@shodha.enet.dec.com's message of 31 Dec 90 05:32:14 GMT


     There is a way to have and use a single meaningful (balanced)
benchmark number, but only to do intial selection:

     Say you choose SPECmarks.  (I use Dhrystones, but that is a minor
detail.)  What you end up with are two things, one a standard single
number rating, and the other a set of comments or annotations telling
the particular strengths and weaknesses of various machines.  First of
all, realize that there are really only three shades of gray where
buying hardware is concerned: More than fast enough, very tight, and
no way Jose.  If you pick a machine in the grey area, there is a major
tradeoff between cost to optimize code for the machine selected, and
the cost of more than adequate hardware.  Unfortunately, supercomputer
useres and real-time people sometimes find that there IS no other
choice, but I digress.

     If your single number correlates well with price, then choosing
the right machine for the job entails first benchmarking your
application on whatever machine you have around.  (I often see
"estimates" of code size and run-time that are off by two orders of
magnitude. If you can't get within a factor of two or three, why
bother spending time benchmarking the hardware?)  At this point we now
have an estimate that say application Y will require a 50 MIPS VAX.
You can now characterize the application in terms such as integer vs.
floating, vectorizeable vs. scalar, I/O intensive vs. compute bound,
or single task instensive vs. multitasking, and check out the machines
in the range you need with the strengths you want.  This will often
come down to a single choice or at least a single processor family
that you need to run your application specific benchmark on.

     This method of choosing hardware DOES require running application
specific benchmarks twice.  But, at least in the applications that I
care about, you are fooling yourself if you don't write and run a good
application specific benchmark to start with.  Using the SPECmark
suite and tailoring it to your particular application might give a
better fit, but the amount of work required to determiine the
coefficients is usually more than that involved in this approach.

--

					Robert I. Eachus

with STANDARD_DISCLAIMER;
use  STANDARD_DISCLAIMER;
function MESSAGE (TEXT: in CLEVER_IDEAS) return BETTER_IDEAS is...