Path: utzoo!utgpu!water!watmath!clyde!att!pacbell!ames!ucsd!ucsdhub!hp-sdd!ncr-sd!ncrlnk!uunet!munnari!bruce!goanna!yabbie!koel!cit5!olympic!jim!jon
From: jon@jim.odr.oz (Jon Wells)
Newsgroups: comp.arch
Subject: Re: Benchmarking
Message-ID: <572@jim.odr.oz>
Date: 13 Oct 88 06:36:22 GMT
References: <4655@winchester.mips.COM>
Organization: O'Dowd Research Pty. Ltd., Melbourne, Australia.
Lines: 61

From article <4655@winchester.mips.COM>, by mash@mips.COM (John Mashey):
> In article <10498@reed.UUCP> mdr@reed.UUCP (Mike Rutenberg) writes:
> ...
>>But it is so hard to make it run and yet be "the same code."
> 
>>The problem is that to get good results with a given benchmark within a
>>given system, you often do have to tweak things to get comparable
>>numbers, often holding your breath that it all works out.
>> [ stuff deleted ...]
> 
> ONE MORE TIME:
> 	use large, real programs as benchmarks.
> 	do NOT use small programs as benchmarks
> 	be especially careful of small synthetic benchmarks
>[ stuff deleted... ]

Seems to me that there are two quite distinct classes of benchmarks
required (maybe three). There are at least three different levels
of information required....

A) The raw execution rate of a processor under ideal conditions.

B) Now fast does the thing go in a particular system (memory config etc.).

C) Now fast does the system, as a whole, go.

   The following comments do not apply to number crunchers which run
   under `ideal' conditions most of the time so A is perhaps the most
   important thing.

A and B could be solved either by simulation or benchmarking, they are
both *very* processor specific things, and as such the same `code' can
and must be used for both. By the same code I mean the same
*instructions*, if you have to write it in assembler because your
compiler optimizes it out then do it. What you're trying find are the
performance limits of a particular processor/configuration, that is, A
gives you the upper limit of B, and B tells you how well our doing.

Both these things are *only* of interest to the architects of processors
and systems, and C is the *only* thing that the people buying these
systems are interested in. I neither know nor care how many wheatstones
this machine does, I do know that it takes a *very* long time to walk
the directory hierarchy, so long that I've never bothered waiting for
such a program to complete. 

C can only be found by running large complex things that approximate
the systems' end use. I can see no better benchmark than the software
that the system will run. Ken MacDonald's unix benchmark suite,
MUSBUS, is one such example. The suite is floating around on various
servers but you'll need a complete and well debugged unix system
before you'll be able to use it.

The point is, that both types of benchmarks are useful and of interest,
just to different classes of people.

jon.




--