Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!watmath!clyde!caip!rutgers!husc6!panda!genrad!decvax!tektronix!orca!tekecs!kendalla@blast.gwd.tek.com (Kendall Auel)
From: kendalla@blast.gwd.tek.com (Kendall Auel)
Newsgroups: net.arch
Subject: Re: Benchmarks in August IEEE Micro
Message-ID: <7696@tekecs.UUCP>
Date: Wed, 1-Oct-86 12:33:12 EDT
Article-I.D.: tekecs.7696
Posted: Wed Oct  1 12:33:12 1986
Date-Received: Sat, 4-Oct-86 09:04:47 EDT
References: <322@oblio.UUCP> <3600003@hplabsb.UUCP>
Sender: news@tekecs.UUCP
Reply-To: kendalla@blast.UUCP (Kendall Auel)
Organization: Tektronix, Wilsonville OR
Lines: 33

In article <3600003@hplabsb.UUCP> wiemann@hplabsb.UUCP (Alan Wiemann) writes:
>                                                   Instead "[they] had the
>same person modify or write all the tests so [they] could be sure that the same
>algorithms would be used for all the processors" (page 56 of the IEEE article).
>Thus the benchmark results reflect not only the individual processors' ability
>to execute instructions but also the cleverness of this programmer in using
>each microprocessor's instruction set and architecture.  The results reported
>should not be considered true measures of the relative performance of these
>microprocessors.

The Computer Family Architecture (CFA) project of the Army and Navy in the
70's attempted to measure the performance of various computers. This is
considered, I believe, to be one of the ``classic'' benchmark efforts.

    "c	Programmers were not permitted to make _algorithmic_ improvements
	or modifications, but rather were required to translate the PDL
	descriptions into assembly language. Programmers were free to
	optimize their test programs to the extent possible with highly
	optimizing compilers. This ``hand translation'' of strictly defined
	algorithms was expected to reduce variations due to programmer skill."

	"Computer Structures: Principles and Examples" pg.58
	_Siewiorek,_Bell,_and_Newell_, McGraw-Hill 1982

If you are measuring the performance of a processor, then it is not
necessarily desireable to compile a standard program. If instead you
are measuring compiler/processor combined performance, then you
should certainly use the same source for all measurements. It is
much easier to rewrite a poor compiler than to redesign a poor architecture.

Kendall Auel
Tektronix, Inc.
(I don't claim or disclaim anything)