Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!lll-crg!nike!sri-spam!sri-unix!hplabs!hplabsb!wiemann
From: wiemann@hplabsb.UUCP (Alan Wiemann)
Newsgroups: net.arch
Subject: Re: Benchmarks in August IEEE Micro
Message-ID: <3600003@hplabsb.UUCP>
Date: Sat, 20-Sep-86 22:45:00 EDT
Article-I.D.: hplabsb.3600003
Posted: Sat Sep 20 22:45:00 1986
Date-Received: Thu, 2-Oct-86 21:02:31 EDT
References: <322@oblio.UUCP>
Organization: Hewlett-Packard Laboratories - Palo Alto, CA
Lines: 18

Benchmark comparisons give valid results only if the same program is presented
to each machine.  The compiler is considered part of the "machine" and its
performance contributes to the overall performance of the machine.  This study
did not present the same program to each machine.  Instead "[they] had the
same person modify or write all the tests so [they] could be sure that the same
algorithms would be used for all the processors" (page 56 of the IEEE article).
Thus the benchmark results reflect not only the individual processors' ability
to execute instructions but also the cleverness of this programmer in using
each microprocessor's instruction set and architecture.  The results reported
should not be considered true measures of the relative performance of these
microprocessors.
   Unfortunately benchmark comparisons often suffer from commercial hype and
unscientific methods.  Let the buyer beware!

                                   Alan Wiemann
                                   Aitchpeelabs
                                   Palowaltocalifornya
                                   hplabs!wiemann