Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!utgpu!water!watmath!clyde!rutgers!rochester!crowl
From: crowl@rochester.UUCP
Newsgroups: comp.arch
Subject: Benchmarking in a High Level Language
Message-ID: <3802@sol.ARPA>
Date: Mon, 2-Nov-87 09:30:46 EST
Article-I.D.: sol.3802
Posted: Mon Nov  2 09:30:46 1987
Date-Received: Wed, 4-Nov-87 20:02:13 EST
References: <861@winchester.UUCP> <2993@phri.UUCP> <864@tut.cis.ohio-state.edu>
Reply-To: crowl@cs.rochester.edu (Lawrence Crowl)
Organization: U of Rochester, CS Dept, Rochester, NY
Lines: 40
Keywords: benchmarks

In article <864@tut.cis.ohio-state.edu> manson@tut.cis.ohio-state.edu
(Bob Manson) writes:
>It means nothing to try to compare my computer with my dumb compiler to a
>well-developed VAX machine with a real compiler.  So why run benchmarks in
>compiled languages???

People run real code in compiled languages, so it is the performance of the
compiled languages that has any meaning.

>It's easier that way, you don't have to write individual programs for each
>machine that might actually show off their abilities and improved instructions.

You also don't introduce the programmer for each machine as another variable.
The competence of the programmer, on the particular machine, can make a large
difference in the resulting speed.  The high level language approach also
prevents the assembly programmer from "cheating" with algorithmic short cuts.

>I'll admit that it does mean something for compiled language users but not
>really for performance comparisons-you're comparing apples and oranges, or
>really the efficiency of the compilers on the machines. 

The effectiveness of compiling a high level language into the particular
architecture affects the performance of the architecture.  After all, people
use high level langauges.  There are two approaches to fair benchmarks.

1) Use whatever _publicly_ available compilers you want on the target machine.
   If one compiler is better than another, tough.  The available compilers
   define the available compute power of the machine.

2) Attempt to equalize the effect of the compiler.  This allows you to get at
   the architectural performance better.  Note that "equivalence" of compilers
   is dependent on their size, their development time, and their execution time.

The first approach is best for customers choosing a machine, while the second
approach is best for designers evaluating an architecture.  In both cases, the
use of a high level language is appropriate.
-- 
  Lawrence Crowl		716-275-9499	University of Rochester
		      crowl@cs.rochester.edu	Computer Science Department
...!{allegra,decvax,rutgers}!rochester!crowl	Rochester, New York,  14627