Path: utzoo!attcan!uunet!husc6!think!ames!oliveb!Ozona!chase
From: chase@Ozona.orc.olivetti.com (David Chase)
Newsgroups: comp.arch
Subject: Recognizing benchmarks
Message-ID: <30653@oliveb.olivetti.com>
Date: 13 Oct 88 19:48:41 GMT
References: <2220003@hpausla.HP.COM> <46500026@uxe.cso.uiuc.edu> <6683@nsc.nsc.com> <6684@nsc.nsc.com> <4263@wright.mips.COM> <6729@nsc.nsc.com> <10498@reed.UUCP> <4655@winchester.mips.COM> <6868@nsc.nsc.com> <1710@eos.UUCP>
Sender: news@oliveb.olivetti.com
Reply-To: chase@Ozona.UUCP (David Chase)
Organization: Olivetti Research Center, Menlo Park, CA
Lines: 29

In article <1710@eos.UUCP> eugene@eos.UUCP (Eugene Miya) writes:
> I discovered it was an
>APL compiler not a Fortran compiler.  The benchmark was a simple
>Gaussian sum (3 APL characters).  The benchmark adds 1 thru n, the compiler
>did what Gauss did: you know n(n+1)/2.  It was placed there by the compiler
>writer who knew the person in the APL community who did this as a
>benchmark.

Actually, some people (Perlis; Cheng, Omdahl, & Strawn) have proposed
the use of "idiom recognition" in APL compilation.  It is claimed that
most APL programs are can be covered by a fairly small set (50 or 100) of
APL idioms, which can in turn be specially compiled.  I don't know
what ever happened along those lines -- Cheng et al had problems with
slow table generation, but (plug for own work) new techniques are
several hundred times faster.

Anyhow, given such compilation technology it is fairly easy to
recognize small stupid benchmarks and deal with them appropriately.
If it were me (and I was using pattern-directed compilation) I'd do it
in a flash; it's an easy way to part fools (those who rely on simple
benchmarks) and their dollars.

You don't win by being too clever, either.  I can imagine a diabolical
benchmarker writing code full of multiple-entry (irreducible) loops,
or code that forces all variables into memory; unfortunately, such
code is not the usual case, and so such pathological benchmarks tell
you nothing about the everyday performance of a compiler + architecture.

David