Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!swrinde!cs.utexas.edu!sun-barr!newstop!texsun!convex!metzger
From: metzger@convex.com (Robert Metzger)
Newsgroups: comp.benchmarks
Subject: Re: benchmarks and interprocedural optimization
Message-ID: <1991Mar23.182154.22395@convex.com>
Date: 23 Mar 91 18:21:54 GMT
References: <1991Mar21.184428.8226@convex.com> <1991Mar21.213144.14636@nas.nasa.gov>
Sender: usenet@convex.com (news access account)
Organization: Convex Computer Corporation, Richardson, Tx.
Lines: 69
Nntp-Posting-Host: bach.convex.com

In article <1991Mar21.213144.14636@nas.nasa.gov> eugene@wilbur.nas.nasa.gov (Eugene N. Miya) writes:
>In article <1991Mar21.184428.8226@convex.com> metzger@convex.com
>(Robert Metzger) writes:
>>One other note:  interprocedural optimization will render many industry 
>>standard benchmarks meaningless.
>
>Just getting back......
>You should be careful about this.
>
>The problem is how companies like yours go about testing fixes and
>optimizations.  Standard test suites exist for language syntax conformance,
>deviance, but no such standard set of tests exists for language optimizations.
>There needs to be some testing consistency for all this fancy software,
>otherwise, how do we have any assurance said features are working.
>
>That's one reason why I just regard this as another form of testing,
>and nothing special.  Of course, no company wants embarassing test results.

I appreciate your kind concern, but I'm not sure what your're warning me
about :-)

I'll elaborate on my point:
1) Compilers that perform interprocedural optimization render meaningless
   the results of simple-minded benchmarks.  For example, one of the first
   things a well-meaning technical-sales-support person at CONVEX did when he
   got a crack at the Application Compiler was to run whetstone through it.
   Sure enough, since whetstone attempts to measure call overhead and the APC
   attempts to minimize call overhead, the results made it look like CONVEX
   had released a new generation of hardware. 
   
   We didn't release those numbers to the public, but there's nothing that
   prevents an unsophisticated buyer from doing the same thing and being
   misled.  (Of course if he's buying million dollar computers based on the
   results of such stupid benchmarks, maybe he deserves what he gets.)

2) Compilers that perform interprocedural optimization cannot be fooled
   by subroutine calls into not performing optimizations.  This is a common
   trick in some widely used benchmarks.   Well, an IPO compiler will
   certainly perform thorough IP side-effect analysis, and it will know
   exactly what is going on inside the procedure.  It may automatically
   perform procedure inlining as well.  Procedure calls no longer prevent
   compilers from moving assignments and uses, or eliminating them altogether,
   where procedure compilers could not do so.  Procedure calls are no longer
   firewalls.  

3) Compilers that perform interprocedural optimization perform a lot more
   compile-time evaluation of your code.   Basically, if you don't read
   your all data from a data file at runtime, the compiler could potentially
   evaluate your expressions at compile time and replace your application
   with a bunch of PRINT statements.  

I will be the first to say that testing interprocedural optimizers is very
challenging.  In some senses, a 100,000 line application is ONE test case.
Which means you have to run hundreds of applications and millions of lines
of code through the compiler before you trust it.

On the other hand, our experience with an interprocedural compiler is that 
the compiler find a dozens or hundreds of bugs in every application for
every one bug the application finds in the compiler.  Logic errors, abuse
of language features, and numerically unstable algorithms are all exposed
by this new technology.  From my perspective, 99.99% of all scientific/
engineering FORTRAN applications are shot through with numerous errors,
and I'm far more concerned about the reliability of the applications than
the compilers at this point.
--
Robert Metzger		CONVEX Computer Corp.  		Richardson, Texas
Generic Disclaimer:	I write software, not CONVEX Corporate Policies.
"The only legitimate function of government is to protect its citizens from
harm or property loss by violence, stealth, or fraud.  All else is tyranny."