Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!swrinde!elroy.jpl.nasa.gov!ncar!gatech!usenet.ins.cwru.edu!eagle!data.nas.nasa.gov!wilbur.nas.nasa.gov!eugene
From: eugene@nas.nasa.gov (Eugene N. Miya)
Newsgroups: comp.benchmarks
Subject: Re: benchmarks and interprocedural optimization
Message-ID: <1991Mar21.213144.14636@nas.nasa.gov>
Date: 21 Mar 91 21:31:44 GMT
References: <1991Mar21.184428.8226@convex.com>
Sender: news@nas.nasa.gov
Reply-To: eugene@wilbur.nas.nasa.gov (Eugene N. Miya)
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Lines: 25

In article <1991Mar21.184428.8226@convex.com> metzger@convex.com
(Robert Metzger) writes:
>One other note:  interprocedural optimization will render many industry 
>standard benchmarks meaningless.

Just getting back......
You should be careful about this.
A company, who shall go unnamed, gave us some of their latest software.
We scheduled an expensive stand-alone test of a super (this is a few years
back) expecting to see several CPUs in use.  One CPU was used.  It was
a mistake, being developed on a uniprocessor system, just a simple default
initialization problem.

The problem is how companies like yours go about testing fixes and
optimizations.  Standard test suites exist for language syntax conformance,
deviance, but no such standard set of tests exists for language optimizations.
There needs to be some testing consistency for all this fancy software,
otherwise, how do we have any assurance said features are working.

That's one reason why I just regard this as another form of testing,
and nothing special.  Of course, no company wants embarassing test results.

--e. nobuo utsunomiya, NASA Ames Research Center, eugene@orville.nas.nasa.gov
  {uunet,mailrus,other gateways}!ames!eugene
  AMERICA: CHANGE IT OR LOSE IT.