Path: utzoo!attcan!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!wuarchive!uwm.edu!ux1.cso.uiuc.edu!csrd.uiuc.edu!sp12.csrd.uiuc.edu!gc
From: gc@sp12.csrd.uiuc.edu (George Cybenko)
Newsgroups: comp.benchmarks
Subject: Re: PERFECT rumor
Message-ID: <1990Dec21.071228.12282@csrd.uiuc.edu>
Date: 21 Dec 90 07:12:28 GMT
References: <6439@mace.cc.purdue.edu> <1990Dec19.190758.8285@ux1.cso.uiuc.edu>
Sender: news@csrd.uiuc.edu (news)
Organization: UIUC Center for Supercomputing Research and Development
Lines: 63

conte@crest.crhc.uiuc.edu (Tom Conte) writes:

>In article <6439@mace.cc.purdue.edu>, abe@mace.cc.purdue.edu (Vic Abell) writes:
>> I have been asked if I have heard that several vendors have had to
>> resubmit their PERFECT club reports.  I haven't heard anything, and
>> I don't even know if the rumor suggests that previously submitted
>> reports were erroneous.
>> 

>Although I haven't heard about this, I want to point out it is difficult
>to `violate' the run rules for the PERFECT club.  Unlike SPEC, the PERFECT
>club allows modification of the code for vectorization, parallelization,
>etc. (including hand modification). It's part of the philosophy: monkey
>around with the code as much as you wish to make it run well on your machine,
>just make sure the results are the same.  For me, this makes PERFECT club
>numbers rather hard to interpret.

There seems to be some confusion about the philosophy and methodology of
the PERFECT Benchmarks.  This may clarify the above remarks:

a.	PERFECT methodology allows baseline (compiler optimizations only)
	and optimized (hand tuning) results.  Both are published in the
	PERFECT reports;

b.	Each of the 13 programs is presented in a single file and at the last
	meeting of the group, it was agreed that baseline means the
	programs should be compiled as single files, not broken out into
	individual subroutines, etc so that different compiler options
	could be applied to different program parts.  This was an
	unanticipated twist that had not been previously discussed
	and everyone was asked to check their results in light of this
	stronger notion of baseline.  Another aspect of baseline
	is that all system components must be released products
	(no beta versions, etc.).

I hope this explains some of the comments of Abell and Conte.  As far as
we know, all results published to date are correct....new data that
has NOT been published is being verified now.  

The PERFECT Benchmarks are 13 FORTRAN applications codes (and datasets) drawn 
from science and engineering totaling about 60,000 lines.  The original
goals (the effort was started in 1987) were to evaluate supercomputer 
performance on full applications, not kernels or algorithms.  Unlike the
SPEC codes which rehash Linpack and NAS Kernels, the PERFECT codes
really do fluid mechanics, seismic migration, molecular dynamics, etc.

Because they are relatively easy to port now and represent codes more 
typical of what you would find scientists and engineers using, they 
are used a lot by compiler writers and architecture researchers in 
simulations and experiments.  The performance numbers are just one aspect
of the effort.

I could go on, but if you are interested in getting some papers and
reports with data, send email to  Cathy Warmbier (warmbier@csrd.uiuc.edu) 
asking for the PERFECT papers and reports.  Requests for the codes 
should also go to Cathy.  I've asked her not to send things out
until the new data is final, so don't expect anything until early
January.

George Cybenko
Center for Supercomputing Research and Development
University of Illinois at Urbana
gc@csrd.uiuc.edu     (217) 244-4145