Path: utzoo!attcan!telly!lethe!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!mips!winchester!mash
From: mash@mips.COM (John Mashey)
Newsgroups: comp.benchmarks
Subject: Re: PERFECT rumor
Message-ID: <44271@mips.mips.COM>
Date: 21 Dec 90 19:23:39 GMT
References: <6439@mace.cc.purdue.edu> <1990Dec19.190758.8285@ux1.cso.uiuc.edu> <1990Dec21.071228.12282@csrd.uiuc.edu>
Sender: news@mips.COM
Reply-To: mash@mips.COM (John Mashey)
Organization: MIPS Computer Systems, Inc.
Lines: 40

In article <1990Dec21.071228.12282@csrd.uiuc.edu> gc@sp12.csrd.uiuc.edu (George Cybenko) writes:
>conte@crest.crhc.uiuc.edu (Tom Conte) writes:
>
>>In article <6439@mace.cc.purdue.edu>, abe@mace.cc.purdue.edu (Vic Abell) writes:
>
>>Although I haven't heard about this, I want to point out it is difficult
>>to `violate' the run rules for the PERFECT club.  Unlike SPEC, the PERFECT
>>club allows modification of the code for vectorization, parallelization,
>>etc. (including hand modification). It's part of the philosophy: monkey
>>around with the code as much as you wish to make it run well on your machine,
>>just make sure the results are the same.  For me, this makes PERFECT club
>>numbers rather hard to interpret.

By the way, in support of PERFECT, note that SPEC and PERFECT club have
legitimately different goals, leading to the different measurement
philosophies.  Due to the desire for SPEC to offer better metrics for
vendors to fight over, we avoid changing the code, to avoid gimmicks
in reporting.
On the other hand, I think it is GOOD that Perfect allows such changes,
as it can be very instructive to see:
	a) How close the default and tuned versions are
	b) How much work it takes to get from one to the other
	c) What one can learn for hardware and software design
In addition, it probably makes a lot more sense in the super-computer
world or near-super-computer world, where people:
	a) Often have their own code
	b) Often have access to source, at least
	c) Can often gain get major gains by tuning, given the nature
	of these algorithms
which is different from the workstation/supermini world, where
more often one is using more third-party software, often not having
access to the source, and where the most common programs do not offer
as much scope for tuning.  (There is always room for tuning, but
not the orders-of-magnitude effects sometimes gainable in the
vector, parallel, vector-parallel world.)
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	 mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash 
DDD:  	408-524-7015, 524-8253 or (main number) 408-720-1700
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086