Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!rutgers!apple!voder!pyramid!prls!mips!mash
From: mash@mips.COM (John Mashey)
Newsgroups: comp.arch
Subject: Re: Benchmarking
Message-ID: <4655@winchester.mips.COM>
Date: 7 Oct 88 04:10:28 GMT
References: <2220003@hpausla.HP.COM> <46500026@uxe.cso.uiuc.edu> <6683@nsc.nsc.com> <6684@nsc.nsc.com> <4263@wright.mips.COM> <6729@nsc.nsc.com> <10498@reed.UUCP>
Reply-To: mash@winchester.UUCP (John Mashey)
Organization: MIPS Computer Systems, Sunnyvale, CA
Lines: 47

In article <10498@reed.UUCP> mdr@reed.UUCP (Mike Rutenberg) writes:
...
>But it is so hard to make it run and yet be "the same code."

>The problem is that to get good results with a given benchmark within a
>given system, you often do have to tweak things to get comparable
>numbers, often holding your breath that it all works out.

>In compiling the dhrystone benchmark, a C compiler I use will remove
>the loop overhead calculation since it is simply a loop that has
>no body and a trivial side effect on the iteration variable.  Even
>procedure calls are eliminated by the compiler if it is clear the
>called procedure does not do anything.

>@BEGIN(Black Magic)
>You can do things to trick the compiler into keeping the loop.  A null
>procedure the loop calls will do the trick if compiled separately.  But
>then you have to also put a call to this null procedure in the main
>dhrystone loop.  But this may do bad things to your numbers, especially
>if it affects your cache hit-rate.  And this will change the numbers
>you get, not in a positive way.
>@END(Black Magic)

>I wish benchmarks would be rewritten to be a ultimately portable &
>really really smart about outwitting too-smart compilers.  It would be
>nice to be able to run a benchmark program totally unchanged.  This
>would avoid the temptation or need to modify the tests.

(back from 3 weeks' Down Under; it will take a while to catch up!)

ONE MORE TIME:
	use large, real programs as benchmarks.
	do NOT use small programs as benchmarks
	be especially careful of small synthetic benchmarks

Two of the most counterproductive things people can be doing are:
	a) Tuning compilers to optimize small benchmarks, especially
	with optimizations that don't really matter much on real
	programs. (Optimizations that actually matter elsewhere are fine.)
	b) Continually reworking synthetic benchmarks to stay ahead
	of advances in compiler optimization.
It is sad how much effort across this business has gone down the rat-holes.
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	{ames,decwrl,prls,pyramid}!mips!mash  OR  mash@mips.com
DDD:  	408-991-0253 or 408-720-1700, x253
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086