Path: utzoo!attcan!uunet!tektronix!reed!mdr
From: mdr@reed.UUCP (Mike Rutenberg)
Newsgroups: comp.arch
Subject: Benchmarking
Message-ID: <10498@reed.UUCP>
Date: 4 Oct 88 05:05:14 GMT
References: <2220003@hpausla.HP.COM> <46500026@uxe.cso.uiuc.edu> <6683@nsc.nsc.com> <6684@nsc.nsc.com> <4263@wright.mips.COM> <6729@nsc.nsc.com>
Reply-To: mdr@reed.UUCP (Mike Rutenberg)
Organization: Reed College, Portland OR
Lines: 32

In article <6729@nsc.nsc.com> grenley@nsc.UUCP (George Grenley) writes:
>some official number, rather than measured data.  Also, there are frequently
>variations in supposedly standard code.  I have different versions of Dhry1.1
>which vary over 40%, even though they are supposedly the same code.

But it is so hard to make it run and yet be "the same code."

The problem is that to get good results with a given benchmark within a
given system, you often do have to tweak things to get comparable
numbers, often holding your breath that it all works out.

In compiling the dhrystone benchmark, a C compiler I use will remove
the loop overhead calculation since it is simply a loop that has
no body and a trivial side effect on the iteration variable.  Even
procedure calls are eliminated by the compiler if it is clear the
called procedure does not do anything.

@BEGIN(Black Magic)
You can do things to trick the compiler into keeping the loop.  A null
procedure the loop calls will do the trick if compiled separately.  But
then you have to also put a call to this null procedure in the main
dhrystone loop.  But this may do bad things to your numbers, especially
if it affects your cache hit-rate.  And this will change the numbers
you get, not in a positive way.
@END(Black Magic)

I wish benchmarks would be rewritten to be a ultimately portable &
really really smart about outwitting too-smart compilers.  It would be
nice to be able to run a benchmark program totally unchanged.  This
would avoid the temptation or need to modify the tests.

Mike