Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!sun-barr!decwrl!decwrl.pa.dec.com!joel
From: joel@pandora.pa.dec.com (Joel McCormack)
Newsgroups: comp.sys.mips
Subject: Variations in benchmark timings, any explanation?
Keywords: benchmarks, MIPS, R2000, R3000
Message-ID: <2247@bacchus.dec.com>
Date: 7 Dec 89 19:09:31 GMT
Sender: news@decwrl.dec.com
Reply-To: joel@pandora.pa.dec.com (Joel McCormack)
Lines: 24

When running x11perf, a program to measure X11 graphics and windowing
performance, I sometimes get measurements that can vary by 10% or more. 
x11perf first runs each benchmark to calibrate how many times total it
should run the benchmark, chooses a number that will result in total
time of about 5 seconds, then run that benchmark 5 times.

The calibration run gets everything into caches, and then the 5 actual
runs usually match up within a few percent of each other.  However, I
have noticed that the order in which I run benchmarks can affect
timings, or if I run another program between two runs of the same
benchmark the timings may be different.  These variations often exceed
10%, which seems rather large.

Does anyone have an explanation for this behavior?  I could understand
cache conflicts that might occur within the X11 server if I relink the
server, but these differences show up using the same binary for both the
server and x11perf.  Pages from the code files might get loaded into
different locations on different runs, but since the caches are much
larger than a page size, I would expect this to have no effect.

Any ideas, however harebrained, would be appreciated.


- Joel McCormack (decwrl!joel, joel@decwrl.dec.com)