Xref: utzoo comp.arch:19985 comp.sys.sgi:7525
Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!usc!sdd.hp.com!mips!lloyd!cprice
From: cprice@mips.COM (Charlie Price)
Newsgroups: comp.arch,comp.sys.sgi
Subject: Re: reliable/reproduceable benchmarks on SGI MIPS box
Message-ID: <44383@mips.mips.COM>
Date: 28 Dec 90 22:24:12 GMT
References: <11737@alice.att.com>
Sender: news@mips.COM
Reply-To: cprice@mips.COM (Charlie Price)
Followup-To: comp.arch
Organization: Your Organization Goes Here
Lines: 40

In article <11737@alice.att.com> andrew@alice.att.com (Andrew Hume) writes:
>
>	I am running some benchmarks on a variety of machines
>and in particular, on a SGI 4D/380, a multiprocesor with 8
>33MHz R3000 cpus.
...
>	my problem is that I see quite large variations over
>multiple runs of the same benchmark, sometimes as much as
>1.26%. Now, the resolution of the timer is .01s and i should se
>an accuracy of about .01/40 or .025%. I am a factor of 50 off this.
>does anyone know how i can run these benchmarks so as to get reproducible
>timings? (i note as an aside that just running the benchmarks on the cray
>in multi-user mode yields variations of the order of .15% which is
>satisfactory).
>
>	andrew hume
>	andrew@research.att.com

One source of variability in benchmark times that nobody else has
mentioned (so I will) is cache conflicts.
Identical exeuctions of a benchmark use the same *virtual* locations
in the same pattern, but these virtual locations get mapped to
physical locations, and in particular cache locations, in some
manner determined by the OS, previous activity on the machine,
the phase of the moon...
If subsequent executions of the program get different patterns
of cache conflict then you can easily see several percent
difference in the execution time due to differences in cache conflict.
This isn't just speculation.
In the early days at MIPS some maddening variability in execution times
was finally traced to variability in page alocation.
The execution variability mostly went away when the OS did page coloring
(matching the physical and virtual address of a page in certain ways)
to remove the cache-use variability.

I suspect that if the OS isn't giving you reproducible use of the
caches that you won't ever be able to get reproducible benchmark times.
-- 
Charlie Price    cprice@mips.mips.com        (408) 720-1700
MIPS Computer Systems / 928 Arques Ave. / Sunnyvale, CA   94086-23650