Path: utzoo!mnetor!tmsoft!torsqnt!news-server.csri.toronto.edu!rutgers!usc!zaphod.mps.ohio-state.edu!mips!sgi!yohn@tumult.asd.sgi.com
From: yohn@tumult.asd.sgi.com (Mike Thompson)
Newsgroups: comp.arch
Subject: Re: reliable/reproduceable benchmarks on SGI MIPS box
Summary: caches colored correctly
Message-ID: <79610@sgi.sgi.com>
Date: 3 Jan 91 02:46:41 GMT
References: <11737@alice.att.com> <44383@mips.mips.COM>
Sender: guest@sgi.sgi.com
Organization: Silicon Graphics, Inc., Mountain View, CA
Lines: 43

In article <44383@mips.mips.COM>, cprice@mips.COM (Charlie Price) writes:
> In article <11737@alice.att.com> andrew@alice.att.com (Andrew Hume) writes:
> >
> >	I am running some benchmarks on a variety of machines
> >and in particular, on a SGI 4D/380, a multiprocesor with 8
> >33MHz R3000 cpus.
> ...
> >	my problem is that I see quite large variations over
> >multiple runs of the same benchmark, sometimes as much as
> >1.26%....
> >
> >	andrew hume
> >	andrew@research.att.com
> 
> One source of variability in benchmark times that nobody else has
> mentioned (so I will) is cache conflicts.
> Identical exeuctions of a benchmark use the same *virtual* locations
> in the same pattern, but these virtual locations get mapped to
> physical locations, and in particular cache locations, in some
> manner determined by the OS, previous activity on the machine,
> the phase of the moon...
> If subsequent executions of the program get different patterns
> of cache conflict then you can easily see several percent
> difference in the execution time due to differences in cache conflict.
> This isn't just speculation.
> In the early days at MIPS some maddening variability in execution times
> was finally traced to variability in page alocation.
> The execution variability mostly went away when the OS did page coloring
> (matching the physical and virtual address of a page in certain ways)
> to remove the cache-use variability.
> 
> I suspect that if the OS isn't giving you reproducible use of the
> caches that you won't ever be able to get reproducible benchmark times.
> -- 
> Charlie Price    cprice@mips.mips.com        (408) 720-1700
> MIPS Computer Systems / 928 Arques Ave. / Sunnyvale, CA   94086-23650

Good guess, but this is probably not the cause.  The SGI OS (a.k.a.
IRIX) manages page/cache coloring.  It is only when memory is very
tight that a process can likely get pages not cache-aligned optimally.

Mike Thompson	yohn@sgi.com
Silicon Graphics Computer Systems