Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!sdd.hp.com!spool.mu.edu!munnari.oz.au!yoyo.aarnet.edu.au!sirius.ucs.adelaide.edu.au!chook!gordoni
From: gordoni@chook.adelaide.edu.au (Gordon Irlam)
Newsgroups: comp.benchmarks
Subject: Running the SPEC benchmarks (pedantic questions)?
Keywords: SPEC, runrules
Message-ID: <2767@sirius.ucs.adelaide.edu.au>
Date: 5 Apr 91 02:29:53 GMT
Article-I.D.: sirius.2767
Sender: news@ucs.adelaide.edu.au
Reply-To: gordoni@chook.adelaide.edu.au (Gordon Irlam)
Lines: 46
Nntp-Posting-Host: chook.ua.oz.au

When running the SPEC 1.2 benchmarks the times reported are supposed to
be "the best times that can be consistently reproduced".

How should the word "consistently" be interpreted?

As meaning "nearly always".  Eg. "We ran the benchmark and consistently
got a time of less than 90.6 seconds."

Or as meaning "more than once".  Eg. "One in every 7 runs gives a time
of 90.6 seconds or better, and thus we are able to consistently reproduce
the reported time."

Is it permitted to run a benchmark several times in succession, rather than
cycle through them in order if this gives better results?  (For instance
the operating system might dynamically tune its memory management algorithms
in response to the decreased demand for new pages).

Running benchrun results in 3 sh's, make, and tee occupying memory as
well as the benchmark.  On a system with little memory this could result
in some extra paging when the benchmark first starts to run.  In addition
a tee is likely to cause several undesirable context switches.  Is it
permissible to run the benchmark from make?  What about typing in the
time command directly to the shell? (This would prevent a cp, which might
make it easier for some operating systems to keep the executable in memory.
It would also prevent the cost of scheduling the write back of the files
that would have been cp'ed - update is likely to perform a sync while the
benchmark is running.  For gcc we are talking of about writing back roughly
1M)?  What about typing in the cp, and then waiting 30 seconds before
starting the benchmark?  If you are very clever you could time when make
started running so that a sync occurs just before the cp has finished, but
before the benchmark starts running.  Is this allowed?  Are the results
always required to be written out to a file, or once the results have been
validated, can /dev/null be used.

Should the SPECmark figure be computed using infinite precision arithmetic,
or should the rounded SPECratios of each benchmark be used?

Which of the above possibilities, while stricly permitted would be considered
to depart from the honest, cooperative spirit in which SPEC was established?

Which of the above possibilities do most vendors employ?

                                            Thanks for any help,

                                                Gordon Irlam
                                                (gordoni@cs.adelaide.edu.au)