Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!usc!apple!netcom!amdcad!dgcad!dg-rtp!webo!cheshirecat!lewine
From: lewine@cheshirecat.webo.dg.com (Donald Lewine)
Newsgroups: comp.benchmarks
Subject: Re: Running the SPEC benchmarks (pedantic questions)?
Keywords: SPEC, runrules
Message-ID: <1991Apr5.200922.4095@webo.dg.com>
Date: 5 Apr 91 20:09:22 GMT
References: <2767@sirius.ucs.adelaide.edu.au>
Sender: usenet@webo.dg.com (Usenet Administration)
Reply-To: uunet!dg!lewine
Organization: Data General Corporation
Lines: 30

In article <2767@sirius.ucs.adelaide.edu.au>, gordoni@chook.adelaide.edu.au (Gordon Irlam) writes:
|> When running the SPEC 1.2 benchmarks the times reported are supposed to
|> be "the best times that can be consistently reproduced".
	[[Many ways to cheat deleted]]
|> Which of the above possibilities do most vendors employ?
|> 
It is fair game for a customer to say, "I want you to reproduce your
claimed SPECmark numbers on *my* configuration as part of the 
acceptance test."  Vendors must be able to say, "No problem" or 
point out that the SPECmark numbers cannot be reproduced because the
customer is buying a smaller configuration than the published one.

In any event, the vendor must be able to tell the customer what
SPECmark (or any other) benchmark result to expect.  The customer
should insist on getting that result prior to paying.  Enough
customers do insist that vendors tend to live by the rules and don't
send output to /dev/null or do other tricks they would be unwilling
to repeat in front of a customer.

On the other hand, Data General uses two different C compilers.
The best compiler for gcc is not the best compiler for expresso.  We
document which compiler is used for which benchmark.  One would need
to buy both compilers to reproduce our exact results.

--------------------------------------------------------------------
Donald A. Lewine                (508) 870-9008 Voice
Data General Corporation        (508) 366-0750 FAX
4400 Computer Drive. MS D112A
Westboro, MA 01580  U.S.A.

uucp: uunet!dg!lewine   Internet: lewine@cheshirecat.webo.dg.com