Adecvax.204
net.works
utzoo!decvax!minow
Wed Apr 14 18:56:28 1982
Re: another view of mips
One problem with published benchmarks, such as Wheatstone's, is that
less than honorable vendors might try to tune their compiler/operating
system to the benchmark.  There are reports of such things happening
for Fortran compilers and the Wheatstone benchmark, by the way.

When Dec first announced Fortran-IV-PLUS and the 11/70 at Hannover
(1974 or 1975), someone went around to all the vendors with a
Fortran benchmark.  The 11/70 ran it in zero time.  Turned out to
be a loop like:

	X = 0
	DO 10  I=1, 100
	DO 10  J=1, 100
10	X = X + 1

Fortran-IV-PLUS did the entire computation at compile-time, generating

	I = 101
	J = 101
	X = 10000
	exit

The person running the benchmark accepted it as valid.

Moral:  benchmarks -- especially computational ones -- are generally
unrepresentative of situations where the customer asks "How many users
can I put on my ..."  (correction: "where the customer really wants to
know...")

Martin Minow
decvax!minow