Path: utzoo!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!sol.ctr.columbia.edu!emory!wuarchive!udel!rochester!pt.cs.cmu.edu!gandalf.cs.cmu.edu!lindsay
From: lindsay@gandalf.cs.cmu.edu (Donald Lindsay)
Newsgroups: comp.arch
Subject: Re: Novice question:  measuring speed
Message-ID: <12388@pt.cs.cmu.edu>
Date: 17 Mar 91 23:08:24 GMT
References: <24780@drilex.UUCP>
Organization: Carnegie Mellon
Lines: 28

In article <24780@drilex.UUCP> dricejb@drilex.UUCP 
	(Craig Jackson drilex1) writes:
>Today, when somebody says they have a 25-MIPS machine, it means
>"processes some benchmark or benchmarks 25 times faster than an
>11/780", unless they are in the IBM world.

There are lies, damned lies, and benchmarks, to quote the title of an
Inmos tech note. However, this enlightened understanding doesn't
prevent Inmos from describing a chip as "20 MIPS" when in fact it
runs some benchmark 3.5 or 3.7 times faster than a 780. The
discrepancy is because they are quoting native MIPS, for a chip
typically executing one one-byte instruction per clock.

This doesn't mean that Inmos is the sleaziest company around:  all
boasts of (MIPS == peak native MIPS) are equally sleazy.  Cray, at
the other end of the spectrum, can get 64 results from a single
opcode. Cray boasts about the 64 results, which is reasonable:
results are the thing you actually want.

Somewhat tricker is the issue of end-to-end results. If a benchmark's
inner loop is recoded to have 3 more FADDs and two fewer FMULs, then
has the benchmark's FLOPs changed? I would argue that it hasn't. The
important thing about a machine is that it can deliver results. I
agree with the benchmark designers who take some fixed count of
results/FLOPs/Whetstones/etc, divide by the runtime-in-seconds, and
call that the results per second.
-- 
Don		D.C.Lindsay .. temporarily at Carnegie Mellon Robotics