Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!uwm.edu!gem.mps.ohio-state.edu!brutus.cs.uiuc.edu!lll-winken!sun-barr!newstop!sun!sunshade!rajivp
From: rajivp@sunshade.Sun.COM (Rajiv Patel)
Newsgroups: comp.arch
Subject: Re: specmarks
Message-ID: <126870@sun.Eng.Sun.COM>
Date: 26 Oct 89 17:50:27 GMT
References: <327@ssp1.idca.tds.philips.nl> <4420015@hpihoah.HP.COM>
Sender: news@sun.Eng.Sun.COM
Reply-To: rajivp@sun.UUCP (Rajiv Patel)
Organization: Sun Microsystems, Mountain View
Lines: 58

In article <4420015@hpihoah.HP.COM> fotland@hpihoah.HP.COM (David Fotland) writes:
>
>You could define an architectural figure of merit as Specmark/Clock freq.
>
>A higher value indicates that this architecture gets more performance for the
>same clock frequency.  I calculate:
>
>
>Prism:	.80
>MIPS:	.56-.67
>HP-PA:	.63
>SPARC:	.45
>88100:	.39-41
>68030:	.09
>
>
>This seems to clearly show the advantage of RISC over 68K type machines.  I
>think it also seems to show the disadvantage of register windows, since
>PRISM, MIPS, and PA don't have them and SPARC and 88K have them.
>
>David Fotland

This is the kind of article which provides the industry with proof that even a
good attempt to standardize the benchmarking process can be really twisted by
individuals and companies who would like to do so.

SPEC decides to publish SPECmark. They explicitly do not mention any other
numbers to confuse people but then everyone would like to extrapolate...

Specmark/Clock frequency as David defines it is by far the furthest figure of
merit for an architecture. It might be used (though I wouldn't suggest it) as
a figure of merit for implementation BUT NEVER an architecture. How could one
explain a changing figure of merit for an architecture (say 68K) with different
chips - 68000, 68020, 68030, 68040 and so on. For an architecture shouldn't the
figure of merit remain constant ?

If one really does want to go with such figure's of merit then how about some
of these :-)
Technical
---------
SPECmark / # instructions
SPECmark / # mem. accesses
SPECmark * highest clock available
SPECmark / your favourite value

Marketing
---------
SPECmark / $$$
SPECmark / # applications
SPECmark / your favourite value


I guess, what I would like to convey is that let us not twist and turn the
numbers published by SPEC. This will only confuse more people and lend a bad
name to the business of benchmarking which as it is no one trusts.


Rajiv.