Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!ncar!boulder!stan!McGuire
From: McGuire@Solbourne.COM (Jim McGuire)
Newsgroups: comp.arch
Subject: Re: specmarks
Message-ID: <2945@anthrax.Solbourne.COM>
Date: 30 Oct 89 15:50:09 GMT
Reply-To: McGuire@Solbourne.com (Jim McGuire)
Organization: Solbourne Computer Inc., Longmont, Colorado
Lines: 66


In article <21569@gryphon.COM> scarter@gryphon.COM (Scott Carter writes:

>In article <4420015@hpihoah.HP.COM> fotland@hpihoah.HP.COM (David Fotland) writes:

>>You could define an architectural figure of merit as Specmark/Clock freq.
>Indeed, this can be a useful value WITHIN "similar" architectures.
>>Prism:	.80
>>MIPS:	.56-.67
>>HP-PA:	.63
>>SPARC:	.45
>>88100:	.39-41
>>68030:	.09
>>This seems to clearly show the advantage of RISC over 68K type machines.  I
>>think it also seems to show the disadvantage of register windows, since
>>PRISM, MIPS, and PA don't have them and SPARC and 88K have them.
>>David Fotland

> ...stuff deleted...
>The Prism has a high FOM because of split instruction mode, which none of the
>other processors in this list have.  Wonder how the i860 will do?
>Scott Carter

On page 68 of the October 16, 1989 Electronic Engineering Times is
a figure entitled "SPEC Benchmark Release 1.0 Summary", which happens
to be for the Apollo DN10010 (PRISM).  Reading the fine print under the
"Notes/Summary of Changes" column:
	* gcc: symout.c not compiled -D_BUILTINS to avoid missing
	  declaration of getcwd()
	* spice2g6: F77 version 10.5 used instead of 10.7: code for DISTO
	  subroutine (not called in benchmark) commented out for F77
	  10.5's benefit
	* fppp: DFLOAT changed to DBLE in fmtest.f and gfloat.f
	# matrix300: code for SAXPY replaced with vec_$dmult_add;
	  compiled with -OPT 3

I found these comments rather interesting.  In particular, replacing
the SAXPY function with a presumably optimized assembly routine.  Also,
the comment "for F77 1.5's benefit" could be interpreted in several
ways, as well.

What's going on here with the SPEC benchmarks?  I thought that this
effort was at last a honest attempt to come up with something that
could be considered a fair comparison between systems.  Naive me :-)!

If people are going to reduce the SPEC numbers to one "FOM", as Mr.
Fotland has done, and ignored the "# ..code for SAXPY.." comments
then it seems to me that this is just more marketing hype. 

Does this mean any vendor can rewrite any portion of the SPEC tests
(or all of it), make a small footnote to the effect:
	* All tests rewritten in KILLER assembly
and then the world will just quote the final SPEC number?

I would prefer to put a stop to this nonsense right at the start.
But then I've already admitted to being naive! :-)
I want to see the SPEC numbers for unmodified benchmark sources.
If "symout.c" doesn't compile with the "-D_BUILTINS" (whatever
that means), I think this should be flagged as a failing to
function in a "standard" environment.  If the standard is not
portable, then fix the standard.

-- 
Jim McGuire, Solbourne Computer			Speaking for
mcguire@solbourne.com				myself only!
...boulder!stan!McGuire