Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!uunet!mcsun!hp4nl!charon!dik
From: dik@cwi.nl (Dik T. Winter)
Newsgroups: comp.arch
Subject: Re: RISC vs CISC simple load benchmark; amazing ! [Not really]
Message-ID: <1652@charon.cwi.nl>
Date: 15 Jun 90 01:01:09 GMT
References: <39319@mips.mips.COM> <675@sibyl.eleceng.ua.OZ> <39397@mips.mips.COM>
Sender: news@cwi.nl
Organization: CWI, Amsterdam
Lines: 76

In article <39397@mips.mips.COM> mash@mips.COM (John Mashey) writes:
 > In article <675@sibyl.eleceng.ua.OZ> ian@sibyl.OZ (Ian Dall) writes:
 > 
 > >I can't help thinking that average speed (over an instruction mix) is,
 > >(like most statistics) an inadequate measure. The trouble is, if you have
 > >a multiply intensive application, it is a pain if it runs dramatically
 > >slower than you would expect for a machine of that class. In a sense, one
 > >would like to know the worst case "speed" as well as the "average" speed
 > >of a machine (lots of hand waving here).
 > 
 > Really, what you want is enough data points that you think you know 
 > not only some measure of centrality but some measure of variation,
 > but those are not enough.  You always really want enough benchmarks to
 > see the patterns of difference: this is why SPEC has alwasy insisted
 > on making ALL of the benchmark nubmers available, because quite
 > different patterns can be found.
 > 
I do not think so.  In my opinion you also want to know the worst case.
But not only the performance for the worst case, but also what is the
worst case.  Every machine has a kind of program where it does not work as
expected.  Well known cases are integer multiply and cache problems (as
outlined by John Mashey).  I think that the interested customer needs to
know the cases where this happens.  The SPEC figures go already in that
direction, as outlined above, but more might be done in this respect.
The alternative is that there are customers that are disgusted at the
performance for their special application (like Robert Silverman for his
integer arithmetic programs some time ago).

What I would like is that the SPEC required also some statement that
mentioned fields where the machine would get below expectations with
respect to the SPEC figures.  I posted already about the SPARC; I have
not had enough time on MIPS processors to get at their weaknesses (but
I understand you might get in problems with the cache); and so on.
All machines I have used upto now have their strong points and their
weak points.  Of course, the trade rags contain the strong points only,
but as SPEC is trying to give some objective information on processors,
in my opinion a mention of weak points might be given.

Of course, this is a very rough idea (and might become controversial),
but some thoughts in this direction are very desirable.  And it might
help the decision process about the machine.

How many customers do know how to benchmark a machine?  Let alone,
how to benchmark a machine for their particular situation?  Of course
the manufacturers make use of this (always lay the blame at the
manufacturer :-)).  As a rough estimate, the numbers for the first and
second case above are (I may be 100% off) respectively: zero and zero.
But I hope those numbers will grow.  (As a parallel example; how many
people do know how to calculate the speedup on a multi-processor
system?)

Currently you read the promotional material, the SPEC figures (if available),
and try your own favourite programs, and decide; taking in account the
computing environment, which might be completely irrelevant.  For larger
machines you may simulate the workload you expect (which is probably wrong, or,
alternatively outdated by the time you have the new system installed).
(And in this context, you is a committee; each member with his own
preferences and so on.)

Alas, in these days the only way to assess the viability of a machine is
to install it, let it run a few years, and than say that it was the
proper decision (or not of course).  (Unless it appears after
two months that the machine is not used; in that case you can make an
early decision; but of course that is too late.)

Do I sound pessimistic?  Yes.
Am I pessimistic?  No.
How come?  In my opinion the SPEC figures are a giant step in the good
direction, and I hope that some time they may become really useful.
Still, the biggest problem is of course that decisions are made by management,
and not by the users (this is not directed at dcab :-)).

Oh well, these were some rambling thoughts...
--
dik t. winter, cwi, amsterdam, nederland
dik@cwi.nl