Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!wuarchive!udel!rochester!kodak!ispd-newsserver!ism.isc.com!ico!rcd
From: rcd@ico.isc.com (Dick Dunn)
Newsgroups: comp.arch
Subject: $/CPUmark is a worthless measure
Message-ID: <1991Apr3.010831.3603@ico.isc.com>
Date: 3 Apr 91 01:08:31 GMT
References: <2004@kuling.UUCP> <32580014@hpcuhe.cup.hp.com>
Organization: Interactive Systems Corporation, Boulder, CO
Lines: 40

I guess I've seen one too many comments like...
> But I wouldn't hesitate to use $/SPECint as a performance measure...

$/SPECmark, or $/any-CPU-benchmark, is about as useful for comparing
systems as lines-of-code/day for comparing programmers, and both are
about as useful as manure output for measuring the work done by horses.
They're no better than order-of-magnitude, and they stink.

People don't buy CPUs.  They buy systems.  This discussion has focused on
workstations.  That means disk, memory, keyboard, display, and maybe a
network card.  (Or does it?  One of the egregious bogus comparisons here
actually compared $/SPECxyzzy for machines with and without disks!)

How can you divide "dollars per system" by "SPECmarks per CPU" and expect
to get anything useful is beyond me.  (The result has the entertaining
dimension of "dollar-CPUs per system-SPECmark".)

The only thing the big numbers-in-lights $/SPECstuff (usually to three
significant figures!!) emphasis produces is more incentive for manufac-
turers to produce low-ball "wheels extra" system configurations--as if
they needed any such incentive.  Face it, if you want to give your product
a cheap boost, which is easier: speed up the CPU or take hardware out of
the bottom-of-the-line?

Sure, you can try to "normalize" the comparisons by insisting that all the
machines you compare have the same configuration.  That's just painting
over the rust.  Why?  The way to win the normalized comparison is to put
your fastest CPU in your most stripped-down machine--thereby comparing
based on maximally-unbalanced and unrealistic configurations!

BTW, I'm NOT saying this is what HP has done!  I'm saying this is why
$/SPECmark is a stupid, worthless measure.  (I can't yet tell what HP has
done.  The one trade rag that quoted figures seemed to say that the $12k
basic machine was diskless, but it also suggested that a 400-Mb add-on disk
would add $22.5k to the machine price!  I assume this was usual "what--me
proofread?" trade rag quality and discounted the whole thing...some real
info on HP's configurations would help.)
-- 
Dick Dunn     rcd@ico.isc.com -or- ico!rcd       Boulder, CO   (303)449-2870
   The Official Colorado State Vegetable is now the "state legislator".