Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!uunet!zds-ux!gerry
From: gerry@zds-ux.UUCP (Gerry Gleason)
Newsgroups: comp.arch
Subject: Neal Nelson Benchmarks
Summary: Do they measure anything?
Message-ID: <196@zds-ux.UUCP>
Date: 22 Feb 90 17:19:19 GMT
Reply-To: gerry@zds-ux.UUCP (Gerry Gleason)
Organization: Zenith Data Systems
Lines: 29

I have just been going through a bunch of marketing hype for Neal
Nelson.  He claims that his "Business Benchmark" measures how
well machines perform on "tasks like word processing, spread sheets,
database management, accounting, programming and CAD," but I have
never seen anything that backs this up with analysis or real data.

Also in the package are quite a few reprints that prominently
feature these benchmarks, including several saying RISC is not
much of a win based on his benchmarks.  (Federal Computer Week,
"Tests Challenge Old RISC, CISC Notions; EE Times, "CISC beats RISC
in test"; Computerworld, "Unearthing RISC worms")  The EE Times
article has the results for his Test 5 (Short integer math) showing
the Sun-3 to be ~10% faster than a Sun-4, which leads me to believe
that the benchmark is bogus.  I thought EE Times was a pretty good
publication, but the article does not even ask the question of what
the benchmark is really measuring.

I was hoping that someone has already done some analysis of these
benchmarks, and can confirm my suspicion that these test not only
are bogus, but don't even measure what they claim to.  Unfortunately,
at least some important fraction of the market uses these benchmarks
to evaluate products, so many of us must apply them to our products
even though we suspect them of being misleading.  If they really are
bogus, what can be done to publicly discredit them, so further harm
is not done?

Gerry Gleason

Gerry Gleason