Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!usc!elroy.jpl.nasa.gov!ames!vsi1!wyse!mips!smsc.sony.com!amir
From: amir@smsc.sony.com (Amir )
Newsgroups: comp.arch
Subject: Re: Neal Nelson Benchmarks
Message-ID: <1990Feb23.223310.6098@smsc.sony.com>
Date: 23 Feb 90 22:33:10 GMT
References: <196@zds-ux.UUCP>
Reply-To: amir@Sony.COM (Amir )
Organization: Sony Microsystems Corp.
Lines: 75

In article <196@zds-ux.UUCP> gerry@zds-ux.UUCP (Gerry Gleason) writes:
>I have just been going through a bunch of marketing hype for Neal
>Nelson.  He claims that his "Business Benchmark" measures how
>well machines perform on "tasks like word processing, spread sheets,
>database management, accounting, programming and CAD," but I have
>never seen anything that backs this up with analysis or real data.
 
Quite true.  I have seen a lot of bad benchmarks in my time but this is
one of the worst.

>Also in the package are quite a few reprints that prominently
>feature these benchmarks, including several saying RISC is not
>much of a win based on his benchmarks.  (Federal Computer Week,
>"Tests Challenge Old RISC, CISC Notions; EE Times, "CISC beats RISC
>in test"; Computerworld, "Unearthing RISC worms")  The EE Times
>article has the results for his Test 5 (Short integer math) showing
>the Sun-3 to be ~10% faster than a Sun-4, which leads me to believe
>that the benchmark is bogus.  I thought EE Times was a pretty good
>publication, but the article does not even ask the question of what
>the benchmark is really measuring.
 
I was quite surprised too but it sort of made sense.  There were a lot
of people who were trying hard for a reason to discredit RISC.

>I was hoping that someone has already done some analysis of these
>benchmarks, and can confirm my suspicion that these test not only
>are bogus, but don't even measure what they claim to.  Unfortunately,
>at least some important fraction of the market uses these benchmarks
>to evaluate products, so many of us must apply them to our products
>even though we suspect them of being misleading.  If they really are
>bogus, what can be done to publicly discredit them, so further harm
>is not done?
 
It is really bogus!  I had the opportunity to meet Mr. Nelson before he
went public with his benchmark.  The story that he gave goes as follows:

He had written/ported an accounting package to an old Unix box (don't remember 
which now -- back in 82-84).  Then his client bought what he thought was
a faster machine and to his surprise, Mr. Nelson's package actually ran
slower.  So, he started to analyze the problem and this led to his
infamous benchmark.

As for the contents of the package, I had pretty strong disagreements with
him.  He takes a simple benchmark that measures something very small (e.g.
speed of add operating) and runs multiple copies to simulate "multi-user"
response.  Then he does the same thing for another simple operationg (e.g.
multiply) and so on.  So, almost all of his arithmetic tests show the same
linear slow down (unless you run out of memory).  The test is showing the
context switch overhead and not add/multiply times...

Then there is the "sync" test.  Apparently, his package used to do a lot
of unneeded syncs so, he tests how fast you can do syncs.  First one copy,
then 2 then...  Well, you get the idea.  Even he agreed that this was
stupid (as sync most systems returns immediately before the data is written
to disk anyways).  But last time I looked, it was still in there.

Also, once you look at the source of the benchmark, you'll realize that he
is not much of programmer either.  It must have more "goto"s and labels
than any other C program that I have ever seen.  It looks more like a 
decomplation of an assembler program than anything else.

There are numerous other flaws in there that I won't go into now.

>Gerry Gleason

To be fair, the benchmark, like any other, does generate a set of "data 
points" that can be useful.  What angers me, is that the results are
interpolated to mean performance of the system for "business" applications.
Since there are no other programs that claim to do this, the benchmark
has become fairly popular....
-- 
Amir H. Majidimehr
Operating Systems Group
Sony Microsystems
amir@smsc.sony.com | ...!{uunet,mips}!sonyusa!amir