Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!tut.cis.ohio-state.edu!ucsd!ucsdhub!hp-sdd!ncr-sd!iss-rb!ivory!steves
From: steves@ivory.SanDiego.NCR.COM (Steve Schlesinger x2150)
Newsgroups: comp.arch
Subject: Re: Neal Nelson Benchmarks
Message-ID: <231@iss-rb.SanDiego.NCR.COM>
Date: 27 Feb 90 22:42:58 GMT
References: <196@zds-ux.UUCP>
Sender: news@iss-rb.SanDiego.NCR.COM
Reply-To: steves@ivory.SanDiego.NCR.COM (Steve Schlesinger x2150)
Organization: NCR Corporation, Rancho Bernardo
Lines: 69


I posted some comments on this subject last week.

In article <196@zds-ux.UUCP> gerry@zds-ux.UUCP (Gerry Gleason) writes:
>I have just been going through a bunch of marketing hype for Neal
>Nelson.  He claims that his "Business Benchmark" measures how
>well machines perform on "tasks like word processing, spread sheets,
>database management, accounting, programming and CAD," but I have

*****************************************************************
	The following is my opinion
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 

This true in the sense that the named applications do add, subtract,
multiply, divide, memory move, compare and disk I/O and the NN
benchmarks also do these operations.

Steve Schlesinger

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 
	The preceding is my opinion and does not reflect the 
	opinion of my employer or any other organization.
*****************************************************************

>never seen anything that backs this up with analysis or real data.
>
>Also in the package are quite a few reprints that prominently
>feature these benchmarks, including several saying RISC is not
>much of a win based on his benchmarks.  (Federal Computer Week,
>"Tests Challenge Old RISC, CISC Notions; EE Times, "CISC beats RISC
>in test"; Computerworld, "Unearthing RISC worms")  The EE Times
>article has the results for his Test 5 (Short integer math) showing
>the Sun-3 to be ~10% faster than a Sun-4, which leads me to believe
>that the benchmark is bogus.  I thought EE Times was a pretty good
>publication, but the article does not even ask the question of what
>the benchmark is really measuring.
>
>I was hoping that someone has already done some analysis of these
>benchmarks, and can confirm my suspicion that these test not only
>are bogus, but don't even measure what they claim to.  Unfortunately,
>at least some important fraction of the market uses these benchmarks
>to evaluate products, so many of us must apply them to our products
>even though we suspect them of being misleading.  If they really are
>bogus, what can be done to publicly discredit them, so further harm
>is not done?
>
>Gerry Gleason


*****************************************************************
	No disclaimer on this part
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 

A while ago, NCR posted the System Characterization Benchmark
on "net.sources."  It measures similar things to the NN benchmark,
but is better and is free.  As an added bonus, you can read the
source and decide for yourself what the results really mean. 
It is not perfect (useful comments will be forwarded to the author,
flames to /dev/null).

My advice to anyone looking for benchmarks is first to use the
applications you currently run, then use SPEC data for computational
results (with your own weightings) and the SCB.

Steve
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
steve schlesinger	steve.schlesinger@sandiego.ncr.com
619-485-2150		NCR - 4010, 16550 W Bernardo Dr, San Diego, CA 92127
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::