Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!ll-xn!ames!amdahl!nsc!grenley
From: grenley@nsc.nsc.com (George Grenley)
Newsgroups: comp.arch,comp.org.usenix
Subject: Re: Benchmarking the 532, 68030, MIPS, 386...at a Usenix!
Message-ID: <4329@nsc.nsc.com>
Date: Fri, 15-May-87 13:06:08 EDT
Article-I.D.: nsc.4329
Posted: Fri May 15 13:06:08 1987
Date-Received: Sun, 17-May-87 00:22:55 EDT
References: <324@dumbo.UUCP> <809@killer.UUCP> <2417@homxa.UUCP> <4294@nsc.nsc.com> <2128@hoptoad.uucp> <826@rtech.UUCP>
Reply-To: grenley@nsc.UUCP (George Grenley)
Organization: National Semiconductor, Sunnyvale
Lines: 63
Xref: mnetor comp.arch:1310 comp.org.usenix:175

A while back I invited other chip manufacturers to join me in a CPU
horse race.  I'm happy to see this request is generating interest.
However, I haven't heard much from Mot or Intel.  Are you guys
listening?  Iknow you're out there...

In article <826@rtech.UUCP> daveb@rtech.UUCP (Dave Brower) writes:
>At last winter's Uniforum, I went around to a number of booths trying to
>run the infamous
>	/bin/time bc << !
>	2^4096
>	!
>At a distressing number of places the sales creatures in the booth would
>say things like, "I don't believe we're interested in running any
>benchmarks today.  Let me show you vi."  Now there are some good reasons
>for this, but it sure sounded like there was something being hidden.

No, they just don't want to risk a system crash, or other malicious use.
Most show-people aren't Unix gurus, and are hesitant to let a hacker play
with the system on the show floor.

>Problem 1 is getting some benchmarks run.  Problem 2 is trying to get a
>straight answer on the price of the system.  What you really want is the
>bang/buck of different benchmarks on different boxes.  The results would
>be an embarrassing to many people wearing suits, which is why it may be
>difficulty to get a lot of cooperation.
>PS:  Given my druthers, I'd like to see:
>
>	* the bc benchmark above
>	* Dhrystone
>	* Whetstones
>	* A paging thrasher.
>	* A system call overhead checker (looped getpid()s maybe).
>	* A process thrasher.

The problem you're experiencing in getting what you call straight
answers lies in your methods.  As a working engineer who knows how to
wear a suit and tie (not common, I know, but some of us manage it) who
has logged many hours of booth duty, I sympathize with the booth person
who is hesitent to allow you to run any program they're not familiar with.
There are a lot of *ssholes at tradeshows who delight in trying (and
succeeding, occasionally) in crashing systems.  Personally, I never let
such a person near a machine, period, no matter how much he protests
the "innocence" of his program.

If you want cooperation, I suggest you work with some of the others on
this net (including myself) to define a reasonable benchmark before the
show, run it under realistic conditions, and let the manufacturers
publish the results jointly.  

I have seen more than one instance where one cooperative manufacturer
ran a "real-world" benchmark, only to be pilloried later by having it
compared to some bs benchmark put out by a less scrupulous competitor.

As to your suggested benchmark list:  Dhrystone has come under fire as
not being very reliable, due to compiler optimization problems.  LIkewise,
most Unix machines aren't floating point oriented.

Consider also that those of us who are chip manufacturers are primarily
interested in CPU benchmarks, not system benchmarks.  Unix is NOT the
entire world, yet.

TO BETTER BENCHMARKS!
George