Path: utzoo!censor!geac!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!usc!zaphod.mps.ohio-state.edu!sol.ctr.columbia.edu!emory!utkcs2!de5
From: de5@ornl.gov (Dave Sill)
Newsgroups: comp.benchmarks
Subject: Re: Don't use bc (was: More issues of benchmarking)
Message-ID: <1990Dec3.204027.16794@cs.utk.edu>
Date: 3 Dec 90 20:40:27 GMT
References: <122@thinc.UUCP> <5042@taux01.nsc.com> <1990Dec3.191756.15280@cs.utk.edu> <39871@ucbvax.BERKELEY.EDU>
Sender: news@cs.utk.edu (USENET News System)
Reply-To: Dave Sill <de5@ornl.gov>
Organization: Oak Ridge National Laboratory
Lines: 60

In article <39871@ucbvax.BERKELEY.EDU>, jbuck@galileo.berkeley.edu (Joe Buck) writes:
>In article <1990Dec3.191756.15280@cs.utk.edu>, de5@ornl.gov (Dave Sill) writes:
>> It [the bc bmark] may not be rigorous, but it does have value.
>> For one thing, it's short enough to be memorized and easily typed
>> at a box at, say, an expo.  
>
>[Amusing story about a drunk looking for his keys under a lamppost,
>even though he'd lost them in an alley, because the light was
>better.] 
>
>The bc "benchmark" is easy to measure, but the number is worthless.

Why do you say that?  It's using a known tool to perform a known task,
right?  No, it won't tell whether machine X is faster than machine Y
with absolute certainty.  But it will tell me *something*.

>> >Of course the biggest problem is that almost no one actually *uses*
>> >`bc' for any large amount of computation, so no vendor has any
>> >incentive to optimize its performance.
> 
>> Ah, but would it be better to benchmark something the vendor has
>> expected and optimized?  If you're looking for actual performance
>> instead of theoretical peak performance, perhaps it better to throw
>> something unexpected at them.
>
>Even if your argument was true, I've seen the bc "benchmark" proposed
>years ago.  Vendors already know about it.

Do they know *my* version of it?  Maybe I've changed the numbers
around, maybe I've changed the operations...

>Here's a valid benchmark: how fast does a standard compiler run, with
>a standard input, compiling for a standard architecture?  (This is the
>"gcc" test in SPEC).  Other valid tests are: how fast does a widely
>used application program, like TeX, troff, SPICE, etc, run given a
>fairly complex input?

I agree that both are valid tests.  I don't agree that the bc test is
fundamentally different.  It's a smaller version of the same idea.

>The idea is to see how fast the computer runs
>programs that people actually use heavily.  If a vendor's machine
>manages to do all these things fast, it will probably be fast on your
>real workload.

Exactly, but you can't carry a SPEC tape with you wherever you go.  I
*can* carry the bc test with me and very quickly determine which end
of the spectrum an unknown machine falls in.  Will I base purchase
decisions on such an admittedly trivial test?  Of course not.

Benchmarks are tools.  Some are higher quality than others.  Some take
more skill to use than others.  The existence of $12,000 table saws
doesn't negate the ability of a $15 hand saw to cut a limb off a tree.
They're different tools for different tasks.  Our job as computing
professionals is to select the appropriate tool for the task.

-- 
Dave Sill (de5@ornl.gov)
Martin Marietta Energy Systems
Workstation Support


Brought to you by Super Global Mega Corp .com