Path: utzoo!censor!geac!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!usc!zaphod.mps.ohio-state.edu!pacific.mps.ohio-state.edu!linac!att!pacbell.com!ucsd!ucbvax!galileo.berkeley.edu!jbuck
From: jbuck@galileo.berkeley.edu (Joe Buck)
Newsgroups: comp.benchmarks
Subject: Re: Don't use bc (was: More issues of benchmarking)
Message-ID: <39871@ucbvax.BERKELEY.EDU>
Date: 3 Dec 90 19:52:26 GMT
References: <122@thinc.UUCP> <5042@taux01.nsc.com> <1990Dec3.191756.15280@cs.utk.edu>
Sender: usenet@ucbvax.BERKELEY.EDU
Reply-To: jbuck@galileo.berkeley.edu (Joe Buck)
Lines: 38

In article <1990Dec3.191756.15280@cs.utk.edu>, de5@ornl.gov (Dave Sill) writes:
> It [the bc bmark] may not be rigorous, but it does have value.
> For one thing, it's short enough to be memorized and easily typed
> at a box at, say, an expo.  

Surely you've heard the one about the drunk looking for his keys under
the lamppost, even though he'd lost them in an alley.  He explains
that he's looking under the lamppost because the light is better.

The bc "benchmark" is easy to measure, but the number is worthless.
The drunk finds it easy to look under the lamppost, but he's not
going to find what he's looking for.

> >Of course the biggest problem is that almost no one actually *uses*
> >`bc' for any large amount of computation, so no vendor has any
> >incentive to optimize its performance.
 
> Ah, but would it be better to benchmark something the vendor has
> expected and optimized?  If you're looking for actual performance
> instead of theoretical peak performance, perhaps it better to throw
> something unexpected at them.

Even if your argument was true, I've seen the bc "benchmark" proposed
years ago.  Vendors already know about it.

Here's a valid benchmark: how fast does a standard compiler run, with
a standard input, compiling for a standard architecture?  (This is the
"gcc" test in SPEC).  Other valid tests are: how fast does a widely
used application program, like TeX, troff, SPICE, etc, run given a
fairly complex input?  The idea is to see how fast the computer runs
programs that people actually use heavily.  If a vendor's machine
manages to do all these things fast, it will probably be fast on your
real workload.


--
Joe Buck
jbuck@galileo.berkeley.edu	 {uunet,ucbvax}!galileo.berkeley.edu!jbuck	


Brought to you by Super Global Mega Corp .com