Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sun-barr!lll-winken!ames!sgi!dragon!bananapc.wpd.sgi.com!ciemo From: ciemo@bananapc.wpd.sgi.com (Dave Ciemiewicz) Newsgroups: comp.benchmarks Subject: Re: Don't use bc (was: More issues of benchmarking) Message-ID: <1990Dec6.193653.1505@relay.wpd.sgi.com> Date: 6 Dec 90 19:36:53 GMT References: <122@thinc.UUCP> <5042@taux01.nsc.com> Sender: news@relay.wpd.sgi.com ( CNews Account ) Reply-To: ciemo@bananapc.wpd.sgi.com (Dave Ciemiewicz) Organization: sgi Lines: 59 In article , dale@convex.com (Dale Lancaster) writes: |> >Amos> Sure it doesn't; I wonder how no one else noted this yet: "bc" |> >Amos> is probably the worst choice of a utility to benchmark by. On |> >Amos> most UNIX systems, it just parses expressions, and forks "dc" to |> >Amos> execute them [...] |> |> >Of course the biggest problem is that almost no one actually *uses* |> >`bc' for any large amount of computation, so no vendor has any |> >incentive to optimize its performance. |> |> For this reason it makes a good a benchmark as any. I suspect that most Unix |> based bc's/dc's work the same. This is a typical dusty-deck, no optimization |> piece of code that shows the performance of the machine from not just the hardware |> side but the software (compiler) side as well. Today you have to benchmark the |> compiler as much as the hardware. I really can't believe that this joke is being perpetuated. The idea of benchmarking is to create a reference standard by which different machines may *MEANINGFULLY* be compared. Since bc may actually be coded differently between systems, the comparisons become weaker. I just diff'ed the sources between BSD and SYSV versions of dc which is the compute engine for bc. There are changes to the SYSV version for robustness that may sway results one way or the other. For all intents and purposes, you might as well be comparing the execution of bc using different stop watches. Since it is purported that AIX is a complete rewrite of UNIX, they may have even rewritten something as simple as dc to avoid licensing which would also be a plausible explanation for the faster performance on the RS6000. It may not have anything to do with the IBM compilers. (Please correct me if I'm wrong.) This mockary of a benchmark does not fall into the category of "unchanged, dusty deck" code as many would believe. We are measuring numbers with different rulers. Not only that, the rulers are the cheap plastic kind we used in grade school. When you put two of them side-by-side, you find out they're different lengths. Meaningless for precise comparison. Possibly meaningless for gross comparisons. |> But of course the only true measure of performance |> is my own application :-) |> |> dml These are the best kind of comparisons of all. When I run my application on this system, what is the performance difference when I try to accomplish some real task. The ANSYS people port their product to a number of architectures and then run the products through scripted sets of tasks for comparisons. This may not be as swell or buzzwordy as MIPS or Dhrystones, yet, these comparisons are better for an individual trying to make comparisons between systems for running the ANSYS products than any of the other "metrics". Imagine buying a car based only on the horsepower, torque, and G ratings without actually taking the care for a test drive. The differences can be astounding. --- David Ciemiewicz P.S. I am speaking for myself. I like to live the delusion that my company supports my opinions however it really depends on what I'm talking about. Brought to you by Super Global Mega Corp .com