Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sun-barr!rutgers!gatech!utkcs2!de5 From: de5@ornl.gov (Dave Sill) Newsgroups: comp.benchmarks Subject: Re: Don't use bc (was: More issues of benchmarking) Message-ID: <1990Dec4.171944.7540@cs.utk.edu> Date: 4 Dec 90 17:19:44 GMT References: <1990Dec3.191756.15280@cs.utk.edu> <5045@taux01.nsc.com> Sender: news@cs.utk.edu (USENET News System) Reply-To: Dave Sill Organization: Oak Ridge National Laboratory Lines: 56 In article <5045@taux01.nsc.com>, amos@taux01.nsc.com (Amos Shapir) writes: >[Quoted from the referenced article by Dave Sill ] >| >|How does that invalidate the results? That's like penalizing an >|optimizing compiler for taking shortcuts the other one didn't. If the >|bc benchmark runs faster on system A than it does on B because vendor >|A took the time to optimize bc, then good for them! The danger is not >|some inherent unreliability in the benchmark, it's in incorrectly >|interpreting the results. > >But all I see in this group is articles posting *one number* for this test; >it may be ok to use it for a CPU-bound test (e.g., running "dc" alone), >but running "bc" means creating two processes, and also measuring the overhead >of context switches, etc. The problem is that most people do not know >that this is happening, and use the results as if "bc" was doing what actually >only "dc" does. The man on the street could not care less how many context switches are involved--or even what a context switch *is*--all he cares about is how long it takes the computer to perform his task. That's one number: elapsed time. *You* might be extremely concerned about what's happening behind the scenes, but not everyone is. >|Bc is bc. If it takes 2^5000/2^5000 and correctly calculates the >|result, what does it matter how it gets there? I.e., this benchmark >|measures bc's performance. Interpreting it as a hardware benchmark is >|fallacious, since hardware performance is only one factor in the >|result. > >But show me one article in this thread that does *not* treat this as a >hardware benchmark; Show me one that does. I haven't seen *any* conclusions drawn from these results, except that one SPARCstation SLC seemed to take entirely too long. (Which points out another use of this trivial benchmark: a quick diagnostic to compare identical systems or to track a system through time.) >after all, that's exactly the charter of this group! No it isn't. I specifically included *all* types of benchmarking. Otherwise it would've been called comp.arch.benchmarks or comp.hw.benchmarks. >"Bc" alone is not useful enough to warrant such an attention, and even if it >was, benchmarks measuring its performance for its own sake would be >posted in comp.bc, not here. First of all, there is no comp.bc. Even if there was, this would be a very appropriate place. Just because the bc benchmark doesn't suit your needs doesn't mean it's worthless. -- Dave Sill (de5@ornl.gov) Martin Marietta Energy Systems Workstation Support Brought to you by Super Global Mega Corp .com