Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!zaphod.mps.ohio-state.edu!julius.cs.uiuc.edu!ux1.cso.uiuc.edu!ux1.cso.uiuc.edu!aglew From: aglew@crhc.uiuc.edu (Andy Glew) Newsgroups: comp.arch Subject: Re: Benchmark performance ratios Message-ID: Date: 20 Nov 90 01:02:58 GMT References: <39896@ut-emx.uucp> <1990Nov19.170400.12437@mozart.amd.com> Sender: news@ux1.cso.uiuc.edu (News) Organization: Center for Reliable and High-Performance Computing University of Illinois at Urbana Champaign Lines: 28 In-Reply-To: tim@proton.amd.com's message of 19 Nov 90 17:04:00 GMT >James Smith, in the paper entitled "Characterizing Computer >Performance With a Single Number" [CACM, October 1988, #10] argues >that the harmonic mean should be used, but only again with absolute >quantities such as MFLOPS (normalization should occur after the mean >has been calculated). Urghhh. That isn't quite what the paper said. You use the harmonic mean to summarize *rates* like megaflops, because the harmonic mean of rates corresponds to an arithmetic mean of run-times. And run-times, after all, is what we are all interested in in the long run (well, some of us are interested in throughputs too as distinct from latencies, but...) How you weight the components of the means is another matter. Averaging normalized values just means that you have chosen a particular weighting function. Hennessy suggests another weighting function. The best weighting function, of course, corresponds to the relative frequency of the application types represented by the benchmarks in your workload - and that statement, of course, is full of assumptions. Another reason to use the harmonic mean is that it tends to be conservative, so it reduces the chance that you will be accused of an overestimate based on a few distorting values. Handwaving there, too. -- Andy Glew, a-glew@uiuc.edu [get ph nameserver from uxc.cso.uiuc.edu:net/qi]