Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!swrinde!cs.utexas.edu!asuvax!noao!arizona!sunquest!venus.sunquest.com!terry
From: terry@venus.sunquest.com (Terry R. Friedrichsen)
Newsgroups: comp.benchmarks
Subject: Re: Which benchmarks are useless?
Summary: statistical correlation
Keywords: benchmarks  date  statistical correlation
Message-ID: <18049@sunquest.UUCP>
Date: 22 Apr 91 21:59:47 GMT
Sender: news@sunquest.UUCP
Followup-To: comp.benchmarks
Distribution: usa
Organization: Sunquest Information Systems, Tucson
Lines: 53

auvsaff@auvsun1.tamu.edu (David Safford) writes:
>mark@mips.com (Mark G. Johnson) writes:

>|>I would suggest that the words "useless" and/or
>|>"meaningless" be reserved for benchmarks that produce results that are
>|>uncorrelated (absolute value of correlation coefficient < 0.2) with
>|>"correct benchmark results".

>You are not seriously saying that statistical correlation signifies meaning,
>are you???
>
>A statistical correlation is necessary, but certainly not sufficient, to
>indicate a meaningful relationship.  Just because the per capita consumption
>of M&M's correlates with the rate of bank failures does not indicate any
>meaningful relationship.  This is an apple and oranges comparison.

No, and he didn't say that.  Read what he wrote:  "useless == uncorrelated".
He did NOT say "correlated == useful".  You yourself admit that correlation
is a necessary, though insufficient, condition.

One of the real problems I note in reading this group is that there are a
LOT of folks who are weak on logic, in interpretation of benchmarks as well
as other postings.

>The "date" "benchmark" is a meaningful measure of only 2 things: (1) whether
>the given system uses dynamic linking, and (2) how fast date runs, given (1).
>
>(1) is more easily determined other ways
>(2) is not important to me

There are a couple of problems here:  first, the benchmark also indicates
the speed of the operating system implementation's fork()/exec() services.
That SHOULD be important to a lot of folks, with the exception of pure
number-crunchers.  Take John Hascall's suggestion and replace date(1) with
true(1), and you'll do even better at measuring fork()/exec() (modulo the
dynamic linking issue, of course).

Second, since date(1) must do integer calculations, how fast it runs MUST
be important to you, unless you do nothing but floating-point arithmetic.
If I handed you a machine that took 47 seconds to run date, I suspect that
you would find other aspects of its performance equally repulsive.  Weak
logic here again.

On the other hand, the matrix300 benchmark is of absolutely no importance
to me, since my matrices are a different size ;-).

Terry R. Friedrichsen

terry@venus.sunquest.com  (Internet)
uunet!sunquest!terry	  (Usenet)
terry@sds.sdsc.edu        (alternate address; I live in Tucson)

Quote:  "Do, or do not.  There is no 'try'." - Yoda, The Empire Strikes Back