Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!husc6!rutgers!clyde!watmath!ccplumb
From: ccplumb@watmath.waterloo.edu (Colin Plumb)
Newsgroups: comp.arch
Subject: Re: myths & magazines
Message-ID: <15580@watmath.waterloo.edu>
Date: Thu, 19-Nov-87 14:13:42 EST
Article-I.D.: watmath.15580
Posted: Thu Nov 19 14:13:42 1987
Date-Received: Sun, 22-Nov-87 10:40:47 EST
References: <56700004@ada-uts> <6745@apple.UUCP> <916@winchester.UUCP> <3425@ames.arpa> <925@winchester.UUCP>
Reply-To: ccplumb@watmath.waterloo.edu (Colin Plumb)
Organization: U of Waterloo, Ontario
Lines: 32
Confusion: U. of Waterloo, Ontario


In article <925@winchester.UUCP> mash@winchester.UUCP (John Mashey) writes:
>Perhaps this leads to something useful: perhaps (either here or in
>some other newsgroup), we should all post examples of what we think
>are inaccurate or accurate reporting, and/or good/bad benchmarking.
>This would at least give other people calibrations on believability.

I, for one, would appreciate this greatly.  While I know that 90% of
everything is bullshit, and 99% of benchmarks, I can't just disregard
them entirely.  Some figures should just be thrown out (like naive
work-loops that the compiler optimizes out), but sometimes I need to
extract some sense from them.  If some experts here could post a detailed
critique, I could learn a great deal.  I know everyone likes to compare
their machines to VAXen on small benchmarks because the frequency of
procedure calls is disproportionately high, which maked the VAX suffer,
but what else should I look out for?  It's a jungle out there...

And in the benchmark skill list:
>5) Wizard: not only does all of 4), but is competent at spotting
>benchmark oddities.  Understands the surprises of caches and optimizing
>compilers.  Understands reasons for skepticism and publishes same.
>Has good idea when somebody sets HZ wrong.  Knows when disk benchmarks
>fit in cache.  Verifies claimed numbers by watching them run,
>and verifies vendor claims regarding other vendor performance by calling
>the other vendors.  Exhorts people to be skeptical.  Trying very hard.

(blush)... What's "HZ"?  The closest I can come is Hertz, but that's
usually not settable by anything but a motherboard swap. :-)
--
	-Colin (watmath!ccplumb)

Zippy says:
Did I say I was a sardine?  Or a bus???