Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!mnetor!uunet!lll-winken!lll-lcc!pyramid!prls!mips!mash From: mash@mips.UUCP (John Mashey) Newsgroups: comp.arch Subject: Re: myths & magazines Message-ID: <925@winchester.UUCP> Date: Wed, 18-Nov-87 01:36:47 EST Article-I.D.: winchest.925 Posted: Wed Nov 18 01:36:47 1987 Date-Received: Fri, 20-Nov-87 20:54:55 EST References: <56700004@ada-uts> <6745@apple.UUCP> <916@winchester.UUCP> <3425@ames.arpa> Reply-To: mash@winchester.UUCP (John Mashey) Organization: MIPS Computer Systems, Sunnyvale, CA Lines: 72 In article <3425@ames.arpa> fouts@orville.nas.nasa.gov.UUCP (Marty Fouts) writes >In article <916@winchester.UUCP> mash@winchester.UUCP (John Mashey) writes: >>0) At the rate of speed this business moves, writers and editors >>are hard-pressed to keep up, even when they try very hard. >I would feel for these editors, except they do a generally bad job,.... Perhaps this leads to something useful: perhaps (either here or in some other newsgroup), we should all post examples of what we think are inaccurate or accurate reporting, and/or good/bad benchmarking. This would at least give other people calibrations on believability. >>Since many of the trade rags are controlled circulation, you >>can't usefully threaten to cancel your subscription! >Actually you can, it just takes a lot more threats for them to do >something.... Unfortunately, if your are a vendor, you MUST continue to get these things in self-defense, if nothing else... > >>5) In general, it is hopeless to improve some of the rags, which are >>little above the National Enquirer. Some of the magazines try very hard, >>even to having their own benchmark suites which they want to watch >>running on a real machine. > >A word about magazine benchmarking suites. Byte magazine had an... >They were all flawed in ways that the readership of this group is well Good point. I'd rate magazines on the following levels (somewhat akin to the old UNIX novice->guru scale): 1) Novice: believes all vendor mips & flops ratings, publishes same without even cursory checks. Thinks whetstones are what you sharpen knives with. Doesn't know difference between single and double-precision. Not really trying, and glad to hype unsupported claims. 2) Beginner: at least labels vendor mips ratings as "claimed". Has heard of LINPACK and other commonly-used ones, and even has some idea of what they measure, at least that some are integer and some are floating point. May still count NOOPS/second. 3) Intermediate: at least has some benchmarks, and wants to see them run on real machines. Benchmarks may have silly flaws, but can at least tell the difference between a 4.7MHz 8088 and a 20MHz 386. A few benchmarks might even be useful, if interpreted carefully. Trying. 4) Advanced: knows the difference between LINPACK and Livermore Loops. Either has own (useful) benchmarks, or gives credence to the more realistic ones that are generally available. Knows when geometric mean should be used. Trying hard. 5) Wizard: not only does all of 4), but is competent at spotting benchmark oddities. Understands the surprises of caches and optimizing compilers. Understands reasons for skepticism and publishes same. Has good idea when somebody sets HZ wrong. Knows when disk benchmarks fit in cache. Verifies claimed numbers by watching them run, and verifies vendor claims regarding other vendor performance by calling the other vendors. Exhorts people to be skeptical. Trying very hard. I'd put some of the Byte stuff in 3). Digital Review I'd put in 4: despite the fact that there are a few silly tests in the 33-test suite, most of it correlates pretty well with some kinds of computing, and it actually has a few real programs in it. Anyway, I'd encourage everybody to write letters to editors, both good and bad: how else is anything going to change if we don't give them feedback. -- -john mashey DISCLAIMER: UUCP: {ames,decwrl,prls,pyramid}!mips!mash OR mash@mips.com DDD: 408-991-0253 or 408-720-1700, x253 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086