Path: utzoo!censor!geac!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!mips!winchester!mash From: mash@mips.COM (John Mashey) Newsgroups: comp.benchmarks Subject: Re: benchmark evaluations Message-ID: <44157@mips.mips.COM> Date: 17 Dec 90 22:56:26 GMT References: <12220@hubcap.clemson.edu> <1990Dec12.070209.3272@murdoch.acc.Virginia.EDU> <1990Dec12.140615.27870@cs.utk.edu> <1990Dec12.182926.14306@murdoch.acc.Virginia.EDU> <1990Dec12.202608.3906@cs.utk.edu> <6391@mace.cc.purdue.edu> Sender: news@mips.COM Reply-To: mash@mips.COM (John Mashey) Organization: MIPS Computer Systems, Inc. Lines: 121 In article <6391@mace.cc.purdue.edu> abe@mace.cc.purdue.edu (Vic Abell) writes: >There seem to be two obstacles to publishing a table of SPEC results: > > 1. Many of the available numbers are copyrighted by SPEC. > > 2. SPEC license holders are constrained to use a comprehensive > reporting protocol that doesn't lend itself to tabulation. > >I've got my own, internal table of SPEC results, gleaned from the SPEC >newsletter, vendor reports, and my own running of the SPEC suite, but I >don't think I can publish it. There seem to be some misconceptions floating around (this was the last of a sequence): 1) SPEC numbers are NOT copyrighted. Go ahead and put together tables and publish them. Most vendors already have, and so have numerous industry analysts. (Even if we wanted to, we COULDN'T copyright the numbers!) 2) There are reporting rules, all right, but there are perfectly reasonable ways to approximate this when compiling tables that fit the spirit of full disclosure. The rules are: In the SPEC Newsletters, numbers are reported for machines by their own vendors, and they have to fill out the full form, which includes: all 10 numbers HW description: clock, cache size, model, memory size, disks SW description: releases of compilers & OS, anything else special any compile-time options used Availability dates If you claim a SPEC number, you are supposed to be able to provide the information on this sheet. (If you cannot, then you probably don't have the configuration pinned down well enough anyway). Now: reasonable ways to approximate this, within the spirit, for a third party, to at least make tables ofthe basic data: 1) Always show all 10 numbers. 2) Either show the source (like SPEC NEwsletter Issue), or if your own measurements, say so, or if the vendor, say so. 3) For the CPU benchmarks, it is probably sufficient to give the clock rate, cache size, and compiler version, plus anything "special". (I.e., I'm thinking of the kind of table you might get to fit on a page.) Alternatively, especially if providing a number for a amchine for which no related machines have had published SPEC results, it would be nice to post the exact details as shown on the upper right quadrant of the SPEC pages. 4) Expect that if you're posting numbers for a machine, and they're lower than what the vendor has posted, you'll probably hear from that vendor. In particular, courtesy implies that especially if you have numbers that look low, you might send them to the vendor first and ask why. REMEMBER: all of the SPEC-published numbers came from vendors, none of whom do anything like: running them on a machine with too-small memory that causes paging running them with year-out-of-date compilers running them unoptimized, or with bad choices of options running them with old makefiles with less-than-optimal options picking the WORST number of N runs... and hence, you can expect that if you publish numbers for a vendor, and they're low, they may have a legitimate claim of "foul". On the other hand, if truly duplicate the environment, you ought to expect to be close, and if you don't get enough information to duplicate it, then that's SPEC's or vendor's fault. On the other hand, one of the whole points of going through all of the agonizing work in this was to get something that people could use to keep us vendors honest. So, go ahead and publish. But: do identify sources of results, so that people don't wake up with lists of numbers that came from who knows where, and do try to be polite, like posting: "I ran SPEC on machine X, with these options, and configuration, and the results seem different from the published ones. Can anybody explain why this is?" rather than: "I ran SPEC on machine X, and vendor X is clearly lying scum...." Anyway, the rules aren't intended to stop people from collecting numbers, they're intended to try to be fair to lots of people. ------------ Some SPEC numbers have been posted already. I have big tables of them, unfortuantely, only in Mac spradsheets. If i weren't still digging out from long trip and just cataching up,. I'd post the lot. I encourage somebody who has a copy of "Your Mileage May Vary" to post the numbers as a service, as I don't have the time right now. Also, having exhaustively analyzed past "bc" benchmarks, and having found them uniformly silly, I wouldn't spend any more time in that direction. (One example I saw spent 99% of it's time in <256 bytes of code on a RISC, and >50% of the time was spent doing integer multiplies and divides, i.e., a profile that fits in the tiniest cache, and whose opcode frequencies resemble few other real programs.....) I encourage the following experiments: 1) Look for correlations between the SPEC integer subset and this bc benchmark. 2) Look for SPEC Integer <-> Dhrystone correlations. 3) Look for SPEC Integer <-> mips-ratings correlations. (BRW: from about 50 machines I've analyzed, to get SPEC Integer estimate, multiply vendor mips by anything from 55% to 97%, i.e., vendors disagree by as much as a factor of 2X in what a mips means. In comp.arch, people were complaing about "28.5 mips"; even worse is price/performance numbering like "$463/mips" in the light of this. -- -john mashey DISCLAIMER: UUCP: mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash DDD: 408-524-7015, 524-8253 or (main number) 408-720-1700 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086