Path: utzoo!censor!geac!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!mips!winchester!mash From: mash@mips.COM (John Mashey) Newsgroups: comp.arch Subject: Re: New SparcStation Message-ID: <44223@mips.mips.COM> Date: 19 Dec 90 20:18:12 GMT References: <1990Dec13> <3300233@m.cs.uiuc.edu> <44137@mips.mips.COM> <11772@ccncsu.ColoState.EDU> <3065@crdos1.crd.ge.COM> Sender: news@mips.COM Reply-To: mash@mips.COM (John Mashey) Organization: MIPS Computer Systems, Inc. Lines: 91 In article <3065@crdos1.crd.ge.COM> davidsen@crdos1.crd.ge.com (bill davidsen) writes: >In article <11772@ccncsu.ColoState.EDU> rro@debussy.cs.colostate.edu (Rod Oldehoeft) writes: > >| Here's a small gotcha. The quoted performance numbers for a SPARC II >| (28.5 MIPS, 21 SPEC) are obtained by using the optional optimizing >| compiler, Sun C 1.1, instead of the bundled compiler. Cute. > > Is there some vendor who doesn't use the best compiler, largest cache, >etc, when running benchmarks? As long as the conditions of test are >given I don't have a problem with it. And neither does SPEC: this is perfectly legitimate on Sun's part, and is properly documented, as far as I can tell. Of course, it IS important for users to understand the differences, and not confuse numbers of various kinds: TIME axis: a) The numbers you get if you run them on the machine you have today. b) Numbers you'll get on a machine available X months from now. c) Numbers you'll get on a machine you have today, but later, when/if you get the next compiler release. VERSION axis: d) Numbers you'll get on a machine you have, but only if you buy a particular compiler. Everybody has to deal with the time axis. Fortunately, after the first few releases of a system, the change rate due to compilers slows down, although the SPEC results do show that even excellent compilers actually continue to improve. The VERSION axis is rather different, as there is a range of distinct cases: 1) One compiler: as in VAX/VMS, other proprietary systems, almost all MIPS-based systems, etc, there is effectively one standard compiler family, and that's what almost every binary in the world has been compiled with, and so the numbers are pretty representative. 2) Two compilers: the Sun case. Clearly, you'd want to get the difference calibrated to use as a rule-of-thumb, and this would probably not be too hard. I'd guess that Sun-provided software would use the better compiler. I'd assume at least some of the ISVs would also. I'd guess 88K case is also close, as I've seen just a few vendors. 3) Many compilers: 68K, and ESPECIALLY X86. (I say this because for whatever reason, there seemed to be less 68K compilers that you might find on a random 68K box, perhaps because many vendors (Sun, Apollo, HP) had their own compilers on own boxes). Maybe somebody can offer some comments here: I'd guess there is quite a large range in the delivered numbers among commercial products. What I can't tell is: a) How big is the range, reallY? b) Are the numbers published from the best ones they could find (probably)? c) Which compilers are mostly used out there, and what kinds of results do THEY give? I.e., if you buy software packages, were they compiled with the best-optimizing compilers, or with ones that were faster compiling, had better debugging environments, or supported various features that might not necessarily be available in the best optimizers? > If some hardware or software not available to the public is used, or >is used and not disclosed, then I think there's reason to complain. The SPEC rules were set up to try to do this, with the caveat that it was OK to gives numbers for something that was not yet available, as long as there was a date given. People should read the dates carefully, of course, but I think we've at least gotten out of the game of seeing numbers for laboratory "hot boxes" with compilers that might never be released. It is a typical practice that people report results for compilers that will actually hit the market in a few months, and I'd argue that this is OK (as long as people watch it), in that a number tends to stick for a while, and if you're going to be able to buy it in a few months, it usually means that: a) The compiler group actually froze it a while back b) It's been in QA for a while c) The compiler group is actually working hard on the next round Anyway, to summarize: Sun IS playing by the rules on this one. If customers also want to get the bundled-compiler numbers, they should tell Sun so, or run them themselves. -- -john mashey DISCLAIMER: UUCP: mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash DDD: 408-524-7015, 524-8253 or (main number) 408-720-1700 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086