Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!mips!mash From: mash@mips.com (John Mashey) Newsgroups: comp.benchmarks Subject: Re: Which benchmarks are useless? Keywords: benchmarks date statistical correlation Message-ID: <2800@spim.mips.COM> Date: 29 Apr 91 23:06:02 GMT References: <1749@marlin.NOSC.MIL> <2717@spim.mips.COM> <1751@marlin.NOSC.MIL> Sender: news@mips.COM Distribution: comp.benchmarks Organization: MIPS Computer Systems, Inc. Lines: 95 Nntp-Posting-Host: winchester.mips.com In article <1751@marlin.NOSC.MIL> aburto@marlin.NOSC.MIL (Alfred A. Aburto) writes: >In article <2717@spim.mips.COM> mash@mips.com (John Mashey) writes: >>I don't have the numbers handy, and am about to go out of town again. >>However, there are a number of combinations where Dhrystone would predict >>that machine A is 25% faster than machine B, but on SPEC integer >achine B is 25% faster than machine A, or equivalent combinations where >>the prediction is 50% off. Combinations like this include RS/6000 vs >>MIPS, or Intel i860 vs MIPS, at appropriate clock rates. A particular >>case is RS/6000 Model 320, which SPECints around 16, but Dhrystone (1.1) >>is around 27.5, versus MIPS Magnum (25Mhz, not the newer 33s), which >>has SPECint at 19.5, but has a lower Dhrystone than the RS/6000. >>If I find time, I'll dig out the numbers, but I've seen enough data over >>the years to have stopped collecting it. What it said was: >> a) Dhrystone ALWAYS gives a higher VAX-mips rating than SPECint. >> (except maybe the VAX-11/780 :-) 1.1 is worse (higher) than 2.1, >> but 2.1 is high also. the raio ranges from about 1.1 up to at >> least 1.6, maybe even as high as 2X. >> b) The Dhrystone:SPECint ratios grossly track with a single >> product line, except that small-cache machines of a family look >> more better on Dhrystone than on SPECint. >Here are some Dhrystone 1.1 and integer SPEC program comparison results >I gathered. > >The Dhrystone 1.1 results came from an article by Walter Price of Motorola >As indicated in the table Dhrystone 1.1 ratio results are greater than the >Integer SPEC ratio results by 14% to 24% with an average of 21% greater. >This is pretty much as you indicated, but I didn't find any really abnormal >results (probably due to a lack of enough data). Dhrystone 2.1 results >would be useful too, but I don't have a data-base ..... For the high ones, look at IBM RS/6000, i860, or maybe Motorola 88K. AMD would probably be high, but doesn't have SPEC result published, to my knowledge. Note that IBM's 27.5/15.8 = 1.7+ ... and I think you'll find the i860 is probably up there as well ... and the DG workstation labeled 17 mips gets around 10 on SPECint ... I don't think we disagree, except in choice of data. Of the data points, the VAX is = 1 by definition. 4 of them are MIPS machines 1 is an HP [which is fairly similar to MIPS, and shares some roots in similar compiler technology] 3 are SPARCs I.e., as I said, within product lines you expect that the major determiner of speed is clock rate, and Dhrystone will show you that. As a minor point, for whatever reason, most of the dhrystone-vax-mips ratings in the world assume VAX-11/780 = 1,757 1.1 Dhrystones, which slightly raises the numbers everywhere. The major issue is (just to make sure people aren't confused by the posted table): IF you pick two machines at random, A and B: a) Dhry(A) and Dhry(B) will both give vax-mipsratings that are high. b) Dhry(A)/Dhry(B) will give reasonably good correlations with SPEC(A)/SPEC(B), especially if A and B are from same family or are related. Unfortunately, there are also plenty of data points, specically, with machines that included instructions to help strcpy, or have done certain optimizations, where you easily pick points where: Dhry(A) > Dhry(B) and SPEC(A) < SPEC(B), by a substantial margin. Thus, one must be careful to distinguish between the 2 statements: a) There is a good correlation between Dhrystone and SPEC (TRUE, in general, especially if you include the vast numbers of X86-based products usually listed). AND b) Dhrystone is a good enough predictor of PEC that you don't need to run SPEC .... FALSE, in practice, because you can get terribly surprised, because a signficant number of recent machines and architectures are OUTLIERs when you do a scatter plot of this kind of data. Now, of course, whether this matters or not depends on whether or not you think SPECint correlates with anything else :-) (As it happens, at least some of us think it does, because it correlates closer than dhrystone does with numerous large internal single-user benchmarks.) Anyway, maybe someone can put together a table with 1 MIPS 1 SPARC 1 RS/6000 1 i860 1 HP 1 88K 1 68K 1 486 and using the more common 1757 number... -- -john mashey DISCLAIMER: UUCP: mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash DDD: 408-524-7015, 524-8253 or (main number) 408-720-1700 USPS: MIPS Computer Systems MS 1/05, 930 E. Arques, Sunnyvale, CA 94088-3650