Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!utgpu!water!watnot!watmath!clyde!cbatt!ihnp4!ptsfa!lll-lcc!pyramid!prls!mips!mash From: mash@mips.UUCP Newsgroups: comp.arch Subject: Re: Dhrystone and Dhampstone Message-ID: <12@winchester.mips.UUCP> Date: Sun, 1-Mar-87 05:59:46 EST Article-I.D.: winchest.12 Posted: Sun Mar 1 05:59:46 1987 Date-Received: Mon, 2-Mar-87 19:57:41 EST References: <500002@uicsg> Reply-To: mash@winchester.UUCP (John Mashey) Organization: MIPS Computer Systems, Sunnyvale, CA Lines: 72 In article <500002@uicsg> patel@uicsg.UUCP writes: > > >As an author of a benchmark DHAMPSTONE (posted earlier on this notes file) >and a researcher in this field I should point out the following >serious misconceptions/myths about the above two benchmarks. I hope >people stop misusing these benchmarks after they have read this. > ....a number of generally reasonable comments.... >Myth 3. It measures the compiler performance. > > The benchmarks were based on high level language statistics. No > information was incorporated regarding optimizing compilers and how > they might alter the statistics. In fact, using a good optimizing > compiler will totally distort the performance figures...... > type of operation and so on. Benchmarks were designed to evaluate > machine performance and not compiler performance. Some > impact of the compiler on the performance is unavoidable. But to > minimize the distortions, DO NOT USE AN OPTIMIZING COMPILER when > using the above benchmarks. > >Myth 4. It measures the absolute machine performance. > > Benchmarks provide only a relative measure of performance not an > absolute one. Thus if a machine X runs with a speed of 100 > Dhampstones (Dhampstone/sec) there is no simple way to predict > the speed of your "troff" or "cc" job. Agree. > However, if another machine Y has a speed of 25 Dhampstones then one > can safely assume that a reasonable mix of systems programs > (troff, edit, compile etc) would run 4 times faster on machine X than > on machine Y. Unfortunately, this last statement is just as misleading as the ways that people often use these numbers. For example, assume that X is a machine designed to be used with optimizing compilers, and it has good ones, that can gain (say) 20% performance (on exactly things like troff, diff, etc), and the normal system uses them. X is machine that is NOT designed for optimizing compilers, or at least doesn't have them. In this case, the "real" performance difference is about 5X, rather than 4X. The point is: if everyone has the same compiler technology, then this sort of benchmark perhaps has a gross correlation with performance. But if there are substantial differences in technology, the benchmarks ought to get fixed up so you can use benchmark the technology you use every day, instead of having to cripple it. This whole argument is like auto racing: some cars have turbochargers and some don't. You then have a race to discover the actual speeds of the cars, but you make those with turbochargers turn them off, and then claim the results tell you real performance. [Obviously, I have an axe to grind, in that we designed our chips as good compiler targets, and we provide serious optimizers.] Finally, on the Dhrystone topic, people may have noticed that the new IBM PC/RT is rated at 6500 Dhrystones, and claimed to be 4.5 RISC MIPS [whatever that is]. However, if you look carefully, you find that those are 1.0 Dhrystones, not 1.1, so subtract 15% right away, giving about 5500. Then, assuming that the new "Advanced C Compiler" is the one we think it is, it's in the same league as ours, and ours can add 15-20% to Dhrystone numbers, although some of that comes from dead code elimination, which (we think) IBM turned off when they were running the benchmark. So now, it's really hard to tell what the number means, but one would guess that the new RT is something like a 3-3.5Mips machine, on the scale where 1Mips = VAX 11/780 running 4.3BSD, although much of the floating point comes out at less than a Sun3/160. -- -john mashey DISCLAIMER: UUCP: {decvax,ucbvax,ihnp4}!decwrl!mips!mash, DDD: 408-720-1700, x253 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086