Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!pacific.mps.ohio-state.edu!linac!att!ucbvax!hplabs!hpfcso!russb From: russb@hpfcso.FC.HP.COM (Russ Brockmann) Newsgroups: comp.arch Subject: Re: Novice question: measuring speed Message-ID: <8840019@hpfcso.FC.HP.COM> Date: 18 Mar 91 19:09:20 GMT References: <645@ssdc?> Organization: Hewlett-Packard, Fort Collins, CO, USA Lines: 97 #define slight_drift Scott Dorsey (kludge@grissom.larc.nasa.gov) writes: > The other problem that we come to is that with >the new RISC machines, the actual MIPS ratings are exceedingly high, much more >so than the performance would indicate if you used CISC processors as a >baseline. If you printed them, people wouldn't believe them, so the marketing >people came up with MIPS-equivalent numbers. VUPS for example ("Very Useless >Promotional Specification") is supposed to measure the compute power of a >machine in terms of 11/780's. It's calculated differently by every company, >and is often promoted as being a "MIPS" number. Surprisingly, this isn't necessarily so, at least in some cases. #define toot_horn_mode Case in point: The HP 835 (a RISC machine, and a quite old one at that) has a clock rate of 15 MHz, and can sustain instruction execution rates of about 11 MIPS (native instructions) (footnote 1). But the machine was rated at around 14 VUPS at intro (actually may have been 15, can't remember now). HP used the "conservative" method of using the geometric mean of 15 integer benchmarks compared to a VAX 11/780 (anyone care to dispute that an 11/780 is a CISC machine :~) ). The geometric mean is a much more conservative way of averaging benchmarks than the arithmetic mean. 835 Native MIPS rating (peak): 15 MIPS 835 Native MIPS rating (sustained): 11 MIPS 835 VUPS rating (geom. mean 15 BM's): 14 MIPS So, believe it or not, at least for this one instance, it was actually better for HP to quote VUPS than to quote sustained MIPS. Even to quote the "nop" rate of 15 MIPS wouldn't have grossly misstated the performance. And I don't think this is that uncommon. And now, back to our regularly scheduled program... #undef toot_horn_mode #undef slight_drift On the original question, MIPS are, for the most part, meaningless. The biggest problem is in attempting to quote a single "MIPS" number without knowing what "MIPS" number is meant. But the more a vendor clarifies _how_ he measured his "MIPS," and the more numbers he provides, the better a potential customer is able to determine a particular machines' strengths and weaknesses. Another problem is that RISC machines often depend heavily on very smart compilers to optimize the instructions needed to get a job done. A given machine can actually improve its "MIPS" rating over time as compilers get smarter. Note that MFLOPS ratings suffer from a slightly less widely known malady. Most chip vendors will quote their peak MFLOPS, which can be a very special case of highly tailored, pipelined operations with numerous assumptions. Most computer vendors tend towards Linpack MFLOPS, which can be compiled or hand-coded to varying degrees, and can be single- or double-precision. This, however, isn't always the case. A vendor may quote a peak rate of inline, non-interlocking FADDS. Or if his hardware supports parallel FADD/FMULT, he can quote an even higher number. This isn't all bad, as long as you know what the number means. A recent trend is to provide many numbers, including SPECmarks, Drystone MIPS, Linpack SP/DP MFLOPS, Whetstones, VAX MIPS, Khornerstones, IOstones, Xstones(?), and a plethora of other benchmark suites. Many manufacturers are now trying to be more careful to specify exactly how they measured the numbers, enough so that the numbers can be duplicated by the customer. This is a positive trend in my opinion, and makes it possible to make limited judgements on the performance of a machine. It's still dangerous, but with care, all "MIPS" numbers are not totally meaningless. MIPS or MFLOPS, by themselves, are incomplete units of measure, somewhat akin to saying $100.00 without saying whether it's US or Canadian. As always, the best test is to run the machines through the kind of workload you intend to use them for. #define nitpick MIPS (Million Instructions Per Second), not MIP (Million Instructions Per ?) MFLOPS (Million FLoating point Operations Per Second) not MFLOP... #undef nitpick -- Russ Brockmann russb@hpfcso.HP.COM (303)229-2489 Disclaimer: echo $MY_OPINIONS >/dev/null -------- footnote 1: J.D. Yetter, et al, "A 15MIP 32-Bit Microprocessor," Digest of Technical Papers, International Solid State Circuit Conference, February 1987. "/tmp/nfa08784" 96 lines, 4455 characters