Path: utzoo!attcan!uunet!mfci!colwell From: colwell@mfci.UUCP (Robert Colwell) Newsgroups: comp.arch Subject: Re: benchmarks Message-ID: <399@m3.mfci.UUCP> Date: 15 May 88 00:53:53 GMT References: <30872@amdahl.uts.amdahl.com> <3460014@hpsrla.HP.COM> <2175@winchester.mips.COM> <31755@amdahl.uts.amdahl.com> <8734@ames.arc.nasa.gov> Sender: root@mfci.UUCP Reply-To: colwell@mfci.UUCP (Robert Colwell) Organization: Multiflow Computer Inc., Branford Ct. 06405 Lines: 35 In article <8734@ames.arc.nasa.gov> eugene@pioneer.UUCP (Eugene N. Miya) writes: >In summary from my paper: > >What makes good benchmarks: > >You require reproducibility, comprehendability, and you would like >simplicity. In fact, benchmarks are too simple. Machines are becoming >more diverse: multiprocessors of different architectures, smart >software, etc. > >--eugene miya, NASA Ames Research Center, eugene@aurora.arc.nasa.gov I think there's one other side to the benchmarking issue. It's currently fashionable to dump on the simple benchmarks (Dhrystone, puzzle et. al.) and I've done my share. But the really simple benchmarks are not without value. I think one should always start with them, and if they run well, it's ok to smile for a few minutes. If they don't, your machine/architecture/toaster may require some adjustments, and the simpler benchmarks are much less ambiguous in terms of causes and effects than larger ones (fewer cache effects, clearer compiler/architecture interactions, controllable I/O and disk accesses...) If what we're looking for is a set of benchmarks that would be useful in comparing diverse machines and architectures, then I second John Mashey's call for better integer codes. I just hope that we don't swing the pendulum all the way from "compare machines on trivial benchmarks and make unwarranted conclusions" (which was all the rage a while back) to "small benchmarks are a joke and anyone who uses them is too." Bob Colwell mfci!colwell@uunet.uucp Multiflow Computer 175 N. Main St. Branford, CT 06405 203-488-6090