Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!mailrus!uwmcsd1!leah!itsgw!steinmetz!uunet!mfci!colwell
From: colwell@mfci.UUCP (Robert Colwell)
Newsgroups: comp.arch
Subject: Re: Checkered Benchmark History (was: NSIEVE C Source File (long))
Message-ID: <513@m3.mfci.UUCP>
Date: 29 Aug 88 01:07:33 GMT
References: <1070@marlin.NOSC.MIL> <1073@marlin.NOSC.MIL> <2899@winchester.mips.COM> <62@radix>
Sender: root@mfci.UUCP
Reply-To: colwell@mfci.UUCP (Robert Colwell)
Distribution: comp.arch
Organization: Multiflow Computer Inc., Branford Ct. 06405
Lines: 33

In article <62@radix> jimv@radix.UUCP (Jim Valerio) writes:
>For several years now, I have been feeling guilty whenever SIEVE was quoted
>as a benchmark.  You see, I'm afraid that I may be indirectly responsible
>for it's use.

Valerio, you slime, you've single-handedly screwed the entire
computing world forever.  (There.  I know you expected that of
someone, and I didn't want to let you down.)

>Over the years, the benchmark has changed enough that I don't see any of
>my code there.  But I'm left with the guilty feeling that with an hour's
>work and a soggy listing, I am responsible for one of the worst of the
>often-quoted benchmarks.

But but but....it's not the benchmark, it's what the benchmarker is
trying to make of it.  The Sieve benchmark has its place.  For
instance, if you store the flags array as a set of bits, and try to
access them a bit at a time on a scientific minisuper, you may find
out just how often such a machine needs to execute extracts and
merges.  That could conceivably be an interesting metric under
certain extreme conditions.

If you're implying that we could or should be trying to make more
bulletproof benchmarks, so that it is more difficult to twist their
results to favor particular machines or architectures, then I'd say
that's a worthwhile effort.  But until that succeeds (IF it succeeds,
which I doubt) then there's a place in the world for toy benchmarks.


Bob Colwell            mfci!colwell@uunet.uucp
Multiflow Computer
175 N. Main St.
Branford, CT 06405     203-488-6090