Path: utzoo!utgpu!attcan!uunet!lll-winken!lll-tis!helios.ee.lbl.gov!nosc!marlin!aburto
From: aburto@marlin.NOSC.MIL (Alfred A. Aburto)
Newsgroups: comp.arch
Subject: Re: NSIEVE C Source File (long)
Message-ID: <1076@marlin.NOSC.MIL>
Date: 1 Sep 88 19:38:07 GMT
References: <1070@marlin.NOSC.MIL> <1073@marlin.NOSC.MIL> <2899@winchester.mips.COM>
Reply-To: aburto@marlin.nosc.mil.UUCP (Alfred A. Aburto)
Distribution: comp.arch
Organization: Naval Ocean Systems Center, San Diego
Lines: 66

--------------

There is of course no denying the the fact that 'nsieve' (really sieve)
and so many other so-called 'benchmark' programs 'bear little resemblance
to most substantial' programs.

The Sieve (Jim Gilbreath, 'A High-Level Language Benchmark', BYTE, Sep 1981,
page 180) is not really a 'benchmark' program.  Gilbreath indicated this
in his article, but none the less he continued to call the program a           
and coded in many languages for the purpose of comparing relative            
performance.  It is certainly not a 'typical' program in terms of what
it does (generate some prime numbers), and it is not typical with respect
to the types of instructions used.

I think alot of confusion happens because (and I'm guilty of this) we keep
calling the Sieve a benchmark when it really isn't anything more than a
simple program to generate some prime numbers.  There is even more confusion
because there is apparently no clear cut set of definitions or specifications
or requirements for a computer benchmark program.  What is a benchmark
program?  What kinds of things should it do?  What kind of outputs should it
provide?  What minimum requirements should be met?  None of these things
are defined and as a result we find ourselves with alot of programs that
claim to be benchmarks of some sort.  The Whetstone and Dhrystone are good
but still I don't think of them as 'benchmarks' or standards --- mostly
because they don't provide much information relative to the performance of
the system under test and because they can be optimized to various degrees.
'nsieve' is I think relativey immune (I'm not positive of this) to high
degrees of optimization, but it provides no weighting for the frequency
of instructions used and it has other biases as your analysis shows.

Still, in my opinion, these programs when run on many systems using a variety
of different languages do provide valuable information with respect to the
capability of these systems (although somewhat limited information).

I don't have alot of information at this time but some nsieve results show
'performance gains' very close to what we see from the Dhrystone. I can't
'prove' that the nsieve and dhrystone provide comparable performance gains
in general as I lack a sufficient amount of data, but some of the results
indicate that they do.

For example, the Amdahl 5890 Dhyrstone/sec is I believe about 30,000. Well,
on the Amiga with 68020 at 14.32 Mhz and Manx Aztec C (with no 'optimizer')
the best Dhrystones/sec figure I've seen is 3034.  So the Amdahl 5890 has a
performance gain of 30000/3034 = 9.9 (roughly) over the Amiga/020. 

Thanks to Chuck Simmons of Amdahl, I also have the nsieve results.  The
Amdahl 5890 ran nsieve (for the 8191 sized array) in 0.05 seconds.  An
average of 5 runs on the Amiga/020 yielded a run time of 0.48+/-0.02 seconds.
These results indicate the Amdahl 5890 has a performance gain of 
0.48 / 0.05 = 9.6 over the Amiga/020.  That is (in this one isolated case)
both the Dhrystone and the nsieve are yielding pretty much the same 
information within about a 4% error.  I would like to investigate these
relationships in more detail, but (right now) there is not sufficient data.

Anyway, thanks again for the nsieve analysis.   I'm not a computer systems
person really --- just a physicist working as an engineer who has somehow
'fallen into' the benchmarking arena.                  

Thanks to Bill Davidson too!  For the Unix timer information.

B A Wichmann of the National Physical Laboratory in England has also sent
me a recent (Mar 88) version of the Whetstone in ISO Pascal --- I posted
it to BIX 'supermicros/listings' as 'whet.arc'.  The main feature of the
new Whetstone is that it provides error checking routines.

Al Aburto
aburto@marlin.nosc.mil.UUCP