Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!cs.utexas.edu!sun-barr!newstop!texsun!convex!rosenkra
From: rosenkra@convex.com (William Rosencranz)
Newsgroups: comp.benchmarks
Subject: Re: benchmark evaluations (was Re: expanded bc results)
Message-ID: <111430@convex.convex.com>
Date: 15 Dec 90 18:01:28 GMT
References: <12220@hubcap.clemson.edu>
Sender: news@convex.com
Organization: Convex Computer Corporation; Richardson, TX
Lines: 48

In article <12220@hubcap.clemson.edu> mark@hubcap.clemson.edu (Mark Smotherman) writes:
> [ list of reasonable bm criteria deleted ]
>In trying to evaluate the bc test (i.e., echo 2^5000/2^5000 | /bin/time bc)
>according to these criteria, I am deeply disturbed by the continuing
>promotion of "bc" in the face of evidence that the benchmarks being run
>on the different systems are not identical.
>
>IMHO, "bc" only has compactness on its side, with representativeness
>questionable and reproducibility totally ruled out.  Why, then, is there
>continuing interest?

why, indeed...

i heartily agree with Mark on this. i have been persuaded that bc can be
a useful bm, PROVIDED (and this is a BIG caveat):

	1) we all use IDENTICAL SOURCE CODE as the basis of comparison
	2) we all have IDENTICAL LOADS ON THE SYSTEM (none is easiest)
	3) we report wallclock time (or wallclock and cputime)

however, this rules out compactness since you will STILL have to carry
a tape around at tradeshows, if you are in fact interested in truly
unbiased results.

i have just run this ditty on a convex C210 and have noted the following:

	1) i ran on a lightly loaded system, though not dedicated.
	2) my times were about 2/3 the time reported for a C2xx here,
	   very nearly the time quoted for a YMP (which has a 6x faster
	   clock and should run at least 2-3x faster than a C210 (and
	   no, i refuse to post our actual numbers)

it is all to obvious that apples and grapes are being compared here.
this is hardly scientific, and reduces the output of this group to
meaningless drivel, hardly what is was set up to do.

if there is a PD version of bc, i suggest that it and it ALONE be used
as a basis of comparison. and run it on a dedicated machine or distribute
a simulated "load" as part of the bm.

if i were cray, i'd really push for this :-)

-bill
rosenkra@convex.com

--
Bill Rosenkranz            |UUCP: {uunet,texsun}!convex!c1yankee!rosenkra
Convex Computer Corp.      |ARPA: rosenkra%c1yankee@convex.com