Path: utzoo!utgpu!watmath!att!dptg!rutgers!network!sdcsvax!ucsdhub!hp-sdd!megatek!jeff!rgs
From: rgs@jeff.megatek.uucp (Rusty Sanders)
Newsgroups: comp.windows.x
Subject: Re: Problems with XSTONES calculations in xbench
Message-ID: <716@megatek.UUCP>
Date: 30 Aug 89 18:34:12 GMT
References: <4344@orca.WV.TEK.COM>
Sender: news@megatek.UUCP
Lines: 52

From article <4344@orca.WV.TEK.COM>, by adamsc@shark.WV.TEK.COM (Chuck Adams):
> About two weeks ago I tried to contact Claus Gittinger with
> the following concerns I have about the XSTONES calculations
> in xbench.  Because I have concerns that someone may be 
> using this program I will post this mail awaiting 
> an answer from Claus.

[description of problem and patches to fix it deleted]

I noticed when the benchmark first came in that the way it actually calculated
the synthetic stones numbers and how it described how it calculated those
numbers didn't quite agree. Without looking too deeply at your patch it does
appear to fix this problem.

However, I'm not sure that the actual problem is in the code. My impression is
that the actual algorithm used is what it should be, and the text description
should be changed to reflect the code, not the other way around.

Remember that the Xstones number is synthetic, and doesn't need to actually
represent any real comparisons. As long as different Xstones numbers can be
compaired in some predictable fasion then all is well.

With the current algorithm, servers are rewarded if they have consistent
performance across all tested areas. Bad performance in any one area can
seriously effect the Xstones number.

With the modified code, servers are rewarded if any significant areas perform
very well, even if others perform absolutly abysmally. For a general benchmark
I suspect the first behaviour is better than the second.

It doesn't help that the benchmark system (Sun 3/50, R3, no fpu) runs arcs very
slowly. This allows any decent server to get arcStones in the hundreds of
thousands, if not the millions. Even though arcStones makes up a small
percentage of the final Xstone, a small percentage of a HUGE number is still a
large number. This seriously skews the Xstones values for such machines.

These problems could be mitigated by using a better benchmark base. But I
really feel that the effect of the current algorithm gives a better comparison
base then using the algorithm described in the text, and implemented in your
patch.

Of course, none of this is to imply the xbench is really a great benchmark. Its
biggest asset is that it comes up with one final number, which can be used as
a quick "general estimation" of a server's speed. A server could have a quite
low Xstones rating, and still be the best price/performance solution to a
particular application. Likewise, a server with a high Xstones rating could be
a real dog for some applications. But, for a quick reference, I believe the
Xstones number is usable. It at least sets a scale as a basis for further
benchmarking efforts.
----
Rusty Sanders, Megatek Corp. --> rgs@megatek or...
         ...ucsd!    ...hplabs!hp-sdd!    ...ames!scubed!   ...uunet!