Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!think.com!spool.mu.edu!uwm.edu!linac!convex!rosenkra
From: rosenkra@convex.com (William Rosencranz)
Newsgroups: comp.sys.atari.st
Subject: Re: What's a fair comparison?
Keywords: Atari Motorola Intel GNU C ZEOS benchmark
Message-ID: <1991May30.233004.24755@convex.com>
Date: 30 May 91 23:30:04 GMT
Article-I.D.: convex.1991May30.233004.24755
References: <1991May30.145023.1684@chinet.chi.il.us>
Sender: usenet@convex.com (news access account)
Organization: CONVEX Computer Corporation, Richardson, Tx., USA
Lines: 34
Nntp-Posting-Host: convex1.convex.com

In article <1991May30.145023.1684@chinet.chi.il.us> saj@chinet.chi.il.us (Stephen Jacobs) writes:
>has had several articles on the subject), but I'll modestly propose as a 
>benchmark the time for GNU C to compile itself, with all temporary files on
>hard disk.  Any other suggestions?

benchmarking is often serious stuff, at least at the level i deal with.
so, you must also post guidelines like 1) all tmp files to single n sized
(empty) partition, 2) hd seed rate should be included in results, 3) non-
standard clock speed, 4) version of GNU C (which can't be enhanced), 
5) compiler switches, etc, etc, etc...

as u can see, to truly make objective comparisons you have to be thorough.
in general, benchmarks including compiles are less desirable (yes, even
SPECmark) since you more often than not compare compilers not systems.
dhrystone is notorious for this (i have seen *widely* varying DS for
exactly the same hardware, just different compilers). i think you are
better off with real or near-real applications and just use the fastest
compiler for that system. report the compiler as well. at least with
real applications, users have a better feel more often than not.

i would also try to exercise different aspects of an architecture: raw
cpu, data/instruction cache, i/o, screen writes, etc. fortunately (???)
the TOS ST is not multitasking (really) so that at least there are no
issues like throughput vs single job times :-)

in general, the more information the better. try to avoid single number
performance indeces. let the user make up his own mind. his/her application
may be totally different than yours...

-bill
rosenkra@convex.com
--
Bill Rosenkranz            |UUCP: {uunet,texsun}!convex!c1yankee!rosenkra
Convex Computer Corp.      |ARPA: rosenkra%c1yankee@convex.com