Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!ll-xn!husc6!endor!reiter
From: reiter@endor.harvard.edu (Ehud Reiter)
Newsgroups: comp.arch
Subject: Benchmark Timing
Message-ID: <1967@husc6.UUCP>
Date: Tue, 12-May-87 16:36:30 EDT
Article-I.D.: husc6.1967
Posted: Tue May 12 16:36:30 1987
Date-Received: Fri, 15-May-87 02:15:38 EDT
References: <4294@nsc.nsc.com> <28200036@ccvaxa>
Sender: news@husc6.UUCP
Reply-To: reiter@endor.UUCP (Ehud Reiter)
Organization: Aiken Computation Lab Harvard, Cambridge, MA
Lines: 33

In regard to the discussion about using real programs with I/O as benchmarks,
I think one question that has to be addressed is what kind of timing should
be used.  The possibilities include:

    1) Elapsed Real Time.  This would seem to be too dependent on the
I/O devices, and on where the files physically reside on the disk (e.g.
how long do the seeks take?).  It also ignores multiprocessing.

    2) Kernal Plus User Time.  This would seem to be a good measure, which
has the advantage of including O.S. overhead in the timings (O.S. overhead is
part of the timing for real applications!).  Problems include
      -- A multitasking, multiuser kernal would have to do a lot more
bookkeeping than a single task, single user system, so the multiuser
system would seem slower.  Is this "fair" - I'm not sure.
      -- An I/O system which went through a lot of effort to optimize disk
accesses and file allocation would look slower than a stupid file system,
since this time measure doesn't include I/O access time.  This does not
seem "fair".

    3) User Time.  This ignores O.S. overhead, and also penalizes a file
system which does most of its work in user mode, against a file system
which does most of its work in kernal mode.

    4) I/O-less operation.  We could rewrite the benchmark programs so that all
I/O is done in big blocks, outside the timing loop.  Totally ignoring
I/O makes our benchmark less representative of real programs, but it
does have the advantage of making the benchmark more controlled and perhaps
less subject to fiddling by marketing types (who, for example, might try running
the program with output directed to the null device instead of a disk!).

					Ehud Reiter
					reiter@harvard	(ARPA,BITNET,UUCP)
					reiter@harvard.harvard.EDU  (new ARPA)