Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!mcnc!ece-csc!ncrcae!ncr-sd!sdcsvax!darrell
From: fouts@orville%ames.arpa (Marty Fouts)
Newsgroups: mod.os
Subject: Submission for mod-os
Message-ID: <2687@sdcsvax.UCSD.EDU>
Date: Tue, 3-Feb-87 18:08:24 EST
Article-I.D.: sdcsvax.2687
Posted: Tue Feb  3 18:08:24 1987
Date-Received: Sat, 7-Feb-87 09:35:55 EST
Sender: darrell@sdcsvax.UCSD.EDU
Organization: NASA Ames Research Center, Mountain View, CA
Lines: 27
Approved: mod-os@sdcsvax.uucp

Direct measurement is very useful, as Gene has pointed out, but suffers
from drawbacks:

1) measurement interfers with the system being measured.
2) measurement is subject to sampling frequency related distortion.
3) measurement can miss transients.
4) measurement still requires analysis and interpretation.
5) Some times you need to know if it will work before you can build it.

In fact, measurement by itself doesn't solve most of Gene's concerns.  It is
easy to cook a set of measurements to obtain the desired results.  After all,
many vendor supplied benchmarks are merely cooked measurements.

You do trade off in reality.  You ignore events that you can't process quickly
enough, you ignore data you don't understand, and you ignore events that you
can't measure adequately.  Direct measurement has as many drawbacks as
modeling.

Measurement is only the first step in understanding.  Building models to
predict future behavior is the second.  Comparing the results of models to
real measurements is the third.  You only understand something if you can
predict its behavior and demonstrate the validity of your predictions.

In the context of computer performance analysis, a set of measured results
will frequently lead to ambiguous interpretation.  It is sometimes not
possible to find a measurement which will distinguish between the alternatives
without first producing at least an analytic model of the system.