Path: utzoo!yunexus!geac!syntron!jtsv16!uunet!seismo!sundc!pitstop!sun!amdcad!ames!ncar!tank!uxc!uxc.cso.uiuc.edu!urbsdc!aglew
From: aglew@urbsdc.Urbana.Gould.COM
Newsgroups: comp.arch
Subject: Re: Bar Wars again (benchmarking)
Message-ID: <28200213@urbsdc>
Date: 17 Oct 88 05:13:00 GMT
Article-I.D.: urbsdc.28200213
References: <4263@wright.mips.COM>
Lines: 84
Nf-ID: #R:wright.mips.COM:4263:urbsdc:28200213:000:3711
Nf-From: urbsdc.Urbana.Gould.COM!aglew    Oct 17 00:13:00 1988


>/* Written  3:39 am  Oct 15, 1988 by eugene@eos.UUCP in urbsdc:comp.arch */
>In article <6005@june.cs.washington.edu> pardo@cs.washington.edu (David Keppel) writes:
>>rik@june.cs.washington.edu (Rik Littlefield) writes:
>>>[ large "real" program benchmarks vs. synthetic benchmarks ]
>>
>>Oh, gee, an opportunity to apply the scientific method :-)
>>
>>(a) Benchmark a bunch of computer systems (hardware/os/compiler)
>>    using synthetic benchmarks.
>>(b) Compare the benchmark performance to observations in the
>>    "real" world.
>>(c) Learn something about benchmarks, refine your synthetic
>>    benchmarks.
>>(d) go to (a)   (Oh no, not a GOTO!)
>
>I am sorry.
>
>I don't see the scientific method in this.  I don't see a theory,
>a hypothesis, a controlled experiment, nor even a control. 8-)
>Actually, don't worry, I get this all the time from the other
>"real" sciences myself.  I do see the beginnings of empirical work.
>Better luck next time.
>
>Another gross generalization from
>
>--eugene miya, NASA Ames Research Center, eugene@aurora.arc.nasa.gov

This has little to do with the discussion, but Eugene has touched
upon a sore point of mine here. 

	SCIENCE >= SCIENTIFIC METHOD

The theory-hypothesis-controlled experiment cycle is only one
part of the scientific process, the part that is applied to
relatively (1) well-understood and (2) easy to manipulate systems.
Eg. biology - at least meets criterion (2); criterion (1) is
often not met (and is the reason why so many biological experiments
cannot really be considered "controlled").
Eg. psychology (with a willing group of undergrads to study).
Etc.

Much of science goes on before the theory-hypothesis-experiment
cycle, and is purely observational. Ie. you must have some information
before you can formulate a theory. Eg. astronomy - it's kind of hard
to set up an experiment out in space. Eg. descriptive biology,
morphology - cut the animal up and describe what you see.
    True, observation can be selective, like an astronomer 
concentrating on a star cloud that he thinks will have a
supernova -- but if such an astronomer throws out evidence of
a black hole, even though he wasn't looking for it explicitly,
he is stupid.

I would even go so far to say that people who think that science
is only the theory-hypothesis-experiment cycle verge on
Aristotelian in their dogmatism.

What does this have to do with computer performance evaluation?
Well, I'll admit that we need more controlled experiments.
But, in a practical sense, the people who apply computer performance
evaluation are system operators and programmers, even individual
workstation users, who aren't in the business of doing experiments.
But they can observe, on a regular and ongoing basis.
How can we make these observations as efficient as possible
-- how can we use this great mass of potential observers to
generate hypotheses, that we can seek to prove or disprove by
experiment, that will ultimately result in improved computer
systems performance?


Andy "Krazy" Glew. 
at: Motorola Microcomputer Division, Champaign-Urbana Development Center
    (formerly Gould CSD Urbana Software Development Center).
mail: 1101 E. University, Urbana, Illinois 61801, USA.
email: (Gould addresses will persist for a while)
    aglew@gould.com     	    - preferred, if you have MX records
    aglew@fang.gould.com     	    - if you don't
    ...!uunet!uiucuxc!ccvaxa!aglew  - paths may still be the only way
   
My opinions are my own, and are not the opinions of my employer, or any
other organisation. I indicate my company only so that the reader may
account for any possible bias I may have towards our products.

PS. I promise to shorten this .signature soon.