Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!bloom-beacon!bu-cs!encore!bzs@xenna
From: bzs@xenna (Barry Shein)
Newsgroups: comp.arch
Subject: Re: Benchmarking
Message-ID: <3913@encore.UUCP>
Date: 19 Oct 88 03:10:45 GMT
References: <4655@winchester.mips.COM> <572@jim.odr.oz>
Sender: news@encore.UUCP
Reply-To: bzs@xenna (Barry Shein)
Organization: Encore Computer Corp
Lines: 31
In-reply-to: jon@jim.odr.oz (Jon Wells)

The short story on benchmarks is: Make a careful hypothesis and design
an experiment which will provide relevant data relating to that
hypothesis.

Hypothesis: This processor/memory combination is faster than that
one on simple integer operations.

Experiment: Design and run a small benchmark which will allow you
to time each.

Hypothesis: This system is faster at running large programs which
stress the virtual memory system.

Experiment: Design and run a benchmark which will allow you to time
each.

Unfortunately one has to know how to relate their hypothesis to the
benchmark design. Instrumentation and careful refinement of the
hypothesis (eg. define "stress the virtual memory system") helps.

Then there's the old rule of thumb that the speed of a computer system
is measured from the moment you get an idea in your head to the moment
you have the answer in your hands, any other measure is superfluous.

I knew a scientist who insisted his Vax730 was many times faster than
the huge campus IBM mainframe based on that rule. He could usually go
from conception to answer on the 730 in less time than it took
standing on line waiting for a user services person to explain what
IEH700104 meant. I think he was right.

	-Barry Shein, ||Encore||