Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!tut.cis.ohio-state.edu!ucsd!ucsdhub!celit!ps
From: ps@fps.com (Patricia Shanahan)
Newsgroups: comp.arch
Subject: Re: Neal Nelson Benchmarks
Message-ID: <7020@celit.fps.com>
Date: 28 Feb 90 16:14:27 GMT
References: <196@zds-ux.UUCP> <231@iss-rb.SanDiego.NCR.COM>
Sender: daemon@fps.com
Reply-To: ps@fps.com (Patricia Shanahan)
Organization: FPS Computing Inc., San Diego CA
Lines: 42

In article <231@iss-rb.SanDiego.NCR.COM> steves@ivory.SanDiego.NCR.COM (Steve Schlesinger x2150) writes:
...
>
>My advice to anyone looking for benchmarks is first to use the
>applications you currently run, then use SPEC data for computational
>results (with your own weightings) and the SCB.
>
>Steve
>::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>steve schlesinger	steve.schlesinger@sandiego.ncr.com
>619-485-2150		NCR - 4010, 16550 W Bernardo Dr, San Diego, CA 92127
>::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


I would reverse the order here. Use "standard" benchmarks to decide which
systems are likely to be useful enough to you to be worth really measuring,
then use multiple jobs of the types you are really going to run to measure
the value of each of those systems to you.

The main value I see in standard benchmarks is that you can get numbers for
them a lot quicker and cheaper than a full scale benchmarking excersize
using actual jobs.

I do think the SPEC approach is likely to be more robust in the face of
architecture changes than the more abstract benchmark approaches. The real
problem with abstract benchmarks is that future architecture changes can
make whatever was changed in doing the abstraction critically important.
For example, the whetstone benchmark was designed to measure, among other
things, the performance of array references that were observed to be a
significant component of scientific computing. At the time is was not
obvious that vector length mattered, so the whetstone only tests arrays
of length 4. By definition an abstract benchmark differs from the real
jobs that it models in some ways. Those differences, for a well-designed
benchmark, will not be significant for CURRENT architecture and compiler
technology. On the other hand, if you select real jobs, they have a better
chance of looking like real jobs in aspects that are unimportant to current
systems but that may be critical to performance on future systems.
--
	Patricia Shanahan
	ps@fps.com
        uucp : {decvax!ucbvax || ihnp4 || philabs}!ucsd!celerity!ps
	phone: (619) 271-9940