Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!husc6!mit-eddie!uw-beaver!ubc-vision!alberta!calgary!radford
From: radford@calgary.UUCP (Radford Neal)
Newsgroups: comp.arch
Subject: Re: Towards A Meaningful Performance Measure
Message-ID: <1203@vaxb.calgary.UUCP>
Date: Tue, 10-Nov-87 16:36:18 EST
Article-I.D.: vaxb.1203
Posted: Tue Nov 10 16:36:18 1987
Date-Received: Fri, 13-Nov-87 21:53:20 EST
References: <861@winchester.UUCP> <2993@phri.UUCP> <864@tut.cis.ohio-state.edu> <881@mips.UUCP>
Organization: U. of Calgary, Calgary, Ab.
Lines: 28
Keywords: benchmarks

In article <881@mips.UUCP>, hansen@mips.UUCP (Craig Hansen) writes:

> As to standarizing on a single compiler/OS, remember that the company
> producing the base architecture has an interest in making their machines
> look competitive. When competitors say their machine is 10X a VAX 780, when
> using trussed up benchmarks and a markedly inferior compiler/OS on the VAX,
> DEC should by all rights be screaming bloody murder. Should DEC claim that
> their VAX 780 is 1.5 times faster than their VAX 780?

This looks like a good argument *for* standardizing on an obsolete
machine You've *got* to standardize all aspects of the base system,
otherwise numbers today aren't comparable with numbers last year. An
obsolete machine's software won't be improving much, so this won't mislead
people. And the obsolete machine's manufacturer won't care what 
people think of its performance any more.

So the ideal benchmark standard is an obsolete machine (with static
operating system and compiler), that is nevertheless still common
and likely to remain so. The Vax/780 and the IBM PC are about as
close as one is likely to get, but the IBM PC has a "typical" architecture
only if you restrict it to "small model" programs, which is not
tolerable. Thus the current use of a Vax/780 seems as good as we
can hope for, though standardizing on some old operating system (e.g 4.2)
is likely to be a problem (who wants to keep it around just for
benchamarking?)

    Radford Neal
    The University of Calgary