Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!husc6!cmcl2!beta!hc!ames!sdcsvax!jww
From: jww@sdcsvax.UCSD.EDU (Joel West)
Newsgroups: comp.sys.mac
Subject: Re: 386 vs. '020, BYTE Benchmarks, Smalltalk
Message-ID: <3928@sdcsvax.UCSD.EDU>
Date: Tue, 22-Sep-87 11:08:32 EDT
Article-I.D.: sdcsvax.3928
Posted: Tue Sep 22 11:08:32 1987
Date-Received: Thu, 24-Sep-87 05:15:28 EDT
References: <1714@tekchips.TEK.COM>
Organization: Palomar Software, Inc., Vista, CA
Lines: 28

In article <1714@tekchips.TEK.COM>, jans@tekchips.TEK.COM (Jan Steinman) writes:
> One problem with most benchmarks is that they are easily twisted to suit some 
> purpose.  The Smalltalk Benchmarks are now considered invalid because one 
> implementation caches compiled code.  

Actually, this shows my main point, which is most 30-line benchmarks
are next to worthless, anyway.  On a real-size benchmark, caching
code will show a significant performance advantage, so this
is a legitimate thing to do.

Of course, it's a lot of work to run small benchmarks; it can require
weeks to run the larger ones across a variety of machines.  So
we should not be suprised that people run around producing and
touting meaningless benchmark results, because those are the only
ones ever run.

My idea of a benchmark is to take three different 20,000 line programs,
written by three different authors from three different styles (it
would be nice if one were machine-generated, such as from C++
or Objective-C), and then see if they produce the correct result,
if they compile or run at all.  Many PC compilers would fail this
test miserably.
w
-- 
	Joel West  (c/o UCSD)
	Palomar Software, Inc., P.O. Box 2635, Vista, CA  92083
	{ucbvax,ihnp4}!sdcsvax!jww 	jww@sdcsvax.ucsd.edu
   or	ihnp4!crash!palomar!joel	joel@palomar.cts.com