Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.1 6/24/83; site hcr.UUCP
Path: utzoo!hcr!mike
From: mike@hcr.UUCP (Mike Tilson)
Newsgroups: comp.arch,comp.org.usenix
Subject: Re: Benchmarking the 532, 68030, MIPS, 386...at a Usenix!
Message-ID: <2641@hcr.UUCP>
Date: Wed, 20-May-87 18:59:51 EDT
Article-I.D.: hcr.2641
Posted: Wed May 20 18:59:51 1987
Date-Received: Fri, 22-May-87 00:35:46 EDT
References: <324@dumbo.UUCP> <809@killer.UUCP> <2417@homxa.UUCP> <15484@gatech.gatech.edu> <401@ralmar.UUCP>
Organization: HCR Corporation, Toronto
Lines: 24

There has been a lot of discussion about running some good benchmarks
at Usenix.  Many people have made suggestions of what should be run,
with no consensus emerging, except a general feeling that benchmarks
at the vendor exhibit would be a good thing, and that a grand comparison
involving all vendors should be conducted.

What people are really trying to do is to generate a single "performance"
number, and to circumvent the "less-than-trustworthy" guys in suits who
prevent ready access to machines at a show.  However, I think a good
benchmark requires a lot of thought, and it can only be run in reference
to an intended application.  That is why some commercial benchmark
products take hours to run -- they gather lots of data that a customer
can interpret depending on the intended application.  (Maybe you want
I/O performance, or MIPs, or FLOPs, or paging rate, etc.)  This kind of
careful analysis can't be done properly on the floor of a trade show.

I think a "Usenix Benchmark Contest" would tend to perpetuate the
misuse of benchmarks, and for that reason I'd suggest it isn't a good idea.
Benchmarks are readily available for use by customers who are serious
about comparing performance.  A trade show benchmark war would encourage
increased marketing hype with very little hard technical information.
I'd like to see less of this at Usenix, rather than more.

/Michael Tilson, HCR Corporation, {utzoo,ihnp4,...}!hcr!mike