Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!csd4.csd.uwm.edu!uakari.primate.wisc.edu!ames!ncar!ico!vail!rcd
From: rcd@ico.ISC.COM (Dick Dunn)
Newsgroups: comp.unix.i386
Subject: _UNIX_Today!_ hits a new benchmarking low
Message-ID: <16054@vail.ICO.ISC.COM>
Date: 29 Aug 89 23:10:47 GMT
Organization: Interactive Systems Corp, Boulder, CO
Lines: 40

Cautious disclaimer:  I'm speaking as an individual, not for ISC.

The August 21, 1989 issue of _UNIX_Today!_ has an article reviewing the
Intel (nee Bell Tech) MPE--a 386-based "workstation".  The discussion
itself is OK, but the comparison to other machines is a mess.  The MPE
with Bell Tech's UNIX is compared for I/O performance to other hardware/
software pairs which are in a different league.  The selection is so badly
biased that the MPE could hardly help but win out.  Now PLEASE NOTE that I
am NOT saying that the bias was deliberate.  I am making NO attempt to
suggest that the author of the article played favorites.  It may well be
that he just made a truly inept set of choices, or tested with whatever
old hardware and software he had at hand.  It doesn't matter; the results
are bad and they don't present other vendors fairly.  I'll illustrate the
problems by comparing the MPE to the box used to test 386/ix.

The Intel/Bell Tech machine is a 25 MHz 386 with 64K cache.  386/ix was
tested on a 16 MHz 386 "Inboard" (a 386 add-in board) which has no cache.
This probably represents about a 2:1 CPU speed difference.

The MPE had a large CDC ESDI drive--which is presumably 10 Mb/s, 1:1
interleave, and under 20 ms average seek.  The 386/ix machine had a Maxtor
XT-1085 on the standard old WD controller--with 5 Mb/s, 2:1 interleave, and
27 ms seek.  It's hard to give an accurate value for the cumulative effect
of these factors, but it's probably in the range of 2:1 to 3:1.

The MPE was tested with V.3.2; the 386/ix version used was 1.0.6, which is
a V.3.0 and was the release before ISC's Fast File System was incorporated.
Why, when ISC was first out of the gate with V.3.2, is _UNIX_Today!_ doing
a test with our V.3.0 and Bell Tech's V.3.2???  The 1.0.6 release is *last
year's* system, and it's particularly annoying that they used such an old
system for an I/O benchmark when the current system has a major I/O
improvement.  (The Microport system used for comparison is 2.2, which is
also pretty old.)

Why is the MPE compared to old software and older, cheaper hardware?
What's the point in publishing such a lopsided comparison?  I'd love to
hear anyone else's insights into this matter.
-- 
Dick Dunn     rcd@ico.isc.com    uucp: {ncar,nbires}!ico!rcd     (303)449-2870
   ...Are you making this up as you go along?