Path: utzoo!attcan!uunet!ncrlnk!ncrcae!ece-csc!ncsuvx!gatech!bloom-beacon!apple!amdahl!pacbell!well!brecher
From: brecher@well.UUCP (Steve Brecher)
Newsgroups: comp.sys.mac
Subject: Re: Reekes on Disktimer (LONG)
Message-ID: <7625@well.UUCP>
Date: 10 Nov 88 22:22:20 GMT
References: <698@mouse.UUCP> <7595@well.UUCP> <20306@apple.Apple.COM>
Reply-To: brecher@well.UUCP (Steve Brecher)
Organization: Software Supply, Sunnyvale, CA
Lines: 127


I persist in the hope that some principles of disk performance testing
may be of interest.  It is not my purpose to defend DiskTimer II as a
superior benchmark.

In article <20306@apple.Apple.COM> posted on his behalf, Jim Reekes writes:

> Steve has defended the  non-use of the File Manager by stating, "I
> developed DiskTimer II precisely to avoid using File Manager calls, i.e.,
> to provide a benchmark that could be run without initializing the disk".
> This is ridiculous statement in my opinion.  How many people can use a 
> hard disk that hasn't been initialized?

A good benchmark that uses the File Manager requires initialization of the
disk followed by loading it with a suite of test files -- the same files
loaded in the same order for each disk under test.  This is necessary
to make the File Manager's view of the disk identical among disks under
test.

> I understand that DiskTimer [II] is actually testing the driver's 
> lowest level of the interface, so then maybe it should be renamed 
> to "DriverTimer".

DiskSubSystemTimer, anyone?  The results are a function of the disk
driver software and the disk hardware, including disk controller
and disk mechanism.  They are also a function of the host Mac
hardware, and any system software (e.g., SCSI Manager) used by
the driver.

> I also understand that DiskTimer is not intended to be a test of
> actual system performance.  But then, of what use are the results
> in that case?

They are of use in many cases in evaluating performance in transferring
24KB blocks of data, and of access time.  (More on the latter below.)
I suspect that the real contribution of DiskTimer II has been its use by
disk system vendors in evaluating and in some cases motivating
improvement in their product performance characteristics.  For example,
the DataFrame XP series grew out of a resolve formed by Steve Edelman
(SuperMac) in response to the results of the original version of DiskTimer
(which was then called DiskBench, I think). That DiskTimer II results have
been abused by advertisers should be cause for complaint about or to the
advertisers, not the program. The most recent abuse, since discontinued,
has been by PLI in claiming miraculous access times (due to cacheing).

> MacWeek has found that the timings of DiskTimer II have no 
> correlation with real world performance.

*No* correlation?  I missed the article referred to, but if Ric
Ford said that he was in error. Consider, from slow to fast,
an Apple serial-port HD20, a typical 20M Seagate N-series based
SCSI subsystem, and a typical large CDC Wren based system. "Slow
to fast" describes both real world performance and DiskTimer II
results.  DiskTimer II's imperfections as a benchmark notwithstanding,
it does actually use the disk in a way that must be correlated with
*some* aspects of real world performance.

> There needs to be a differentiation in the test between 
> multiple vs. single block transfers.  A large multiple block reads 
> will always perform best on a drive formatted with a 1:1 
> interleave.  In the real world of Macintosh, most drives are 
> interleaved.  This causes the multiple block read/write tests to 
> be weighted towards any drive formatted with no interleave and 
> will handicap a drive that is interleaved. [as we move to higher 
> capacity drives, the 1:1 interleave is more common since these 
> drives often do not support interleaving -JR]

Multiple block reads perform best on a drive with 1:1 interleave
provided that interleave is suitable for the subsystem, i.e., that
the subsystem keeps up (does not miss sectors).  If the subsystem
is too slow to keep up with 1:1, then best results will be at
whatever interleave is suitable for it, and 1:1 will not be best.
Drives that require interleaving for best performance are of
course "handicapped" relative to drives that don't -- that's the
whole point.  It's not that the big, fast drives don't *support*
interleaving; it's that they don't *require* it for maximum
throughput.  (Whether they support changing the interleave by
the user is not relevant.)

> Consideration must be given to the drive's interleave, number 
> of read/write heads, number of bytes per block, number of blocks 
> per track, and number of tracks per cylinders.  Without this 
> information, no test will be fair to drives with dissimilar 
> geometries.
> ...
> With DT's access test, a 
> slower seeking drive may test better than a faster seeking drive.  
> How?  The answer is in the drive's geometry.  As an example; if a 
> drive with 15 read/write heads is matched against a drive with 
> only 4 heads, then the bigger drive has access to nearly four 
> times as much data without performing any head seeks!  This is 
> exactly why DT's access testing methods are not a valid test of 
> head seeks.  The drive with only 4 heads will need to travel a lot 
> more distance seeking in DT's test.  Any access test needs to 
> insure that an actual head seek across a specific number of tracks 
> has incurred.

The idea is not to measure only the speed of head movement; the
idea is to measure the speed of access to data.  That speed is
affected by head movement speed, rotation speed, and geometry.
If two users load the same data on two drives, the user whose drive
has more heads, larger tracks, etc. will, other things being equal,
have better performance.  In other words, "the bigger drive [having]
access to [more] data without performing any head seeks" gives
it a performance advantage.

> The current  access time test reads a block and then another block one
> Megabyte into the drive, therefore adding to this result the time it takes 
> to read the two blocks it is seeking to.  Steve believes "this 
> [confounding] is not significant".  But in fact it could very 
> well be, considering the interleave of the drive.  The heads may 
> have found the proper track, but the block was not in position 
> under the heads.  This will add the time it takes to spin the 
> platter(s) to get the block under the heads.  So the problem 
> described here is testing seeks across so many Megabytes.  Without 
> knowledge of the drive's geometry, this testing method is not 
> useful and potentially misleading.

Rotational latency is part of access time.  DiskTimer II avoids
penalizing certain geometries by inserting a random time delay
between the single-block requests, so that the total latency will
approach the drive's average.  Unfortunately, as Ephraim Vishniak has
pointed out, I introduced a design error when I inserted
the random delays, namely synchronization of the requests with
the Mac's 60hz clock.  But if that were corrected, the access
time test would be a reasonably good benchmark for subsystems
not providing block cacheing.