Path: utzoo!attcan!uunet!husc6!bloom-beacon!apple!mjohnson From: mjohnson@Apple.COM (Mark Johnson) Newsgroups: comp.sys.mac Subject: Re: Reekes on Disktimer (LONG) Message-ID: <20306@apple.Apple.COM> Date: 9 Nov 88 23:55:32 GMT References: <698@mouse.UUCP> <7595@well.UUCP> Organization: Apple Computer Inc, Cupertino, CA Lines: 188 I'm requesting this message to be posted to this net. I am not a member. -JR My original comments that I've heard about being published, once again, contains some inaccuracies. My intent at that time, some 18 months ago, was to stir up some controversy and present the 'con' view point regarding the results being published in popular magazines and on local BBSs. That may or may not have been a good idea, since I'm still trying to explain myself. At any rate, I also recently read S. Brecher's comments dated Nov. 8 on yet 'another network' rebutting the old feud once again. Steve's corrections to my statements are accurate. The old discussion that Steve and I had from times of old gave me more to think about. His rebuttal corrected some of my inaccuracies. I've since learned of the errors in that original article. This original article taken out of context cannot be appreciated unless the entire discussion was also presented. I wish to withdraw the original article due to mistakes it contained, and after my re- examining of the Disk Timer source code. Although, this has not changed my opinion of Disk Timer. I do have more than 'no idea' of what I'm talking about, but my misunderstandings republished recently may lead one to believe otherwise. Then about 12 months ago, another fellow decided to revise Disk Timer and bring forth a new version. I had a long series of discussions on yet 'another network'. The following is my comments made at that time, which I still believe true. [I made a few new edits, shown in brackets] I still stand my following statements made during the second discussion of the continuing Disk Timer "Pro vs. Con" series. ________________________________________ Date: Mon Sep 8, 1987 10:45 am EST I wish to express my sentiments regarding the resent proposal of revising DiskTimer. Before I make my comments, I'll add that I have great respect for Steve Brecher's work. My first reaction was, "Why do we need another DiskTimer?" The earlier versions caused more misunderstandings and misrepresentations than anything else. Benchmarks are not to be considered lightly. As an example, consider how BYTE magazine can't produce a valid set of tests for the IBM vs. Macintosh benchmarks. There are issues that must be addressed if we are going to be publishing the results of any benchmark test. It must consider the methods of how a system operates in real life. So, if DiskTimer is to be considered at all then it needs to test a drive under "real world" situations. This means it is going to require standard file I/O (e.g. using the Mac's File Manager). Without the use of the test files, the results are too far removed from reality. In other words, the purpose of a drive is to read and write files. Any benchmark of a drive's performance must test its ability to do this, which is done via the File Manager. Steve has defended the non-use of the File Manager by stating, "I developed DiskTimer II precisely to avoid using File Manager calls, i.e., to provide a benchmark that could be run without initializing the disk". This is ridiculous statement in my opinion. How many people can use a hard disk that hasn't been initialized? [text deleted -JR] I understand that DiskTimer is actually testing the driver's lowest level of the interface, so then maybe it should be renamed to "DriverTimer". I also understand that DiskTimer is not intended to be a test of actual system performance. But then, of what use are the results in that case? The fact is that DiskTimer is going to be scooped up by everyone with a hard disk and the results are going to be publicized at every user level. (Witness the 'special' version of DiskTimer that SuperMac Technologies has produced. They even claim that their drives are three times faster than everyone else's based on these results.) Some magazine editors have stated that they will no longer publish performance tests based on DiskTimer results. (refer to Ric Ford's recent article of drives for the Mac SE printed in MacWeek) MacWeek has found that the timings of DiskTimer II have no correlation with real world performance, and therefore the results have little or no value in their comparisons. The following list is requirements that the new DiskTimer (DT) must include, if it is to be rewritten at all. 1. There needs to be a differentiation in the test between multiple vs. single block transfers. A large multiple block reads will always perform best on a drive formatted with a 1:1 interleave. In the real world of Macintosh, most drives are interleaved. This causes the multiple block read/write tests to be weighted towards any drive formatted with no interleave and will handicap a drive that is interleaved. [as we move to higher capacity drives, the 1:1 interleave is more common since these drives often do not support interleaving -JR] 2. In the real world of Macintosh, all file I/O requests are given by the File Manager. If DT is to ignore reading and writing files stored on the disk, then its results are invalid [in my opinion -JR]. Testing the performance of a drive requires the testing of the drive under typical conditions. Since the biggest percentage of the over-head in file I/O is caused by the File Manager, then DT III must use it. DT should create, read, and write files. Yes, DT results will then be subjected to a drive's fragmentation. But then again, the FileManager will try to use a contiguous space on the drive, which is another reason to use the FileManager. This is the typical situation. 3. Consideration must be given to the drive's interleave, number of read/write heads, number of bytes per block, number of blocks per track, and number of tracks per cylinders. Without this information, no test will be fair to drives with dissimilar geometries. 4. DT needs to address drive configurations that can invalidate its results, such as a drive with integral caching. I feel there is an inconsistency in the philosophy of DT. Steve says, "the results in such cases will almost certainly be so low (so 'good') as to immediately identify them as invalid." But DT is being distributed to the public at large, which does not have the technical knowledge to determine when the results are too 'good' to be true. The typical user of DT is Joe Blow trying to beat the guy next door. [this is the reason I take an issue with Disk Timer -JR] 5. I recommend a larger than 24K file to be used in the test. The majority of files on a Macintosh system are considerably larger than this. 6. During the test, a warning should be presented to the user telling him that interrupting it may cause loss of data, and possibly the entire drive. Although this is true with any and all programs that write to a drive, DT is unique. It quietly sits there performing its test. If a less than knowledgeable user gets bored with the test, he may be tempted to abort it by resetting or powering off the computer. 7a. I've saved the criticism of the access time test for last, because it has the biggest problems. With DT's access test, a slower seeking drive may test better than a faster seeking drive. How? The answer is in the drive's geometry. As an example; if a drive with 15 read/write heads is matched against a drive with only 4 heads, then the bigger drive has access to nearly four times as much data without performing any head seeks! This is exactly why DT's access testing methods are not a valid test of head seeks. The drive with only 4 heads will need to travel a lot more distance seeking in DT's test. Any access test needs to insure that an actual head seek across a specific number of tracks has incurred. If DT III is going to report access times, then it better make certain that the heads have actually moved a certain number of tracks. I've been able to alter the results of the access test by changing the interleave of the drive. Obviously, the drive's access time could not have actually changed by this. Also, consider the SCSI Seek command in the access test. 7b. Steve defends his access time results by stating, "Do not confuse DiskTimer II's access time test with an attempt to simulate hardware vendors' average access time statistic." Then I beg, please, do not call it an "access time" test. The current access time test reads a block and then another block one Megabyte into the drive, therefore adding to this result the time it takes to read the two blocks it is seeking to. Steve believes "this confoundment (sic) is not significant". But in fact it could very well be, considering the interleave of the drive. The heads may have found the proper track, but the block was not in position under the heads. This will add the time it takes to spin the platter(s) to get the block under the heads. So the problem described here is testing seeks across so many Megabytes. Without knowledge of the drive's geometry, this testing method is not useful and potentially misleading. In conclusion, the reason I'm so adamant to the proposed revision to DT is that it has been exploited by far too many people. It has established its name as 'the' test to use. It is the de-facto standard, but it is a loaded weapon and in the wrong hands it is dangerous. Mis-information is worse than no information. I feel if DT is to be re-written then it needs to be re-philosophized. As Ephraim Vishniac has pointed out, DT has been exploited by certain vendors. Certainly everyone will agree, the purpose of a drive is to store files. With this in mind, any performance results of a drive should test the drive's ability to transfer files to and from a Macintosh. DiskTimer in the past has only tested the driver's interface. This is too far removed from the real world of hard disk usage. My final words regarding this 'benchmark war' is this. People interested in the evaluation of SCSI drives on the Macintosh need a copy of SCSI Evaluator. Send $20 to "Digital Microware, PO Box 3527, Mission Viejo, CA 92690. In return you will receive a 60+ page manual discussing benchmarks for SCSI drives. Before continuing any further discussion of SCSI test programs, everyone concerned needs to read the information in the SCSI Evaluator manual. That is all. -JR