Path: utzoo!utgpu!attcan!uunet!eplrx7!mouse!lad From: lad@mouse.UUCP (Larry Deleski) Newsgroups: comp.sys.mac Subject: Reeks on Disktimer (LONG) Message-ID: <698@mouse.UUCP> Date: 5 Nov 88 02:37:53 GMT Organization: The MouseTrap 3B1 (pnet71) Lines: 344 This is a file which I downloaded from the MouseHole BBS about a year ago. At that time, Jim Reekes wrote an article on Disktimer, our beloved disk performance benchmark, and essentially exposed Disktimer for was it was (and still is), a fraud. Ephraim Vishniac would have us beleive that Reekes is a lunatic. Ephraim exhausted many hours at the keyboard disputing that article. I, for one, beleive what I read in that article. A large portion of what Reekes wrote is fact. I would like to take this opportunity to point out an error in something I said earlier. I stated that Disktimer did not use the Device Manager. It does, but not in the same way a typical SCSI driver does. What Disktimer fails to do is use the FileManager, something every driver has to do. Please read the following and decide for yourself. Reekes *does* know what he's talking about. ------------------------ DISKTIMER II vs. THE REAL WORLD DiskTimer II has been mistakenly used as the de facto standard for hard drive performance. Before I attempt to discredit any results of S. Brecher's program, I'd like to point out that I don't blame him. He is a good programmer, but I think there are some serious flaws in his benchmarkcalculations. Normally I would let this utility fade off into the sunset, but a certain hard diskvendor has taken DiskTimer and exploited it. Today, nearly every hard disk user is aware ofDiskTimer results for nearly every given drive. Brecher disclaims any similarities to realworld performance, but then what do DiskTimer results really mean? Well, they aremeaningless. Let me try to explain what disk interleaves and DiskTimer results really mean in thereal world. INTERLEAVES The ONLY reason to interleave blocks on a disk is to compensate for an interface that cannot handlethe full data transfer rate of the disk. A given disk drive has a native transfer rate that depends onthe bit density, data coding and spindle speed of the drive. If the interface of the computer is too slow for the drive's data rate, then the drive's controller chokes on reads (from disk) and starves the disk on writes (to disk). When this happens, the transfer must be suspended until thecomputer catches up. This kind of error is called a data late error. Suppose the computer was almost as fast as the disk. After the first block is read from the disk,the computer will be a bit behind in getting the data into memory. But come hell or high water,here comes the second block on the hard disk. Well, the computer just isn't ready yet ... DATA LATE ERROR ... and now we have to wait for the disk to spin completely around until the second block isagain under the drive's head. By this time the computer has been waiting for the second block. Since there are normally 26 blocks on each track, this means for every block read the computerwill have to wait for 26 more blocks to pass!!! Now THIS IS IMPORTANT!!! The block interleave is NOT a measure of performance! Just becausethe Dataframe XP use 1:1 interleave and the CMS S40 uses 6:1 does not mean the S40 is 6 timesslower. Actually, the native transfer rate of the S40 is much faster than the Dataframe XP. TheDataframe can get away with a 1:1 interleave because its transfer rate is slower than the S40. The Macintosh cannot keep up the high speed of the S40, so it is relegated to the substandardperformance of the Macintosh SCSI interface. Furthermore, 1:1 interleaving is disadvantageous due to the fact that the Macintosh only makessingle-block requests. In fact, an interleave of approximately 6:1 is optimal for handling suchrequests. Mr. Brecher himself tested this by modifying his DiskTimer (which issues multi-blockrequests) to issue single-block (or sector) requests. He ran this single-sector version ofDiskTimer, SSDiskTimer, as well as the original DiskTimer on the same drive at variousinterleaves. Below are the actual results that Mr. Brecher found. (lower times are better) INTERLEAVE CHART Interleave SSDiskTimer DiskTimer 1:1 6801 507 2:1 7206 812 3:1 7511 1116 4:1 7518 1523 5:1 6513 1827 6:1 2538 2233 7:1 2538 2537 8:1 2943 2943 As Steve says, This result clearly confirms that the about 6:1 estimate for optimal handling ofconsecutive single-block requests. It is also interesting to note that DiskTimer (withmultiple-block requests) favors a drive with a 1:1 interleave, whereas the SSDiskTimer (withsingle-block requests) demonstrates better performance at 6:1. Beside the fact the Brecher hadproven to himself that 6:1 interleave was best, he should have realized something else about hisprogram. Why do the DiskTimer results go from good to bad on the interleave scale and DiskTimer go bad to good? I'll explain this in a moment. Tom Reilly of SuperMac states; The Mac Plus is set up for a 3:1 disk interleave,... This is not accurate information. Maybe the Apple hard disk for the Macintosh Plus is set up with a 3:1interleave, but every drive manufacturer will need to determine their own interleave factor. The best interleave factor for a drive is not determine by the host computer alone. After a developerhas completely optimized his software/hardware interface, he can then test all possibleinterleaves until he achieves the desired performance. Different drive configurations will havedifferent optimal interleaves. Furthermore Tom states; The SCSI interface on the Plus requiredyou to do direct ROM calls to gain performance. Again, this is not accurate information. This isSuperMac's opinion, it is not my opinion and certainly not recommended by Apple. (The Macintosh developer guidelines clearly states never do direct ROM address calls.) I am reminded of the early Macintosh programmers that considered QuickDraw too slow. Instead ofconforming to Apple's guidelines, many decided to draw directly on the screen (ignoring windows)or jump to ROM addresses. This software blew up when Apple changed the ROMs. These programs went from high performance to no performance. We can expect Apple to continueto increase the performance and capabilities of the Macintosh ROMs, and those that continue to follow the developer's guidelines will reap the benefits without having to go back to the drawingboards to write another revision (while the customer waits). When I sit in front of a Macintosh Plus with a Dataframe XP20 or a CMS S20, I can not observeany performance differences. Both drives appear to run my applications at the same rate, eventhough DiskTimer says that the XP20 is 5.48 times faster than my CMS S20. Gary Eisenstien,Supermac's national sales manager, states the Dataframe XP is three timer faster than any otherSCSI drive available for the Macintosh. This is absolutly false! How did he ever come to believethis? He probably considers DiskTimer the final word for a drive's performance. One should not put much faith into interleave factors alone. One should understand how they affectthe user. But I would also like to point out that the FileManager of the Macintosh will only requestsingle block transfers. It is the FileManager that is responsible for all file I/O. The Macintosh will request one block, digest it, and maybe then request another block. The time the Macintoshtakes to digest a block is about the same time as it takes two blocks to pass underneath the drive's read/write heads. So, a drive with 3:1 interleave is closely in tune with the Macintosh's FileManager. Examine the chain of command from application to hard disk. Application>>FileManager>>DeviceManager>>DeviceDriver>>SCSIManager>>HDController>>HardDisk (NOTE: In some cases, such as Dataframes, the HDController is a completely separate interface board. CMS uses embedded SCSI drives, where this controller is integrated into the hard drive.) In a real world situation, an application will talk to the FileManager and request a file. TheFileManager needs to determine which blocks that the file occupies by passing a message throughthe entire chain of command to get the hard disk and read some directory informationone-block-at-a-time. Once the FileManager has found the blocks it needs, it then requests a singleblock of that file, again through all the managers, to the disk. That's one block at a time folks. DiskTimer, on the other hand, talks directly to the DeviceManager with multi-block requests. Refer to the interleave chart and look at the difference between single and multiple block requests. Clearly the results of this chart demonstrate that DiskTimer favors a drive with 1:1 interleave,and handicaps a drive that isn't. I wish the other version, DiskTimer, had been released instead. Its results are more of a real world test because of its single-block accessing method. DISKTIMER RESULTS Steve Brecher has written a benchmark program, DiskTimer, for testing the performance ofSCSI hard disks on the Macintosh. Although I respect his work, I have to critize his methods oftesting. Determining which hard disk is the better performance drive based upon DiskTimerresults can be very misleading for many reasons. For one thing, DiskTimer uses device managercalls directly. It never issues calls through the FileManager, which is the method of allapplications in the real world. DiskTimer goes directly to the device manager and requestsmultiple blocks by their block number. Applications never know what block numbers they need. This test is unfair and does not behave as a real application would. Worst of all, DiskTimer usesthe Macintosh's internal SystemTicks to calculate its resulting times. SystemTicks are in1/60th of a second and DiskTimer is attempting to measure times in 1/1000th of a second! (ie: Atypical 20MB drive's access time is 65ms and transfer rates are greater than 1MB/sec.) One very important issue that is not even being considered in DiskTimer results is whether or notthe manufacturer's SCSI drivers are performing any verification of the data that was transferred. If someone were to ignore verifying their read/writes, then they could reduce their DiskTimerresults in half! Personally, I'd rather have my data verified and ignore DiskTimer readings. Most importantly, DiskTimer should never be run on a disk containing user data. This program firstreads a block of your data, holds it in RAM, performs its test on that block, and finally writes yourdata back to the drive. It continues along with this sequence block-by-block. If you were to reset,power off, or get the bomb dialog during a DiskTimer test, then chances are you've lost data. DISKTIMER SEEK TEST The seek test does not utilize the number of data surfaces on the hard disk, the number of blocksper surface, or their size. DiskTimer can not determine if an actual seek has occurred. DiskTimer's seek test, according to the source code that I have; performs a seek by reading a block,calculates 1MB worth of (512 byte) blocks, and reading that last block. This adds two block readsto the seek time, which has nothing to do with seeks. And even more importantly, this seek testdoes not consider rotational latency. If the disk platter were required to complete an additionalrotation just to bring that second block under the disk heads, then it would add additional time tothe seek results. Moreover, the DiskTimer seek test does not consider the method of bad block allocation. Some drives store its allocated blocks on every track, others store allocation blocks at one end of the disk. If DiskTimer were to test a seek to (or across) spared blocks, it would impose unfair times for that drive. In order to perform a valid seek test, it would require a SCSI Seek command, which positions the heads without attempting to read data. At best, the read time needs to be subtracted along with the actual incurred latency. If one examines the DiskTimer results of the DataFame 20, you'll see seek times ranging from 38 to 80. How can the same hard disk display such variances? The Dataframe SCSI driver software could not possible change the true seek performance of a hard disk. The results should remain constant, because that drive has an average access time of 65ms. The only explanation for seeing such a variance is because DiskTimer's testing methods are affecting the results. FACTS ABOUT SYSTEMTICKS Consider the Macintosh's hardware limitations. In the Macintosh Plus world, the maximumtransfer rate is 320K/second. Disregarding any software limitations for the moment, theMacintosh hardware can not possibly keep up the a 1:1 interleave from the typical hard diskoperating at greater than 1MB/second. The theoretical limit is about a 3:1 interleave, as Applehas recommended. Here is a quote from Inside Macintosh (Apple's bible for the Macintosh); WARNING: Don't rely on the tick count being exact; it will usually be accurate to within one tick, but may be off by more than that. A SystemTick is 1/60th of a second. A hard disk rotates at 3600 RPM or 60 times per second. Therefore, a tick count is worth one rotation of a hard disk HYPOTHESIS What if we tested the average drive set up with the following format. 1:1 interleave 512 bytes per blocks 26 blocks per track - or - 13k per track ERROR FACTORS Tick count error is +/-1 or more. Timing a transfer of 13k, or one rotation, with SystemTickscould introduce a error of 100% or more. Because the number of ticks could be 0S, R1 or more! A tick is approximately 16.7ms while some hard disks can seek track to track in 4ms. A drive could seek across 4 or more tracks before TickCount changed. As an analogy, using ticks tomeasure a 32k read/write, would be like using a car's odometer to measure lengths of feet. CONCLUSION I understand that DiskTimer disclaims any comparisons with real world performance, but considerthe inherent errors built into the results it does present us. A 32k file or 64 blocks wouldrequire 2.46 tracks (not cylinders) of storage on the average drive. To read or write a 32k filewould require at least 2.46 rotations. SystemTicks after 2.46 rotations could be 2S or R3, andeither of these figures may be off +/- 1 or more. DiskTimer will introduce a significant errorfor each 32k read/write test. DiskTimer does 200 read/write tests and averages them to thenearest 60th of a second. Then this figure is converted to the nearest 10th of a second. Thereforeany accuracy resulting with this program, in the real or un-real DiskTimer world, would bepurely coincidental. To summarize, all DiskTimer results have been invalidated because of; 1. Multi-block requests 2. Favors a drive with 1:1 interleave 3. Ignores the FileManager calls 4. Does not perform a true seek test 5. Uses SystemTicks for calculating timing data Q.E.D. Jim Reekes (Acknowledgements to Bob Denny's material on disk interleave, Steve Brecher's source code, and other documentation that I have quoted and used as my source material.) ----------------------------