Path: utzoo!attcan!uunet!husc6!think!ephraim From: ephraim@think.COM (Ephraim Vishniac) Newsgroups: comp.sys.mac Subject: Re: Reeks on Disktimer (LONG) Message-ID: <30643@think.UUCP> Date: 8 Nov 88 15:56:31 GMT References: <698@mouse.UUCP> Sender: news@think.UUCP Reply-To: ephraim@vidar.think.com.UUCP (Ephraim Vishniac) Organization: Thinking Machines Corporation, Cambridge MA, USA Lines: 220 Summary for anybody who cares: Jim Reekes presents many arguments against DiskTimer, some good, many bad, some really dumb. Larry Deleski endorses them all, but I don't. In article <698@mouse.UUCP> lad@mouse.UUCP (Larry Deleski) writes: >This is a file which I downloaded from the MouseHole BBS about a year ago. >At that time, Jim Reekes wrote an article on Disktimer, our beloved disk >performance benchmark, and essentially exposed Disktimer for was it was >(and still is), a fraud. >Ephraim Vishniac would have us beleive that Reekes is a lunatic. Ephraim >exhausted many hours at the keyboard disputing that article. I, for one, >beleive what I read in that article. A large portion of what Reekes wrote >is fact. [and the rest...] A key point here is that I think Reekes is loony not for disliking DiskTimer, but for the stupid arguments he presents against it. >I would like to take this opportunity to point out an error in something I >said earlier. I stated that Disktimer did not use the Device Manager. It >does, but not in the same way a typical SCSI driver does. What Disktimer >fails to do is use the FileManager, something every driver has to do. Still wrong. No driver uses the File Manager. The File Manager uses the drivers. Even Reekes says that! >Please read the following and decide for yourself. Reekes *does* know >what he's talking about. Let the readers decide... > DISKTIMER II vs. THE REAL WORLD >INTERLEAVES >The ONLY reason to interleave blocks on a disk is to compensate for >an interface that cannot handle the full data transfer rate of the >disk. A given disk drive has a native transfer rate that depends >onthe bit density, data coding and spindle speed of the drive. If >the interface of the computer is too slow for the drive's data rate, >then the drive's controller chokes on reads (from disk) and starves >the disk on writes (to disk). When this happens, the transfer must be >suspended until thecomputer catches up. This kind of error is called >a data late error. This is true, but becomes irrelevant to our discussion when the disk controller has a local cache which exceeds the size of a disk track. Then the crucial interface is between the disk and the controller, not between the disk subsystem (disk + controller) and the host. >Furthermore, 1:1 interleaving is disadvantageous due to the fact that the >Macintosh only makes single-block requests. Since this premise is false, all of the discussion that followed it is pointless. The Mac does make multi-block requests; not to do so would be incredibly stupid. >When I sit in front of a Macintosh Plus with a Dataframe XP20 or a >CMS S20, I can not observeany performance differences. Both drives >appear to run my applications at the same rate, eventhough DiskTimer >says that the XP20 is 5.48 times faster than my CMS S20. This is hardly surprising. As I mentioned in an earlier posting, data transfer rate is highly overrated as a performance measure. When running most applications, seek time is much more important. Data transfer rate becomes most visible in activities such as copying files. >One should not put much faith into interleave factors alone. One should >understand how they affectthe user. But I would also like to point out that >the FileManager of the Macintosh will only requestsingle block >transfers. Again, this is simply wrong, and all the discussion based on it is pointless. >DISKTIMER RESULTS >Worst of all, DiskTimer usesthe Macintosh's internal SystemTicks to >calculate its resulting times. SystemTicks are in1/60th of a second >and DiskTimer is attempting to measure times in 1/1000th of a second! >(ie: Atypical 20MB drive's access time is 65ms and transfer rates are >greater than 1MB/sec.) I very nearly agree with Jim here. These are real problems, but they can be overcome statistically. DiskTimer attempts to overcome them by repeated meaurements to gain a large sample. Brecher's error was that he starts each measurement by synchronizing with the system clock, so that (assuming a random distribution of results) he has an average underestimate of half-a-tick per measurement. For more accurate results, he should deliberately randomize his timing with respect to the clock so that over- and under-estimates are equally likely. When Brecher first came out with DiskBench (DiskTimer's predecessor) I made this same point and even wrote the code for him. He disagreed, evidently, as the change didn't survive into later versions. >One very important issue that is not even being considered in >DiskTimer results is whether or notthe manufacturer's SCSI drivers >are performing any verification of the data that was transferred. True, but how can it? If someone is silly enough to buy a disk "by the numbers" without considering quality, support, reliability, and the like, then he deserves what he gets. This point is basically irrelevant. >Most importantly, DiskTimer should never be run on a disk containing >user data. This program firstreads a block of your data, holds it in >RAM, performs its test on that block, and finally writes your data >back to the drive. It continues along with this sequence >block-by-block. If you were to reset,power off, or get the bomb >dialog during a DiskTimer test, then chances are you've lost data. Actually, DiskTimer's test consists of reading and writing the data that was already on the drive - it never overwrites the user data with anything else. So, a system failure during a write could corrupt your disk, but this is no more likely than with any other application. >DISKTIMER SEEK TEST This is prety much true. DiskTimer does an awful job with the seek test. >FACTS ABOUT SYSTEMTICKS >Disregarding any software limitations for the moment, theMacintosh >hardware can not possibly keep up the a 1:1 interleave from the >typical hard disk operating at greater than 1MB/second. The >theoretical limit is about a 3:1 interleave, as Apple has >recommended. The distinction completely lost here is between "keeping up with" and "gaining some benefit from." First, the Mac Plus can keep up with a non-cached drive (ST225N) at 2:1 interleave. Since it does this reliably, there must be some margin. Now, think about a cached drive running at 1:1. The controller can read a full track in one pass. The Mac can't read the data in that same time, but it can read it in less than two rotations. So, it gets an effective performance that's intermediate between what you see at 2:1 and what you would hope for at 1:1. Let's go back to the purpose of interleaving: matching performance. When there's no intermediate buffer (or very little), this means matching performance with the host system. When there's a sizable intermediate buffer, it means matching performance with the controller. Many sophisticated controllers don't even support variable interleaving because there's no point. The controller's internal cache effectively hides the interleaving from the host system. [The other hazard of mentioning a specific interleaving is that it implies different transfer rates for different disks. Apple's recommendation related to MFM formatted disks, which were dominant at the time. RLL formatted disks are another story.] >Here is a quote from Inside Macintosh (Apple's bible for the >Macintosh); > WARNING: Don't rely on the tick count being exact; it will usually > be accurate to within one tick, but may be off by more than that. The context, which Reekes thoughtfully omitted, states that "TickCount returns the current number of ticks since the system last started up." >A SystemTick is 1/60th of a second. A hard disk rotates at 3600 RPM >or 60 times per second. Therefore, a tick count is worth one rotation >of a hard disk >ERROR FACTORS >Tick count error is +/-1 or more. Timing a transfer of 13k, or one >rotation, with SystemTickscould introduce a error of 100% or more. >Because the number of ticks could be 0S, R1 or more! A tick is >approximately 16.7ms while some hard disks can seek track to track in >4ms. A drive could seek across 4 or more tracks before TickCount >changed. As an analogy, using ticks tomeasure a 32k read/write, >would be like using a car's odometer to measure lengths of feet. This is exactly why DiskTimer does 100 repetitions of each test. The smallest unit on my car's odometer is .1 mile (528 feet). If I drive some short distance once, I'll put little faith in what the odometer says. If I drive it a hundred times and divide the odometer reading by 100, I'll feel pretty good about the accuracy. Still, Brecher's mistake of synching to the tick count does introduce a systematic error which is easily visible on a Mac II. >CONCLUSION >To summarize, all DiskTimer results have been invalidated because of; > 1. Multi-block requests The premise was false, you'll remember. > 2. Favors a drive with 1:1 interleave If it has a significant cache. Let's not confuse different problems. > 3. Ignores the FileManager calls Irrelevant. > 4. Does not perform a true seek test True. > 5. Uses SystemTicks for calculating timing data It isn't using them that's bad, but the way they're used. So, I'd give Reekes 1.5 points out of 5, or 30%. With all its problems, DiskTimer is more accurate than that! Ephraim Vishniac ephraim@think.com Thinking Machines Corporation / 245 First Street / Cambridge, MA 02142-1214 On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?"