Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!wuarchive!wugate!uunet!munnari.oz.au!sharam From: sharam@munnari.oz.au (Sharam Hekmatpour) Newsgroups: comp.software-eng Subject: The Errors of TEX Keywords: testing debugging structured-programming OOD Message-ID: <2402@munnari.oz.au> Date: 12 Oct 89 07:30:44 GMT Sender: news@cs.mu.oz.au Lines: 90 A number of people have commented about D. Knuth's "The Errors of TEX". I read the paper and I think that he should be praised for such extreme honesty. I find some of his suggestions very useful, but there also ones which I find inappropriate if not shocking. This one more than any other: "I found that structured programming greatly increased my confidence in the correctness of the code, while the code still existed on paper. Therefore I could wait until the whole program was written, before trying to debug any of it. This saved a lot of time, because I did not have to prepare 'dummy' versions of non-existent modules while testing modules that were already written. I could test everything in its final environment..." -- D. Knuth, The Errors of TEX, Software P&E, Vol 19(7). This is the worst testing technique I have ever heard of. Recently I finished writing a 40,000 line event-driven C program. I used object-oriented design + structured programming. I agree that these methods increase one's confidence in the 'correctness' of a program (and I should also point out that OOD does much more in this respect than SP), but to suggest that these methods justify the use of the Big Bang testing approach is naive. For my own program, I tested most modules first in isolation and then during incremental integration. Of the 2 years that I spent on this program about 6 months of it was solid, systematic testing. Had I used Knuth's approach I could have perhaps cut down this time to 3 months, but I doubt very much if the whole thing would have worked in the end. That Knuth's testing approach worked for TEX may be because of other reasons: the nature of the application, its size, its complexity ??? But let's not forget that he decided to scrap the original version and rewrite it from scratch. I wonder if Big Bang testing had something to do with this. I must admit that I have done Big Bang testing in the past. Perhaps that's why I'm so much against it. It invariably lead to rewriting everything from scratch. A Big Bang scenario is usually something like this: [1] Design and code with care and get more and more confident. [2] Defer testing until everything can be put together. "Why should I waste 30 minutes writing stubs and testing 1 module when I can test 800 modules in one go in 5 minutes?" [3] Get real excited when the system is complete. "This is gonna work first time." [4] "Shit! It doesn't even compile." [5] "At last! Compiled. Now it will run." [6] "Damn! What could I have done wrong? I think I need a debugger cause I have no idea what's causing these segmentation faults." In the many weeks/months that the programmer spends wrestling with the bugs, confidence goes downhill and impatience reaches its peak. He would try anything just to see it working. His impatience forces him to debug in front of a terminal, redesign and recode many of the faulty pieces there and then. He has to see it working. Tomorrow won't do. It has to be tonight. But one fix leads to another and another and another... [7] "At last! It runs." But not for long. New bugs emerge everyday and it becomes increasingly more difficult to patch them. What started as a good SP exercise is now a beast. The design has become so unstable that it scares anyone to touch it. [8] "No, this won't do. I think now I have a pretty good idea how I should have written it in the first place. Let's rewrite the whole thing." The worst aspect of Big Bang is that it gives you no time to reconsider your design. When it comes to testing, a whole army of design faults hit you dead. You'll have no idea where to begin. The bugs are so many and so interdependent that you don't know what's causing what. From this point on it's all hacking. This is where false confidence gets you. Systematic testing, however, gives you justified confidence. When it comes to integration, at least you have a good idea how reliable each component is. There will be less errors and these can be quickly located and corrected. To sum up my argument: no matter how good your design and coding techniques, NEVER have confidence in a component until you have fully tested it. Good design and good testing go hand-in-hand. Trying to cut corners with one damages the other. +--- | Sharam Hekmatpour | Melbourne University ---+