Path: utzoo!attcan!utgpu!watmath!att!cbnewsj!edschulz
From: edschulz@cbnewsj.ATT.COM (edward.d.schulz)
Newsgroups: comp.software-eng
Subject: Re: The Errors of TEX
Summary: TeX experience is consistent with
	 Cleanroom Software Engineering!
Keywords: testing debugging structured-programming OOD Cleanroom
Message-ID: <1422@cbnewsj.ATT.COM>
Date: 13 Oct 89 20:29:59 GMT
References: <2402@munnari.oz.au>
Organization: AT&T Bell Laboratories
Lines: 77

In article <2402@munnari.oz.au>, sharam@munnari.oz.au (Sharam Hekmatpour) writes:
> 
> A number of people have commented about D. Knuth's "The Errors of TEX".
> I read the paper and I think that he should be praised for such extreme
> honesty. I find some of his suggestions very useful, but there also ones
> which I find inappropriate if not shocking. This one more than any other:
> 
>   "I found that structured programming greatly increased my confidence in the
>    correctness of the code, while the code still existed on paper. Therefore
>    I could wait until the whole program was written, before trying to debug any
>    of it. This saved a lot of time, because I did not have to prepare 'dummy'
>    versions of non-existent modules while testing modules that were already
>    written. I could test everything in its final environment..."
> 	-- D. Knuth, The Errors of TEX, Software P&E, Vol 19(7).
> 
> This is the worst testing technique I have ever heard of.

This is not shocking; it's consistent with the experiences of people who
use one of the best testing techniques I have ever heard of.

I suggest you read the papers on Cleanroom Software Engineering (some
references follow).  I spent a week this summer studying with Harlan
Mills and Richard Linger, who are at the leading edge of this
technology.  My organization has not yet used Cleanroom Engineering, but
I might be able to answer any questions beyond my notes here.

The Cleanroom process does not call for a "big-bang" test of the entire
system (more like a few medium bangs?), but does emphasize the
prevention of errors to begin with, rather than removing them later.
Using formal design methods and mathematics-based functional
verification by humans, people have demonstrated the ability to create
nearly defect-free software before any execution or debugging, less than
five defects per thousand lines of code.  Several significant software
systems at IBM Federal Systems Division (one example: 80 KLOC) have been
developed in the Cleanroom discipline, with no debugging before usage
testing and reliability certification.  Smaller student Cleanroom
projects have been successful at the Universities of Maryland,
Tennessee, and Florida.

> To sum up my argument: no matter how good your design and coding techniques,
> NEVER have confidence in a component until you have fully tested it.

"Testing can show the presence of bugs, not their absence." - E. W.
Dijkstra

"No matter how intelligently conceived, reliability evidence of coverage
testing is entirely anecdotal, not scientific." - Harlan Mills

The Cleanroom method employs statistically based independent testing of
each increment of the program, where an increment is typically 5K to 20K
new source lines. The interfailure times during such testing are fed
into a reliability model to track the reliability growth of the program
during development. You can never "fully test" any interesting piece of
software.

Here are some references for those who wish to learn more:

[1] H. D. Mills, M. Dyer, and R. C. Linger, "Cleanroom Software
    Engineering," IEEE Software, September 1987, pp. 19-25.

[2] R. W. Selby, V. R. Basili, and F. Terry Baker, "Cleanroom Software
    Development: An Empirical Evaluation," IEEE Transactions on Software
    Engineering, Vol. SE-13, No. 9, September 1987, pp. 1027-1037.

[3] Harlan D. Mills and J. H. Poore, "Bringing Software Under Statistical
    Quality Control," Quality Progress, November 1988, pp. 52-55.

[4] R. C. Linger and H. D. Mills, "A Case Study in Cleanroom Software
    Engineering: The IBM COBOL Structuring Facility," Proceedings of
    COMPSAC '88, IEEE Computer Society Press, 1988.

Does anyone else on the net have any experience, impressions, opinions,
etc. on this topic?  I'd like to hear some discussion...
-- 
--
Ed Schulz, AT&T, Room 2P276 200 Laurel Ave., Middletown, NJ 07748
+1 201 957 3899     e_d_schulz@att.com    or    eds@mtdcb.att.com