Path: utzoo!utgpu!news-server.csri.toronto.edu!mailrus!uwm.edu!zaphod.mps.ohio-state.edu!samsung!cs.utexas.edu!sun-barr!olivea!apple!well!jjacobs
From: jjacobs@well.sf.ca.us (Jeffrey Jacobs)
Newsgroups: comp.software-eng
Subject: Re: Specification Tools and Code Testing
Message-ID: <19578@well.sf.ca.us>
Date: 16 Aug 90 19:39:38 GMT
References: <5456@stpstn.UUCP> <1990Aug12.134735.22528@cbnewsm.att.com> <1990Aug13.140347.9441@nixtdc.uucp>
Distribution: usa
Organization: Whole Earth 'Lectronic Link, Sausalito, CA
Lines: 52


> testing a big system is *hard*

It shouldn't be "hard", but it is a major cost factor.

Some general comments on testing large systems:

1.  There are various stages and types of testing, ranging from "unit testing"
to systems testing.  Testing and the resources required must be part of
the initial planning.  Designing the system for testability is very
important.

2.  During the requirements (or product) definition phase, each requirement
(or feature) should have an associated means of testing it defined
or suggested.  (Subject to latter revision, of course).

3.  Most phases of testing should be conducted by a separate test
organization.
This test organization should include some senior people; one of the most
common mistakes is assuming that testing (and Q/A) is something that only
requires a BS/CS and a couple of years of experience.

4.  An effective configuration managment and reporting system is critical
to the success of a large project.  It must be automated, and should
provide *support* to all groups concerned.

5.  Automated testing tools are a necessity; manual inputs should be
eliminted wherever possible.  We have built numerous tools for testing
large systems over the years.  We have built testing languages which
generate test code; however, I can't think of any case where a
"specification language" would have been much use (see above).  The
amount of work required to write a rigourous specification would
indeed be roughly equal to that required to write the deliverable code.
It would be more productive for such a specification language (SL) to generate
the end code instead of "test" code.

6.  Despite all the emphasison testing, the single most effective
technique for eliminating and preventing defects is "inspection".

There is an excellent book from Auerbach called "A Standard for
Application Testing" or something similar (I don't have it handy
at the moment); although oriented toward more classic DP COBOL
shops, it contains a thorough discussion of testing which is
applicable to any type of project.  Also, look for articles by
William Howden in IEEE Transaction on Software Engineering circa
1986-1988.  (I'll try to post a more complete bibliography in
a few days).

Jeffrey M. Jacobs
ConsArt Systems Inc, Technology & Management Consulting
P.O. Box 3016, Manhattan Beach, CA 90266
voice: (213)376-3802, E-Mail: 76702.456@COMPUSERVE.COM