Path: utzoo!utgpu!news-server.csri.toronto.edu!mailrus!cs.utexas.edu!samsung!crackers!jjmhome!smds!rh
From: rh@smds.UUCP (Richard Harter)
Newsgroups: comp.software-eng
Subject: Re: Specification Tools and Code Testing
Summary: Some further notes on testing
Message-ID: <163@smds.UUCP>
Date: 19 Aug 90 07:18:16 GMT
References: <5456@stpstn.UUCP> <1990Aug12.134735.22528@cbnewsm.att.com> <19578@well.sf.ca.us>
Distribution: usa
Organization: SMDS Inc., Concord, MA
Lines: 47

Here are just a few notes on testing in addition to Jeffrey M.
Jacobs execellent article on testing.

(1)	There are at least three rather general categories of testing:
specification testing, performance testing, and pathology testing.
Each category requires different kinds of testing techniques and scripts,
and each has different objectives.  Specification testing is the most
straightforward of these, at least in principle, since (given good
specs) one can simply check to see that the output of the software
matches the specs.  In practice the problem is that software specifications
are often determined ex post facto.  The major problem in performance
testing is simply to determine what it is that should be measured.

	The hardest type of testing is pathology testing -- i.e.
beat on the software in a variety of ways to reveal unexpected bugs.
This is an area where experience pays.  Essentially what you are trying
to find are errors that were not found in the development process.
Some of these are the usual logic errors -- fence post problems,
boundary problems, unchecked system error returns, etc.  However
the most perplexing and hardest to evoke are what I call mystery bugs.
Typically these come from storage overwrite problems.  They manifest
only under specific combinations of conditions and the resulting errors
appear later in the course of execution without apparent connection
to the original error.  Some languages, C and Fortran among them,
are particularly prone to these errors.

(2)	Good instrumentation in the software is very important.
My view is that instrumentation should be hard wired in so that the
delivered software is the same as the tested software.  Examples:
Every module contains a counter that counts the number of times
it is called.  The software contains a call history trace scheme.
Storage allocation is encapsulated with overwrite checking on
boundaries.  (I would very much like to see a discussion of software
instrumentation techniques.)  The software should be structured so
that the testing procedures have access to the internal instrumentation
data.

(3)	Testing procedures should be under configuration management.
They should also be tied to change control.  Ideally there should
be a test procedure associated with every functional change to the
software.

-- 
Richard Harter, Software Maintenance and Development Systems, Inc.
Net address: jjmhome!smds!rh Phone: 508-369-7398 
US Mail: SMDS Inc., PO Box 555, Concord MA 01742
This sentence no verb.  This sentence short.  This signature done.