Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!asuvax!ncar!elroy.jpl.nasa.gov!usc!wuarchive!m.cs.uiuc.edu!marick
From: marick@m.cs.uiuc.edu (Brian Marick)
Newsgroups: comp.software-eng
Subject: Re: Personal growth and software engineering!
Message-ID: <1991Apr12.134908.20245@m.cs.uiuc.edu>
Date: 12 Apr 91 13:49:08 GMT
References: <9233@castle.ed.ac.uk> <1991Mar25.164133.29674@unislc.uucp> 	<JGAUTIER.91Apr3131954@vangogh.ads.com> <549@tivoli.UUCP> <JGAUTIER.91Apr8184934@vangogh.ads.com>
Distribution: comp.software-eng
Organization: University of Illinois, Dept. of Comp. Sci., Urbana, IL
Lines: 43

jgautier@vangogh.ads.com (Jorge Gautier) writes:

>... By effectively I mean that the metric should give you a
>nonambiguous message of what needs to be done given your intentions.
>For example, a coverage measuring tool measures the amount of code
>exercised by tests.  If you want 90% coverage and you are getting 30%,
>the message is clear: write more and/or better tests.  Testing is a
>well formalized process and can be meaningfully measured in this way.

Bad analogy.  It is by no means clear what 90% coverage means in terms
of the measure that really counts: what proportion of bugs are left in
the program.  Or, wait: what *really* counts is how many *failures*
the average customer sees.  No, that's not quite it: following Deming
and Taguchi, what counts is also the variability from customer to
customer.

And what's the difference between 90% coverage, 80% coverage, and 100%
coverage?  Does it depend on the program?  

Suppose you have two test suites.  The first was developed from the
specification (black box), and achieved 90% coverage.  The second was
developed from the specification, achieved 50% coverage, and then was
augmented to reach 90% coverage.  Which is a better test suite?  Which
is more likely to detect bugs due to missing code?  Which signals a
better test process?


I don't mean to say that measuring test coverage is useless, because I
happen to find it very useful -- when interpreted in context.  The
best numbers focus your attention, say, "Hey!  Look over here; there's
something odd going on."  Without them, problem solving is next to
impossible; with them, it's merely difficult.  (Shewhart control
charts, used in manufacturing quality control, are a perfect example
of what I mean.)

Now, it's debatable whether complexity metrics focus our attention in
the right place.  But it's not fair to complain because they require
interpretation.


Brian Marick
Motorola @ University of Illinois
marick@cs.uiuc.edu, uiucdcs!marick