Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!swrinde!elroy.jpl.nasa.gov!decwrl!ads.com!saturn!jgautier From: jgautier@vangogh.ads.com (Jorge Gautier) Newsgroups: comp.software-eng Subject: Re: Personal growth and software engineering! Message-ID: Date: 12 Apr 91 16:46:24 GMT References: <9233@castle.ed.ac.uk> <1991Mar25.164133.29674@unislc.uucp> <549@tivoli.UUCP> <1991Apr12.134908.20245@m.cs.uiuc.edu> Sender: usenet@ads.com (USENET News) Distribution: comp.software-eng Organization: Advanced Decision Systems, Mountain View, CA 94043, +1 (415) 960-7300 Lines: 37 In-Reply-To: marick@m.cs.uiuc.edu's message of Fri, 12 Apr 91 13:49:08 GMT In article <1991Apr12.134908.20245@m.cs.uiuc.edu> marick@m.cs.uiuc.edu (Brian Marick) writes: > >... By effectively I mean that the metric should give you a > >nonambiguous message of what needs to be done given your intentions. > >For example, a coverage measuring tool measures the amount of code > >exercised by tests. If you want 90% coverage and you are getting 30%, > >the message is clear: write more and/or better tests. Testing is a > >well formalized process and can be meaningfully measured in this way. > Bad analogy. It is by no means clear what 90% coverage means in terms > of the measure that really counts: what proportion of bugs are left in > the program. Or, wait: what *really* counts is how many *failures* > the average customer sees. No, that's not quite it: following Deming > and Taguchi, what counts is also the variability from customer to > customer. > And what's the difference between 90% coverage, 80% coverage, and 100% > coverage? Does it depend on the program? Percent coverage does not tell you how good your test suite is or how well you have tested your code. It measures exactly what it says: how much of your code was exercised during execution. Any other interpretation is a likely self-deception. I used this metric as an example of one of the less bogus metrics. Actually, I have found % coverage less useful in practice (except for giving numbers to the boss ;-) than the number of times that specific branches were taken. By examining the coverage for specific branches you can sometimes determine whether you have tested important situations. Overall, coverage metrics are some of the least bogus ones around, but I'm sure that they can also be misused. As for measuring the number of bugs left in a program, it seems impossible without extensive use of formal and effective behavioral specifications in the development process. -- Jorge A. Gautier| "The enemy is at the gate. And the enemy is the human mind jgautier@ads.com| itself--or lack of it--on this planet." -General Boy DISCLAIMER: All statements in this message are false.