Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!uunet!dg-rtp!farmhand!cole From: cole@farmhand.rtp.dg.com (Bill Cole) Newsgroups: comp.software-eng Subject: Re: Personal growth and software engineering! Message-ID: <1991Apr15.202323.12024@dg-rtp.dg.com> Date: 15 Apr 91 20:23:23 GMT References: <9233@castle.ed.ac.uk> <1991Mar25.164133.29674@unislc.uucp> <549@tivoli.UUCP> <1991Apr12.134908.20245@m.cs.uiuc.edu> <1991Apr12.183600.15528@dg-rtp.dg.com> <1991Apr15.141457. Sender: cole@farmhand (Bill Cole) Distribution: comp.software-eng Organization: Data General Corporation, Research Triangle Park, NC Lines: 48 |> I was referring to code coverage. For example, 100% branch coverage |> means that running the program against your test suite forces every |> branch in the program to be taken in both the TRUE and FALSE |> directions. 100% statement coverage means that every statement has |> been executed. 100% loop coverage is usually taken to mean that every |> loop has executed 0, 1, or many times. |> |> Code coverage is a measure of how thoroughly your test suite exercises |> the actual text of the program. The problem with coverage is that it |> does NOT measure how thoroughly you've explored the program's |> specification. That is, it doesn't directly address your worry: |> |> >... that we haven't covered some specific, but minor, situation |> >that will eventually bite a user in the ear -- or lower. |> |> In general, there's no way to measure specification coverage, so |> people resort to using code coverage as an approximation. |> |> The next step is then to think a bit about why you missed that |> specification area in the first place and how you can avoid missing |> ones like it next time, when you might not have a coverage clue. See |> also Scott McGregor's recent note in this newsgroup. |> I don't think that covering every branch, loop or case statement will assure you of 100% coverage. I'll admit to being unsure on this one. It seems to me that, by the time I get to the point where each bit of functionality is pretty sound on its own, it's the way my tests wander through the code paths that make the coverage 100%. For instance: You've got a working bit of software that rarely fails (say if takes 3 weeks of running flat out to reproduce the problem) and you finally discover the bug was due to 'timing' or 'circumstances you couldn't foresee' -- perhaps a failure condition that the software didn't deal with particularly well. Experience -- I won't tell you how much -- tells me that most really evil bugs in released software are either in timing situations or they occur at the interface between pieces of functionality. We try to learn from our 'mistakes' -- bugs that we could have found but didn't. We have a set of tests that used to run for two weeks on two machines; that test suite has been enhanced so that it now runs for three weeks on seven machines. The good thing, though, is that the bugs that get by us now are much more obscure. Thanks for the reply. I represent these opinions to be my own, /Bill