Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!usc!wuarchive!emory!mephisto!mcnc!rti!bcw
From: bcw@rti.rti.org (Bruce Wright)
Newsgroups: comp.windows.ms
Subject: Re: Microsoft Support (or lack of)
Summary: Not quite that simple ...
Message-ID: <4053@rtifs1.UUCP>
Date: 7 Sep 90 16:29:19 GMT
References: <1990Aug28.195832.1853@chinet.chi.il.us> <2684@dataio.Data-IO.COM>
Distribution: na
Organization: Research Triangle Institute, RTP, NC
Lines: 43

In article <2684@dataio.Data-IO.COM>, bright@Data-IO.COM (Walter Bright) writes:
> In article <1990Aug31.142549.21662@jarvis.csri.toronto.edu> west@turing.toronto.edu (Tom West) writes:
> <I've bought 6.0, have found one code generation bug 
> <through pure luck and am petrified that I am going to distribute another
> <code generation bug that I haven't found in a commercial product.  
> 
> You need a test suite for your product. If your livlihood depends on it,
> or lives do, you must test your product thoroughly, as you are responsible
> for it, not the tool vendors.

To some extent I agree with this sentiment (that you need a test
suite for a commercial product), but I think you are putting too
much confidence in what a test suite can find.  It isn't reasonable 
to think that a test suite can find all possible bugs - by definition
a test suite is finite, and many sorts of software can be subjected
to a countably infinite number of possible inputs.

You may also run into situations where the generation of the test
suite runs into the same sorts of tool-related bugs that the product
ran into;  in the limit you would have to have a complete test suite
for all the software and hardware that you were using.  This can be
very difficult to put together (_really_ good test suites are non-
trivial to generate).

Along this line, consider the number of problems that Intel had with
early versions of their 386 and 486 chips, and we can be sure that
they had done lots of testing of their chips before they shipped.  But
in the case of the 486, it wasn't until later that some of the problems
were found - when Compaq found some of their diagnostics didn't always
work properly!

I think the best approach is to attack the problem from many different
angles, including test suites, proofs, and field testing with users.
(Users can often think of things to try that none of the product or
test suite developers think of, so they never make it into the test
suite).  You'd certainly want to know if anyone _else_ had found that
one of the tools you were using was suspect, since (as noted above)
it can be difficult to verify that they work in all possible cases.

If you're trying to build a reliable product I think you'd want to
take anything you can get -

						Bruce C. Wright