Xref: utzoo comp.sys.mac.misc:10533 comp.windows.ms:11164
Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!swrinde!elroy.jpl.nasa.gov!decwrl!mcnc!rti!bcw
From: bcw@rti.rti.org (Bruce Wright)
Newsgroups: comp.sys.mac.misc,comp.windows.ms
Subject: Re: software testing
Summary: Good software development is hard
Keywords: software test simple - funny...
Message-ID: <1991Apr4.183135.2136@rti.rti.org>
Date: 4 Apr 91 18:31:35 GMT
References: <4482@orbit.cts.com> <12911@goofy.Apple.COM>
Followup-To: comp.windows.ms
Organization: Research Triangle Institute, RTP, NC
Lines: 56

In article <12911@goofy.Apple.COM>, jyen@Apple.COM (John Yen) writes:
> Rick Allard writes:
> > Yes, software is complicated, but it doesn't
> > vary statistically like an auto nor is it difficult to test.
> 
> '- nor is it difficult to test'?!  Have you ever tested non-trivial commercial
> software?  Think about all the equivalence classes of input and output,
> matrixed with possible hardware configurations, matrixed with interactions
> with other software, matrixed with different uses and users of the software.
> That's just off the top of my head.

I'd second the comment that software testing is a difficult subject.
I suspect that the original author meant that each individual test
is not too difficult (possibly true with a lot of software).  But as
you say you have to run lots of tests;  it's simply impossible to run
every possible input under every possible hardware configuration for
software of any useful degree of complexity.  The problem is trying
to choose which tests would be the most useful and performing them so
as to get the maximum amount of information from them - and this
selection process is hard.

There are, of course, a few celebrated bugs that one wonders "how can
that have ever made it out the door?"  And sometimes it is obvious
that an option was never really tested.  But there can be surprising
hardware differences that can foul up something that appeared to work
just fine on another piece of hardware - maybe trash in a memory
location, maybe a different CPU model handled exceptions differently,
or a video card that behaves in a nonstandard fashion, etc.  If you've 
never developed software used by a lot of sites, you may not realize 
how UNhomogeneous the computing environments are.

> There should be a full range of users across several
> hardware/software configurations in varying environments, with engineers
> watching, listening, but saying NOT 1 DAMN THING -- this is an experiment;
> interaction destroys validity.

Possibly a step like this is required for some types of software - I'm
not at all sure that it's a general requirement.  I _do_ think you need
to have a dialogue between the engineers and the users - in my experience
many users can't verbalize exactly what they want but "know it when they
see it".  Just watching user interaction won't necessarily get you there,
and may not lead to the sorts of brainstorming that you can get when the
user and the engineer can interact more closely.  The sort of experiment
you describe can be useful for quality control or to see if there are 
things that are "left out", at least if the software is highly interactive.

Customer testing ("beta testing") can also uncover unforseen problems,
and it is rarely practical to have an engineer watch every individual
doing beta testing ...

> this is a statement of incredulity that at least some users seem to believe
> good software is crunched out like so many cookies.  

A not uncommon belief I'm afraid ...

							Bruce C. Wright