Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!csd4.csd.uwm.edu!cs.utexas.edu!uunet!pilchuck!dataio!bright
From: bright@Data-IO.COM (Walter Bright)
Newsgroups: comp.lang.c
Subject: Re: ANSI Validation (And Validation in general).
Keywords: Testing
Message-ID: <2112@dataio.Data-IO.COM>
Date: 25 Aug 89 20:18:37 GMT
References: <20900002@inmet> <20500@sequent.UUCP>
Reply-To: bright@dataio.Data-IO.COM (Walter Bright)
Organization: Data I/O Corporation; Redmond, WA
Lines: 41

In article <20500@sequent.UUCP> paulr@crg3.UUCP (Paul Reger) writes:
>I was wondering what experiences anyone has had with test suites for C.
>What I would be interested in is responses from your experiences with
>testing a C compiler:
>1. What suites do you consider the best ones ??
>2. Are there automatic tools available which produce and evaluate
>random sequences of 'oddball' C expressions.
>3.  How successful are such automatic test case generators ???

Test suites come in 3 flavors:
1. Testing from the user's point of view. This means going through the manual
   and trying out all the combinations of each behavior described in the
   manual. I haven't seen the Plum Hall suite, but I assume this is the
   essence of it.
2. Testing that the optimizer is really functioning, that is, testing that
   specific code is generated for specific input (this is not the same
   as 1, which only specifies that the output code *behaves* correctly).
3. Testing every line of code in the compiler. Coverage testers are
   normally used to verify that every line of code was executed at least
   once by some function.

Unfortunately, only (1) is available as a product, and those general
test suites cannot hope to test compiler-specific features.

Another major failing in commercial test suites is lack of testing of
floating point and transcendental function accuracy. In their effort to
make the test portable, the details of floating point are not addressed.

I've also seen 'test suites' which consist of simply feeding millions of
lines of C code into the compiler. This is largely ineffective because:
1. The C code needs a test suite itself to verify that it compiled
   correctly, not just that it compiled.
2. Different people have different styles. A million lines of C written
   by Joe Programmer probably won't test the compiler any better than
   10,000 lines written by him, and any atypical things he does could be
   boiled down to <100 lines.

What I mainly rely on is that when I fix a bug, I add the bug report as
a test case to my test suite. Over the years, this has resulted in a
very comprehensive test suite for C and C++. (Please don't ask me for it,
I regard that suite as a major competitive advantage for me!)