Path: utzoo!utgpu!news-server.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!pt.cs.cmu.edu!sei!bwb From: bwb@sei.cmu.edu (Bruce Benson) Newsgroups: comp.software-eng Subject: Re: Specification Tools and Code Testing Message-ID: <8353@fy.sei.cmu.edu> Date: 25 Aug 90 20:53:08 GMT References: <1990Aug13.140347.9441@nixtdc.uucp> <19578@well.sf.ca.us> <8316@fy.sei.cmu.edu> <20013@well.sf.ca.us> Reply-To: bwb@sei.cmu.edu (Bruce Benson) Distribution: usa Organization: Software Engineering Institute, Pittsburgh, PA Lines: 92 In article <20013@well.sf.ca.us> jjacobs@well.sf.ca.us (Jeffrey Jacobs) writes: > >I agree that inspection and so-called "unit"/structural testing should be the >developer's responsibility. However, this does not eliminate the >need for an independent testing organization. a) Programmers still >tend to be blind to their own mistakes. b) Large systems involve "blind to their own mistakes", applies to everyone, even testers who should use a methodical method since "intuitive testing" doesn't work well. My real point is that if the programmer (or anyone) is blind to their own mistakes (and everyone is, try writting a letter) then the "system" should help overcome this (i.e., methodical methods, tools, *peer* reviews, etc) in a way that helps one to improve. Sounds corny, but we've used it and it worked for us. Another way to phrase this is the primary purpose of finding errors is to help the individual (programmer, tester, configuration tech, etc) not repeat the error. We knew the strengths and weakness of each of our programmers and the type of errors they made (and what they did well!). This change in perspective and reason for audits, reviews, and tests, is why I feel we eliminated code errors detected by test and the ultimate user. >a level of complexity that cannot cost effectively tested by the >programmers, e.g. "integration testing", "systems testing", etc. In >most cases, the programmers frequently don't even have a sufficiently >broad knowledge of the system to know where to start, nor can they >be expected to acquire such knowledge and still do any development work. > >"Code working well" is only a part of building a large system. I've seen >more than one project where all of individually tested piece "work well", >yet when put together, the whole thing falls apart. I admit to having background in maintenance of as opposed to development of large systems (well, 1/4 to 1/2 million lines of code - you judge what size that is. The military contracts for development, but then maintains the code). The only point I would make here is that getting the written code to work well is a big step forward - this allows us to concentrate on improving requirements, design and project management. >No argument there! But I happen to thing that the "first time" should >mean the first time it gets *executed* (or close to the first time). This Absolutely! We had a simple philosophy (or challenge) which was for the work (again, maintenance) to work the first time it is executed. We backed this up with a well defined process to approach each maintenance activity. I found that programmers got addicted to this approach and the results. >means inspecting before testing. And, for the individual programmmer, >it means *desk checking* the code before running their unit (or >structural) tests. I also strongly advocate an independent review of the >code prior to execution. In the ideal environment, these independent >reviews would be informal and an ingrained part of the "culture" of >the company. I have to disagree. This whole process must be formal. "Coding" is not something that should be a discretionary activity. It should be done methodically with planned approaches (follow architecture of program, code general solutions - not *fixes*, commments follow a pattern, etc) planned reviews and testing. Since our approach emphasized improving the programmer and was not some arbitrary *text book* process (ever see military regulations?), it was adopted with less resistance than we expected (programmers helped define the process). >> Most code written is correct. The trick is to increase this percentage. > >Make up your mind! :-) Frankly, I'm afraid my experience forces me to >disagree. Most code written is not correct. Even if the system doesn't >crash or corrupt data, if it isn't functionally correct, then the code >isn't correct. But I've seen far too much code that did crash/corrupt >data/return bad results to come close to accepting the word "most". We looked at the code and analyzed the errors caught. In our work the vast (how do you like that for fuzzy) amount of code was correct. This I believe explains why we so quickly achieved and sustained "zero defects" (code errors detected by test and users). >> Independent test did the functional test, and we argued a lot over if the >> function was implemented correctly... > >This is indicative of a major flaw in the process!!! If everybody >is arguing a lot over the functionality of the system then the requirements >and analysis were obviously done poorly!!! Yup, requirements part of the process stunk. For that matter, independent test was a "window". If we turned buggy code over to test, then buggy code eventually went to the users. We realized that our test function could only give us an idea of how well we did. Can't fix everything at once! * Bruce Benson + Internet - bwb@sei.cmu.edu + + * Software Engineering Institute + Compuserv - 76226,3407 + >--|> * Carnegie Mellon University + Voice - 412 268 8469 + + * Pittsburgh PA 15213-3890 + + US Air Force