Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!sharkey!dino!uxc.cso.uiuc.edu!uxc.cso.uiuc.edu!m.cs.uiuc.edu!marick From: marick@m.cs.uiuc.edu Newsgroups: comp.edu Subject: technology transfer and testing (lo Message-ID: <4200009@m.cs.uiuc.edu> Date: 11 Jul 89 23:04:00 GMT Lines: 252 Nf-ID: #N:m.cs.uiuc.edu:4200009:000:11727 Nf-From: m.cs.uiuc.edu!marick Jul 11 07:58:00 1989 Here's a position paper I wrote for a workshop. I'd be interested in seeing what you think about it. Bidirectional Technology Transfer and Testing A Position Paper for the Workshop on Directions in Software Analysis and Testing Brian Marick Motorola Inc.[1] 1101 E. University Urbana, Illinois 61801 Internet: Marick@urbana.mcd.mot.com UUCP: uunet!uiucuxc!mcdurb!marick (217) 384-8521 In much of industry, testing is an entry-level job filled with uncreative drudgework. The testing profession is caught in a vicious cycle - many of the talented, aggressive people escape as quickly as possible to product development, instead of staying where their talents could have the most leverage. They learn to ignore testing, instead of using it. They work apart from the testers that could help them, so they produce untestable software whose bugs are discovered by customers. This will change; economics dictates it. Manufacturing com- panies are learning that quality is a prerequisite to profitabil- ity. Software companies will follow. As they do, testing will receive more attention. In this position paper, I discuss how this change might happen, what we might do to hasten it, and what research directions might produce the results industry will need in the coming years. I think it likely that the evolution of software quality will mirror that of manufacturing quality. In manufacturing, testing is no longer just statistical analysis of products at the end of the assembly line. Instead, it is integral to the careful design, monitoring, and continual refinement of the development and manufacturing processes. Errors must be understood, not just detected. That understanding is used to reduce the number of errors, but also to detect them earlier, more cheaply, and more reliably. Such understanding must be based on teamwork; it appears only when designers, process engineers, and quality con- trol people work together throughout the life of the product. Software has not reached this level of maturity, partly because the need is not understood, but also partly because organizations haven't been growing their testers. What can be done to increase the chance of testers growing into full contri- butors, into people who are sought out both to plan and perform testing of programs, designs, specifications, requirements, docu- mentation, and the software development process itself? A bad beginning makes a bad ending. - Euripides Your first job has a profound effect on the rest of your career. First positions in testing often teach only that testing ____________________ [1] The opinions in this paper should be considered my own. is miserable and to be avoided. My position is that we must pro- duce more college graduates who know better: who are well versed in general software engineering, in sophisticated testing tech- niques and tools, and - most importantly - who have the testing attitude, who understand the role of "designated pessimist", someone who always asks the question, "how can this not work?" My analogy is with UNIX[2]. UNIX enjoys its current popu- larity for many reasons. One is that so many graduates knew it. They entered industry agitating for its use. (UNIX is hardly an isolated case, of course - why else do companies donate hardware and software?) We need to produce bright-eyed and enthusiastic students who will enter industry expecting to do testing right. These students must come from universities that teach them test- ing right. Right now, few universities have the experience to do that. Testing is best learned by apprenticeship and iteration. Because universities have not concentrated on testing, they don't have enough mentors to teach the apprentices. Practitioners must be imported from industry. I envision a university "center of testing excellence" with faculty, industry practitioners on sabbatical, and both undergra- duate and graduate students. The first role of the industry tes- ters will be technology transfer into the university. In partic- ular, they will transfer the techniques and perspectives that are used in large-scale testing, and they will transfer an under- standing of some of the important problems. They will help develop the coursework and research program. Once the center has a clear direction and is beginning to create new technology, the practitioners will take on a new role. Having facilitated the creation of appropriate technology, they will begin facilitating its transfer. I will discuss that transfer after discussing what the center will produce. The Program The center of excellence will produce people, techniques, and tools. People: It will produce good testers by apprenticeship and classwork. Testing classes will heavily emphasize projects. The ideal project is testing another student's class project - the tester and developer work in a team throughout the entire development process. Not only does the tester learn through experience, the developer (one hopes) will learn the value of having a tester. More advanced students will test other univer- sity research projects, often those funded by industry - which will certainly be interested in seeing well-tested research. Techniques: The center will produce techniques as the result of research conducted by faculty and by Master's and PhD candi- dates. A very important area of research is the relationship between testing and the early phases of software development. A good question to answer is, "What does design for testability ____________________ [2] UNIX is a trademark of Bell Laboratories. mean?" In the case of structural testing, it means that the tests are not only based on the structure of the system, but that they also influence that structure. In the case of functional testing, it means exposing more function that can be used to detect errors more readily. But these are too simple answers - design for testability is a subset of "good design", and we're far from understanding that. However, existing research areas have not yet had their testing potential mined: (1) Highly integrated environments encourage the creation of reusable, sensibly partitioned components that are more likely testable. Some of those components are debugging tools that are readily adaptable into test- ing tools. (2) Object-oriented design and programming is a way of decomposing a system into metaphors of concreteness (objects). We're used to manipulating "things" in various ways, so we're more likely to design an object such that testing manipulations can take place. In- heritance makes the modification of systems for test- ing purposes simpler. Active values allow another simple way to insert testing probes into a system. (3) Formal methods of specification and verification in- teract with testing. How does one test the adequacy of a large specification? How can specifications be used to drive automated test generation? How can for- mal methods be merged into the tester's repertoire? (4) Also important are time-motion studies. What do pro- grammers really do? If there's a bug, what caused it? - and why wasn't the test to catch it written? What went wrong? Further, there are hard kinds of systems whose testing is not well understood. The center should concentrate on these. Optimizing compilers, for example, are quite difficult to test, largely because the set of classes of inputs is so large. Operating systems and embedded systems are also hard, for an additional reason - the inputs (including clock interrupts, power failures, and the like) are very hard to control. With graphical user interfaces, the output is the problem: it's often difficult to express the expected results of an operation in a way that allows automatic comparison to the actual results (which are, themselves, often difficult to capture). Tools: The center should create a freely-available tester's toolchest, so that its graduates can quickly demonstrate their skills with proper tools. Like the easy availability of DoD pro- tocols in Berkeley UNIX, this could dramatically improve the state of the practice. One of the most important tools would be a general-purpose test harness for running tests and gathering them into test suites. This might be built on a flexible hyper- text system. My experience is that hypertext links work well for integrating tests with production code. They allow developers to easily examine tests, run them, and perhaps write new ones. Technology Transfer The center will produce the techniques and tools incremen- tally, by applying them to projects. Each project will begin by discarding the failures and incorporating the successes of previ- ous projects. Because universities are free to fail, to try promising approaches that don't work out, they are more efficient at this cycle than industry can be. But what's the point of pushing the state of the art if the state of the practice remains constant? Developing technology is hard, but effective technology transfer is harder yet. Most typically, techniques are transferred indirectly. Companies hire graduates who know the techniques; these graduates apply them, often in a quite limited way. They're limited because the techniques are based on tools that are not transferred. The center is designed to be more efficient at technology transfer of both techniques and tools: (1) The center will produce expert graduates. Experts are more likely to be allowed to use ambitious new tech- niques and tools. Some of the "graduates" will be practitioners, people who already have strong track records. (2) The practitioners will guide the center toward the creation of appropriate and adoptable technology, based on iterated practical projects and the idea of a portable tester's toolkit. (3) Part of the technology transfer problem is that researchers and development organizations are usually separated - just as old-style product development separates design and manufacturing engineers. So, just as old-style designs are difficult to manufac- ture, research results are difficult to apply. The solution is the same: closer teamwork. The testing center implements this solution by putting practition- ers into both worlds: they work at the center, but for their company. A major problem will be finding exceptional practitioners: good at testing and teaching, firmly practical while understand- ing academic and theoretical issues, and with good visibility into their parent corporation. If we can find these people, per- suade them to join the center, and persuade their companies to loan them, we stand a good chance of transferring technology, both directly and indirectly.