Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!usc!cs.utexas.edu!ut-emx!tivoli!alan From: alan@tivoli.UUCP (Alan R. Weiss) Newsgroups: comp.software-eng Subject: Re: Personal growth and software engineering! Message-ID: <599@tivoli.UUCP> Date: 18 Apr 91 17:43:05 GMT References: <9233@castle.ed.ac.uk> <1991Mar25.164133.29674@unislc.uucp> <581@tivoli.UUCP> <1991Apr15.221119.6242@cbnewsm.att.com> Reply-To: alan@tivoli.UUCP (Alan R. Weiss) Organization: Tivoli Systems Inc., Austin, TX Lines: 216 In article <1991Apr15.221119.6242@cbnewsm.att.com> lfd@cbnewsm.att.com (Lee Derbenwick) writes: >>In article <581@tivoli.UUCP>, alan@tivoli.UUCP (Alan R. Weiss) writes: >> While I believe that this *IS* an open area of research, as you say, >> I TRULY believe that people-who-are-doing-the-work ("workers") >> MUST be involved in setting their own standards. How are they to >> know? Management outlines the broad parameters of the problem >> and/or improvement goals, and then provides leadership. [ ... ] > >This begs the question. Of course the workers must be involved in >setting the standards. Good. Then we agree on something. >But, right now, we know of _no_ metrics that >measure some of what we really want to measure. You are too vague here, so I'll give you an idea of just *some* of the things we are measuring: Gilb Inspection Process Metrics -------------------------------- # of Sev 1 defects found during inspections # of Sev 2 defects found during inspections # of Sev 3 defects found during inspections # of Sev 4 defects found during inspections Total of all defects logged Total minutes spent in Defect Logging Meeting Total Defects Logged/Minute Total Pages Inspected % Pages Inspected Total Defects/Page Average Defects/Page Pages/Hour Total Reported Minutes Spent Preparing For Inspections Total Cost of Preparation Cumulative Total Time Spent in Inspection Process Cumulative Total Cost of Inspection Process Average Cost to Fix A Defect During Test Phase Total SAVINGS (estimated) Due To Inspections Cost of Quality Test Execution Metrics ---------------------- Total Test Cases Generated (automatically) Total Test Assertions Generated (automatically) % Test Cases to Specifications (Function Points) Approximate % Test Coverage Total Test Assertions Executed Total Test Assertions Succeeded (PASSED) Total Test Assertions Failed (FAILED) Total Test Assertions Blocked (BLOCKED) % Sucess to Total Total Sev 1 Defects Found in Testing Total Sev 2 Defects Found in Testing Total Sev 3 Defects Found in Testing Total Sev 4 Defects Found in Testing Performance Measurements etc. etc. etc. Source Code Analysis Metrics ---------------------------- Relative Cyclomatic Code Complexity Total Lines of Code Defects per KLOC's Defect per Function Point Schedule Metrics ---------------- Time to Code Time to Build Time to Integration Test Time to Functional Test (Component/LPP Test) Time to System Test Number of schedule slips (if any) etc. etc. etc. Now then, Lee, I fail to see how you could *possibly* say that "we" can't measure software, because "we" are doing it every single day. Please note that not all of these measurements have equal weight, and different metrics have different audiences and purposes. This is where management leadership comes into play. NO WHERE are these metrics used to beat people up. They are used to improve both the product and the process, and are used by individuals interested in improving their own performance. Managers only get to see aggregates, not individual measurements. Whether its called "metrics", or "measurements", or just simply "running the numbers", we like to use BOTH halves of our brains: the creative half, and the logical half. (Thanks to Tom Gilb, wherever the hell you are, for your work on Inspections. Thanks also to Michael Fagan, William Howden, Boris Belzer, Kerry Kimbrough, Robert Choate, C.A.R. Hoare, Tony DeMarco, Ed Yourdon, and literally thousands of other people who are building Software Science. Also, thanks to my agent, my mom .... :-) >No amount of >management leadership is going to allow anyone to create those >metrics. But enough management pressure will force people to create >bogus ones. Balderdash :-) I guess you just have had unfortunate experiences. Management creates the space that allows for a process that results in metrics. If that sounds too "California-speak", then forgive me: I *came* from California. Talk to Tom Peters and Ken Blanchard if you don't dig this. And maybe W. Ed Demings, Phil Crosby, and J.M. Juran, too. > >> >Here are a couple of quality metrics I would _like_: they seem to capture >> >two key areas of software quality -- faults and maintainability. Both, >> >unfortunately, violate causality: >> > >> >1. Total number, severity, and time-to-discovery of remaining faults that >> > will be experienced by customers as software failures. >> > >> >2. Cost of introducing the next several enhancements that will be required >> > of this code. >> >> The first one should be "mom and apple pie" for all software orgs. >> The second one is basic risk analysis, and its good, too. > >I'm afraid Alan missed the tense in #1. It is (reasonably) easy to >measure what _did_ happen: but that tells you what your software >process was doing one or two years ago. Yeah yeah, except that we do this kind of stuff daily and weekly and monthly and yearly, and we don't wait. Besides, we can calculate the Escape Rate by knowing the incoming rate, the find rate, the fix rate, and the code coverage rate. Sure, its just an approximation, but so what? Management is NEVER precise, because PEOPLE aren't, and software is a People-business. >And if you haven't changed >your process at all in that time, you haven't been paying attention >even to qualitative information. To improve your process _now_, you >need to know what faults your customers will experience as future >failures. So it's far from "Mom and Apple Pie." No, I believe that if you are still doing things algorithmically rather than constantly adjusting your process to improve quality, you don't deserve to live economically. Which is exactly what Tom Peters has been preaching for about 10 years. >(In cases where you can accurately characterize your customers' usage >patterns, John Musa's work on software reliability gives you ways of >estimating MTBF for customers, which is close to what I want. But >with new or significantly changed software, you often have no better >than guesses about the usage.) Could you please send me some information on John Musa's work? Sounds useful. Right now, we go to customer sites, sit down with them, learn their business, and try to learn their work habits. >The second metric also requires prediction of the future. Measures >of structure, etc., can be useful at a crude level, but they can't >capture "What is the chance that the customer will decide they >desperately need something that violates a fundamental assumption of >this code, so that it will have to be totally rewritten?" And I don't >know of any good way to measure fundamental assumptions. Again, you >can find out later whether you had assumed too much -- but that tells >you about your process in the past. "Basic risk analysis" isn't even >close, though it _is_ a first step in that direction. Hey, I haven't a clue. We manage this process by getting as close to our customers as possible, by trying to learn their business, and by creating a business relationship. But, if a customer changes their minds, they're the customer. We try and stay light on our feet enough to either modify the current product or come out with a follow-on fast. If you think this is just small-company think, then look at 3M, H-P, etc. >or more out of date. But you can make significant process changes >within a few months. In terms of control theory, you have a process >with a certain time constant, but your observations of that process have >a significantly longer time constant. By attempting to use those >observations to control the process, you are likely to introduce >oscillatory or chaotic behavior. > > -- Speaking strictly for myself, > -- Lee Derbenwick, AT&T Bell Laboratories, Warren, NJ > -- lfd@cbnewsm.ATT.COM or !att!cbnewsm!lfd Continuous quality improvement, Lee. Say it like a mantra, or we'll all be learning to speak Japanese Real Soon Now. :-} Companies are NOT like market systems, in which changes to the process (like market intrusion) causes increases in the oscillations. This may be true for the Federal Reserve Board, but falls apart for firms. The reason: communication of information can be speeded-up (velocity of price information, as it were). Besides, what you are advocating is to just leave well enough alone. It appeals to my libertarian nature, Lee, but the manager in me sees disaster. And hey, what DO you advocate? _______________________________________________________________________ Alan R. Weiss TIVOLI Systems, Inc. E-mail: alan@tivoli.com 6034 West Courtyard Drive, E-mail: alan@whitney.tivoli.com Suite 210 Voice : (512) 794-9070 Austin, Texas USA 78730 Fax : (512) 794-0623 _______________________________________________________________________