Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!usc!cs.utexas.edu!ut-emx!tivoli!alan From: alan@tivoli.UUCP (Alan R. Weiss) Newsgroups: comp.software-eng Subject: Re: Personal growth and software engineering! Message-ID: <701@tivoli.UUCP> Date: 24 Apr 91 00:52:34 GMT References: <9233@castle.ed.ac.uk> <1991Mar25.164133.29674@unislc.uucp> <599@tivoli.UUCP> <1991Apr21.181153.17062@cbnewsm.att.com> Reply-To: alan@tivoli.UUCP (Alan R. Weiss) Organization: Tivoli Systems Inc., Austin, TX Lines: 123 In article <1991Apr21.181153.17062@cbnewsm.att.com> lfd@cbnewsm.att.com (Lee Derbenwick) writes: >[ long list deleted ] I would have liked to have had your feedback on the long list I posted. >Yes, you are measuring lots of things. Do you know what they mean, >or are you measuring them because they are things you can measure? At this stage in our development as a company we are measuring for a number of reasons, including (but not exclusively) the need to establish some baselines. Yes, I know what they mean. There are many more things that we do NOT measure, too long to post here. >To pull one out in isolation, it's not clear to me whether you want >to increase or decrease "# of Sev 3 defects found during inspections." >An increase could mean your coding is getting sloppier, or that your >inspections are getting more effective. (If your coding is getting >enough sloppier, your inspections could even be getting _less_ >effective.) "No change" might mean that your coding and inspections >are both getting better, or it might mean that they're both getting >worse. And you won't know which till you have enough feedback from >system test and from customers: maybe months or more from now. Correct. Right now, we are measuring to build up our empirical data and to start to form our heuristics. In point of fact, though, defects logged per minute tends to look like a bell curve: you start by not getting very many, then you get good at inspections, then the number of actual defects decreases as the rework is accomplished PER deliverable. You are correct that these measurements will be VERY useful in the NEXT release of this product, but since we are on a VERY rapid development pace this will happen soon anyway. >Also, with so many things to measure, you need to do _something_ to >condense them down to a relatively few composite metrics to use in >optimizing your process. You may well be doing this already... >(Indeed, I must assume you are.) Yes indeed, eventually. Right now we're still learning which levers and dials (engine room metaphor) are important in our environment. >And you have at least one non-metric that you list as a metric: > >> Total SAVINGS (estimated) Due To Inspections > >This is not measureable -- you even note that it is an estimate; at >worst, it's purely subjective; at best it's derived statistically from >other metrics, in which case it's not a metric in its own right. First, the fact that it is an estimate does not mean it is not measurable, only that it has a relevant range. It IS derived statistically both from other metrics AND using other estimates and assumptions. We are performing some sensitivity analysis now to encourage a step-wise refinement (specifically on Cost-to-Fix, Cost-Per-Defect-If-Escaped, etc). Some of this is proprietary technology (sigh ... setenv COP-OUT). Mostly we don't want our competitors to know our productivity rates (only our customers!). >You are collecting lots of data. Most of these metrics have ambiguous >interpretations. The conversion of this data into information is very >much an open area. Yes, and as I stated we don't beat people up over them. We encourage the developers to look at them, throw spears at them, and help refine them. They are posted on a big whiteboard in the kitchen, with lots of space for flames and comments. By doing so, we help focus their thoughts for some amount of time on Quality. All quality studies I've seen suggest that this is A Good Thing. >John D. Musa, Anthony Iannino, Kazuhira Okumoto, >_Software Reliability: Measurement, Prediction, Application_ >McGraw-Hill, 1990. Thank you! >To make best use of it, you have to know the actual statistics of >customer inputs. (A reasonable approximation will do, but even that >is often not available.) Yes, we do this, too. We have been going to customer sites, soliciting feedback, filming (well, taping) customer responses to our graphical user interface environment (kinda like the Mazda Kansei idea), etc. We're not wiring customers up to feedback machines yet (EEG, EKG, galvanic skin response, etc.) ... dunno, think this is a good idea? :-) Kinda violates their privacy, eh>? >I advocate working on metrics, but I recognize that at the moment we >are more capable of collecting large amounts of data than we are at >understanding what those data really mean. And I _don't_ advocate >waiting until we have fully reliable metrics before making changes. I >advocate using an out-of-fashion concept called engineering judgment >to work on continuous improvement _now_, making use of what we can >measure, but not treating it as a religion (or as the science it isn't, >yet), while we learn how to measure more meaningfully. > > -- Speaking strictly for myself, > -- Lee Derbenwick, AT&T Bell Laboratories, Warren, NJ > -- lfd@cbnewsm.ATT.COM or !att!cbnewsm!lfd OK, I can dig that. So, in order to reach a stage where the measurements are more meaningful, we are actually doing The Right Thing by gathering the data and stepwise refining it into nuggets of information. We use LOTS of engineering judgement: this IS a startup company, after all! _______________________________________________________________________ Alan R. Weiss TIVOLI Systems, Inc. E-mail: alan@tivoli.com 6034 West Courtyard Drive, E-mail: alan@whitney.tivoli.com Suite 210 Voice : (512) 794-9070 Austin, Texas USA 78730 Fax : (512) 794-0623 _______________________________________________________________________