Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!dali.cs.montana.edu!uakari.primate.wisc.edu!zaphod.mps.ohio-state.edu!wuarchive!uunet!stanford.edu!unix!CRVAX.Sri.Com!hlavaty From: hlavaty@CRVAX.Sri.Com Newsgroups: comp.software-eng Subject: Re: bridge building and discipline Message-ID: <24653@unix.SRI.COM> Date: 23 May 91 17:17:09 GMT Sender: news@unix.SRI.COM Organization: SRI International Lines: 152 In article , jgautier@vangogh.ads.com (Jorge Gautier) writes... >In article <24563@unix.SRI.COM> hlavaty@CRVAX.Sri.Com writes: >> more than one person enters the discussion. I do not have your values, >> experiences, or knowledge (at least, not necessarily) so I can not possibly >> accept qualitative information/decisions from you with the same impact that >> the information has on you. How do *I* know your right? Why do I care? What >> if it's my job to care? Am I supposed to accept your "feelings" on a multi- >> million dollar development? If you had a fine track record on a similar >> project in the past, I just might do that. But if I am not familiar with >> your past history, or this project is new for you, I cannot in good conscience >> accept your unsubtatianted opinion. > >If you don't trust me, why the heck did you hire me to do the job? >Why don't you just do it yourself if you're the only one who can >interpret information and make decisions? Don't you think that I can >substantiate my "feelings"? (Or does everything have to look like a >number? :) Who's got a problem here, the person who is reporting >something or the person who refuses to accept the report? If the >report is irrelevant to the project, just say so. If you don't want >to accept it because it doesn't come in "metric normal form," that is >your problem. I almost certainly *didn't* hire the person who did the job. The contract was awarded to a company, who decides who will manage the software development as they see fit. Alternatively, if I am a manager repsonsible for you and your project directly, metrics offer a way for you to "substantiate" your opinion of the health and status of your development. I don't like to use the phrase "don't you trust me" because it turns the issue into one of personal integrity. I don't view it that way at all; things are assumed to be at a professional level. What were talking about here is merely an engineering judgement, which I don't see as related to your presonal integrity. As a rule, I assume everyone has personal integrity until proven otherwise. Also as a rule, I question everyone's engineering judgement until I have worked with them enough to trust them implicitly. Unfortunately, I seem to have conveyed the impression that I view metrics as the sole way of knowing the health/status of a development. This is not my position. I view metrics as a usefull tool to try and cut through the opinions/values/subjectivenss of a discussion and lead discussion to the areas that warrant attention (see my post Metric Example). If a metric is showing something negative, does it mean the program is failing? Not necessarily, but it alomst always points to a possible problem that may (or may not) need fixing. One of the interesting things about metrics is the more you use them together, the more powerfull they each become. One metric by itself doesn't mean much, but five of them each measuring something different about a development start to give a good picture of the health/status when looked at in total, and form a good basis for a management discussion. >> Where qualitative judgements really break down is when two people with >> different opinions view the same situation differently. Who's right? Many >> times you just wind up in a pissing contest (to think how many meetings I've >> sat in and watched that happen!). Qualitative information is by definition >> only reliable to yourself. If that's all that matters in your organization >> (if so, I would assume it to be small), you can get away with it. I would >> still say your missing an oportunity, however. > >I would say you're the one who's missing an opportunity if you're >ignoring qualitative information. Many things that happen at the >reality level have not been quantified, and yet they can have a >dramatic impact on the project. For example, how do you measure a bad >design? This can be very problem-dependent; are you including a model >of the problem in your metrics? Or would you wait until it is >implemented so you can measure the defect density, because this is >*something* (in your words) that can be measured? What is your early >warning for this situation? Suprise! I agree with you 100%. I certainly don't ignore the qualitative information. That is another piece of the puzzle. But it does have to be taken with a grain of salt (or metrics). :=) Measuring a bad design is very difficult. So is finding out that you have one when your not measuring it (unless you are the developer. The problem arises when the developer either isn't aware of it or chooses to not do anything about it). I would try to avoid at all costs finding out that I had a bad design before I got to code testing. Errors are a lot easier to fix up front than later. If I am a customer rep, I do spot checks on design units to give me some indication (I pick the units - get's around the phenomenon you describe below). If I am a manager, I make sure my team (or manager(s) below me) are doing smart things like design walkthroughs. I am not aware of any well established metric here except for token/node analysis, which can require a significant amount of extra work. I suppose the spot sampling of design units could be a metric (if all four that I looked at had problems, it's time to look VERY CLOSELY at what's going on). > >> The advantage of metrics is that they facilitate a common ground for discussion >> between people and organizations. You may disagree with me what the numbers >> mean, but we now have a common reference point. The real trick to metrics is >> really just to start measuring *something*. After trial and error you will >> arrive at "things" to track that will work for you and your organization (all >> of which are peculiar, IMHO). What you are after are "early warnings" of >> impending problems that allow you time to fix them up front - before they >> are problems. I would argue that someone with a lot of experience that isn't >> using metrics consciously is actually using them unconsciously (or intuitively). > >"When searching for the least common denominator, beware of the >occassional division by zero." > >A disadvantage of metrics is that it is easier to LIE with them. I >recall a meeting with a previous manager during which he "re-weighted" >the severity of the outstanding defects in a system because, well, >they weren't really that bad, were they. These were cases were the >system clearly did not satisfy the requirements, and worse. Our >common ground for discussion simply facilitated the lie. Nobody had >the guts to say "we don't care if the system doesn't work sometimes," >they simply changed the numbers to make everything look good. Similar >problem with other metrics: An unreported defect is not a measured >defect. The number of reported defects does not tell you how many >defects the system has. The rate of defect discovery depends on how >hard you look for defects, and this is not constant over time. The >amount of time spent looking for defects and/or the reporting and >classification of defects can be fudged to make the rate look better >or worse. Lines of code or other size metrics can be twisted to make >defect density look better or worse. Test coverage metrics don't tell >you how good your test suites are. Etcetera. If you think that the >numbers correspond to the reality of the project and that a change in >one implies a change in the other, think again. Again, I have to agree with you for the most part. But this behavior is exactly why metrics are necessary. People with the attitude you describe are the bane of the manager. They purposely distort things for the short term gain, leaving the fall guy later down in the development stream. Without metrics, how can you ever get a straight answer out of them? They will tell you everything is fine, and you have nothing to go on to get a better answer out of them. As you point out, even by using metrics a person bent on distorting the picture can still muddy up the waters. My contention is that they can't do it as much. With metrics, you now have a "rope" on them that they have to deal. Over time, you can keep throwing ropes on them untill you do get the real picture, despite their best efforts. For example, let's take defect density, which you correctly point out is directly dependent on how much effort you spend looking for problems. Well, let's measure that too in conjunction with defect density and plot them both over time on the same graph! "Gee, Jorge, you found a lot of errors last month, yet this month you spent almost no time looking for new errors. No wonder you didn't find any new ones! Why did you make that decision?" Unfortunately, the above discussion focuses on the "metrics as oppression" approach whereby a manager can use them to control someone below him that he dosn't trust. This is possible (over time), but misses the best use of metrics. When the developers do not have the attitudes described above, but are really interested in the long term quality of a project, the metrics become an invaluable control mechanism to highlight areas that MAY need attention, and can communicate these areas quite effectively to upper management. "Look, boss, see how our unit testing is falling behind our projected rate? If we don't fix this soon, we're not going to make schedule. We've traced the problem to the fact that we don't have enough time on the mainframe. Do you think you could get us more time?" More food for thought. And let me say that I am thoroughly enjoying these discussions. (yes, this is a thinly veiled attempt at flame reduction) :=) Jim Hlavaty