Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!uwm.edu!linac!att!cbnewsm!lfd
From: lfd@cbnewsm.att.com (Lee Derbenwick)
Newsgroups: comp.software-eng
Subject: Re: Personal growth and software engineering!
Summary: Lots of data, but how much information?
Message-ID: <1991Apr21.181153.17062@cbnewsm.att.com>
Date: 21 Apr 91 18:11:53 GMT
References: <9233@castle.ed.ac.uk> <1991Mar25.164133.29674@unislc.uucp> <599@tivoli.UUCP>
Organization: AT&T Bell Laboratories
Lines: 78

In article <599@tivoli.UUCP>, alan@tivoli.UUCP (Alan R. Weiss) writes:
> In article <1991Apr15.221119.6242@cbnewsm.att.com> lfd@cbnewsm.att.com (Lee Derbenwick) writes:
> >But, right now, we know of _no_ metrics that
> >measure some of what we really want to measure.
> 
> You are too vague here, so I'll give you an idea of just *some*
> of the things we are measuring:
> 	
[ long list deleted ]

Yes, you are measuring lots of things.  Do you know what they mean,
or are you measuring them because they are things you can measure?

To pull one out in isolation, it's not clear to me whether you want
to increase or decrease "# of Sev 3 defects found during inspections."
An increase could mean your coding is getting sloppier, or that your
inspections are getting more effective.  (If your coding is getting
enough sloppier, your inspections could even be getting _less_
effective.)  "No change" might mean that your coding and inspections
are both getting better, or it might mean that they're both getting
worse.  And you won't know which till you have enough feedback from
system test and from customers: maybe months or more from now.

Also, with so many things to measure, you need to do _something_ to
condense them down to a relatively few composite metrics to use in
optimizing your process.  You may well be doing this already...
(Indeed, I must assume you are.)

And you have at least one non-metric that you list as a metric:

> 	Total SAVINGS (estimated) Due To Inspections

This is not measureable -- you even note that it is an estimate; at
worst, it's purely subjective; at best it's derived statistically from
other metrics, in which case it's not a metric in its own right.

> Now then, Lee, I fail to see how you could *possibly* say that
> "we" can't measure software, because "we" are doing it every
> single day.  [ ... ]

You are collecting lots of data.  Most of these metrics have ambiguous
interpretations.  The conversion of this data into information is very
much an open area.

> >(In cases where you can accurately characterize your customers' usage
> >patterns, John Musa's work on software reliability gives you ways of
> >estimating MTBF for customers, which is close to what I want.  But
> >with new or significantly changed software, you often have no better
> >than guesses about the usage.)
> 
> Could you please send me some information on John Musa's work?
> Sounds useful.  Right now, we go to customer sites, sit down with
> them, learn their business, and try to learn their work habits.

John D. Musa, Anthony Iannino, Kazuhira Okumoto,
_Software Reliability: Measurement, Prediction, Application_
McGraw-Hill, 1990.

To make best use of it, you have to know the actual statistics of
customer inputs.  (A reasonable approximation will do, but even that
is often not available.)

> Besides, what you are advocating is to just leave well enough
> alone.  It appeals to my libertarian nature, Lee, but the
> manager in me sees disaster.  And hey, what DO you advocate?

I advocate working on metrics, but I recognize that at the moment we
are more capable of collecting large amounts of data than we are at
understanding what those data really mean.  And I _don't_ advocate
waiting until we have fully reliable metrics before making changes.  I
advocate using an out-of-fashion concept called engineering judgment
to work on continuous improvement _now_, making use of what we can
measure, but not treating it as a religion (or as the science it isn't,
yet), while we learn how to measure more meaningfully.

 -- Speaking strictly for myself,
 --   Lee Derbenwick, AT&T Bell Laboratories, Warren, NJ
 --   lfd@cbnewsm.ATT.COM  or  <wherever>!att!cbnewsm!lfd