Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!think.com!zaphod.mps.ohio-state.edu!van-bc!ubc-cs!uw-beaver!Teknowledge.COM!unix!CRVAX.Sri.Com!hlavaty
From: hlavaty@CRVAX.Sri.Com
Newsgroups: comp.software-eng
Subject: Metrics Example
Message-ID: <24600@unix.SRI.COM>
Date: 22 May 91 15:26:50 GMT
Sender: news@unix.SRI.COM
Organization: SRI International
Lines: 91

I guess the best way to illustrate my position on metrics (previously discussed
in several of the Re:bridge buildings...) is to use an example.  Kers. Caravan:
sorry if my original response to your post was "flamelike" - it was not 
intended to be.  From your repsonse, I think you missed my point entirely.  
Anyway...

Several years ago, my job as a customer representative was to ensure that the
software developments currently underway were progressing smoothly (i.e. the
customers money was being spent correctly).  By progressing smoothly I mean
the advertised schedules were indeed accurate, and no "short-term" gains were
being taken in place of "long-term" overall health (such as skimping on unit
testing to keep to the schedule, for now).  This was a cost-plus contract
(which means the customer pays for overruns) so the customer was very 
inetersted in looking over the developers shoulder (bitter experience in the
past has taught them this hard lesson). 

I decided that we would have a monthly status meeting where I would meet with
the managers of the different software projects and talk with each of them for
about an hour on how their efforts were going.  At the first meeting, all of the
discussions were what I call qualitative (and yes, I have confused qualitative
with subective.  I welcome discussion on how they're different).  Each manager
showed his schedule chart (the schedule that they themselves managed to - it was
not my format.  I wanted the *real* story), and proceeded to discuss why their
development was progressing just fine.  The only thing I had to go on to agree
with them was their qualitative opinion (and since it wasn't necessarrily my
opinion, I considered it subjective - meaning it was their opinion and I didn't
see any evidience other than their experience/feelings/motivations that led me
to believe this to be true).  Now, after this meeting I had to go to the 
customer and tell them what I thought of the development efforts.  I quickly
realized I had a problem, since I was very uncomfortable telling them what I
thought; it was really only what the software managers thought (and the customer
already new that!).  What to do...What to do...

I seized on metrics as the answer to my problems.  If I could get the managers
to show me various metric data, then *I* could interpret it my own way and make
my opinions!  This is what I was referring to when I previously posted about a
"common reference point".  Before, since I was not part of the software 
development, I had no common reference point.  I had to accept what the managers
were telling me at face value (and my customers experience has shown this to be
dangerous in a cost-plus contract).  Once we were using metrics, at least we
could have some numbers to discuss.  So, here is the set of metrics I got them
to start presenting:

	1) A graph of unit test cases complete over time.  Both projected and
	   completed test cases were plotted.
	2) The same graph for integration test cases
	3) The number of errors detected, fixed, and still open each month
	4) The location of these errors
	5) SLOC coded, integrated, and projected each month

Why did I pick these?  I acquired information from the industry that talked 
about what these metrics had meant on OTHER programs done by other companies.
I therefore had other data to compare the contractors results to, and 
deteremine for myself if it compared favorably to what the contractors were
saying (things were on schedule and fine).  None of these metrics were in use
by the software managers at the time.  Well, several interesting things
happened...

After several months, the unit/integration test case graphs clearly showed that
the existing schedule for one of the developments was not accurate.  The rate
at which the development was accomplishing these was not matching the projected
this metric, say by an experienced manager following his own qualitative 
opinion?  Yes, it could have.  But the manager in question *didn't* catch it.
He didn't realize his problem untill confronted with the question "Since your
demonstrated rate of testing is not matching your original projection, how are
you going to finish on time?"   

A big advantage gained using metrics is that it gave me a window into the 
development process through which I could look to determine if things
were in control.  With the metric charts, now my questions would be
"Gee, you had a large error spike last month.  What caused that?" or "You
didn't get your normal amount of unit testing in last month.  Why was that?"
Suddenly we were discussing *real* issues.  The answers to these questions
invariably led to the identification of specific problems the manager was
having (not enough computers, someone quit unexpectedly, and the occassional
"that's a problem we don't know how to solve just yet").  Now I had what I
needed - a window into the process that communicated to me enough information
to ask the right questions.

After a while, the chief software manager at the company and the program
manager started using these charts themselves for their own purposes.  They
realized that they couldn't afford to miss this information!
other million things your job entails.  Once you have found a metric that
indicates future success, by measuring it you are encouraging the development
to conform to practices that have proven successfull in the past.  You also
have a common framework from which to ask questions and also compare your
development to others that have also used the metric.

I hope this illustrates my point more concretely.  

Jim Hlavaty