Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!think.com!mintaka!spdcc!dirtydog.ima.isc.com!ism.isc.com!b1!ico!rcd
From: rcd@ico.isc.com (Dick Dunn)
Newsgroups: comp.software-eng
Subject: metrics and the SAT example
Summary: metrics two levels deep
Message-ID: <1991May22.222646.10571@ico.isc.com>
Date: 22 May 91 22:26:46 GMT
References: <24563@unix.SRI.COM> <1991May21.223401.27023@netcom.COM>
Organization: Interactive Systems Corporation, Boulder, CO
Lines: 30

jls@netcom.COM (Jim Showalter) writes a bunch of generally on-target stuff,
but he uses one example that should caution us in our quest for decent
metrics...

> ...Think of metrics like the SAT college admissions test. It doesn't
> purport to measure intelligence, it just claims to be a reasonably accurate
> predictor of success in college. The evidence supports this claim: SOMETHING
> that has some bearing on success in college is being measured by the SAT's,
> since those with lower scores tend to do worse in college...

OK, I don't argue with the SAT's success there, but consider:  What is
"success in college"?  Generally it's a matter of satisfying another set of
metrics which purport to be related to the acquisition of knowledge and
skills.  HOWEVER, these metrics (grades, oversimplifying a bit) are also
indirect measures.  So, for the SAT to work, all it has to do is measure a
student's ability to perform well according to the college metrics; it may
not mean squat about what a student is actually going to get out of
college.  In particular, both the SAT and college grades often reflect
one's ability to take multiple-choice tests.  (One of my favorite examples
is that I've done well on a multiple-choice test in basic French, in spite
of never having learned the language...hell, I can barely read a Bordeaux
label.)

Now, note that this doesn't make the SAT inaccurate--it DOES predict what
it's supposed to predict (for whatever reason), just as Jim said.  But we
have to be careful that the metrics, particularly if two levels deep,
predict something useful in the end result.
-- 
Dick Dunn     rcd@ico.isc.com -or- ico!rcd       Boulder, CO   (303)449-2870
   ...Simpler is better.