Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!mips!prls!pyramid!athertn!hemlock!mcgregor
From: mcgregor@hemlock.Atherton.COM (Scott McGregor)
Newsgroups: comp.software-eng
Subject: Re: use of metrics (use of Prescient Agents)
Keywords: Prescient Agents
Message-ID: <35430@athertn.Atherton.COM>
Date: 8 Jun 91 00:13:01 GMT
References: <35364@mimsy.umd.edu> <4794.284cfad3@iccgcc.decnet.ab.com> <35346@mimsy.umd.edu> <1991Jun7.074226.12105@netcom.COM>
Sender: news@athertn.Atherton.COM
Reply-To: mcgregor@hemlock.Atherton.COM (Scott McGregor)
Organization: Atherton Technology -- Sunnyvale, CA
Lines: 118

>BUT you cannot automagically collect data which is invisible.  Kindly explain
>to me how a simple tool will determine the amount of unpaid (unrecorded) time
>spent?  Or decide how much of the 40 hours spent on error correction last week
>was spent on each of the 17 error fixes that were performed?  This is highly
>valuable data for the project metrics program, and it only comes from
humans.  
If we accept that absolute precision in unnecesaary (and with typical
corner cases impossible to uniquely predefine, then *rough estimates* of how
much time were spent on each error might be good enough.  Estimates can
be hypothesized, by systems that pay attention to what work the individual
is doing.  For instance, work done while a copy of the bug report A is being
concurrently displayed in a window, might be more likely to associated with
error A than with error B.  An agent might therefore estimate that more
time was spent working on A than B.  The programmer might take these estimates
and review them for general correctness, making corrections to any
misestimates made by the system (note this keeps ownership of the data
with the engineer--they are free to alter it if they feel data would
result in punishment, which encourages managers to not use the data that
way, since
such use would lead to unusable numbers in the future).  With such
estimates, predictive estimates can be made, of the complexity of
various pieces of work, of their cost to fix, etc.  They won't be
perfect, but may be better than total ignorance.

The point is that very little research has looked at what can be learned
from watching the work process itself.  The past assumption is that the
computer must be explictly told everything, or it must be derivable from
static representations of code, or tests dones in special harnesses. Little
attention to the way people work at a keyboard, mouse and monitor, on a
particular task has been applied to this sort of estimation process.

However, in fact a great deal of information can be gained from watching
users at work.  People doing UI testing have long known this, and have
frequently taken video tapes of people at work.  From watching the
video tape, it is easy to see about how long someone spends on a
task, or how frequently they switch from task to task, and when they seem to
be confused, thinking, etc. You can tell this even without their
explictly saying anything about what they are doing. Interestingly
enough, while watching
facial movements, body movements, etc. helps a lot, I have found
you can discover a lot of information MERELY from watching a video
of the display they are looking at (with no sound, no body movements,
etc.).  But this is just people watching the person at work from outside.

Yet, IF PROPERLY INSTRUMENTED, the same information can be monitored
from the inside of the system. Once the data is collected, it can
not only help generate metrics, but it can also be used to do
pre-caching of menus, databases, automatic raising of help, etc. that
assists the user directly in a way similar to the way company clerk Radar
O'Reilley of M*A*S*H helped his colonel by preparing files in advance, having
information available just before it was needed, etc.  This use of
instrumented UIs to provide additional assitant style help has been
the focus of the work that I have done and presented in my articles
and lectures on "Prescient Agents."  Many other figures in CHI and CSCW
areas have made similar observations.

Of course, almost no one does such monitoring today, and almost no one
thinks about it, or how they would use such a system. Yet
characteristics of X-Windows ICCCM conventions, NFS file systems, ptrace
system calls, ps and "accounting"
monitoring capabilities in today's Unix systems provide a reasonable base
for building such agents. Such agents can be aware of such things as
when windows and sprite locations change, when "X rooms" change,
locations and durations of windows, accesses to files, spawning and
death of child processes, etc.  People work in certain habitual ways,
determined by their neural wiring systems.  For example, if people are
working on multiple projects,
and they have a system with X rooms, then the most common organization of
a project is to have closely related objects in the same  rooms  at the
same time.  On the other hand, it is often inconvenient for people to
work on related things if they are overlapping, so non overlapping
windows in the same "room" are even more likely to be strongly related. 
Not every
window in a room is necessarily related (e.g. a clock window) to the
other objects, but most are, and often the exceptions can be easily
handled (often just by ignoring them!).  Exceptions often even have patterns
that allow them to be easily recognized (for instance a window that is
shown in ALL rooms is probably less likely to be related to any of them.)
Norman, Schneiderman, Malone, Engelbart, Kay and many others have
long noted that people use spatial and temporal proximity and cues to help them
manage complex tasks.  Secretarys, clerks, and others use these
heuristics without concious awareness to try to understand other people's
work, but these observations can also be applied conciously, and even
mechanically by agents to achieve surprising results.

> Let me add that I also believe that a metrics program absolutely needs an
> error / metrics analyst to audit reports for sanity.  Bias, subjectivity,
> and other problems will taint metrics reporting, so the reports cannot be
> taken verbatim - some careful checking of reports shortly after submission
> serves as much-needed validation.

Despite the fact that people are already observed doing their work  by other
people today, I suspect that some people will be wary of their computer
doing this sort of monitoring.  Despite how the computer may actually be able
to use this to actively assist in the tasks being observed, people may be
concerned with other uses of the monitored data by others.  It is clear
that like many other technologies, such monitoring can be quite beneficial
but in other hands can be quite threatening.  This can be especially true
of computer information which many people quite uncritically presume to
be ultra precise and accurate even in situations where data accuracy
are known to be only approximate.     I have recommended that
the individual being monitored/assisted be in control of their data, and
that they be free to provide it, deny it, or alter it before giving it
to others such as managers or metrics analysts.  I perceive this as being
like credit information, that you might want to share with some one you
want credit from, but which you also might not want to share.  You might
also want to correct some mistakes or misleading random features in the data.
It is clear that organizations can be created where the user does not control
nor own their data, and in such a situation monitored data can be used
for tyrannical purposes.  I do not recommend that individuals submit
themselves to such organizations if they have the choice, and I feel badly for
those who do not have the choice, and who may be monitored in more ways
than mechanically (e.g. secret police, etc.).  Information technology
seems to always be a two edged sword.

Scott McGregor
Atherton Technology
mcgregor@atherton.com