Path: utzoo!utgpu!water!watmath!clyde!rutgers!mit-eddie!bloom-beacon!oberon!cit-vax!elroy!mahendo!jplgodo!wlbr!scgvaxd!trwrb!aero!venera.isi.edu!smoliar
From: smoliar@vaxa.isi.edu (Stephen Smoliar)
Newsgroups: comp.ai
Subject: interviewing experts
Message-ID: <4662@venera.isi.edu>
Date: 30 Jan 88 17:34:31 GMT
Sender: daemon@venera.isi.edu
Reply-To: smoliar@vaxa.isi.edu (Stephen Smoliar)
Organization: USC-Information Sciences Institute
Lines: 112


I just read the article, "How to Talk to an Expert," by Steven E. Evanson
in the February 1988 issue of AI EXPERT.  While I do not expect profound
technical insights from this magazine, I found certain portions of this
article sufficiently contrary to my own experiences that I decided a
bit of flaming was in order.  Mr. Evanson is credited as being "a
practicing psychologist in Monterey, Calif., who works in the expert
systems area."  Let me being with the observation that I am NOT a
practicing psychologist, nor is my training in psychology.  What I
write will be based primarily on the four years of experience I had
at the Schlumberger-Doll Research Laboratory in Ridgefield, Connecticut
during which I had considerable opportunity to interact with a wide
variety of field experts and to attempt to implement the results of
those interactions in the form of software.

Mr. Evanson dwells on many approaches to getting an expert to explain
himself.  For the most part, he address himself to the appropriate sorts
of probing questions the interviewer should ask.  Unfortuntely, one may
conclude from Mr. Evanson's text that such interviewing is a unilateral
process.  The knowledge engineer "prompts" the expert and records what
he has to say.  Such a practice misses out on the fact that experts are
capable of listening, too.  If a knowledge engineer is discussing how an
expert is solving a particular problem;  then it is not only valuable, but
probably also important, that the interviewer be able to "play back" the
expert's solution without blindly mimicking it.  In other words, if the
interviewer can explain the solution back to the expert in a way the
expert finds acceptable, then both parties can agree that the information
has been transferred.  This seems to be the most effective way to deal
with one of Mr. Evanson's more important observations:

	It is very important for the interviewer to understand
	how the expert thinks about the problem and not assume
	or project his or her favored modes of thinking into the
	expert's verbal reports.

Maintaining bilateral communication is paramount in any encounter with an
expert.  Mr. Evanson makes the following observation:

	Shallowness of breathing or eyes that appear to defocus
	and glaze over may also be associated with internal
	visual images.

Unfortunately, it may also indicate that the expert is at a loss at the
stage of the interview.  It may be that he has encountered an intractable
problem, but another possibility it that he really has not processed a
question from the interviewer and can't figure out how to reply.  If
the interviewer cannot distinguish "deep thought" from "being at a loss,"
he is likely to get rather confused with his data.  Mr. Evanson would have
done better to cultivate an appreciation of this point.

It is also important to recognize that much of what Mr. Evanson has to say
is opinion which is not necessarily shared "across the board."  For
example:

	As experts report how they are solving a problem, they
	translate internal experiences into language.  Thus
	language becomes a tool for representing the experiences
	of the expert.

While this seems rather apparent at face value, we should bear in mind that
it is not necessarily consistent with some of the approaches to reasoning
which have been advanced by researchers such as Marvin Minsky in his work
on memory models.  The fact is that often language can be a rather poor
medium for accounting for one's behavior.  This is why I believe that it
is important that a knowledge engineer should raise himself to the level
of novice in the problem domain being investigated before he even begins
to think about what his expert system is going to look like.  It is more
important for him to internalize problem solving experiences than to simply
document them.

In light of these observations, the sample interview Mr. Evanson provides
does not serve as a particularly shining example.  He claims that he began
an interview with a family practice physician with the following question:

	Can you please describe how you go about making decisions
	with a common complaint you might see frequently in your
	practice?

This immediately gets things off on the wrong foot.  One should begin with
specific problem solving experiences.  The most successful reported interviews
with physicians have always begun with a specific case study.  If the
interviewer does not know how to formulate such a case study, then he
is not ready to interview yet.  Indeed, Mr. Evanson essentially documents
that he began with the wrong question without explicitly realizing it:

	This question elicited several minutes of interesting
	unstructured examples of general medical principles,
	data-gathering techniques, and the importance of a
	thorough examination but remained essentially unanswered.
	The question was repeated three or four times with
	slightly different phrasing with little result.

From this point on, the level of credibility of Mr. Evanson's account
goes downhill.  Ultimately, the reader of this article is left with
a potentially damaging false impression of what interviewing an expert
entails.

One important point I observed at Schlumberger is that initial interviews
often tend to be highly frustrating and not necessarily that fruitful.
They are necessary because of the anthropological necessity of establishing
a shared vocabulary.  However, once that vocabulary has been set, the
burden is on the knowledge engineer to demonstrate the ability to use
it.  Thus, the important thing is to be able to internalize some initial
problem solving experience enough so that it can be implemented.  At
this point, the expert is in a position to do something he is very good
at:  criticize the performance of an inferior.  Experts are much better
at picking apart the inadequacies of a problem which is claiming to
solve problems than at giving the underlying principles of solution.
Thus, the best way to get information out of an expert is often to
give him some novice software to criticize.  Perhaps Mr. Evanson has
never built any such software for himself, in which case this aspect
of interacting with an expert may never have occurred to him.