Path: utzoo!attcan!uunet!samsung!uakari.primate.wisc.edu!dali.cs.montana.edu!milton!lisbon!almond
From: almond@lisbon.stat.washington.edu (Russel Almond)
Newsgroups: comp.ai
Subject: Re: Verification of KB Systems
Message-ID: <13943@milton.u.washington.edu>
Date: 8 Jan 91 05:34:14 GMT
References: <1991Jan4.180056.20917@evax.arl.utexas.edu>
Sender: news@milton.u.washington.edu
Organization: U.W. Department of Statistics
Lines: 36


I'm comming in from left field on this issue, but let me describe my
approach to this problem.  As far as I know it has never been
implimented.

I tend to view an expert system as a giant probabilistic model.
Certain variables are observed and on the basis of those others (of
importance to some decision) are predicted.  This view arrises
naturally from the graphical model/belief net concept supported by
Judea Pearl or David Spiegelhalter.  It can even be used for models
based on logic or other inference principles.

As a statistician, I reguard the proof of a probabilistic model as its
predictive ability.  The proof of a knowledge based system would
therefore be its ability to make the same decisions as the "expert."
A validation set of possible inputs would be selected and the KB
system's predictions or decisions would be compared with the expert's
decisions or, in the cases where it was known, the true state of
nature.  Various statistical methods, such as log-linear modelling or
logistic regression could be brought to bear on the problem of
estimating the program's predictive error rate.

Of course such a procedure would be sensitive to the choice of
validation set.  The validation set could easily miss a sample input
for which the system would make erronious and irrelevant predictions.
There is no simple way to deterimine this.  On the other hand, I don't
feel that an expert system is robust unless it can throw up its
hands and say it doesn't know when it reach a point in its input space
about which it has no knowledge and can learn from its mistakes.

I hope these ideas are of some help.

	--Russell Almond
	University of Washington
	Department of Statistics
	almond@stat.washington.edu