Path: utzoo!utgpu!news-server.csri.toronto.edu!bonnie.concordia.ca!uunet!mcsun!unido!uklirb!shell
From: srt@aero.org (Scott TCB Turner)
Newsgroups: comp.ai.shells
Subject: Re: KES Timing Result
Message-ID: <7708@uklirb.informatik.uni-kl.de>
Date: 5 Apr 91 20:08:32 GMT
References: <7642@uklirb.informatik.uni-kl.de> <7700@uklirb.informatik.uni-kl.de>
Sender: shell@uklirb.informatik.uni-kl.de
Organization: The Aerospace Corporation, El Segundo, CA
Lines: 30
Approved: shell@dfki.uni-kl.de
Posted-Date: Mon Apr  8 07:30:32 GMT 1991

Like some previous posters, Anurag Acharya recently wrote in this
newsgroup that he doesn't think much of very simple benchmarks of
inferences engines.  He pointed out the importance of testing the
performance of rules with multiple conditions.  Other posters have
suggested other important features: pattern matching, kb lookup, and
so on.

All these are valid points.  You can't understand the performance
characteristics of an inference engine without knowing these things.
Unfortunately, I don't have the time to develop a realistic benchmark
suite and apply it to a variety of inference engines, so I've had to
make do with very simple benchmarks.  So far, the benchmarks have
corresponded fairly well with "real-life" performance of the engines
I've looked at, so perhaps the simple benchmark is more useful than it
appears.  And judging from the comments I've received, others have
found these benchmarks useful, and I think most of the readers of this
newsgroup are knowledgeable enough to recognize these benchmarks for
what they are.  I don't really need anyone else to tell me how simple
and useless these benchmarks are.

In the meantime, there's a big demand out there for thorough
benchmarking of expert system tools.  I've had several requests for an
article along those lines, and I've had companies soliciting me for
evaluations.  I encourage anyone who is interested to begin work on
developing a benchmarking suite, and to share your progress with this
group.  I'm sure you'll get plenty of suggestions of what's important
to test :-).  Once you've made some progress, I can put you in touch
with some people who'd be interested in an article on your results.

					-- Scott Turner