Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!ncar!boulder!bill
From: bill@boulder.Colorado.EDU
Newsgroups: comp.ai.neural-nets
Subject: Re: : Step Function
Keywords: learning,generalization
Message-ID: <11308@boulder.Colorado.EDU>
Date: 3 Sep 89 18:56:49 GMT
References: <1060@rex.cs.tulane.edu> <6980@sdcsvax.UCSD.Edu> <17538@bellcore.bellcore.com> <1727@cbnewsl.ATT.COM> <7011@sdcsvax.UCSD.Edu>
Sender: news@boulder.Colorado.EDU
Reply-To: bill@synapse.Colorado.EDU (Bill Skaggs)
Organization: University of Colorado, Boulder
Lines: 27


  Is it necessary to have a bias in order to be able to learn?

  Not always.  If the training set includes an example of every
possible input, then the device only needs to be able to "memorize"
the correct responses -- it doesn't need a preset bias.

  If, though, there are inputs not included in the training set,
then the device can only produce responses by somehow "generalizing"
from what it has seen, and the rules it uses for generalization
constitute a bias.  (I actually prefer the word "heuristic" to
"bias".  What I am saying here is that any device that learns from
incomplete data must use some sort of heuristic.)

  For devices that learn solely from examples -- which includes all
the neural networks I've seen -- that's all there is to it.  It's
not a deep question at all.

  It may be more interesting to wonder about more "intelligent"
devices, which are capable of learning from commentary as well as
examples.  ("Commentary" is, e.g., your English teacher telling you
that all proper nouns should be capitalized.)  Even this, though,
does not change the general principle.  A comment may be thought of
as a constraint upon the set of possible responses.  If all of the
examples, plus all of the constraints, do not force a unique response
to every possible input, then the device, in order to pick a response,
must use some sort of bias.