Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!ames!lll-winken!arisia!kanga!chrisley
From: chrisley@kanga.uucp (Ron Chrisley UNTIL 10/3/88)
Newsgroups: comp.ai.neural-nets
Subject: Re: : Step Function
Keywords: bias in learning,generalization
Message-ID: <2798@arisia.Xerox.COM>
Date: 6 Sep 89 04:34:37 GMT
References: <1060@rex.cs.tulane.edu> <6980@sdcsvax.UCSD.Edu> <17538@bellcore.bellcore.com> <1727@cbnewsl.ATT.COM> <7011@sdcsvax.UCSD.Edu> <11308@boulder.Colorado.EDU> <7024@sdcsvax.UCSD.Edu>
Sender: news@arisia.Xerox.COM
Reply-To: k.karn@macbeth.stanford.edu (Ron Chrisley)
Organization: Xerox Palo Alto Research Center
Lines: 30


In article <11308@boulder.Colorado.EDU> bill@synapse.Colorado.EDU (Bill Skaggs)
writes:
>
>  Is it necessary to have a bias in order to be able to learn?
>
>  Not always.  If the training set includes an example of every
>possible input, then the device only needs to be able to "memorize"
>the correct responses -- it doesn't need a preset bias.
>

This assumes, as has much of the discussion on this topic, that one is dealing
with detertministic classification problems, that is, one in which the proper
response to an input is constant.  But many problems, such as the
classification of natural signals like speech, are statistical in that at
different times, the same point will be mapped to different classes.  Then you
can at best model the discriminant function for each of the categories, and
then, given a point, choose the class that is most probable.  This requires
more than one instance per sample in order to use the memorization technique.
So either you abandon memorization (since there are usually an infinite # of
inputs anyway, and you don't know in advance what an effective discretization
of the input space would be) and therefore use a bias, or, in the cases where
memorization is possible, you will have to look at more than one instance of
each input (I'm wondering:  Isn't this how KNN or k-means classifiers, or even
codebooks, work?).

But what will be the proper number of instances to look at?  That cannot be
determined in adavance, so that may be yet another way bias can creep in...

Ron Chrisley