Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!ncar!boulder!bill
From: bill@boulder.Colorado.EDU
Newsgroups: comp.ai.neural-nets
Subject: Re: : Step Function. Biases are necessary
Keywords: learning,generalization
Message-ID: <11627@boulder.Colorado.EDU>
Date: 13 Sep 89 17:52:30 GMT
References: <1060@rex.cs.tulane.edu> <6980@sdcsvax.UCSD.Edu> <2934@arisia.Xerox.COM> <1851@cbnewsl.ATT.COM>
Sender: news@boulder.Colorado.EDU
Reply-To: bill@synapse.Colorado.EDU (Bill Skaggs)
Organization: University of Colorado, Boulder
Lines: 50


  Well, the discussions on this topic are getting long-winded,
with lots of nested quotations and such, so it's probably
pretty much exhausted its vitality -- but I can't resist taking
one more fling, and then I shall remain resolutely silent.

>I believe a couple of points have been brought out in our discussion over the
>past few weeks. In my *opinion*,
>1) learning and memorization are two very different things.

    I bet there isn't a single psychologist in the whole world who doesn't
  think that memorization is a kind of learning.

>2) learing implies generalization and rule-extraction. Memorization does not.

    "Learning", as the word is usually used, is a more-or-less enduring
  change in behavior, caused by experience.  It implies generalization
  only in the sense that no two stimuli are ever exactly the same.  (As
  Heraclitus put it:  you can't step twice into the same river.  The
  second time, it's a different river, and you're a different person.)
  Whether it implies rule-extraction depends on what you mean by a "rule".
  If a rule is simply an association between some inputs and outputs, then
  you're right; if it is more than that, you're not.

>3) Biases of some sort are required to learn anything.

    If I understand this statement, what it means is:  In order to be
  able to generalize, a device must be capable of inferring responses
  to inputs it has not experienced, _and_there_is_no_uniquely_correct_
  way_of_doing_that_.

>4) Learning is fastest with borderline patterns that require the machine
>to differentiate subtle differences in classes. But, it also seems reasonable
>that strikingly different examples also play an important role in learning.

    Learning (of a categorization task) is usually _fastest_, at least in 
  the early stages, with inputs that are "typical" of their categories:
  if you want to teach someone "mammal", you start with a mouse or a
  dog, not a dolphin or bat.  Borderline inputs are useful later on,
  after the categories have been roughly sketched out, because they give
  precise information about where the borders are.
   
>5) Learnablility should be defined in terms of a particular set of biases,
>perhaps dependent on network architecture. (e.g. some things are just not
>learnable by a particular network or machine)

    This point seems completely correct to me, and it is the most 
  important point of the whole discussion.

>Not bad.