Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!ncar!gatech!mcnc!uvaarpa!murdoch!hudson.acc.Virginia.EDU!aam9n
From: aam9n@hudson.acc.Virginia.EDU (Ali Minai)
Newsgroups: comp.ai.neural-nets
Subject: Re: Backpropagation...  What is it?
Message-ID: <1990Nov15.042207.29026@murdoch.acc.Virginia.EDU>
Date: 15 Nov 90 04:22:07 GMT
References: <10087@helios.TAMU.EDU>
Sender: news@murdoch.acc.Virginia.EDU
Organization: University of Virginia
Lines: 35

In article <10087@helios.TAMU.EDU> guansy@cs.tamu.edu (Sheng-Yih (Stanley) Guan) writes:
>
>A problem that once plagued error-correction learning Artificial Neural
>Systems was their inability to extend learning beyond a two-layer ANS.
>Specifically, the amount of error each hidden layer processing element
>should be credited for the ouput processing elements' errors was not
>defined.  Fortunately this problem, known as the credit assignment
>						 ^^^^^^^^^^^^^^^^
>problem has been solved by using backpropagation algorithm.
>^^^^^^^
>
>Quoted from Artificial Neural Systems by P. K. Simpson

Well, I wouldn't go so far as to say that the credit assignment problem is
"solved" by back-propagation. As someone pointed out earlier, back-propagation
is just a neat application of the chain rule to calculate derivatives in the
network. The gradient values returned by the algorithm are, of course, a
strictly *local* and *numerical* estimate of the "true" gradient ---
though this notion is itself somewhat shaky. All in all, back-propagation is
beautiful; I love it; I spend many hours a day working with it; but the
final solution it ain't. P.K. Simpson is obviously indulging in what
Hecht-Nielsen rather politely describes as "hype".

One point that I seldom see noted in connection with back-propagation is that
Werbos' original procedure, which he called "dynamic feedback", is a very
general method, and is directly applicable to all sorts of optimization problems
(see, for example, Werbos' papers in recent issues of Neural Networks and IEEE
Trans. on Systems, Man, and Cybernetics). Numerous researchers (including
Mozer & Smolensky, Le Cun & Becker, Chauvin etc.) have utilized the generality
of the algorithm to back-propagate various kinds of gradient information as
part of neural net learning algorithms. 

Ciao,

Ali Minai