Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!helios!cs.tamu.edu
From: guansy@cs.tamu.edu (Sheng-Yih (Stanley) Guan)
Newsgroups: comp.ai.neural-nets
Subject: Re: Backpropagation...  What is it?
Message-ID: <10136@helios.TAMU.EDU>
Date: 16 Nov 90 03:01:49 GMT
Sender: usenet@helios.TAMU.EDU
Organization: Computer Science Department, Texas A&M University
Lines: 58

In article <1990Nov15.042207.29026@murdoch.acc.Virginia.EDU> 
aam9n@hudson.acc.Virginia.EDU (Ali Minai) writes:

>>A problem that once plagued error-correction learning Artificial Neural
>>Systems was their inability to extend learning beyond a two-layer ANS.
>>Specifically, the amount of error each hidden layer processing element
>>should be credited for the ouput processing elements' errors was not
>>defined.  Fortunately this problem, known as the credit assignment
>						 ^^^^^^^^^^^^^^^^
>>problem has been solved by using backpropagation algorithm.
>>^^^^^^^
>>
>>Quoted from Artificial Neural Systems by P. K. Simpson

Minai>Well, I wouldn't go so far as to say that the credit assignment problem 
Minai>is "solved" by back-propagation. (the rest deleted)

My apology for quoting the text of P. K. Simpson out of context which leads
the conclusion from A. Minai saying that the credit assignment problem is
not really solved by back-propagation.

My intention of making the quote is to explain the backpropagation from
another point of view.  Indeed, backpropagation was introduced to solve
the credit assignment problem which the ultimate purpose is to try to 
assign the weights associated with links appropriately.

As also noted in Simpson's book, backpropagation is not quaranteed to find
the global error minimum during training, only the local error minimum.
Even that, there are numerous applications of backpropagation which all
of them have claimed at least they have achieved moderate success in 
using backpropagation to do the training.  When I talk about backpropagation
here, I mean the original backpropagation alg. and all its variants.

Minai>P.K. Simpson is obviously indulging in what Hecht-Nielsen rather politely 
Minai>describes as "hype".

Well, I wouldn't go so far as to say that.

Minai>One point that I seldom see noted in connection with back-propagation is 
Minai>that Werbos' original procedure, which he called "dynamic feedback", is
Minai>a very general method, and is directly applicable to all sorts of  
Minai>optimization problems

Coincidently, in Simpson's book there is a note of Werbos's dynamic
procedure in the section 5.4.3 Backpropagation.  "Werbos (1974) independently
discovered the backpropagation algorithm and several variants - calling
the algorithm dynamic feedback - while working on his
thesis in statistics."


The nice thing about Simpson's book is that he has tried to compare
and summarize different Artificial Neural Systems using similar notations
and criteria which is itself not an easy job (the reader
should be cautioned that there are some errors in his book.)

Ciao,

Stanley Guan