Path: utzoo!news-server.csri.toronto.edu!cs.utexas.edu!usc!wuarchive!udel!chester
From: chester@udel.edu (Daniel Chester)
Newsgroups: comp.ai.neural-nets
Subject: Re: Are Conjugate Gradient algorithms any good?
Keywords: Conjugate Gradient algorithms, Back-propagation
Message-ID: <47034@nigel.ee.udel.edu>
Date: 9 Mar 91 03:59:40 GMT
Sender: usenet@ee.udel.edu
Reply-To: chester@udel.edu ()
Organization: University of Delaware
Lines: 24
Nntp-Posting-Host: dewey.udel.edu

In his March 6th reply to Denis Anthony, Mark Plutowski made the assertion
that "the backpropagation update is the special case of the Gauss-Newton
update obtained by setting the Hessian to the identity matrix."  This is
incorrect; to get something like the backpropagation update, the Hessian
has to be set to the 0 matrix. Even then it is not the same because it
does a line search where backpropagation does one step.  If you set the
Hessian to the identity matrix, the Gauss-Newton update becomes a
conjugate-gradient method.  See the following reference for details.

David F. Shanno. "Conjugate gradient methods with inexact searches", in
Mathematics of Operations Research, Vol. 3, No. 3, August 1978, 244-256.

This reference claims that the resulting memoryless BFGS algorithm 
"substantially outperforms known conjugate gradient methods on a wide class
of problems", though I haven't tried it.

Daniel Chester
Univerisity of Delaware
Department of Computer and Information Sciences
Newark, DE 19716

chester@udel.edu
--
Daniel Chester