Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!uunet!mcsun!ukc!cf-cm!cybaswan!eeoglesb
From: eeoglesb@cybaswan.UUCP (j.oglesby eleceng pgrad)
Newsgroups: comp.ai.neural-nets
Subject: Re: Using Newton's Method to speed up backpropagation.
Message-ID: <2124@cybaswan.UUCP>
Date: 13 Nov 90 11:34:27 GMT
References: <7556@uwm.edu>
Reply-To: eeoglesb@cybaswan.UUCP (j.oglesby eleceng pgrad)
Lines: 45

In article <7556@uwm.edu> markh@csd4.csd.uwm.edu (Mark William Hopkins) writes:
>
>   A typical intro. to backpropagation that goes into any kind of analysis in

...... the rest of the message

>   But generally, using Newton's Method seems to provide a natural way to
>accelerate learning when and only when it needs acceleration.

I would like to add a bit to this last message:

I believe Mark talks about a 1D Newton line search to set the step size, to
which I would add the following:

There are other more robust and just as fast 1D line search that can be used.
Performing an accurate line search on each search is generally a wast of
computation and can in some cases slow you significantly. Producing a good
search direction ie something like conjugate gradient rather than steepest
descent, with an approximate but fast line search, will in general yield
a faster solution than an accurate line search with stepest descent directions.

I have recently seen a paper on Newtons method for ID line searches with
a variety of stratigies for deriving search directions, which leads me to
the conclusion : Yes any line search is generally better than setting a 
constant step size (ETA), but if it is performed accuratley then in general
you will be wasting time. Unfortunately this is paper is still in review
so I can't reference it.

I use a quadratic interpolation line search using a couple of points in the
search direction. This involves a couple of function evalutions in the search
direction from which an estimate of the step size to the minimum is calculated.
This potentially shuld be good at finding LOCAL MINIMUM but I haven't found
this a problem in 3 years of experience. 4 D EXOR's are generally solved in 
< 10 line searches.

Hope my experience is helpful to some of you folks out there,
JO.

------------------------------------------------------------------------------
 John Oglesby.                        UUCP  : ...!ukc!pyr.swan.ac.uk!eeoglesb 
 Speech Processing Group,             JANET : eeoglesb@uk.ac.swan.pyr         
 Electrical Engineering Dept.,        Phone : +44 792 205678  Ex 4564         
 University of Wales,                 Fax   : +44 792 295686                  
 Swansea, SA2 8PP, U.K.               Telex : 48358                           
------------------------------------------------------------------------------