Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!ucsd!pacbell.com!lll-winken!dweasel!loren
From: loren@dweasel.llnl.gov (Loren Petrich)
Newsgroups: comp.ai.neural-nets
Subject: Re: Using Newton's Method to speed up backpropagation.
Message-ID: <86170@lll-winken.LLNL.GOV>
Date: 15 Nov 90 02:44:08 GMT
References: <7556@uwm.edu> <2124@cybaswan.UUCP> <6873@jhunix.HCF.JHU.EDU>
Sender: usenet@lll-winken.LLNL.GOV
Organization: Lawrence Livermore National Laboratory
Lines: 42
Nntp-Posting-Host: dweasel.llnl.gov

In article <6873@jhunix.HCF.JHU.EDU> ins_atge@jhunix.HCF.JHU.EDU (Thomas G Edwards) writes:
>In article <2124@cybaswan.UUCP> eeoglesb@cybaswan.UUCP (j.oglesby eleceng pgrad) writes:
>>I use a quadratic interpolation line search using a couple of points in the
>>search direction. This involves a couple of function evalutions in the search
>>direction from which an estimate of the step size to the minimum is calculated.
>>This potentially shuld be good at finding LOCAL MINIMUM but I haven't found
>>this a problem in 3 years of experience. 4 D EXOR's are generally solved in 
>>< 10 line searches.
>
>Have you tried the 2 intertwined spirals problem?  It really gives my
>conjugate-gradient backprop neural net program a rough time with
>local minima.  But it is easily solved with cascade-correlation.

	Does anyone know of any readily-available Cascade-Correlation
source?

	I tried writing a Cascade-Correlation routine, without very
much success.

	I think that the problem of local minima will necessarily
plague ANY localized search algorithm, because if it finds one
minimum, it will know nothing about the other minima. The only
practical way of getting around that difficulty, at least that I know
of, is by using a stochastic approach, using several different
starting points. And even that is not guaranteed to find the global
minimum.

	I know that, in some cases, the minimization problem is known
to be NP-complete, which means that there is no known algorithm which
solves the problem in some power of the problem size -- all known
solution of NP-complete problems go either exponentially or
factorially with the problem size. However, all NP-complete problems
appear to be fundamentally equivalent, and a solution of one may well
apply to all.


$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Loren Petrich, the Master Blaster: loren@sunlight.llnl.gov

Since this nodename is not widely known, you may have to try:

loren%sunlight.llnl.gov@star.stanford.edu