Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!rphroy!caen!uwm.edu!bionet!agate!usenet.ins.cwru.edu!tut.cis.ohio-state.edu!retina.cis.ohio-state.edu!kolen-j From: kolen-j@retina.cis.ohio-state.edu (john kolen) Newsgroups: comp.ai.neural-nets Subject: Re: The first few epochs in BP Message-ID: Date: 4 Apr 91 16:44:51 GMT References: <6882@rex.cs.tulane.edu> Sender: news@tut.cis.ohio-state.edu Organization: Ohio State Computer Science Lines: 28 In-reply-to: georgiou@rex.cs.tulane.edu's message of 3 Apr 91 00:11:33 GMT For those who worked with Back-Propagation: Have you notice any chaotic behavior in the graph of the (usual) error function vs epochs? Specifically, during the first 2 of 3 epochs the value of the error would jump all over the place, but afterwords becomes smooth. This is phenomena arise from an interaction between the shape of the error function and the initial weight selection. The error surface near the origin is relatively bumpy, giving rise to the "chaotic" appearence of the error measure. This can be attributed to the relatively large step sizes (to the bumps) taken by backprop as it traverses these divets in error space. As back-propagtion continues to change the weights, there is a tendency for these weights to move away from the origin into a relatively smoother region (wrt to step size). The same sort of behavior can be seen when bp is started in a region of weight space where small changes can make a drastic change in network functionality. To see this phenomena in action see J. Kolen and J. Pollack. (1990) Backpropagation is Sensitive to Initial Conditions. Complex Systems, 4:269-280. -- John Kolen (kolen-j@cis.ohio-state.edu)|computer science - n. A field of study Laboratory for AI Research |somewhere between numerology and The Ohio State Univeristy |astrology, lacking the formalism of the Columbus, Ohio 43210 (USA) |former and the popularity of the latter