Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!att!pacbell.com!ames!eos!shelby!portia.stanford.edu!portia!bellido From: bellido@elrond.world (Ignacio Bellido) Newsgroups: comp.ai.neural-nets Subject: Re: Backprop Weight Initialization Message-ID: Date: 8 Dec 90 03:55:31 GMT References: <1990Dec6.161422.5314@cs.utk.edu> <268@daedalus.albany.edu> Sender: news@portia.Stanford.EDU Reply-To: bellido@psych.stanford.edu Organization: /user/bellido/.organization Lines: 57 In-Reply-To: berg@cs.albany.edu's message of 7 Dec 90 23:35:04 GMT In article <268@daedalus.albany.edu> Berg@daedalus.albany.edu.UUCP (George Berg) reply me: >>I saw that poster [Kolen and Pollack: "Back-propagation is Sensitive to >> Initial Conditions"] also in last NIPS, but I don't believe this is very >>important, I think all of us who work with backpropagation have these > It may not fit your agenda, but a blanket dismissal of this work is utterly >inappropriate. Since backpropagation is a widely-used technique, the *fact* Ok, may be I wrote something wrong. This job [Kolen and Pollack: "Back-propagation is Sensitive to Initial Conditions"] IS important. What I was trying to say is that this is obbious to anyone who has worked with backpropagation, or just studied it. Also markh@csd4.csd.uwm.edu (Mark William Hopkins) says: >I know this may not sound like much, but one sure bet is to initialize the >weights close to their final values... :) > >Other than that, it really sounds like an impossible problem. Think of what >would happen if the error function looked like a lunar surface with billions >of craters deep and shallow all over the place. > >Doing backpropagation on it would be like trying to find a real 'deep' crater >on the moon by riding a lunar rover constantly downhill from where the lunar >lander set down. You really have to be near the crater to find it, else it >could be on the other size of the moon... I like this analogy about the moon surface. More than that, its worse because we search on a different moon eachtime we begin a search. But I believe that if we try to find the nearest point to our objetive to land, we can not do that unless we have a global view of our space, and that we are trying to do is to find a way without this kind of global view, that is, we are searching in a local environment (we have little elements looking only their own space). How can we alwais find the right hole with only a local view? Thats the question, I'd like to have the answer but I'm sorry, I haven't. My point of view is that if you find a local hole, what you have to do is to change the space and pray to fall better the next time. Another question is how to know (locally) that you are into a local minima. Ignacio -- -------------------------------------------------------------------------- Ignacio Bellido Fernandez-Montes -1z Visiting Scholar at Stanford University e-mail: bellido@psych.stanford.edu Psychology Department Graduate Student Madrid University of Technology Department of Telematic Engineering e-mail: ibellido@dit.upm.es --------------------------------------------------------------------------