Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!wuarchive!rex!spool.mu.edu!uunet!mcsun!hp4nl!fwi.uva.nl!smagt
From: smagt@fwi.uva.nl (Patrick van der Smagt)
Newsgroups: comp.ai.neural-nets
Subject: Re: State of the Art Feed-Forward Network Training Algorithms
Message-ID: <1991May17.090435.9180@fwi.uva.nl>
Date: 17 May 91 09:04:35 GMT
References: <AJ3U.91May17010658@opal.cs.virginia.edu>
Sender: news@fwi.uva.nl
Organization: FWI, University of Amsterdam
Lines: 62
Nntp-Posting-Host: wendy.fwi.uva.nl

aj3u@opal.cs.virginia.edu (Asim Jalis) writes:

>What is the state of the art in training feed-forward networks.
...
>did not see anything that improved performance over plain
>Backpropagation drastically for the general case.

The answer to this question is really very, very simple.  Remember
that teaching a feed-forward network is nothing but minimisation
of a function

	E = 1/2  \sum_p (\vec d_p - \vec a)^2

by varying the parameters W.  This has been investigated a zillion of
times.  Look up any book on numerical analysis, e.g.,

	%A W. H. Press
	%A B. P. Flannery
	%A S. A. Teukolsky
	%A W. T. Vetterling
	%T Numerical Recipes: The Art of Scientific Computing
	%I Cambridge University Press
	%C Cambridge
	%D 1986

	%A J. Stoer
	%A R. Bulirsch
	%T Introduction to Numerical Analysis
	%I Springer-Verlag
	%C New York--Hei\-del\-berg--Ber\-lin
	%D 1980

to read about improvements to steepest descent (aka gradient descent)
minimisation.  Amongst the best methods is conjugate gradient optimisation.

I myself haven't used error back-propagation for over a year, but CG
instead.  It sizzles.

						Patrick van der Smagt 

		`But wait a bit!' the oysters cried
		`Before we have our chat.'
		`For all of us are out of breath'
		`And some of us are fat.'
		`No hurry,' said the carpenter.
		They thanked him much for that.
								    /\/\
                                                                    \  /
Organisation: Faculty of Mathematics & Computer Science             /  \
              University of Amsterdam, Kruislaan 403,            _  \/\/  _
              NL-1098 SJ  Amsterdam, The Netherlands            | |      | |
Phone:        +31 20  525 7524                                  | | /\/\ | |
Fax:          +31 20  525 7490                                  | | \  / | |
                                                                | | /  \ | |
email:        smagt@fwi.uva.nl                                  | | \/\/ | |
                                                                | \______/ |
                                                                 \________/

								    /\/\
``The opinions expressed herein are the author's only and do        \  /
not necessarily reflect those of the University of Amsterdam.''     /  \
                                                                    \/\/