Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!think.com!zaphod.mps.ohio-state.edu!wuarchive!uunet!mcsun!hp4nl!fwi.uva.nl!smagt
From: smagt@fwi.uva.nl (Patrick van der Smagt)
Newsgroups: comp.ai.neural-nets
Subject: Re: Adding 0.1 to logistic
Keywords: Logistic
Message-ID: <1991May23.141446.28619@fwi.uva.nl>
Date: 23 May 91 14:14:46 GMT
References: <{HV_2H$@warwick.ac.uk>
Sender: news@fwi.uva.nl
Organization: FWI, University of Amsterdam
Lines: 36
Nntp-Posting-Host: chris.fwi.uva.nl

esrmm@warwick.ac.uk (Denis Anthony) writes:

>I attended a seminar yesterday, at which it was stated that adding 0.1
>to the logistic function in back prop speeds up learning by 50% (in one
>application anyway).

>If this a known phenomenum, and if so is there any reason for it ?

Well, maybe this is not so hard to explain.  When initial weights are
very small the input to each hidden unit (i.e., the parameter of the
logistic function) is situated around 0.  Then the behaviour of the
network is almost linear, since around 0 the logisitic function is
almost linear.  The network will then not be able to solve a non-linear
problem with linear hidden units, and the weights will tend to 0.

Adding a small value to the input of the hidden unit will, of course,
shift its value to a less linear region, and thus the initial phase of
training will be faster.

							Patrick van der Smagt
								    /\/\
                                                                    \  /
Organisation: Faculty of Mathematics & Computer Science             /  \
              University of Amsterdam, Kruislaan 403,            _  \/\/  _
              NL-1098 SJ  Amsterdam, The Netherlands            | |      | |
Phone:        +31 20  525 7524                                  | | /\/\ | |
Fax:          +31 20  525 7490                                  | | \  / | |
                                                                | | /  \ | |
email:        smagt@fwi.uva.nl                                  | | \/\/ | |
                                                                | \______/ |
                                                                 \________/

								    /\/\
``The opinions expressed herein are the author's only and do        \  /
not necessarily reflect those of the University of Amsterdam.''     /  \
                                                                    \/\/