Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!rutgers!apple!ames!ucsd!sdcsvax!desi!gary
From: gary@desi.ucsd.edu (Gary Cottrell)
Newsgroups: comp.ai.neural-nets
Subject: Re: Does back-propagation work with a wider dynamic range...?
Keywords: Back-propagation, dynamic range....
Message-ID: <6511@sdcsvax.UCSD.Edu>
Date: 28 May 89 21:32:21 GMT
References: <18635@vax5.CIT.CORNELL.EDU> <2166@blake.acs.washington.edu>
Sender: nobody@sdcsvax.UCSD.Edu
Reply-To: gary@desi.UUCP (Gary Cottrell)
Organization: Computer Science and Engineering, UC San Diego
Lines: 43

You have some bugs in your math.
You have:

>sig(y) = {2n/(1-exp[-y])} - n
>       = n*(1+exp[-y])/(1-exp[-y)
>James Taylor
>
>james@uw-isdl.ee.washington.edu

The sigmoid is sq(x) = 1/1+exp(-x), not 1/1-exp(-x).

I will use k instead of n, since for any simulation, it is
a constant.

We want the derivative of  2k*sq(x) - k.

Let's call sq(x) OLD, and 2k*sq(x)-k NEW, and derivative DER.
So we want
	
DER(NEW)=DER(2k*OLD-k)		since NEW = 2k*OLD-k
	= 2k*DER(OLD)
	= 2k*(1-OLD)*OLD,	since DER(OLD) = (1-OLD)*OLD

But if in your simulation, your squash is the new one,
you need the derivative in terms of the new squashing function.

OLD = (NEW + k)/(2k),
so we have
DER(NEW)= 2k*(1-OLD)*OLD
	= 2k*[1 - (NEW + k)/2k]*(NEW + k)/2k
	=    [1 - (NEW + k)/2k]*(NEW + k)
	=    [(2k - (NEW + k))/2k]*(NEW + k)
	=    [(k-NEW)/2k]*(k + NEW)
	=    (k^2 - NEW^2)/2k, if you prefer.


gary cottrell	619-534-6640
Computer Science and Engineering C-014
UCSD, 
La Jolla, Ca. 92093
gary@cs.ucsd.edu (ARPA)
{ucbvax,decvax,akgua,dcdwest}!sdcsvax!gary (USENET)
gcottrell@ucsd.edu (BITNET)