Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!uunet!mcsun!hp4nl!dutrun!duteca4!kooijman
From: kooijman@duteca (Richard Kooijman)
Newsgroups: comp.ai.neural-nets
Subject: Re: Finite word length BP
Keywords: Back-propagation, number respersentation
Message-ID: <1171@duteca4.UUCP>
Date: 10 Jan 91 08:26:15 GMT
References: <0022@calsci> <1991Jan9.205143.27257@eng.umd.edu>
Organization: Delft University of Technology, Netherlands
Lines: 25

yunshu@eng.umd.edu (Yun-Shu Peter Chiou) writes:

>Does anyone out there have any references or have done any works
>on the effects of finite word length arithmetic on Back-Propagation.

I don't have references, but I have done it myself.
I wrote a BP simulator for IBM compatibles and wanted some more speed.
So I replaced all floating-point arithmetic by fixed-point arithmetic.
The speed gain was > 3.5. The convergence appeared to evolve somewhat less 
smoothly, but I can't say there were any more problems. Everything appeared to
work OK and a lot faster.
The only important thing to do is to limit the value of the weights. If a
neuron has many inputs and they have large values for their weights and they
are all 'on', then the weighted sum of the inputs may overflow for the 
fixed-point type you choose for the activation of the neuron.
Another solution is to check the values you add to the activation. If the
value and activation are positive and the sum is negative then there was an
overflow (mind you, there are no automatic overflow checks on integer types
on a IBM compatible).
One more important thing: I worked with numbers accurate to four decimals.
This seemed to be enough and real live neurons have also been reported to
be not so accurate.  >8-)


Richard.