Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!wuarchive!uwm.edu!ux1.cso.uiuc.edu!honda.ece.uiuc.edu!andy
From: andy@honda.ece.uiuc.edu (Andy Bereson)
Newsgroups: comp.ai.neural-nets
Subject: Re: BP input scaling, normalization
Keywords: BP, scaling
Message-ID: <1991Apr20.214657.29195@ux1.cso.uiuc.edu>
Date: 20 Apr 91 21:46:57 GMT
References: <2533@fornax.UUCP>
Sender: usenet@ux1.cso.uiuc.edu (News)
Reply-To: bereson@cs.uiuc.edu
Distribution: na
Organization: University of Illinois at Urbana
Lines: 43

> From: mcguire@fornax.UUCP (Michael McGuire)
> Subject: BP input scaling, normalization
> 
> I have been using back-propagation to combine two sets of 11 parameters 
> (22 inputs) into 11 output classes (there are 275 training patterns and 275 
> test patterns). Therefore the net has 22 inputs, 11 outputs and possibly some
> hidden layers. The inputs for each set were scaled by a respective constant
> so that the input values were in the range 0 to 1 (this was a requirement of
> the BP software).
> 	1. Different scaling constants resulted in very different
> 	   classification performances.
> 	2. A network with no hidden-layers outperformed nets with 1 hidden
> 	   layer (both nets had near perfect classification on the training
> 	   patterns).
> 
> 	1. What are the effects of scaling the inputs to a BP net and is
> 	   there an optimal way to do this (especialy since I have 2 sets of
> 	   inputs that need to be scaled differently).

Scaling the inputs to back-propagation has almost no affect, usually.
Such scaling corresponds trivially to scaling the weight matrix.  The
initial weights (which are usually random) will overcome any good that
scaling the inputs does.  I've been experimenting with scaling the
initial weights, but I haven't found anything interesting yet.

> 	2. Why would a single-layer net outperform a two-layer net (2-layer
> 	   net only had 5 hidden units). I would expect the two-layer net to
> 	   at least do as well.

If you are getting near perfect classification with no hidden units, then
you may not need hidden units.  Generally speaking, generalization is
improved by minimizing the number of hidden units in the network.

> 	3. Do output activations of 0.1 and 0.9 (as opposed to 0.0 and 1.0)
> 	   help the generalization process.

Some people believe this is true.  It sets more realistic goals for the
error measure since the sigmoid used most frequently in back-prop
asymptotes towards 0.0 and 1.0 and therefore, even when the net outputs
the correct answer it will sense some error and try to make further
_improvements_.  I do this usually, however, I don't really know if it's
best.  I'd like to see results that answer that answer this question, but
I'm not aware of any.