Path: utzoo!utgpu!news-server.csri.toronto.edu!bonnie.concordia.ca!thunder.mcrcim.mcgill.edu!snorkelwacker.mit.edu!usc!sdd.hp.com!spool.mu.edu!uwm.edu!linac!att!ucbvax!ucsd!nprdc!jones
From: jones@nprdc.navy.mil (David Ryan-Jones)
Newsgroups: comp.ai.neural-nets
Subject: Re: Backprop with additional noisy inputfeature
Message-ID: <15347@arctic.nprdc.navy.mil>
Date: 15 May 91 16:26:03 GMT
References: <3559@wn1.sci.kun.nl> <1991May14.182932.21193@convex.com>
Sender: news@nprdc.navy.mil
Reply-To: jones@nprdc.navy.mil (David Ryan-Jones)
Organization: Navy Personnel R&D Center, San Diego
Lines: 34

In article <1991May14.182932.21193@convex.com> dyes@convex.convex.COM (Tim Dyes) writes:
>In article <3559@wn1.sci.kun.nl>, mderksen@sci.kun.nl (M. Derksen) writes:
>|> Dear neural netters,
>|> 
>|> Can someone explain me why an additional noisy inputfeature results in a
>|> better prediction performance (generalization). 
>|> 
>
>Try this on for size...
>Adding a noisy input feature in effect adds a noise term to the input of each
>node within the 1st hidden layer.  Successful training forces the network 
>to adjust its weights to overcome this noise level.  That will result in there
>being more distance between the 1st hidden layer's output vectors that
>correspond to different network outputs.  This in effect raises the 
>discrimination capability of the 3rd layer to the 2nd layer's output, and so
>allows it to more correctly classify an input pattern.
>
>- Tim Dyes

The effect of adding noise to the input upon generalization is very 
interesting.  I have noticed with my own research that I can improve 
generalization to the test set by about 10% if I add a very small amount 
of gaussian noise to each of the input variables.  Has anyone else reading 
this newsgroup been able to improve generalization to a greater degree 
(say 30-40%) by this technique?  Does this technique work by reducing the 
degree to which the network "learns" the data in the training set instead 
of the underlying relationship between input and output?  Are there any 
studies that have been published as tech reports or journal articles which 
have investigated this effect. 

Thanks, 

David Ryan-Jones