Path: utzoo!utgpu!news-server.csri.toronto.edu!mailrus!wuarchive!usc!ucla-cs!alexis@oahu.cs.ucla.edu
From: alexis@oahu.cs.ucla.edu (Alexis Wieland)
Newsgroups: comp.ai.neural-nets
Subject: Re: Why the more input neurons, the faster the convergence..?
Message-ID: <38361@shemp.CS.UCLA.EDU>
Date: 26 Aug 90 17:27:44 GMT
Sender: news@CS.UCLA.EDU
Organization: UCLA Computer Science Department
Lines: 42


> From a recent experiment on Guassian data classification 
> using Bp Algorithm, I found that the higher dimensions
> ones( so need more input neurons) converge much much faster
> than those of lower dimensions.

> The order of difference is nearly 100 folds!

Since bp neural nets work with a weighted sum of the inputs, and since
the variance of a sum of independant Gaussian distributed random
variables tends to 0 as the number gets large, *any* classifier
working on Gaussian data should perform better with more inputs.  This
is characteristic of Gaussian classifiers.

The behaviour you report is often even more true for neural nets.  It
is simple to create examples (even inadventently) where the noise free
(and effectively also the high dimensional) case is linearly separable
(i.e., a net will learn quickly) and the high noise case is not
(i.e., learning will be comparatively slow).  A 100 fold difference 
is quite believable.

Actually, our experiences in the past shows there's often more to it than
that.  Four or so years ago we (like everyone else) did a character
recognition system (ours was independant of rotation).  To make a long
story short, 8x8 images took about 10x the wall clock time to learn as
16x16 images.  The difference was that smaller images were so bleary
that it really was hard to distinguish some characters, say a 'C' and
a 90 degree rotated 'A', (this is all in a INNS '87 paper) The moral
is "know your data" ....  (re another discussion,  Russ Leighton the 
co-author of that work, later extended those techniques, used lots of 
limited receptive fields and a handful of tricks and found objects in 
computer generated composite images of up to 1024x1024).

	- alexis.


><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
  Alexis Wieland                     also part-time/on-call 
  grad student at                    lead scientist at
  UCLA CS Department                 The MITRE Corporation, Washington
  alexis@CS.UCLA.EDU                 (don't ask, it's a long commute).
><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><