Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!bloom-beacon!oberon!orion.cf.uci.edu!ucsd!sdcsvax!beowulf!demers
From: demers@beowulf.ucsd.edu (David E Demers)
Newsgroups: comp.ai.neural-nets
Subject: Re: NN Question
Message-ID: <6041@sdcsvax.UCSD.Edu>
Date: 3 Mar 89 05:27:09 GMT
References: <32125@gt-cmmsr.GATECH.EDU>
Sender: nobody@sdcsvax.UCSD.Edu
Reply-To: demers@beowulf.UCSD.EDU (David E Demers)
Distribution: usa
Organization: EE/CS Dept. U.C. San Diego
Lines: 82

In article <32125@gt-cmmsr.GATECH.EDU> kirlik@hms3.gatech.edu (Alex Kirlik) writes:
->Has anyone else been puzzled by the following phenomenon?
->Why should a net with only a few dozen neural units be
->successful at mimicking human behavior that is presumably
->the result of the activation of a tremendous number of
->neurons?  That is, why should a small number of units
->be successful at simulating the behavior of a large 
->number of neurons?

I don't believe that much is known about how human behavior
results from the action of neurons or collections of neurons.
The fact that connectionist systems can do pattern recognition
does not mean that they are doing it in the way humans do.
Thus it shouldn't necessarily be surprising that "similar"
tasks can be done with nets and brains.  Many pattern
recognition/mapping networks appear to be doing interpolation;
is that what WE do?  Maybe... 
But you do ask a question worthy of study.

->I know that the validity of this question depends upon the
->"level" at which we interpret our models, but, after all,
->these units are modeled to mimic the behavior of individual
->neurons, aren't they?  

Not generally.  Some are (on a crude scale), but again,
very little is known about the way nets built from meat work.
 
->I am aware of the drastic simplifications
->that are made but this doesn't change the intended referents of
->our theoretical objects.

Many if not most researchers are not attempting to model the
brain, but are trying to see if highly parallel and distributed
processing can produce useful and interesting computational
systems.  It is known, for example, that networks with one
hidden layer and feedforward architecture can approximate
any Borel-measurable function from R^n to R^m to any degree
of accuracy (given sufficiently many hidden units). [Hornik,
Stinchcombe & White, 1988]  Can brains do that?  Anyone know?

->One answer would seem to be that there is a tremendous amount
->of additional processing in the brain that is extraneous to
->the processing critical to the task being modeled, yet we are
->only modeling this "critical" segment. For many reasons (that
->could be discussed if necessary) I do not find this answer
->particulary compelling.

Or perhaps the brain just has a lot to do, with a lot of
redundancy built in for safety.  The brain is built
from material that is not robust and does not have high
precision, and does not operate faster than maybe 10ms/step.
But there are perhaps 10^10 neurons with about 1000-10000
connections each.  Our models can be built from pretty reliable
and fast stuff, operating 1000 or more times faster per step.

->A second answer might be that that neural processing has 
->self-similar properties.  That is, the behavior of neural
->collectives share properties with the behavior of individual
->neurons. I find this answer to be interesting and attractive,
->yet I know of no evidence for it.

I suppose a "collective" could
be considered to be a higher order unit, processing a more
sophisticated function than threshold logic.  This is an
efficiency issue, I believe, not a fundamental issue of
computational complexity.

Jack Cowan recently suggested at a workshop in San Diego that
we should all read (or re-read) David Marr's early work.  I
plan to do so soon... even if I'm not trying to model the
brain, nature sure did build some wonderful mechanisms to 
learn from.

->Alex Kirlik
->UUCP:	kirlik@chmsr.UUCP
->        {backbones}!gatech!chmsr!kirlik
->INTERNET:	kirlik@chmsr.gatech.edu

Dave DeMers			demers@cs.ucsd.edu
Computer Science & Engineering 
UCSD
La Jolla, CA 92093