Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!rutgers!njin!princeton!phoenix!mbkennel
From: mbkennel@phoenix.Princeton.EDU (Matthew B. Kennel)
Newsgroups: comp.ai.neural-nets
Subject: Re: request for philosophic reactions to connectionism
Keywords: connectionism philosophy materialism representations
Message-ID: <7903@phoenix.Princeton.EDU>
Date: 22 Apr 89 06:53:20 GMT
References: <370@eurtrx.UUCP| <18496@gatech.edu> <7894@phoenix.Princeton.EDU> <18504@gatech.edu>
Reply-To: mbkennel@phoenix.Princeton.EDU (Matthew B. Kennel)
Organization: Princeton University, NJ
Lines: 65

In article <18504@gatech.edu) myke@gatech.UUCP (Myke Rynolds) writes:
)Matthew B. Kennel writes:
)|That's exactly the point.  For linear problems, than I have no doubt that
)|classical algorithms (linear systems of equations) should work better than
)|gradient descent (BP), with the whole shebang of nice rigorous results, but
)|the whole point is that back-prop tries to learn general non-linear
)|transformations that AREN'T matrix multiplications.
)|For some kinds of associative memory something like ART
)|may be fine, but associative memory isn't the whole story.  It's
)|generalization (i.e. high-dimensional interpolation) which is the the most
)|interesting aspect of multi-layer perceptrons.
)
)Ah! It is if you are dealing with real valued neurons, which BP gives the
)fascade of doing, but infact it only uses the high and low end of the range
)and is thus binary. With binary neurons, non-linear models are not one iota
)more powerful. Infact, they only increase the complexity of the alg.
             ????????????????
Huh? Wasn't the inability to learn non-linear
transformations the fatal stake through the heart of the single-layer 1960's
perceptrons?  Can ART learn parity?  What makes it different from a
classical perceptron?  You said ART was basically matrix multiplication; if
so, I have serious doubts about its power.

)I'm
)working on a sparse linear equation solver with binary compaction to function
)as a BAM type associative memory. I want to beat the masters at chess.
)(ALL my facualty are excited by my ideas! Thats to say, I'm not a crank)
)
You can beat Kasparov with _linear_ tranformations?  Or maybe what you
mean by linear isn't the same as what I'm thinking of.  By linear
I mean linear in the input vectors: a single-layer classical perceptron.
I can easily deal with algorithms that are linear in the _free parameters_,
but still represent _nonlinear_ transformations, like radial basis functions.

)|Can something like a BAM network be more efficient than an "encoder"
)|type of perceptron in terms of the number of connections?  
)Its an associative memory, not an encoder. Night and day.
)
What I mean is some network that has a small internal layer that then
fans back out to an output layer.  As a purely contrived example, consider
associating 128 bit binary numbers where only a single bit in each string
is on.

For a single layer linear system:
128 neurons -) 128 neurons = 16,384 connections.

For a multi-layer "encoder":
128 neurons -) 7 neurons -) 128 neurons = 1,792 connections.

This is something like what I'm trying to get at.

)|)Connectionists are generally psychologists and computer scientists who do not
)|)appreciate the deeper simplicity of math under the outer tremendous diversity.
)|I've never been able to discern the deeper simplicity of math in any ART paper
)|that I've seen (which is very few, I must admit); back-prop is 
)Thats because the dude is clueless about people.

Do you think you could try to make a simple mathematical description of
ART?  It would be enlightening.

)-- 
)Myke Rynolds
)School of Information & Computer Science, Georgia Tech, Atlanta GA 30332
)uucp:	...!{decvax,hplabs,ncar,purdue,rutgers}!gatech!myke
)Internet:	myke@gatech.edu