Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!ukma!rex!uflorida!gatech!myke
From: myke@gatech.edu (Myke Rynolds)
Newsgroups: comp.ai.neural-nets
Subject: Re: request for philosophic reactions to connectionism
Keywords: connectionism philosophy materialism representations
Message-ID: <18509@gatech.edu>
Date: 22 Apr 89 17:55:39 GMT
References: <370@eurtrx.UUCP| <18496@gatech.edu> <7894@phoenix.Princeton.EDU> <18504@gatech.edu> <7903@phoenix.Princeton.EDU>
Reply-To: myke@gatech.UUCP (Myke Rynolds)
Organization: School of Information and Computer Science, Georgia Tech, Atlanta
Lines: 71

Matthew B. Kennel writes:
>Myke Rynolds (me)  writes:
>)Ah! It is if you are dealing with real valued neurons, which BP gives the
>)fascade of doing, but infact it only uses the high and low end of the range
>)and is thus binary. With binary neurons, non-linear models are not one iota
>)more powerful. Infact, they only increase the complexity of the alg.
>             ????????????????
>Huh? Wasn't the inability to learn non-linear
>transformations the fatal stake through the heart of the single-layer 1960's
>perceptrons?  Can ART learn parity?  What makes it different from a
>classical perceptron?  You said ART was basically matrix multiplication; if
>so, I have serious doubts about its power.
You neglect the fact that any set of vectors can be made orthoganal. If you
tried to teach a BAM to count in binary, it would fail of course. That hardly
means that you can't teach it to count. By expanding the number of elements in
a vector, fewer of them need to be set, which still further decreases the % set.
This isn't ART, this was my realization that ART isn't really limited. You can
add higher order combinations of the inputs to get better multi-d taylor aproxs
of a function, but that is conceptually the same as expanding the input from
binary to unary.

>You can beat Kasparov with _linear_ tranformations?  Or maybe what you
>mean by linear isn't the same as what I'm thinking of.  By linear
>I mean linear in the input vectors: a single-layer classical perceptron.
>I can easily deal with algorithms that are linear in the _free parameters_,
>but still represent _nonlinear_ transformations, like radial basis functions.
You're blowing me away :-) I don't know what you mean here, but I spent two
quaters talking to mathematicians about multi-dimensional taylor expansions,
that seemed to me to be where the most power was. But I was always disturbed
by the fact that preforming such expansions on binary inputs left with just a
lot more binary inputs! Its simply another form of binary input expansion.
>
>)|Can something like a BAM network be more efficient than an "encoder"
>)|type of perceptron in terms of the number of connections?  
>)Its an associative memory, not an encoder. Night and day.
>What I mean is some network that has a small internal layer that then
>fans back out to an output layer.  As a purely contrived example, consider
>associating 128 bit binary numbers where only a single bit in each string
>is on.
Encoders are in some ways the reverse of what I'm talking about. You have
an unary input (one element set) that activates a binary rep in a much smaller
internal layer. Whats the internal layer for? Why not just associate the unary
input to the output? That is a much simpilar problem for a linear transform.

>Do you think you could try to make a simple mathematical description of
>ART?  It would be enlightening.
Sure! Input signals go through the LTM matrix and get contrast enhanced.
The extreme of contast enhancement is winner takes all with only one output
element set. The LTM is an adaptive linear filter with associative decay.

The output F2 activation gets sent back through the LTM traces to F1, and if
the error is too great, the system resets and another F2 activation pattern
is attempted. The reset is self scaled, if a pattern with three 1's set has
an error it means much more than if a pattern with fifty 1's set only had one
bit wrong. I don't understand how the search for a new F2 activation pattern
works unfortunely.

Finally, there is a gain control, giving you three things that can be active
or inactive. The input, the top down F2 pattern, and the gain control. F1 only
sends its activation pattern up when two of these three sources are active.
The presence of an input pattern activates the gain control while the presence
of a top down F2 pattern inhibits it.

But all this stuff makes ART able to deal with a constantly changing input
environ, which a production system needs to deal with. My environ is constant,
thus I prefer BAM's simplicity.
-- 
Myke Rynolds
School of Information & Computer Science, Georgia Tech, Atlanta GA 30332
uucp:	...!{decvax,hplabs,ncar,purdue,rutgers}!gatech!myke
Internet:	myke@gatech.edu