Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!ucbvax!decwrl!labrea!rutgers!psuvax1!shire.cs.psu.edu!ian
From: ian@shire.cs.psu.edu (Ian Parberry)
Newsgroups: comp.ai.neural-nets
Subject: Re: Neuron Digest V5 #17
Message-ID: <4461@psuvax1.cs.psu.edu>
Date: 12 Apr 89 14:02:57 GMT
References: <5935.608322790@hplpm>
Sender: news@psuvax1.cs.psu.edu
Reply-To: ian@theory.cs.psu.edu
Organization: Penn State University
Lines: 102

Bruce, thanks for your interesting reply.  I have been away from the net
for a while (system installation), sorry if I am a bit out-of-date.

>Subject: Re: Re: bottom-up (was Re: NN Question)
>From:    brp@sim.uucp (bruce raoul parnas)
>Organization: University of California, Berkeley
>Date:    Wed, 15 Mar 89 17:28:45 +0000 

I think we are basically agreed that a statement like "the world is
discrete" or "the world is analog" gives us little reason to model
neural networks as discrete or analog.

>>Real numbers, continuous functions etc., are abstractions which help
>>us deal with the fact that the number of discrete units is larger
>>than we can deal with comfortably.
>
>right.  and in most physical systems we may, for our understanding, treat them
>as essentially analog since we simply can't deal with the complexity presented
>by the true (?) discrete nature.

I'm not convinced.  Computational complexity theory gives us tools for
dealing with discrete resources (time, memory, hardware) which are
too large to handle individually.  There is no need to treat them as
continuous.

>>There are (at least) two objections to the classical automata-
>>theoretic view of neural systems.  One is that neural systems
>>are not clocked (I presume that this is what you mean by
>>"continuous time"), and that neurons have analog behaviour.
>
>that is precisely what i meant.  neurons each evolve on their own, independent
>of system clocks.

Yes?  I didn't think the evidence was in on that.  I recently heard of
a paper that claimed a large amount of synchronicity in neuron firings.
I don't remember the author.  I'll send you email if I remember.

>i believe that a system clock would be more of a hindrance that a help.  
>studies with central pattern generators and pacemaker activity (re: the heart)
>show clearly that system clocks are not unavailable.  if evolution had found
>a neural system clock advantageous, one could have been created.  i feel,
>however, that the continuous-time evolution of neural systems imbues them
>with their remarkable properties.

You are entitled to your opinion.  You are reasoning by analogy here.
Could there REALLY be a wetware system clock?  You may be missing
implementation details that make it impossible.  For example, could
the correct period (milliseconds) be achieved?  And could it be
communicated reliably and in small hardware to all neurons?
I think the remarkable properties of neural networks come from other
sources; or perhaps we have different definitions of "remarkable".

Here is another way of looking at it.  When one neuron fires and its
neighbour is not receptive (building up charge) there is a fault.
Faults are relatively infrequent (receptive time is larger than
nonreceptive time).  The architecture is fault-tolerant.  That's
why we observe that the brain is fault-tolerant when some of its
neurons are destroyed.  It has to be in order to get around the lack
of system clock.  Neural architectures are better at fault-tolerance
than von-Neumann ones (at least, we can prove this when the thresholding
is physically separated from the summation of weights, as seems to be
the case for biological neurons).

>>2.  Is analog behaviour important?  If I restrict neuron excitation
>>    values to 6 decimal places, will the networks still function
>>    correctly?  More importantly, how does the precision scale with
>>    the number of neurons and/or connections?
>
>I don't think that such a fine level of precision is necessary in neural
>function, i.e. six places would likely be enough.  but since digital circuitry
>is made actaully from analog circuit elements limited to certain regions of
>operation, why go to this trouble in real neural systems when analog seems
>to work just fine?

If six decimal places is enough, then we can model everything as integers.
Why do this?  It is easier to analyze.  Combinatorics is easier than
analysis (despite Hecht-Nielson's claim in the first San Diego NN
conference that the opposite is true).  I don't care if the real neural
systems seem to behave in an analog fashion.  If it seems that the
_computationally important_ things going on are really discrete (and
you seem to have agreed that this is the case), then our model should
reflect this.  I'm not necessarily saying that we should _build_ them
that way.  That's another question.  But perhaps we ought to _think_ of
them that way.  To use an analogy, we don't usually think of a computer
as having infinite memory, but it certainly helps to program them as
if it were the case.  For a complexity theorist, infinite means "adequate
for day-to-day use".  This is where the classical attack on theoretical
computer science (my TRaSh-80 is not a Turing machine) breaks down.

I think that, despite the bad press that theoretical computer science
gets from some NN researchers (I've heard many unprofessional statements
made in conference presentations by people who should know better),
complexity theory has something to contribute.  So do other disciplines.
I'm just a little tired of people closing doors in my face.  It has
become fashionable to disparage TCS (following the bad examples mentioned
three sentences ago).  Sorry if my knee-jerk reaction to your posting
was a little harsh.
-------------------------------------------------------------------------------
			Ian Parberry
  "The bureaucracy is expanding to meet the needs of an expanding bureaucracy"
  ian@theory.cs.psu.edu  ian@psuvax1.BITNET  ian@psuvax1.UUCP  (814) 863-3600
 Dept of Comp Sci, 333 Whitmore Lab, Penn State Univ, University Park, Pa 16802