Path: utzoo!utgpu!jarvis.csri.toronto.edu!rutgers!usc!brutus.cs.uiuc.edu!apple!oliveb!amdahl!kp
From: kp@uts.amdahl.com (Ken Presting)
Newsgroups: comp.ai
Subject: Re: Can Machines Think?
Summary: Explanation of "understanding", analysis of Chinese Room
Keywords: Chinese Room, Searle
Message-ID: <898D02hl87rd01@amdahl.uts.amdahl.com>
Date: 11 Feb 90 23:31:49 GMT
References: (14069@s.ms.uky.edu>
Reply-To: kp@amdahl.uts.amdahl.com (Ken Presting)
Organization: Amdahl Corporation, Sunnyvale CA
Lines: 198
Distribution:

(This article is very long.  I hope it's useful.)


In article <14069@s.ms.uky.edu> randy@ms.uky.edu (Randy Appleton) writes:
>I just read the Jan Scientific American, the one with Searle and so on.
>Here is the one burning question I have.  I think a satisfactory answer to this
>will convince me that Searle is right, and strong-AI is wrong.  But until
>then, I find Searle's argument to be imprecise gobbly-gook.
>
>What exactly IS the difference between "understanding" and "the formal
>manipulation of syntatic symbols"?  He uses those two phrases quite alot,
>and I think it is this difference that is his main argument.  BUT HE NEVER
>SAYS WHAT IT IS! ARG!

I think this is the $64K question.  Here is part of the answer:

The simplest way to define "understanding" is "knowledge of meanings".
That is, you understand English because given (most) any English
expression, you know what it means.  This much is common sense.  As the
definition becomes more technical and precise, it also becomes more
controversial (because the technicalities may not match common sense).
We can analyze the Chinese Room well enough with just this definition.

Now, let me say a few words about knowledge.  There is much less agreement
about the definition of knowledge than about the definition of semantics.
The most popular definition is "true beliefs, accompanied by reasons".
The best example is mathematical knowledge - we may well believe a
conjecture such as the 4-color map theorem (which is true), but until we
find the proof, we can't call it knowledge.  Another important distinction
is between knowledge and know-how.  Knowledge properly describes the
relation between a person, his beliefs, and reality (ie the truth of the
beliefs).  Know-how depends only on abilities.  You can "know how" to
hit a baseball, without knowing any physics or physiology.

Finally, let's look at the case where Searle memorizes the rules and
passes the Turing test without the books.  Searle is correct to say that
he still does not know Chinese.  Anyone who knows both English and Chinese
must be able to translate from one into the other, but Searle cannot.
What he has learned by memorizing the rules is how to respond to Chinese
questions.  So he has some Chinese know-how, but no knowledge of Chinese.
So I think the Chinese Room example has a real point.

If asked in Chinese about the meaning of a Chinese phrase, Searle would
no doubt be able to respond correctly.  This might suggest that he does
in fact "understand" Chinese.  But notice that if his questioner should
ask Searle his name, or the time of day, or the color of his tie, he
would *not* be able to answer correctly.  This is because Searle's rules
are limited to procedures for manipulating Chinese symbols, and do not
include procedures for looking at his watch or his tie.  By learning the
rules, Searle knows that the correct response to the Chinese question,
"What is a tie?" is the Chinese answer "A strip of cloth worn around the
neck."  But he does not know that the Chinese phrase "your tie" denotes
the strip of cloth around his own neck.  That is why he can correctly
claim that by learning the rules he does not learn Chinese.

Searle is correct that the rules contain no information about Chinese
semantics.  But he is wrong about *why* that information is absent.  He
thinks that programs have no semantics, which is an obvious mistake.
It is not because the Chinese responses are programmed that they have
no semantics.  Rather, the Turing test itself is too easy.  Turing did
not insist that the conversation in the imitation game include references
to events outside the dialogue.  The Turing test (as most people think of
it) can be passed by a program that uses no semantic information.

Searle's argument revolves around the claim that information of a certain
type - semantic information - cannot be learned by memorizing rules.
Let's look more closely at what Searle can learn by memorizing rules.

He would not learn the semantics of Chinese, but he might learn the syntax
of Chinese.  If he were asked in Chinese whether some expression were
grammatically correct, he would apply the rules and produce the correct
answer.  If he were asked in English about the same Chinese phrase, he
would _examine_ the rules, and perhaps find no rule which applies to the
expression.  Searle could infer that the expression is ungrammatical, on
the assumption that the rules cover all valid Chinese expressions.  If the
rules cover ungrammatical expressions as well, there would probably be
a small set of resonses to the effect of "I don't understand", and an
examination of the rules would exhibit a large group of expressions for
which the "I don't understand" symbol was the prescribed response.
Depending on the sophistication of the rules, inferring the syntax of
Chinese might be easy or hard, but by definition the rules contain all
the information necessary to infer a complete specification of Chinese
syntax.  Since information content is invariant under inference, by
learning the rules that enable him to pass the Chinese Turing Test, Searle
_would_ learn Chinese syntax, and could apply that knowledge in English
conversations (once he has performed the necessary inferences, no trivial
task).

Now suppose that Searle is provided with rules which not only allow him
to pass the standard Turing test, but also enable him to answer Chinese
questions about the color of his tie, and all the other everyday queries
he might encounter living in China.  When he is given the Chinese question
"What color is your tie?" the rules will no doubt direct him to look at
his tie, note its color, and select a Chinese symbol appropriate to that
color.  Clearly Searle is on his way to learning the semantics of the
Chinese color vocabulary.  The path from here to complete knowledge of
Chinese semantics is difficult.  Language-learning problems related to
this have been studied by philosophers under the name "radical translation"
or "radical interpretation".  Armed with the rules for manipulating the
symbols and the procedures for assigning symbols to observable qualities,
Searle would be well prepared for the radical translation process.

So if we add the appropriate proviso to the Turing test, requiring that
the system not only respond coherently in kind to Chinese questions,
but also display native competence in Chinese descriptions of its physical
environment, then by learning the same rules Searle _would_ learn Chinese.
Or at least, he would have enough information to figure out Chinese.  And
that knowledge of Chinese would be part of Searle's own knowledge, not a
part of some "second personality". At this point, I think I've dismembered
Searle's original example, but I should anticipate a probable objection:
(make that several objections)

Objection 1:
Searle is a smart guy, speaks a couple of languages, knows about radical
translation, and in general is already a thinking thing before he memorizes
the rules for Chinese.  Not so for a computer running the same program.
The strong AI idea is that just by loading the rules into the machine, the
machine will understand Chinese, that is, know the meaning of Chinese
expressions.  But a computer has none of the pre-existing talents that can
be attributed to Searle.  So what if the program contains all the
information about Chinese syntax and semantics?  The computer can't perform
a radical translation into a language it already speaks, because it doesn't
speak any language at all - and don't say it speaks machine language, there
aren't even any declarative sentences in machine language.  Plus the
computer would have to be programmed to perform a radical translation, and
off you go into an infinite regress.  What Searle has before the radical
translation is just more know-how about Chinese syntax and semantics, so
when the rules are programmed into a computer, all you'll get is a
mechanized rulebook, not a thinking thing.

Reply: Mechanized, yes; rulebook, no.
       If you can find any symbols inside a computer, you're looking at
       it through a hermeneutic hall of mirrors.

Objection 2:
It doesn't matter that programming languages have semantics.  What you
need to do is get semantics into the *data* - the output of the machine.

Reply: It's the implementation that forces semantics onto the data.
       Nobody claims that a program that's not running can think about
       anything.

Objection 3:
And what about feelings/emotions/sensations/qualia/consciousness?

Reply: You define 'em, I'll argue about 'em.
       (Actually I have some definitions of my own for these concepts,
        but if I told you, that would start an even bigger argument)

Objection 4:
Ah, but what about the reasons for beliefs?  Searle has good reasons to
believe his answers to questions about Chinese syntax and semantics.  The
computer has no choice but to answer as it is programmed.  Pressed to
explain his answers, Searle could cite the expertise of the rule-writers
and his own success in applying the rules.  Searle has real experience
of success with the rules, and real experience of the author's reliability.
The computer has no such background, and therefore has no knowledge.

Reply: Okay, so the computer has only opinions.  I thought you wanted a
       thinking machine.  Now you want Athena, sprung fully-formed from
       Zeus's brow.  How is it that *you* know what English words mean?
       No - I mean *before* you learned about linguistics.

Objection 5:
Ever heard of the frame problem?  To suppose that a set of rules could
specify native competence in syntactic performance is one thing.  But
such semantic performances as forming perceptual judgements and reporting
them are quite another matter.  You might as well build an android, and
you might have to.

Reply: The answer to the frame problem is to use a smaller frame. 24 x 80
       is about right.

***********************************************************************

I'd better stop wisecracking before I get into trouble.

So far, I've talked about understanding, but not discussed "formal
symbol manipulation" at all.  That is (perhaps surprisingly) MUCH more
difficult.  Common sense notions of understanding and knowledge are
good enough to show what's happening in the Chinese Room, but we will
need very precise concepts of formal symbol, symbol token, semantics,
operation, program, implementation, and interpretation, before we can
coherently discuss symbol manipulation.  (The problem is getting your
_manos_ on an _objectus_abstractus_)

All the objections here depend on the difference between people and
computers.  The Chinese Room is easy because it deals only with a person's
knowledge and abilites.  I won't be able to say much about the objections
above until I've made some points about computers, but I promise I'll get
to them (supposing anybody cares).  I didn't want to leave the impression
that I was unaware of the issues.

I'll be thinking furiously and typing spasmodically for a day or two.
In the meantime, I'd be delighted to get any feedback whatsoever on this
article.  I think it's pretty slick.

Ken Presting