Path: utzoo!attcan!uunet!bu.edu!rpi!zaphod.mps.ohio-state.edu!uakari.primate.wisc.edu!caen!marky
From: marky@caen.engin.umich.edu (Mark Anthony Young)
Newsgroups: comp.ai.philosophy
Subject: Re: Searle, Strong AI, and Chinese Rooms
Summary: ELIZA never passed the Turing Test and never will
Message-ID: <1990Dec2.070746.18705@engin.umich.edu>
Date: 2 Dec 90 07:07:46 GMT
References: <15798@venera.isi.edu> <1990Nov30.231103.17041@cs.umn.edu> <15878@venera.isi.edu>
Sender: news@engin.umich.edu (CAEN Netnews)
Organization: University of Michigan Engineering, Ann Arbor
Lines: 71

In article <15878@venera.isi.edu> smoliar@vaxa.isi.edu (Stephen Smoliar) writes:
>
>On the other hand, David is quite right that a computer which is mistaken for
>a human is not necessarily a "winner" at Turing's original imitation game.  I
>would guess, however, that he would have been content to accept an example of
>ELIZA being confused for a human as a reasonable alternative solution to his
>original problem.  After all, the scenario is not that different:  Humans are
>communicating through typewriters, and the question is one of whether or not
>a computer could be successfully substituted for a human.  
>
While it's largely fruitless to argue about what someone would or would not
have been content with, I have to disagree with the statement that ELIZA's 
being taken for human constitutes a "reasonable alternative solution".  While
the surface structure may be similar to the Turing Test, ELIZA's "test" is
missing the two most important parts:

	(1) Direct comparison of human and non-human.  The subject must be 
		aware that it is possible s/he is talking with a non-human.
		Otherwise the natural assumption is that one is talking to 
		a human (this is the assumption we all make on the net).  
		Only when it becomes usual to talk with non-humans will this 
		assumption go away.  

		The direct comparison is important because research shows
		that raising suspicions about lying doesn't increase 
		accuracy in detecting lies, it only makes people more 
		suspicious of everyone (Toris & DePaulo, JPSP 47-5, 1985).  
		By always having one truth-teller and one liar, the
		interviewer can concentrate on differences between the
		interviewees, and our measurements will thus be more
		meaningful.

	(2) The non-human must be able to fool a significant proportion of
		people, not simply a few here and there.  There will always
		be a part of the population that has no idea how to tell
		a (simple but clever) computer simulation from the real
		thing.  These people will be reduced to guessing, and the
		non-human will get half of them simply by chance.

		When we say significant proportion, we must have some
		comparable task carried out by humans (known intelligence).
		The humans' rate of success here sets the base rate
		against which the non-human is measured.  Turing
		suggested the Imitation Game (man pretending to be a
		woman) as a comparable task for humans.

I have often heard it said that ELIZA passed the Turing Test (or a version
thereof).  I've heard two stories describing this amazing feat.  In one a
person insisted that there must be someone on "the other side," otherwise
who were they talking to?  The other story involved a person who asked
someone else to leave as the conversation with ELIZA was getting personal.
In the second case, it's not even clear that ELIZA was mistaken for a 
person.  In the first, the simple rejection that anything but a person was
even possible seems the best explanation.

Tests with PARRY come closer to the Turing test (though still not there).  
The only version I've seen in an actual journal (sorry, I can't remember
where) involved having psychiatrists rate transcripts for degree of
paranoia.  PARRY did rather well, scoring "mildly paranoid" in its "low"
setting and "very paranoid" in its "high" setting, nicely bracketing the
actual paranoids used as control.  Psychiatrists were not told, however,
that they might be reading a transcript generated by computer.  

Apparently there was a later version of this experiment in which the
psychiatrists were able to interview PARRY, and were actually told that
it might be a computer on the other end.  If anyone has any information
on this experiment I'd be interested in seeing it.  I'd be particularly
interested in knowing whether any base rate measures were taken, and, if
so, how PARRY compared to humans in their task.

...mark young