Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!uwm.edu!rutgers!aramis.rutgers.edu!athos.rutgers.edu!nanotech From: josh@cs.rutgers.edu Newsgroups: sci.nanotech Subject: Review of Penrose's review of Moravec Message-ID: Date: 22 Jan 90 22:03:46 GMT Sender: nanotech@athos.rutgers.edu Lines: 124 Approved: nanotech@aramis.rutgers.edu There is a review in the Feb 1 New York Review of Books by Roger Penrose of Hans Morevec's Mind Children. What follows is a review of that review, and my two cents on the subject of the "AI controversy". * * * * * * * It used to be an item of the conventional wisdom, not so long ago, that there were tribes of savages living in various unexplored regions, who took great exception to being photographed. They believed, or so the story went, that the camera would capture their souls, and that various dire consequences would result. Thus far have we advanced: Penrose appears to have taken great alarm over the prospect of the far more sophisticated recording and duplication process Moravec advocates, precisely because he is afraid that it *won't* capture his soul. Penrose claims that there are three cases to consider, first, the "Strong AI" position with which he labels Moravec; secondly, the anti-AI position of Searle et al, and finally his own hypothesis that consciousness consists of the brain's performing some non-computable calculation with the aid of some unexplained phenomenon of quantum mechanics. Penrose takes Moravec to task for failing to consider the question of AI in his book; Moravec does essentially assume his position and build from there. However, it seems a bit thick for Penrose to object, given the extremely shallow coverage of substantive recent AI work in his own book, where that was putatively the point at issue. What is the "Strong AI" position and why should Penrose be worried about it? (Moravec apparently isn't worried about it.) There are various formulations, and it usually part of a position that is taken in objection to expostulations by an AI researcher. However, this aspect needn't concern us here, because the distinction implied by the phrase "Strong AI" is only significant if the AI researchers succeed! Suppose they do succeed, and write a program that talks, and if in control of a robot, acts, as if it had feelings and intentions and free will and consciousness. Strong AI is the contention that it then has feelings, intentions, will, and consciousness by definition. If no such "Turing test" program appears, the question is moot. Searle illustrates his claim that "Strong AI" is false by the famous Chinese Room example. Searle, in a gedanken experiment, is placed in a sealed room with an instruction book and scratch paper. Through a slot come sheets with odd symbols; Searle follows the instructions through a lengthy and apparently meaningless calculation and issues other symbols through the slot. It turns out that he has been asked a question in Chinese and has produced an appropriate answer in the same language. How does this refute "Strong AI"? Searle claims (a) he doesn't understand Chinese, (b) books, scratch paper, and rooms don't understand Chinese, and therefore (c) no understanding of Chinese has taken place. And, presumeably, if a mere symbol manipulation system isn't understanding, it surely isn't feeling or intending or any of those things. Searle is using a classical technique of argumentation known as "begging the question". He is trying to establish that mere symbol manipulating systems cannot "truly understand" but only appear to. He does this by exhibiting such a system that in common experience is not thought of as capable of understanding, but of course in common experience is also incapable of the question-answering task he ascribes to it. By this device Searle attempts to induce us to assume what he has set out to prove: that the symbol-manipulation system cannot "truly understand". What is the difference between a mountain and a molehill? Is it qualitative or merely quantitative? One can argue that both are elevations of the surface of the Earth, varying merely in size; but one can also argue that mountains can be ascribed properties-- grandeur in the seeing, hazard in the crossing-- that molehills simply do not have in any degree. Let us call this kind of quantitative difference that makes a qualitative one, a *quantum* difference. Let us note that, as in the case of mountains and molehills, a quantum difference is particularly to be noted where the varying property, in this case size, is moved completely across the commonly encountered human scale. There is a quantum difference between the kind of systems we are used to seeing implemented by people using pencil and paper, and a symbol manipulation system of the scale necessary to claim "understanding" of Chinese or any other human tongue. Using Moravec's figures, some personal experimentation, and assuming that only 1 tenth of the brain is used in understanding language, we can estimate that it would take Searle over 60,000 years to answer a simple question like "Which way to the men's room?" Thus Searle's example, whose intent is to show us like processes and have us adduce like properties, would have us equilibrate phenomena whose quantitative difference is that in size between a grain of sand, and the sun. I believe that Penrose himself has gotten hung up on the same "quantum" difference, though at a more reasonable level. The more you study computers, the more they don't look like minds. Indeed, if Moravec is right, a Sun-4 should bear the same resemblance to the human mind that a molehill does Mount Everest. Penrose appears unsettled by the prospect of "surrendering our superiority" to more- than-human robots; one feels silly bowing to molehills, but there is no discredit in being awed by a mountain. Penrose paints two pictures, one he believes horrific in which Searle is correct, but AI succeeds, and we all upload into machines which are not conscious. The machines continue to act in grotesque parody of real people, but true consciousness has died. He doesn't actually believe this, however, but rather that because (as he contends in tENM) consciousness depends on non-computational properties of QM, "computers will never be able to achieve genuine understanding, insight, or intelligence, [and that] human beings will [continue to] supply the guidance, the motivation, and the 'being' of society." This I can live with. Searle's position poses no terror for me as a candidate uploader; it is too obviously the rationalization of an anthropocentric epistemology. Penrose, on the other hand, has simply posed a testable hypothesis: if we can build 'em smart, he's wrong. --JoSH