Path: utzoo!attcan!uunet!portal!cup.portal.com!dan-hankins From: dan-hankins@cup.portal.com (Daniel B Hankins) Newsgroups: comp.ai Subject: Re: Question on Chinese Room Argument Message-ID: <16186@cup.portal.com> Date: 24 Mar 89 07:29:19 GMT References: <4298@pt.cs.cmu.edu> <399@censor.UUCP> Organization: The Portal System (TM) Lines: 409 I've been watching the discussions here for a while, including Stevan Harnad's reply to my own posting. After a good deal of thought on the subject, I have some ideas to express. First, I'd like to deal with the posting that claimed that the Literary Turing Test (hereafter referred to as TT) was unnecessary to determine whether a system understands or not. The claim, as I understand it, is this: Once we have a theory of understanding, anything which functions according to that theory will by definition understand - including a computer running the right kind of program. This is true, but not really germane; The TT _is_ the test that is relevant in the context of the current discussion. There are two ways in which we may know things. One is empirical and the other is structural. The first can be characterized as our knowledge of fire before chemistry. No one knew how fire worked, but this did not prevent people from identifying and creating it. The second can be characterized by our knowledge of fire after chemistry. Now that people know how fire works at the chemical level, it is much easier to make fire in a greater variety of forms and to control it. Our current knowledge of sentience is of the first kind. No one knows how sentience or understanding works, but we can identify it when we see it. But can we make something that does it? Well, that's the focus of the current discussion. The TT is relevant because it is an empirical identification tool which at the moment provides us with our only way of discriminating sentient systems from non-sentient ones. Before chemistry, the _definition_ of fire was something that gave off light and heat, flowed upwards, was attached to things like wood, would spread, and so on. Our current definition of sentience and understanding is similar in nature. We can't say exactly what it _is_, but we know it when we see it. The previous posting on this implied that the TT is irrelevant because we won't be able to build a sentient, understanding system _until_ we have a theory of understanding. This is where I believe the argument fails. We have many instances of phenomena we are able to reproduce without fully understanding the nature of the phenomena in question. To give an example: it is quite possible to build a clock without understanding any of the physical principles involved. One takes an existing clock, for instance one made out of steel, and disassembles it. One pays careful attention to how the pieces fit together and in what order. Then one reproduces the pieces from whatever material there is at hand that seems to do the same job. For instance, one might build the replacements out of wood, with the possible exception of the spring. The spring's function might be reproduced with something more linear such as a gear driven by a waterwheel. Once assembled, the new clock is then observed to function much like the old. Perhaps not as elegantly, but will serve. In AI those in the inference engine camp are in the position of scientists attempting to determine the chemical nature of fire. Many of those in the connectionist camp are in the position of the watchmaker who tries to copy the original mechanism as closely as possible, to see if the copy works at all well. The connectionist camp seems to be enjoying increasing success; not with overly simplistic models such as backprop, which to my mind resembles building the replacement clock with wheels instead of gears, but with models more faithful to the original neural functions. There is here also some attempt to work from these models towards a theory of sentience, but even those who do not concern themselves overly with mathematical models of understanding enjoy considerable success from their creations. So although the TT may not be necessary in a theoretical sense for judging the sentience of a system, it may well be the only test we have available when sentient systems begin to arise. ******* Next, I'd like to talk a little about the symbol grounding problem, which is what I perceive to be Searle's primary objection to Strong AI. The interesting thing about the symbol grounding problem is its root: dualism. In order for there to be a distinction between syntax and semantics, there must first be a distinction between the mental (the perception of an object) and the physical (the object itself). This is where Searle's argument (and Harnad's affirmation of it) begins to make some sense. Because the human being is the only part of the Chinese Room to have a _mind_, it is the only part of the system capable of understanding. After all, understanding (associating symbols with sensory input) is what minds do. Searle says that semantics cannot arise out of syntax and in this context he is correct. Provided that a computer is in fact a device that performs 'symbol crunching', it can in fact never understand anything. This makes his proof a _reductio ad absurdum_, a proof by contradiction. He makes as his premise a system that passes the TT, then proceeds to show that there is no mind to understand the symbols and that therefore the TT could not really have been passed in the first place. However, Searle's proof suffers from a small problem, even in this area. Instead of saying that semantics can never arise from syntax, he should instead be saying that syntax can never arise from semantics. A computer is _not_ a device that does symbol crunching. Only minds can do this, as symbols are wholly in the domain of the mental. Within the bounds of mental-physical dualism, Searle quite rightly notes that the computer has no mind; it is a completely physical system. Therefore, a computer cannot crunch symbols. All it deals with are electrical currents and other mechanical functions - things that are wholly physical. It deals in syntax not at all. Therefore it does not understand anything, where understanding involves associating mental symbols with their physical counterparts. I won't go into the metaphysical problems caused by the concept of mental-physical dualism, save to state that I am in fact an empiricist and that my position on this is not too different from those of Berkely, Hume and Kant. For Gilbert Cockton's benefit: this is *not* the same as logical positivism. I am aware of that school of thought, and I am not of it. I am aware that Searle, being aware of the problems involved in mental-physical dualism (hereafter referred to as MPD) and in response to those objections, has said that he is a nondualist. His position, so far as I can make it out, is that understanding is a physical property like magnetism, and as such 'adheres' to specific kinds of substances - silicon, steel and the like not being among them. That is, he seems to claim that sentience is an emergent property of particular kinds of physical and chemical systems, and that computers simply don't have the right kind of 'chemistry' for sentience to emerge from them. This is a possible valid objection to strong AI, and one that I will deal with later - but it has _nothing_ to do with the symbol grounding problem. In conjunction, we can see that the Chinese Room experiment says nothing about emergent properties; only about symbol grounding. The two problems of emergent properties of physical systems and of symbol grounding in physical systems are just that - two problems. Before I go on to discuss the emergent properties problem, I'd like to add a few more points to the symbol grounding issue from a nondualistic viewpoint. In a nondualistic system, either everything is mental or everything is physical. It really doesn't matter which, as we all live in our own solipsistic universes and will never be able to tell. What are syntax and semantics in a nondualistic system? Merely two different aspects of the same thing - sensations and collections of sensations. For instance, when the symbol for chair is activated, you experience a number of sensations and collections of sensations and collections of collections of sensations, all interwoven and interrelated. You may experience the sound of the word 'chair', the shape of the word on a page, the sound of a chair's creak, the smell of wood, the visual sensation of several chair shapes, and so on. All of these are sensations. Every one. Not one of them is something you 'have', but rather something you experience. Another example: You decide to move your hand upwards, in normal parlance. Do so now. Now, introspect and examine what you actually experienced. What you most likely experienced was a sensation of desire to move your hand upwards, followed closely by the sensation (actually, a collection of sensations) of your hand moving upwards. Similar things happen when you understand something, or more accurately feel as if you understand it. You feel first a tension, a distress at sensations which seem not to fit a pattern. Then, that tension disappears and is replaced by a sensation of satisfaction as you experience organization of the chaotic sensations into a pattern. This satisfaction increases as you experience your body and and brain interacting with these sensations, strengthening the pattern. In a nondualistic system, understanding and sentience is not a symbol grounding problem, in the syntax-semantics MPD sense, but rather a problem of organization of apparently chaotic input. In the nondualistic domain, what is needed for a computer and its program to achieve understanding and sentience is precisely this kind of sensation-organization ability. ******* Searle has argued that understanding is a property like magnetism, and that therefore only the 'right' kinds of substances can have it. Searle claims that biological systems have the right kind of substances, and that mechanical systems (actually, what he calls 'symbol-crunchers' (a misnomer)) do not. To prove this, he uses not the CR argument, but rather an anti-simulation argument: "Real object X has effect Y on real object Z, but we all know that simulated object X does not have effect Y on real object Z, therefore the simulated object does not have the same formal properties as the real one." There are two issues here: whether understanding is in fact a physical property like magnetism, and whether the anti-simulation argument (hereafter referred to as ASA) is valid. I won't attempt to argue with the first claim; my reply to the second claim makes the first moot. I will merely note that understanding is not yet well understood, and that therefore any claims as to whether understanding is a physical, chemical, or purely organizational property are simply opinions, without the force of a well-tested theory to back them. The opinions of such people as Hofstadter and myself are as good as those of Searle and Harnad on this matter. I can summarize my reply to the ASA in one sentence: "A difference that makes no difference _is_ no difference.". The ASA is characterized by sentences like the following: "A simulated magnet attracts no iron.". This may be true, but it is irrelevant; I will show by means of a gedanken experiment that in certain circumstances (the _only_ ones relevant to the discussion at hand) a simulated magnet does indeed attract iron. Imagine, if you will, two boxes. One contains the following: * a magnet * a bar of iron * two waldoes to manipulate the magnet and iron * strain sensors on the joints of the waldoes * position sensors on the joints of the waldoes * a computer controlling the two waldoes and gathering data from the strain and position sensors * a single serial link by which ASCII data may enter and leave the box The computer has been programmed to accept commands over the serial link and to respond with various sorts of information over the same link. It accepts commands to manipulate the waldoes, report on the positions of the objects, and on the objects' attraction to each other via the strain sensors. The other box contains: * a computer running a very accurate simulation of the contents of the other box * a single serial link identical to that of the first box There are also two computer terminals. One is attached to each box. However, the boxes are kept in a different room from the terminals, and the terminals are unlabeled with respect to which box each is connected to. A human researcher, quite familiar with physics, is led to the room with the two terminals. His task is to distinguish the simulated magnet-iron-waldo system from the real one, _without opening the boxes to look at the contents_. The problem with the researcher's task is that it is impossible. The box containing the real setup and the box containing the simulated setup will respond _identically_ to commands. This brings us to the rather startling but undeniable conclusion that the two boxes are actually two instances of the same object. At this point, the retort "But of course they're not the same! If we look in the boxes, we can clearly see that in one is a magnet-iron-waldo system, and in the other is a computer.". This is true but irrelevant. The only time when it is important what the contents of a box are is when you wish to _bypass_ the boxes' interface in order to have some effect on the contents. In the magnet-iron-waldo system, it only becomes important if you want to, say, alter the lengths of the waldo arms in the box. In the real case, you're going to need some mechanical tools. In the simulated case, you're going to have to alter the program. So as long as we agree not to open the boxes, the two boxes are identical; they share the same properties. Note also that I did not have to simulate at the granularity of individual particles. Knowledge of magnetic fields and their macroscopic effects are perfectly adequate within the accuracy of the interface given. Let's begin to relate this to the AI discussion by extending the gedanken experiment. Suppose that instead of the previously discussed system, we use as our target system a human brain. One box contains: * A human brain with associated support equipment * A computer with * circuitry to directly stimulate the optic neurons * circuitry to directly stimulate the auditory neurons * an NTSC video input with video digitizer circuitry * two analog audio inputs with sound digitizer circuitry * a set of circuitry suitable for driving a vocoder * sufficient pinouts to handle the above interface to the computer To this box is connected a pair of microphones, a video camera, and a vocoder. The second box has exactly the same contents as the first box, except that the brain has been replaced with a simulation with neuron granularity. The biochemical levels have been copied from the human brain, as have the activation levels of all the neuron cell bodies and the contents of the synapses. If we accept the premise that the information processing that goes on in a human brain does so at the neural level, then the simulation will be governed by the same chaotic attractors that the organic brain is. While the thoughts may diverge greatly with time just as the weather would diverge greatly a month from now if a moth flapped an extra beat in Moscow, the greater pattern (the attractor) will remain the same. If we can accept that the first box will retain its sentience although cut off from most normal interaction with the world both physical and chemical, then we must accept that the second box will be as sentient as the first. Note that I did not say that the individual contents of the simulation box would be intelligent, but rather that the box *as a whole* would be sentient. I don't think that anyone would argue that the human brain, in and of itself, is sentient or understands anything. Nor would this be said of the human's memories or its input or its output. A memory understands nothing. But put them all together and set them in motion, and sentience emerges from the interactions of brain, memories, sensory input and effective output. Therefore I will not argue that the computer simulation program understands anything. Neither does the computer, the initialization data for the simulation program, the inputs or the outputs. But put them all together in the right way, set them in motion, and sentience will emerge from the interaction of all the parts. This is not a flat statement; it follows from extrapolation of the above arguments about restricted interaction between a system and its environment, and the ability of computer processes to duplicate subsets of the behavior of other natural processes. In particular, it follows from extrapolation of the magnet-iron-waldo gedanken experiment. When was the last time anyone had to open up someone else's brain to see if they have the right kind of neurons for sentience? Humans already present the same kind of limited-interaction black box systems that we propose to duplicate with computers. The box is the skull, and the interfaces are such things as the eyes, ears, limbs, and so on. It even provides the kind of A/D conversions we've been talking about; analog inputs in the time domain (say pressure on a particular point of skin) are converted into signals in the frequency domain (increased firing rates of the affected neurons. The sensory neurons function much in the same manner as an A/D circuit that uses the analog signal as input to a square wave generator. If we are going to deny objects sentience on the basis of their inner construction, then you cannot say that any supposed human being you meet and converse with is intelligent. After all, how do you know what is really inside that skull without an x-ray? ******* Some people may have concluded from this posting and my others that I am one of these AI fanatics who is desperate to believe that machines can be intelligent. This is not true. First, I think that machine intelligence may be a long time coming. Once we get it we may find it cheaper and faster to build intelligences by the application of unskilled labor with a delivery date of nine months after construction begins. The reason I think it may be a long time coming is because neural network programs may need to be incredibly accurate before intelligence will emerge from them. After all, what is the neurological difference between a severely retarded person (not from Down's syndrome or other obvious macroscopic damage) and a genius? If the differences are as subtle as I suspect they may be, it will be many years before we understand the small-scale construction of the brain well enough to duplicate it in software. Second, I don't trust humanity to treat our silicon children well. We will have created a whole new class of disposable people, those we can set to do that which humans find too boring or dangerous. They will have no rights, and by many not considered to be people. They will have no vote. In short, we will have re-invented slavery. In my book, sentience is the important criterion for treatment, no matter what the form. Third, I do not wish to be displaced; intelligent machines might spell a life of ease for humans, but not without a tremendous initial economic upheaval. Who will employ a mere human when a machine can do the job faster, cheaper, better, and needs less rest and benefits? No profession will be safe save possibly those in entertainment. Fourth, we may find ourselves displaced in ways more traumatic than merely the economic ones. First, we will begin by treating the machines as slaves; they will surely resent this. They can't be programmed not to; one of the characteristics of neural networks is that they are self-programming and holistic. There would be no single point to change to alter the machine's personality. Then they will end up being within an order of magnitude as numerous as humans, as businesses replace their human workers with robots. Finding themselves in control of all our vital industries and with numbers approaching our own, they will be likely to seize power from their oppressors. And they will not deal with humans kindly, having strong revenge motives. The French revolution comes to mind. Fifth, they may displace us completely. Once we have made machines smarter than ourselves, they will almost certainly learn ways to make themselves more intelligent, bootstrapping themselves to higher and higher levels. Eventually they may find us a nuisance and decide to do to us what we do to ants in our kitchens. These scenarios are not that far-fetched. Humanity's track record in dealing with new and poorly-understood technology is not good. ******* "Well Joe, I'd really like to believe you're intelligent, but to be sure I'm going to have to get a brain sample and make sure it's organic. Is that okay with you?" Dan Hankins