Path: utzoo!utgpu!water!watmath!clyde!att!pacbell!ames!think!bloom-beacon!Xerox.COM!Hoffman.es From: Hoffman.es@Xerox.COM (Rodney Hoffman , Rodney Hoffman) Newsgroups: comp.ai.digest Subject: Re: does AI kill? Message-ID: <19880721201606.2.NICK@HOWARD-JOHNSONS.LCS.MIT.EDU> Date: 21 Jul 88 20:16:00 GMT Sender: daemon@bloom-beacon.MIT.EDU Organization: The Internet Lines: 77 Approved: ailist@ai.ai.mit.edu Sender: Hoffman.es@Xerox.COM Date: Tue, 19 Jul 88 12:13 EDT Subject: Re: does AI kill? To: AIList@AI.AI.MIT.EDU From: Rodney Hoffman The July 18 Los Angeles Times carries an op-ed piece by Peter D. Zimmerman, a physicist who is a senior associate at the Carnegie Endowment for International Peace and director of its Project on SDI Technology and Policy: MAN IN LOOP CAN ONLY BE AS FLAWLESS AS COMPUTERS. [In the Iranian Airbus shootdown,] the computers aboard ship use artificial intelligence programs to unscramble the torrent of infor- mation pouring from the phased array radars. These computers decided that the incoming Airbus was most probably a hostile aircraft, told the skipper, and he ordered his defenses to blast the bogey (target) out of the sky. The machine did what it was supposed to, given the programs in its memory. The captain simply accepted the machine's judgment, and acted on it.... Despite the fact that the Aegis system has been exhaustively tested at the RCA lab in New Jersey and has been at sea for years, it still failed to make the right decision the first time an occasion to fire a live round arose. The consequences of a similar failure in a "Star Wars" situation could lead to the destruction of much of the civilized world. [Descriptions of reasonable scenarios ....] The advocates of strategic defense can argue, perhaps plausibly, that we have now learned our lesson. The computers must be more sophisticated, they will say. More simulations must be run and more cases studied so that the artificial intelligence guidelines are more precise. But the real lesson from the tragedy in the Persian Gulf is that computers, no matter how smart, are fallible. Sensors, no matter how good, will often transmit conflicting information. The danger is not that we will fail to prepare the machines to cope with expected situa- tions. It is the absolute certainty that crucial events will be ones we have not anticipated. Congress thought we could prevent a strategic tragedy by insisting that all architectures for strategic defense have the man in the loop. We now know the bitter truth that the man will be captive to the computer, unable to exercise independent judgment because he will have no indepen- dent information, he will have to rely upon the recommendations of his computer adviser. It is another reason why strategic defense systems will increase instability, pushing the world closer to holocaust -- not further away. - - - - - I'm not at all sure that Aegis really uses much AI. But many lessons implicit in Zimmerman's piece are well-taken. Among them: * The blind faith many people place in computer analysis is rarely justified. (This of course includes the hype the promoters use to sell systems to military buyers, to politicians, and to voters. Perhaps the question should be "Does hype kill?") * Congress's "man in the loop" mandate is an unthinking palliative, not worth much, and it shouldn't lull people into thinking the problem is fixed. * To have a hope of being effective, "people in the loop" need additional information and training and options. * Life-critical computer systems need stringent testing by disinterested parties (including operational testing whenever feasible). * Many, perhaps most, real combat situations cannot be anticipated. * The hazards at risk in Star Wars should rule out its development. -- Rodney Hoffman -------