Path: utzoo!attcan!uunet!lll-winken!lll-tis!helios.ee.lbl.gov!pasteur!ucbvax!bloom-beacon!IRISHMVS.BITNET!GKMARH
From: GKMARH@IRISHMVS.BITNET (steven horst 219-289-9067)
Newsgroups: comp.ai.digest
Subject: Does AI kill? (long)
Message-ID: <19880726053140.1.NICK@HOWARD-JOHNSONS.LCS.MIT.EDU>
Date: 26 Jul 88 05:31:00 GMT
Sender: daemon@bloom-beacon.MIT.EDU
Organization: The Internet
Lines: 73
Approved: ailist@ai.ai.mit.edu

X-Delivery-Notice:  SMTP MAIL FROM does not correspond to sender.
Date: Sat, 23 Jul 88 18:36 EDT
To: ailist@ai.ai.mit.edu
From: steven horst 219-289-9067 <GKMARH%IRISHMVS.BITNET@MITVMA.MIT.EDU>
Subject: Does AI kill? (long)

I must say I was taken by surprise by the flurry of messages about
the tragic destruction of a commercial plane in the Persian Gulf.
But what made the strongest impression was the *defensive* tone
of a number of the messages.  The basic defensive themes seemed to be:
  (1) The tracking system wasn't AI technology,
  (2) Even if it WAS AI technology, IT didn't shoot down the plane
  (3) Even if it WAS AI and DID shoot down the plane, we mustn't
      let people use that as a reason for shutting off our money.
Now all of these are serious and reasonable points, and merit some
discussion.  I think we ought to be careful, though, that we
don't just rationalize away the very real questions of moral
responsibility involved in designing systems that can affect (and
indeed terminate) the lives of many, many people.  These questions
arise for AI systems and non-AI systems, for military systems and
commercial expert systems.
     Let's start simple.  We've all been annoyed by design flaws
in comparatively simple and extensively tested commercial software,
and those who have done programming for other users know how hard
it is to idiotproof programs much simpler than those needed by the
military and by large private corporations.  If we look at expert
systems, we are faced with additional difficulties: if the history of
AI has shown anything, it has shown that "reducing" human reasoning
to a set of rules, even within a very circumscribed domain, is much
harder than people in AI 30 years ago imagined.
     But of course most programs don't have life-and-death consequences.
If WORD has bugs, Microsoft loses money, but nobody dies.  If SAM
can't handle some questions about stories, the Yale group gets a grant
to work on PAM.  But much of the money that supports AI research
comes from DOD, and the obvious implication is that what we design
may be used in ways that result in dire consequences.  And it really
won't do to say, "Well, that isn't OUR fault....after all, LOTS of
things can be used to hurt people.  But if somebody gets hit by a car,
it isn't the fault of the guy on the assembly line."  First of all,
sometimes it IS the car company's fault (as was argued against Audi).
But more to the point, the moral responsibility we undertake in
supplying a product increases with the seriousness of the consequences
of error and with the uncertainty of proper performance.  (Of course
even the "proper" performance of weapons systems involves the designer
in some moral responsibility.)  And the track record of very large
programs designed by large teams - often with no one on the team
knowing the whole system inside and out - is quite spotty, especially
when the system cannot be tested under realistic conditions.
     My aim here is to suggest that lots of people in AI (and other
computer-related fields) are working on projects that can affect lots
of people somewhere down the road, and that there are some very
real questions about whether a given project is RIGHT - questions
which we have a right and even an obligation to ask of ourselves, and
not to leave for the people supplying the funding.  Programs that
can result in the death or injury of human beings are not morally
neutral.  Nor are programs that affect privacy or the distribution
of power or wealth.  We won't all agree on what is good, what is
necessary evil and what is unpardonable, but that doesn't mean we
shouldn't take very serious account of how our projects are
INTENDED to be used, how they might be used in ways we don't intend,
how flaws we overlook may result in tragic consequences, and how
a user who lacks our knowledge or uses our product in a context it
was not designed to deal with can cause grave harm.
     Doctors, lawyers, military professionals and many other
professionals whose decisions affect other people's lives have ethical
codes.  They don't always live up to them, but there is some sense
of taking ethical questions seriously AS A PROFESSION.  It is good to
see groups emerging like Computer Professionals for Social
Responsibility.  Perhaps it is time for those of us who work in AI
or in computer-related fields to take a serious interest, AS A
PROFESSION, in ethical questions.
     --Steve Horst     BITNET address....gkmarh@irishmvs
                       SURFACE MAIL......Department of Philosophy
                                         Notre Dame, IN  46556