Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!uwm.edu!uakari.primate.wisc.edu!gem.mps.ohio-state.edu!usc!orion.oac.uci.edu!uci-ics!nancy From: nancy@ics.uci.edu (Nancy Leveson) Newsgroups: comp.software-eng Subject: Re: Information on current state of software safety desired Message-ID: <1989Oct12.184705.11620@paris.ics.uci.edu> Date: 12 Oct 89 18:47:05 GMT References: <1321@cs.rit.edu> <195@cherry5.UUCP> <1989Oct4.055359.15145@paris.ics.uci.edu> <10901@riks.csl.sony.co.jp> Sender: news@paris.ics.uci.edu (Network News) Reply-To: Nancy Leveson Organization: University of California, Irvine - Dept of ICS Lines: 95 In reading the responses of both Norman Diamond and Neal Murphy to my posting about the Therac, it appears that I have been unclear. I meant to say that the design of the linear accelerator itself (as opposed to the design of the software controlling parts of it) is not the responsibility of the software engineer but of the nuclear, mechanical, electronic, system, or other engineer that designs the linear accelerator hardware. Of course the software engineer or anyone else that sees a flaw in the hardware should speak up. But the software engineer is usually not involved in the actual design of the hardware being controlled by the computer so should not take the blame for any deficiencies in it. Accidents are usually the result of multiple failures and design deficiencies. It does not help to point the finger at someone else and say "the hardware should have been built to compensate for all possible software errors." The Therac should have included hardware interlocks; these have now been added. But most systems are such that it is not possible to build hardware to protect against every incorrect software behavior. Furthermore, hardware interlocks can also fail. For systems that can potentially kill people, it is necessary that the software be as safe as possible (which means building special safeguards into it and using special hazard analysis techniques on it) as well as building in special hardware protection devices when possible. Unfortunately, what I see continually is the hardware people assuming that the software will be correct and thus not building in the interlocks and the software people assuming that the hardware will provide protection against software errors and not performing the extra software engineering procedures necessary for safety-critical software (as partially outlined in my Computing Surveys paper). BOTH need to be done in order to reduce the risk of killing people as much as possible. Neal Murphy writes: >A terrible tragedy it is. But I will not persecute the designers of the >system. I imagine they feel pretty rotten anyway... What is done, is done. >What can we learn from our mistakes? We are only human. We WILL make >mistakes. I will never claim that any software I create will be so perfect >that it won't need to be independently checked. Try as I might to avoid >them, I DO make mistakes. I HAVE hurt people in my lifetime. And I have >felt that hurt. >All we can do is our best. And when that best is not good enough, we hurt, >because we realize that we could have done better. Those of us who care >stand up and accept responsibility for our actions. To mis-quote the Good >Book, "Let him, who has made no mistakes, cast the first stone." If there >were more of us working together instead of attacking each other, mayhap we >would make some progress. No? I do not want to persecute anyone or to point blame. But we do need to learn from accidents that occur so that they do not happen again for the same reasons in the future. In too many instances, people are building software that is safety-critical who should not be. I am sure that they are doing their best, but they are unqualified to do the job. Would we say that a person involved in an auto accident who does not know how to drive and does not possess a driver's license is "only human" and did his best? It is not a matter of casting stones; it is a matter of ensuring public safety by requiring only the best practices on projects that can endanger innocent people. Norman Diamond says: >The suggestion that software engineers do not have this responsibility >is also hard to understand. Morally we should behave with a certain >amount of responsibility too, and ask "what if...". And we are >certainly obligated to test our code.... >So why did you say that software engineers >did not have responsibility in this particular case? Let me attempt to be very clear. I did not say that software engineers do not have responsibility or that they did not have responsibility in this particular case. I said that they did not have responsibility for any deficiencies in the design of the linear accelerator itself. That is the responsibility of the people who designed it. The software engineers have responsibility for the safety of the software that provides control instructions to that linear accelerator. Everyone has the responsibility to communicate with each other and with the system safety engineers to make sure that all parts of the system working together will be as safe as possible. Relying on testing of software or independent review by someone else to ensure the safety of software is inadequate. Building safety-critical software in the same way you would build other software is inadequate. Special effort and techniques must be employed by both the software engineers and system engineers to minimize risk. Software engineers who work on such projects should be highly qualified in basic software engineering and in these special software safety techniques. Accidents will happen; there are no such things as risk-free systems. But we need to be able to say that the systems we build are as safe as it is possible to make them given the current state-of-the-art knowledge. And we need to stand up and argue against using computers to control systems when even this state-of-the-art knowledge is not adequate to provide acceptable risk, especially when, as is often the case, the primary reason for introducing computers into these systems is to save money. nancy -- Nancy Leveson