Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!rutgers!ucsd!ogccse!orstcs!ube.CS.ORST.EDU!ruffwork
From: ruffwork@ube.CS.ORST.EDU (Ritchey Ruff)
Newsgroups: comp.software-eng
Subject: Re: Information on current state of software safety desired
Message-ID: <13097@orstcs.CS.ORST.EDU>
Date: 13 Oct 89 15:45:31 GMT
References: <1321@cs.rit.edu> <195@cherry5.UUCP> <1989Oct4.055359.15145@paris.ics.uci.edu> <10901@riks.csl.sony.co.jp> <1989Oct12.184705.11620@paris.ics.uci.edu>
Sender: usenet@orstcs.CS.ORST.EDU
Reply-To: ruffwork@CS.ORST.EDU (Ritchey Ruff)
Organization: Oregon State Univ. - CS - Corvallis, OR
Lines: 47

Nancy Leveson writes:
}[...] But we do need to learn
}from accidents that occur so that they do not happen again for the same reasons
}in the future.  
}[...]
}Accidents will happen; there are no such things as risk-free systems.  But 
}we need to be able to say that the systems we build are as safe as it is 
}possible to make them given the current state-of-the-art knowledge.  And we
}need to stand up and argue against using computers to control systems when
}even this state-of-the-art knowledge is not adequate to provide acceptable
}risk, especially when, as is often the case, the primary reason for introducing
}computers into these systems is to save money.

Two must-read books on engineering and safety in general:
	(1) To Engineer is Human: the role of failure in successful design,
	    Henry Petroski, NY:St. Martin's Press, 1985. ($10.95 USA).

	(2) Normal Accidents, Charles Perrow, Basic Books, 1984. ($11.95 USA).

(1) points out something we all know but don't tout too much: you
only learn limits from failures, and no matter how careful you are
when you do something never done before it's impossible to know it
will work as you think until you try it out.  Each new piece of
engineering is an experiment! (boy, would Joe Public have fun with this
one ;-)  All you can do is be as careful as humanly possible and
be prepared to learn if it fails.

(2) points out that one major source of failure is non-linearity
in the coupling of different parts of a system (or between systems),
especially as the systems become too complex for a single person to
understand fully.  (by non-linear coupling I mean the failure of part a
and part b together is MUCH worse that the failure of part a or part
b added together).

So, what's the point?  I think that one has to try to stick to
"appropriate technology" (32 bit, 1Meg computers to control a simple
car motor is what I'd call "inappropriate").  Mechanical systems are much
better understood and often are much easier to analyze w.r.t. error
and failure modes---if a simple mechanical system does the job, why
use a complex electronic system? (money...)  Also we have to
try to decouple interactions as much as possible (making analysis
of error and failure modes easier) and try to make sure interactions
are as linear as possible.  Finally we must accept that things will
fail and be willing to accept the results of the failure, and LEARN
from that failure.

--ritchey ruff					ruffwork@cs.orst.edu