Path: utzoo!utgpu!jarvis.csri.toronto.edu!cs.utexas.edu!usc!elroy.jpl.nasa.gov!decwrl!shelby!portia!matheson From: matheson@portia.Stanford.EDU (David Matheson) Newsgroups: comp.society.futures Subject: Re: Retrospective Forecasting Keywords: forecast, forecasting Message-ID: <9625@portia.Stanford.EDU> Date: 28 Feb 90 17:09:05 GMT References: <5473@bgsu-stu.UUCP> Sender: David Matheson Reply-To: matheson@portia.Stanford.EDU (David Matheson) Organization: Stanford University Lines: 57 There are (at least) two things to be learned by looking at past forecasts: 1. Knowledge about the domain of interest (e.g. computers and so on) 2. Knowledge about the forecasting procedures. In this second domain, there has been quite a bit of research in the area of probabilistic forecasting. Most of this research (that I know of) comes from psychology. A recent book, "Judgement under uncertainty: Heuristics and biases" by Daniel Kahneman, Paul Slovic, and Amos Tversky provides a nice summary of this literature. Let me site two well-know biases as examples. People are overconfident in their judgements, and tend to anchor their assessments on reference points. If one has to make a numerical assessment, for example, it is likely that first a reference point will be identified (such as a linear extrapolation, current level, or whatever), and then adjusted to account for the future uncertainty. This procedure systematically leads to too little adjustment. Furthermore, the assessee will be overconfident about his judgement. These biases are pervasive, effecting the judgements of everyone from laymen through experts. Careful practitioners are careful to use procedures designed to decrease the importance of such biases. In complex environments (such as technology forecasting), biases in hindsight will likely hinder the efforts of those trying to determine how good the forecast was. See chapter 23 in the book cited above. For example, there is a strong bias of creeping determinism, which is the conscious beleif that whatever happened had to have happened. Another effect is fellacious learning, in which people find meaning in random sequences of events. There are environments where probabilistic forecasts are extremely accurate, in some sense. In the probability world, accuracy is usually measured in terms of calibration, which is does the event occur with the same frequency as the level of probability you assigned? Weathermen, for example, are extremely well-calibrated. Their domain is the prime example of domains where researchers find good learning. The feedback is unambigious, fast, and there is lots of it. The more interesting types of forecasts are about events that occur only once, such as "will memory technology reach 64M by 2000?". Here there is only one event, the feedback is ambiguous and takes years to get. The only practical way to judge the a forecast (the probabilty that the event is true) is by examining the procedure whereby the probability was elicited. I would certainly agree with the sentiment of wanting to look at past forecasts and find such efforts laudible. We may learn less than we hope from such activity. David -- ______________________________________________________________ David Matheson matheson@portia.stanford.edu 376 College #5, Palo Alto, CA 94306-1545 (415) 328-3515