Superforecasting The Art and Science of Prediction

Philip E. Tetlock and Dan Gardner

We cannot avoid forecasting. Everything that we do is based on what we expect the outcome to be. Some forecasting is short-term and primarily preconscious such as planning a movement while taking into consideration the positions and movements of others around you. We have been making these kinds of predictions for millions of years and we apply the same processes to skills for which we have not evolved such as driving.

asignsWe have immediate feedback on our accuracy of these forecasts enabling us to gradually improve. The transition to making conscious, long-range forecasts in our complicated world has not eliminated the often bewildering influence of our subconscious. Since these influences are subconscious, we are unaware of how they steer our decisions. And we can be deceived about our ability to make these forecasts accurately because we do not have immediate feedback. “Human thought is beset by psychological pitfalls, a fact that has only become widely recognized in the last decade or two.” P. 23

Philip Tetlock provides a readable and understandable review of these pitfalls at the beginning of Superforecasting and gives examples of how they influence our thinking and our forecasts. What makes this book different from dozens of similar books in this genre is that it goes on to demonstrate that we are not doomed to our fallibilities. His research has disclosed ways in which forecasting can be improved.

cant

Tetlock spent decades tracking the accountability and accuracy of the predictions of experts in various fields. When the results were published, people humorously summarized the findings with the statement: “The average expert was roughly as accurate as a dart-throwing chimpanzee.” P. 4 What was overlooked by considering only the majority is that some people were consistently better at making forecasts.

The credibility of our Intelligence Community was severely compromised by its prediction of the existence of Weapons of Mass Destruction in Iraq which triggered our invasion and all that has followed. Recognizing that they had to try to improve their predictions, Tetlock was invited to participate with his Good Judgment Project (GJP) in a research effort sponsored by the Intelligence Advanced Research Projects Activity (IARPA) which is the research arm of our Intelligence Community. “Thanks to IARPA, we now know that a few hundred ordinary people and some simple math can not only compete with professionals supported by a multibillion dollar apparatus but also beat them.” P. 91 The Intelligence Community of the United States employs about 100,000 people at an annual cost of $50 billion in an effort to keep us safe.

A surprising fact about superforecasters is that they are not distinguished by who they are but by what they do. “Superforecasting demands thinking that is open-minded, careful, curious, and – above-all – self-critical. It also demands focus. Only the determined can deliver it reasonably consistently, which is why our analysis consistently found commitment to self-improvement to be the strongest predictor of performance.” P. 20

download

It is important to recognize that there are limits to forecasting. Even those things that can be forecasted to a high degree of probability, like the weather, become much less predictable the farther the prediction is made into the future. “One of twentieth century sciences’ great accomplishments has been to show that uncertainty is an ineradicable element of reality.” P. 127  “For superforecasters, beliefs are hypotheses to be tested, not treasures to be guarded.”P. 141 “Chance and fate do not mix. And to the extent that we allow our thoughts to move in the direction of fate, we undermine our ability to think probabilistically.” P. 149  “Knowing what we don’t know is better than thinking we know what we don’t.”  P. 245 “Meaning is a basic human need. As much research shows, the ability to find it is a marker of a healthy, resilient mind.” P. 148

athinking

“Superforecasters may have a surprising advantage: they’re not experts or professionals, so they have little ego invested in each forecast.” P. 163 “John Maynard Keynes operated on a higher plane than most of us, but that process – try, fail, analyze, adjust, try again – is fundamental to how all of us learn, almost from the moment we are born.” P. 178 “The humility required for good judgment is not self-doubt – the sense that you are untalented, unintelligent, or unworthy. It is intellectual humility. It is a recognition that reality is profoundly complex, that seeing things clearly is a constant struggle when it can be done at all, and that human judgment must therefore be riddled with mistakes. This is true for fools and geniuses alike.” Pp 228 – 229 “Forecasters who see illusory correlations and assume moral and cognitive weaknesses run together will fail when we need them the most.” P. 229

As an example, Tetlock uses what I feel is one of the most powerful and beautifully written speeches ever created. “With firmness in the right, as God gives us to see the right, let us strive to finish the work we are in.” (From Abraham Lincoln’s Second Inaugural Address) The traps he discusses are traps that are common to all of us. The techniques to overcome them are not theoretical but have been proven through thousands of forecasts by hundreds of people who participated in the research. Forecasting is a learned skill just like reading, algebra, driving, manners, morality, attention, mindset, and grit. “I believe that it is possible to see into the future, at least in some situations and to some extent, and that any intelligent, open-minded, and hardworking person can cultivate the requisite skills.” P. 6

Lincolns_Second_Innaugaral_Address_large

Stumbling on Happiness

How We Learn: The Surprising Truth About When, Where, and Why It Happens

Grit

Visual Intelligence

 

 

Advertisements