The Economist
WEAPONS of mass destruction will be found in Iraq. It will rain tomorrow. Jeremy Corbyn cannot possibly become leader of the Labour Party. The Japanese rugby team will never beat South Africa. Human beings cannot resist trying to scry the future. If soothsaying is not the oldest profession, it is certainly one of them.
The Chinese had the I-Ching; the Romans peered at the entrails of sacrificed animals. These days, anyone wanting to know what the future holds can consult everything from telephone psychics to intelligence agencies, bookies, futures markets and media pundits. Their record is far from perfect. But it is difficult to say just how imperfect: for all the importance people attach to forecasting, hardly anyone bothers to keep score.
Philip Tetlock is a rare exception. His most recent book, “Superforecasting”, (written with Dan Gardner, a Canadian journalist with an interest in politics and human psychology) is a scientific analysis of the ancient art of divination. Mr Tetlock, who teaches at the Wharton School of Business, became famous for concluding, on the basis of a 20-year forecasting tournament that ran between 1984 and 2004, that the average expert is “roughly as accurate as a dart-throwing chimpanzee”. His findings were more subtle than that, and his new book is an attempt to set the record straight. It shows that the future can indeed be foreseen, at least in the near term. More interestingly, it shows that some people are much better at it than others. And, crucially, it shows that prophecy is not a divine gift, but a skill that can be practised and improved.
The book describes another contest, this time run by America’s spies in the wake of the disastrous misadventure in Iraq. Begun in 2011, it posed hundreds of geopolitical questions (“Will Saudi Arabia agree to OPEC production cuts in November 2014?” for instance) to thousands of volunteer participants. A small number of forecasters began to pull clear of the pack: the titular “superforecasters”. Their performance was consistently impressive. With nothing more than an internet connection and their own brains, they consistently beat everything from financial markets to trained intelligence analysts with access to top-secret information.
They were an eclectic bunch: housewives, unemployed factory workers and professors of mathematics. But Mr Tetlock and his collaborators were able to extract some common personality traits. Superforecasters are clever, on average, but by no means geniuses. More important than sheer intelligence was mental attitude. Borrowing from Sir Isaiah Berlin, a Latvian-born British philosopher, Mr Tetlock divides people into two categories: hedgehogs, whose understanding of the world depends on one or two big ideas, and foxes, who think the world is too complicated to boil down into a single slogan. Superforecasters are drawn exclusively from the ranks of the foxes.
Humility in the face of a complex world makes superforecasters subtle thinkers. They tend to be comfortable with numbers and statistical concepts such as “regression to the mean” (which essentially says that most of the time things are pretty normal, so any large deviation is likely to be followed by a shift back towards normality). But they are not statisticians: unlike celebrity pollsters such as Nate Silver, they tend not to build explicit mathematical models (after all, questions such as “Will Russia officially annex Ukranian territory in the next three months?” are less suitable for the data-heavy, historical approach that Mr Silver prefers).
But superforecasters do have a healthy appetite for information, a willingness to revisit their predictions in light of new data, and the ability to synthesise material from sources with very different outlooks on the world. They think in fine gradations. Rather than assigning something a probability of 60 to 40, for instance, a superforecaster might, after careful consideration and many small revisions to take account of newfound subtleties, settle on odds of 62 to 38.
Most important is what Mr Tetlock calls a “growth mindset”: a mix of determination, self-reflection and willingness to learn from one’s mistakes. The best forecasters were less interested in whether they were right or wrong than in why they were right or wrong. They were always looking for ways to improve their performance. In other words, prediction is not only possible, it is teachable.
Talk of growth mindsets, statistical fluency and a complicated world may sound dry and technical. It is not. Mr Tetlock’s thesis is that politics and human affairs are not inscrutable mysteries. Instead, they are a bit like weather forecasting, where short-term predictions are possible and reasonably accurate.
What tomorrow brings
The implications of this are far-reaching, and not just for governments and spies. The book opens with a discussion of Archie Cochrane, a Scottish doctor born in 1909, who did more than perhaps anyone else to transform medicine from a black art into a fully fledged science. His insight—deeply controversial half a century ago—was that a doctor’s qualifications, eminence and confidence are irrelevant and that the only test of a treatment’s effectiveness was whether it could be shown, statistically and rigorously, to work. Mr Tetlock hopes to bring about a similar rigour to how people analyse forecasts of the future.
That will be an uphill struggle. Prediction, like medicine in the early 20th century, is still mostly based on eminence rather than evidence. The most famous forecasters in the world are newspaper columnists and television pundits. Superforecasters make for bad media stars. Caution, nuance and healthy scepticism are less telegenic than big hair, a dazzling smile and simplistic, confident pronouncements. But even if the hoped-for revolution never arrives, the techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone.
WEAPONS of mass destruction will be found in Iraq. It will rain tomorrow. Jeremy Corbyn cannot possibly become leader of the Labour Party. The Japanese rugby team will never beat South Africa. Human beings cannot resist trying to scry the future. If soothsaying is not the oldest profession, it is certainly one of them.
The Chinese had the I-Ching; the Romans peered at the entrails of sacrificed animals. These days, anyone wanting to know what the future holds can consult everything from telephone psychics to intelligence agencies, bookies, futures markets and media pundits. Their record is far from perfect. But it is difficult to say just how imperfect: for all the importance people attach to forecasting, hardly anyone bothers to keep score.
Philip Tetlock is a rare exception. His most recent book, “Superforecasting”, (written with Dan Gardner, a Canadian journalist with an interest in politics and human psychology) is a scientific analysis of the ancient art of divination. Mr Tetlock, who teaches at the Wharton School of Business, became famous for concluding, on the basis of a 20-year forecasting tournament that ran between 1984 and 2004, that the average expert is “roughly as accurate as a dart-throwing chimpanzee”. His findings were more subtle than that, and his new book is an attempt to set the record straight. It shows that the future can indeed be foreseen, at least in the near term. More interestingly, it shows that some people are much better at it than others. And, crucially, it shows that prophecy is not a divine gift, but a skill that can be practised and improved.
The book describes another contest, this time run by America’s spies in the wake of the disastrous misadventure in Iraq. Begun in 2011, it posed hundreds of geopolitical questions (“Will Saudi Arabia agree to OPEC production cuts in November 2014?” for instance) to thousands of volunteer participants. A small number of forecasters began to pull clear of the pack: the titular “superforecasters”. Their performance was consistently impressive. With nothing more than an internet connection and their own brains, they consistently beat everything from financial markets to trained intelligence analysts with access to top-secret information.
They were an eclectic bunch: housewives, unemployed factory workers and professors of mathematics. But Mr Tetlock and his collaborators were able to extract some common personality traits. Superforecasters are clever, on average, but by no means geniuses. More important than sheer intelligence was mental attitude. Borrowing from Sir Isaiah Berlin, a Latvian-born British philosopher, Mr Tetlock divides people into two categories: hedgehogs, whose understanding of the world depends on one or two big ideas, and foxes, who think the world is too complicated to boil down into a single slogan. Superforecasters are drawn exclusively from the ranks of the foxes.
Humility in the face of a complex world makes superforecasters subtle thinkers. They tend to be comfortable with numbers and statistical concepts such as “regression to the mean” (which essentially says that most of the time things are pretty normal, so any large deviation is likely to be followed by a shift back towards normality). But they are not statisticians: unlike celebrity pollsters such as Nate Silver, they tend not to build explicit mathematical models (after all, questions such as “Will Russia officially annex Ukranian territory in the next three months?” are less suitable for the data-heavy, historical approach that Mr Silver prefers).
But superforecasters do have a healthy appetite for information, a willingness to revisit their predictions in light of new data, and the ability to synthesise material from sources with very different outlooks on the world. They think in fine gradations. Rather than assigning something a probability of 60 to 40, for instance, a superforecaster might, after careful consideration and many small revisions to take account of newfound subtleties, settle on odds of 62 to 38.
Most important is what Mr Tetlock calls a “growth mindset”: a mix of determination, self-reflection and willingness to learn from one’s mistakes. The best forecasters were less interested in whether they were right or wrong than in why they were right or wrong. They were always looking for ways to improve their performance. In other words, prediction is not only possible, it is teachable.
Talk of growth mindsets, statistical fluency and a complicated world may sound dry and technical. It is not. Mr Tetlock’s thesis is that politics and human affairs are not inscrutable mysteries. Instead, they are a bit like weather forecasting, where short-term predictions are possible and reasonably accurate.
What tomorrow brings
The implications of this are far-reaching, and not just for governments and spies. The book opens with a discussion of Archie Cochrane, a Scottish doctor born in 1909, who did more than perhaps anyone else to transform medicine from a black art into a fully fledged science. His insight—deeply controversial half a century ago—was that a doctor’s qualifications, eminence and confidence are irrelevant and that the only test of a treatment’s effectiveness was whether it could be shown, statistically and rigorously, to work. Mr Tetlock hopes to bring about a similar rigour to how people analyse forecasts of the future.
That will be an uphill struggle. Prediction, like medicine in the early 20th century, is still mostly based on eminence rather than evidence. The most famous forecasters in the world are newspaper columnists and television pundits. Superforecasters make for bad media stars. Caution, nuance and healthy scepticism are less telegenic than big hair, a dazzling smile and simplistic, confident pronouncements. But even if the hoped-for revolution never arrives, the techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone.
RSS Feed
