Expert Political Judgment

The New Yorker does a good rundown on “Expert Political Judgment: How Good Is It? How Can We Know?�, a new book by Berkeley academic, Philip Tetlock.

As the title suggests, it’s all about how accurate the predictions the political and economic pundits we see and read actually turn out to be. And Tetlock isn’t just some crank, shooting off his mouth. He’s pretty rigorous:

[H]is conclusions are based on a long-term study that he began twenty years ago. He picked two hundred and eighty-four people who made their living “commenting or offering advice on political and economic trends,� and he started asking them to assess the probability that various things would or would not come to pass, both in the areas of the world in which they specialized and in areas about which they were not expert. …By the end of the study, in 2003, the experts had made 82,361 forecasts

And his conculsion? The ‘experts’ are full of it.

[P]eople who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. …

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable.

And the big shots are the worst:

Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote.

Tetlock points to a number of explanations, most centering on the psychological foibles inherent in assessing uncertainty, along with simple stuff like the fact there’s really no puditocracy penalty for ‘getting it wrong.’

Anyway, the piece is relatively short, not one of those New Yorker book/articles and well worth the read.

(N.b. My predictions were not part of the study, and remain, of course, rock solid certainties. You can continue to bet the farm on the Good Doctor's forecasts)

(h/t Jane Galt)

posted by Dr. Reo Symes at 01:00 PM

| Access Comments