In recent days, there has been a lot of discussion about Sarah Palin's lack of experience in foreign policy. These criticisms all depend on the same assumption: that knowing more about foreign policy is always better. (Experience is typically used as a stand-in for knowledge, so when people say that you're inexperienced what they're really saying is you're ignorant.) But is that true? What is the payoff of expertise when it comes to political judgment?
Philip Tetlock has conducted the gold-standard study of political expertise. In the early 1980's, he picked two hundred and eighty-four people who made their living "commenting or offering advice on political and economic trends" and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be re-elected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the experts were asked to rate the probability of several possible outcomes. Tetlock then interrogated the pundits about their thought process, so that he could better understand how they made up their minds. By the end of the study, Tetlock had quantified 82,361 different predictions.
After Tetlock tallied up the data, the predictive failures of most experts became obvious. Although they were paid for their keen insights into world affairs, they tended to perform worse than random chance. Most of Tetlock's questions had three possible answers; the pundits, on average, selected the right answer less than 33 percent of the time. In other words, a dart-throwing chimp would have beaten the vast majority of professionals. Tetlock also found that the most famous pundits tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.
Furthermore, Tetlock found that knowing a lot about a place - being, say, an expert on Iraq - can actually interfere with one's judgment on questions involving Iraq. "We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly," Tetlock writes. "In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals--distinguished political scientists, area study specialists, economists, and so on--are any better than journalists or attentive readers of the New York Times in 'reading' emerging situations."
To summarize: I'm not sure there's good evidence that Sarah Palin's lack of experience or knowledge when it comes to foreign affairs will actually be a handicap. (Many experts, after all, seriously misjudged the cost and duration of the Iraq war.) Let's just hope she attentively reads the New York Times.
That said, Tetlock's study should make us worry about some aspects of Palin's decision-making process, at least as revealed in her recent ABC News interview. Although his study is best known for its demonstration of expert failure, Tetlock also found that a few experts performed above average. Tetlock explained the difference between successful and unsuccessful pundits with an allusion to an ancient metaphor, made famous by Isaiah Berlin, who distinguished between two types of thinkers: hedgehogs and foxes. (Tetlock did not find any significant correlation between political ideology and thinking style.) A hedgehog is a small mammal covered with spines that, when attacked, rolls itself into a ball, so that its spines point outwards. This is the hedgehog's only defense. A fox, on the other hand, doesn't rely on a single strategy when threatened. Instead, it adjusts its strategy to fit the particulars of the situation. Foxes are also cunning hunters. In fact, they are one of the hedgehog's few predators. According to Tetlock, the problem with people who think like hedgehogs is that they are prone to bouts of certainty - their big idea is irrefutable - and this certainty causes them to misinterpret the evidence.
So when Palin gave this answer in response to a question:
...you have to be wired in a way of being so committed to the mission, the mission that we're on, reform of this country and victory in the war, you can't blink.
Is it bad to blink? When did second-guessing yourself become a sin? As Tetlock writes, "The dominant danger [for experts] remains hubris, the vice of closed-mindedness, of dismissing dissonant possibilities too quickly." Palin may not be a foreign-policy expert, and that may not be a problem, but if the last eight years have taught us anything it's that the world is a little too complicated for a hedgehog in the White House.
- Log in to post comments
Why doesn't somebody just ask the big "STEM CELL" question, so they can inadvertently in their comments prove the necessity of separating political control over science on this issue. Are you aware of the possibilities of the latest research in Down Syndrome? As a mother, were I in such a situation, I would move the earth to uncover any scientific advances for my child. I wouldn't find a serenity prayer enough comfort in any situation where I might need to become involved and make changes for the future of many physically and mentally ill people.
Blinking became bad when "flip-flopper" became one of the main insults thrown at Presidential candidates...
Much about the future seems to be largely unpredictable.I have for many years followed the news very closely, and have made at least my share of mistakes. I didn't see the sudden, peaceful breakup of the Soviet Union. Or the turning of China away from Maoism and toward moderation. Or the sudden rise of militant radical Islam. Or the sudden explosion of the Internet. Or the shrinking power of the American trade unions. I'm not sure it makes me feel better or worse that most experts didn't see those things either, and possibly none foresaw all of them.
To me, the preciction powers of the experts is one of the most undercovered stories in our society. I congratulate Tetlock on his work. And I hope you will devote more posts in the future to this subject, because it is a subject worthy of much more attention. A subject that maybe even deserves a blog of its own.
Remember, Palin's success or failure in foreign policy does not just depend on whether she is good at predicting long-term outcomes. The study that you point out describes predictions that are made over a long period of time, yet making a 10 year prediction is less like foreign policy and more like making a wager on the stock market; you can't be certain of the outcome because the underlying processes are stochastic.
In my mind, a good foreign policy comes from someone who performs well under pressure, is not overly reactive, has a broad knowledge of current international politics, can easily reference historical examples, and who has good judgment about the psychology of people. Good international policy is not about making judgments for the future (those are often rather obvious, or can come out of advisory committees) -- it's about performing well under pressure and having the necessary tools inside your head. So, in that sense, a lack of experience just means that we don't know. This kind of uncertainty is unacceptable. We have no idea how Palin performs under pressure and we have no idea what her knowledge base is like. Given Obama's intellectual base and, frankly, his people skills, I'm actually not that worried about his so-called "lack of experience". Obama gives every impression of performing well under pressure and being open to the advisement of others. Palin, on the other hand, has never been in the scene before and we still don't know what she knows or how she acts when she's not being coached by McCain's team.
That said, I totally agree about the hedgehog issue. That's a great analogy, and a good way to think of the kind of leader that we'd like to have. Most dictators are pretty single-minded, right? Let's avoid that!
Very interesting and I'm so glad you're relating what you know to politics. It's what keeps me coming back to read what you have to say. Thanks very much for posting.
PS, props to Rachael for her commentary too. Interesting, though I disagree that Obama performs well under pressure. He performs well under controlled pressure. But under uncontrolled pressure - situations he's not firmly in control of, without the luxury of prepared remarks or time to think things through, he sticks his foot in his mouth. Which doesn't bode well for time in the White House, either.
Every situation is unique. Experience is not as valuable as the ability to grasp immense problems in their entirety and have the intellectual agility to see the various potential causes of the situation as well as many solutions and the unintended consequences of each solution. One must be flexible. Every situation requires different experience. Palin possess neither the Intellectual agility nor the intellectual curiosity.
Predicting the future is not the issue. Knowing what the hell is going on right now is the issue. Making decisions based on that knowledge is the issue. No one can predict the future, unless it's a case when anyone can predict it.
And besides, the people in the study are commentators. Everyone knows that they don't have a clue.
Tetlock's study was of pundits, not policy professionals.
Would his results have been different if his research population had included diplomats, business executives, politicians, intelligence analysts, etc? (My personal suspicion: such a study would be impossible, as the better the potential subjects, the more likely they would feel a negative interest in participation (the exact opposite of the punditariat).
I clicked through from my feed reader to respond, and found Rachel had already made all the points I was formulating in my head. So, thanks Rachel. I agree!
I think the issue is not her lack of experience in foreign policy, economic policy, science policy or military affairs, I think the issue is her false certitude, poor judgment and constant lying.
She is a good politician: ruthless, self-absorbed, photogenic and glib.
As a leader though her credentials are mostly bad: use of office for personal attacks, disregard for the environment, use of government funds for personal expenses, borrowing money for big spending and playing footsie with the oil companies.
Now she did impose a special tax on oil companies, but instead of using it to fix infrastructure or develop alternative energy sources, she gave her voters an extra oil check....so off course they like her.
Palin: Inexperienced but Cocksure - a Bush-Clone with Lipstick!
Considering some of her answers to the recent ABC interview, I suspect that 'reading the New York Times' may not anywhere near the top of her 'to-do' list. Indeed, it seems that she has little interest in anything to do with foreign policy, including even bothering to think about Iraq. Since every American must have at least had some thoughts about Iraq at some point (even if your son has not just joined up), you seriously wonder about her capacity to understand the issues involved.
Yes, and I think you also have to factor in her fundamentalist worldview. A hedgehog may not be the right critter for the White House, but a hedgehog who thinks she has a mandate from God is a downright dangerous beast. My impression is that Palin is part political animal, part religious devotee - I just can't tell which part is more promiment, and neither is appealing. And I've known enough people from Assembly of God churches and the like to know that I'd not want any one of them in the #2 position in the land. If only Richard Hofstadter could see the anti-intellectualism on display in this election, he'd probably regret how right his predictions were.
I might be able to throw a dart and predict as well as my grandmother's oncologist how fast she'll recover from her surgery. But a dart won't tell me how to perform the operation itself. Predicting isn't the same as doing. Even if this study tells us that decision-making doesn't benefit as much as from expertise as we might expect, the evidence of the last eight years suggests that successfully implementing those decisions certainly does.
Thus the neocon argument that the Iraq War was a good decision incompetently executed.
As you say, when you have a personality type like Bush or Palin, who know the answers without having to think about them, then expertise plays no part in the equation. And of course this is rather a false dichotomy is it not? Isn't this prediction base on all things being equal with no new information? What if the experts are allowed to change their opinions if they receive new information compared to the person who has no information and never gets any more? Is not the essence of a scientist being able to say "I can change my mind" as compared to the ideologue who will always find justification for their continuing wrong position?
On a purely political note, I'm not worried about Palin's lack of experience. I am worried about her apparent complete lack of understanding (though she seems to have memorized some facts) of world affairs and of the US economy. The Bush Doctrine? Fannie and Freddie got too expensive for taxpayers? Everything I have heard this woman say has led me to believe that I myself would be more qualified for the job than she is--and I have no experience whatsoever.
From now on, when she's asked about foreign experience and responds with a speech about energy, I will refer to that as "the hedgehog defence."
@ Jonah: I read the Tetlock book a year or two back, and agree it's excellent, and actually readable for a non-academic.
I think you might be misinterpreting Palin and Tetlock. You should play more poker. "Blinking" is showing outward signs of your inward thoughts, feelings and strength of your position. Poker players are second-guessing and analyzing all the time, but they don't "blink" when they are in a tight spot. Furthermore, Tetlock devoted an entire chapter in his book to "The Hedgehogs Strike Back," so let's not be too quick to cast on the ash heap people who believe in One Big Idea. What if the One Big Idea is "democracy is the least bad form of government"?
@ Rachael: your commentary of Tetlock's book is good, but has a couple flaws. You're right that foreign policy success as President or VP doesn't require accurate forecasting. But you're wrong to think that Palin doesn't perform well under pressure. Does taking on an incumbent governor, or an entrenched, cozy oil-and-regulators government body qualify as performance under pressure? Indeed, Aug 29-Nov 4 will be a great tryout for Palin. Lastly, to use Tetlock's/Berlin's/Archilochus' terms, you're right that most dictators are hedgehoggish --about self-preservation -- but are very foxlike in how they achieve that goal. Kim Jong-Il uses nuclear threats (show of strength) while also pleading for economic and food assistance (show of weakness/plea for mercy).
@Pierce R. Butler: Tetlock interviewed hundreds of professional forecasters over a decade. Some appeared in media more frequently than others. So they weren't all just "pundits" in the 24-hour-cable-news sense of the word. You are on to something, however, in that Tetlock finds a statistcally significant NEGATIVE correlation between forecasting accuracy and number of media appearances. So listen to the forecaster who works in obscurity or only for paying clients, not the publicity hound.
GWB was clueless about foriegn affairs, didn't even know who was president of Pakistan. - 'nuff said!
I enjoyed this article immensely. Certainty is indeed a handicap in any objective field. I took a few political science classes in college and was astounded to learn that there is nothing remotely scientific about it. Scientists are open to new evidence and constantly re-evaluate their hypotheses. Pseudoscientists and politicians stick unswervingly to their positions. Of course that's the easiest way to win people over quickly, so I don't expect to see a fox in the White House any time soon.