It seems that whenever a measure of realism regarding the U.S. engagement in Afghanistan sobers the debate among policymakers and observers, the Asia Foundation brings forth a new—predictably sunny—“Survey of the Afghan People.” Year in, year out, these doubtless well-meaning and clearly painstaking efforts portray a population that is essentially happy with the direction the country is going, confident in government institutions, and doing better economically than the previous year. The findings, which defy logic, never fail to amaze Afghans or Westerners who spend significant time in Afghanistan. Nevertheless they are cited with numbing regularity by current and former U.S. officials as reason to “hope.”

Other Resources
Afghanistan in 2012: A Survey of the Afghan People
The Asia Foundation
There Is Hope yet for Afghanistan
Karl Eikenberry, Financial Times
At a crucial inflection point in the Afghanistan mission, U.S. policymaking should not be based on foundations so unsound. A serious strategy for withdrawal from Afghanistan must take reality, not wishful thinking, as its starting point.

What makes surveys so attractive to decisionmakers is their apparent neutrality and scope. While people who undergo daily life in Afghanistan tell stories, surveys tabulate numbers. They stake implicit or explicit claim to representative, nationwide coverage. So their findings are accorded the stature of “empirical” research, while experience is often disparaged as “anecdotal.”

And yet polling, a notoriously complex art, is almost impossible to conduct meaningfully in Afghanistan. Surveys are plagued by methodological flaws that corrode the value of their results. Though polling reports’ dense texts often acknowledge these flaws, the caveats are not reflected in the numbers. And it is the numbers that so many officials, experts, and journalists love to cite.

In the case of the 2012 Asia Foundation survey, perhaps the most significant flaw is that nearly one-third of the planned “sampling points” could not be accessed by interviewers, largely for security reasons. So people in safer neighborhoods or villages were interviewed instead. One result is a series of polls that cannot be used to track trends over time, since the sample population is never the same.

But the implications of this defect go deeper—to undermine the report’s fundamental conclusions about Afghan confidence. “Public optimism about the overall direction of Afghanistan is currently at its highest point since 2006,” declares the preface. Inside pages, however, acknowledge that “restrictions on the movement of survey researchers due to security concerns considerably increased in 2012.”

Afghans are ever happier about a country in which it is ever more life threatening to get around? On the bare face of it, such a proposition seems absurd—an absurdity the report reduces to a clinical observation: “The opinions of those living in insecure areas are likely to be underrepresented in survey findings.”

In other cases, the answers to one question may have invalidating implications for the answers to another. For example, a total of 64 percent of respondents agreed that “it is generally not acceptable to talk negatively about the government in public.” Could that result have a bearing on the number of respondents (75 percent) who professed to feel the Afghan government is doing a good job? In a similarly illuminating inconsistency, the survey authors acknowledge that 42 percent of those who said they had confidence in the Afghan National Army also said they feared to encounter it.

A broader problem with this and other surveys has to do with how Afghans tend to use language in the complex and threatening environment they have learned to navigate over the past several decades. I have rented out offices to polltakers—and watched them sit at the desks and fill in the answers they were supposed to be getting from respondents. A friend who worked as a field researcher a few years ago told me his supervisor dictated the desired responses. But even if research staff were truly pounding the dirt roads and impartially recording respondents’ answers, those answers are more unlikely in Afghanistan than in most other countries to reflect what the speaker actually thinks.

Afghans are survivors. For thirty years marked by invasion and civil war, saying the wrong thing to the wrong person could get them killed. So most Afghans have grown skilled at discerning the likely orientation of the person they are talking to, and—if no overwhelming interest dictates otherwise—saying what they think he or she will want to hear.

In the West, the notion of “independent” polling is broadly accepted, because no significant segment of society rejects the intrinsic principle, and pollsters are typically seen as embodying broadly accepted scientific principles. In Afghanistan, by contrast, only the government and the international community conduct polls. The Taliban or nonviolent dissidents do not. So surveyors are likely to be seen as tied to the Afghan government and its international backers.

While the Asia Foundation researchers are instructed to tell respondents, “Your answers will be kept confidential and your name will not be given to anyone,” there is no reason a random Afghan should believe that. Out of self-preservation, random Afghans are likely to answer pollsters in ways they believe will be pleasing to the government and its international backers.

It is time to stop deluding ourselves with such patently distorted information, and using it as a basis for analysis or for placating the public with a comforting message. It is dangerous to build strategy on such quicksand.

Recent conversations with ordinary Afghans indicate that weapons are rapidly being bought up, at least in the north, and business is slowing as fewer people are willing to incur risk in a situation they judge to be increasingly unpredictable. Such factors provide more eloquent indications about prevailing conditions than do opinion surveys.

A more candid appreciation of Afghan realities would allow for more accurate assessments of Afghans’ likely contingency planning and that of their neighbors, which in turn might induce U.S. decisionmakers to consider how to avert or at least mitigate the significant dangers of the momentous upcoming international withdrawal. A measure of realism is a minimal starting point.