In Afghanistan, It’s Not All in the Numbers

Source: Getty
U.S. policy should not rely on unsound opinion surveys. A serious strategy for withdrawal from Afghanistan must take reality, not wishful thinking, as its starting point.
Related Media and Tools

It seems that whenever a measure of realism regarding the U.S. engagement in Afghanistan sobers the debate among policymakers and observers, the Asia Foundation brings forth a new—predictably sunny—“Survey of the Afghan People.” Year in, year out, these doubtless well-meaning and clearly painstaking efforts portray a population that is essentially happy with the direction the country is going, confident in government institutions, and doing better economically than the previous year. The findings, which defy logic, never fail to amaze Afghans or Westerners who spend significant time in Afghanistan. Nevertheless they are cited with numbing regularity by current and former U.S. officials as reason to “hope.”

Other Resources
Afghanistan in 2012: A Survey of the Afghan People
The Asia Foundation
There Is Hope yet for Afghanistan
Karl Eikenberry, Financial Times
At a crucial inflection point in the Afghanistan mission, U.S. policymaking should not be based on foundations so unsound. A serious strategy for withdrawal from Afghanistan must take reality, not wishful thinking, as its starting point.

What makes surveys so attractive to decisionmakers is their apparent neutrality and scope. While people who undergo daily life in Afghanistan tell stories, surveys tabulate numbers. They stake implicit or explicit claim to representative, nationwide coverage. So their findings are accorded the stature of “empirical” research, while experience is often disparaged as “anecdotal.”

And yet polling, a notoriously complex art, is almost impossible to conduct meaningfully in Afghanistan. Surveys are plagued by methodological flaws that corrode the value of their results. Though polling reports’ dense texts often acknowledge these flaws, the caveats are not reflected in the numbers. And it is the numbers that so many officials, experts, and journalists love to cite.

In the case of the 2012 Asia Foundation survey, perhaps the most significant flaw is that nearly one-third of the planned “sampling points” could not be accessed by interviewers, largely for security reasons. So people in safer neighborhoods or villages were interviewed instead. One result is a series of polls that cannot be used to track trends over time, since the sample population is never the same.

But the implications of this defect go deeper—to undermine the report’s fundamental conclusions about Afghan confidence. “Public optimism about the overall direction of Afghanistan is currently at its highest point since 2006,” declares the preface. Inside pages, however, acknowledge that “restrictions on the movement of survey researchers due to security concerns considerably increased in 2012.”

Afghans are ever happier about a country in which it is ever more life threatening to get around? On the bare face of it, such a proposition seems absurd—an absurdity the report reduces to a clinical observation: “The opinions of those living in insecure areas are likely to be underrepresented in survey findings.”

In other cases, the answers to one question may have invalidating implications for the answers to another. For example, a total of 64 percent of respondents agreed that “it is generally not acceptable to talk negatively about the government in public.” Could that result have a bearing on the number of respondents (75 percent) who professed to feel the Afghan government is doing a good job? In a similarly illuminating inconsistency, the survey authors acknowledge that 42 percent of those who said they had confidence in the Afghan National Army also said they feared to encounter it.

A broader problem with this and other surveys has to do with how Afghans tend to use language in the complex and threatening environment they have learned to navigate over the past several decades. I have rented out offices to polltakers—and watched them sit at the desks and fill in the answers they were supposed to be getting from respondents. A friend who worked as a field researcher a few years ago told me his supervisor dictated the desired responses. But even if research staff were truly pounding the dirt roads and impartially recording respondents’ answers, those answers are more unlikely in Afghanistan than in most other countries to reflect what the speaker actually thinks.

Afghans are survivors. For thirty years marked by invasion and civil war, saying the wrong thing to the wrong person could get them killed. So most Afghans have grown skilled at discerning the likely orientation of the person they are talking to, and—if no overwhelming interest dictates otherwise—saying what they think he or she will want to hear.

In the West, the notion of “independent” polling is broadly accepted, because no significant segment of society rejects the intrinsic principle, and pollsters are typically seen as embodying broadly accepted scientific principles. In Afghanistan, by contrast, only the government and the international community conduct polls. The Taliban or nonviolent dissidents do not. So surveyors are likely to be seen as tied to the Afghan government and its international backers.

While the Asia Foundation researchers are instructed to tell respondents, “Your answers will be kept confidential and your name will not be given to anyone,” there is no reason a random Afghan should believe that. Out of self-preservation, random Afghans are likely to answer pollsters in ways they believe will be pleasing to the government and its international backers.

It is time to stop deluding ourselves with such patently distorted information, and using it as a basis for analysis or for placating the public with a comforting message. It is dangerous to build strategy on such quicksand.

Recent conversations with ordinary Afghans indicate that weapons are rapidly being bought up, at least in the north, and business is slowing as fewer people are willing to incur risk in a situation they judge to be increasingly unpredictable. Such factors provide more eloquent indications about prevailing conditions than do opinion surveys.

A more candid appreciation of Afghan realities would allow for more accurate assessments of Afghans’ likely contingency planning and that of their neighbors, which in turn might induce U.S. decisionmakers to consider how to avert or at least mitigate the significant dangers of the momentous upcoming international withdrawal. A measure of realism is a minimal starting point.

End of document

About the South Asia Program

The Carnegie South Asia Program informs policy debates relating to the region’s security, economy, and political development. From the war in Afghanistan to Pakistan’s internal dynamics to U.S. engagement with India, the Program’s renowned team of experts offer in-depth analysis derived from their unique access to the people and places defining South Asia’s most critical challenges.


Comments (8)

  • Jeffrey Laurenti
    Sarah Chayes has cogently and convincingly explained why the Asia Foundation surveys in Afghanistan have to be treated with considerable reserve--especially by policy analysts trying to understand what will be viable going forward. On top of all the methodological issues that Ms. Chayes cites (access, respondent reticence, etc.) there is the surveys' curious failure to pose questions directly about the elephant in the room: what Afghans think about the foreign military presence in their country, those troops' behavior and actions, and Kabul's relations with it.
    Yet there are data buried within the survey--not spotlighted by the Foundation's summaries--that allow suggestive inferences about Afghans' attitudes toward the "foreign friends" and the war, some of which I teased out in a recent article on Huffington Post and on the Century Foundation blog: The Foundation's obviously expensive research may not be entirely in vain.
    Reply to this post

    Close Panel
  • bcarreau
    1 Recommend
    Bingo. Ms. Chayes is exactly on target. Despite the best of intentions, and even with supposedly scientific polling standards, western-style polling will almost always produce unreliable results. As Ms. Chayes points out, this is for the simple reason that most Afghans will be extremely reluctant to speak their minds to someone they do not know, especially a foreigner, or a local employed by a foreigner, and especially one they assume is working for the Afghan government or the international community that is supporting the Afghan government. And in this part of the world, no one, absolutely no one, would dare criticize the government in public or to a stranger.
    Reply to this post

    Close Panel
    • shafi replies...
      You obviously dont know much about this country saying nobody dares to critisize the government. Afghan media daily have these numerous talks and each and everyone participates in them. They include the former Communists, the former Mojahiddin, the former Talibans and people of no specific links. I can gurentee that we have better freedome of speach in that sense that many of our neighbors do. But whether the government responsds or is hold accountable for the claims made and speaches given is another story. But in Afghanistan it is quite common actually to dare to speack against the government....
  • rjsr
    One is reminded of the studies of the "elections" to the South Vietnamese parliament in the late Sixties. An earnest PhD student of Samuel Huntington did one of these and was then hired and eventuallty made chair of a Poli Sci Dept at a reasonabloy good university. Hah! He then made a career as a CIA researcher. Ms. Chayes is polite and understated. It's Bunkum.
    Reply to this post

    Close Panel
    • LIndsay Dorman replies...
      Sarah is correct instating that reality, not wishful thinking is required in a serious withdrawal strategy.
      A cold shower might be the order of the day
  • Mark Kryzer, Country Rep, Asia Foundation, Afghanistan
    1 Recommend
    Conducting public opinion surveys in conflict zones is never easy. The Asia Foundation has been in Afghanistan since 1954 and we know the landscape well. We think the Survey is well worth the effort while fully recognizing its limitations. It’s one data point among many to help understand the changing needs of the Afghan people and where we can best direct our programming. While it is important to listen to anecdotal stories of individual Afghans, it is also important to listen to the opinions of 6,290 randomly selected Afghan citizens from all 34 Provinces as they respond to open ended questions about their lives. Both have their uses. Our partner ASCOR’s Afghan interviewers (742 this year) walk as much as two hours to each interview site. This is part of an overall Foundation effort to increase commitment and survey capacity for further research.

    This is our eighth survey and as in the past includes optimistic and pessimistic opinions. Some challenged my assumptions, probably a good thing. Having lived and worked here for nearly ten years it’s easy to get complacent about what I think I know about Afghanistan. As I continue to sift through the raw data to better understand the responses, I find it useful to refer to the demographics of the survey population described in the back of the report. Broken down by region, gender, occupation, urban versus rural, education, income levels, language, etc. it provides much-needed context.

    I also find the cross-tabulation possibilities of the data revealing. For example, 42% of those who reported confidence in the Afghan National Police and Afghan National Army also reported some fear of those institutions. How did Dari speaking, urban dwelling women with a high school education living in the Western region of Afghanistan respond to this question? It’s in the survey.

    Another piece of contextual information is that this year about 16% of the originally selected sampling points could not be reached because of fighting, the same percentage as in 2010. If, on the day an interviewer is attempting to reach a designated village, there is fighting going on, a replacement village is selected randomly in the same region. The sampling points replaced are detailed in the Survey report. Again, an important element, but not one that invalidates the usefulness of the survey when taken together with other reliable resources.

    Mark Kryzer
    Country Representative
    The Asia Foundation
    Kabul, Afghanistan

    Reply to this post

    Close Panel
  • Jamil Chaudri
    The questions Asia Foundation, AF, posited are not as important as the questions it chose NOT to ask:
    1.     What percentage of Afghans ENJOY foreign occupation of their country?
    2.     What percentage of Afghans would like to perpetuate and institutionalize foreign occupation?
    3.     How many civilians, especially women and children, does an average Afghan think have been killed by Foreign Occupiers in the last 12 months?
    4.     How do Afghans feel about Foreign induced corruption in, and of, their country?
    5.     Was corruption more under the Taliban or is it more under Foreign occupiers?
    Even if AF had attempted to elicit such data, the question about its veracity and reliability would have remained.
    And here is a question for the readers, if America were under the iron first of occupation, how many Americans would tell their true feelings to the occupiers and their locally recruited agents.
    Reply to this post

    Close Panel
  • Emil
    Sadly Ms Chayes joins the ever swelling ranks of those who stand on the sidelines, then snipe and boo - but without explaining and arguing logically what else could replace surveys apart from a one-line throwaway at the bottom.
    Just as we all have to be exact and responsible in our analyses as we approach transitioning, so too must the journos reporting on this process. Sadly this piece is neither, lacking, as it does, in setting out any reasoned alternative to surveys.
    Reply to this post

    Close Panel

Stay in the Know

Enter your email address to receive the latest Carnegie analysis in your inbox!

Personal Information
Carnegie Endowment for International Peace
1779 Massachusetts Avenue NW Washington, DC 20036-2103 Phone: 202 483 7600 Fax: 202 483 1840
Please note...

You are leaving the website for the Carnegie-Tsinghua Center for Global Policy and entering a website for another of Carnegie's global centers.