Source: Stanford Social Innovation Review
“The proposal asked us to improve access to justice in Somalia, in nine months. With quantifiable results,” exclaimed one horrified aid implementer. I was giving briefings around the world on improving program design and measurement for economic development programs. Government aid officials nodded and described new initiatives in problem-driven enquiry, political-economy analysis, adaptive management, and iterative programming. But implementers, like the woman above, sighed: little of this thinking was changing how programs are actually delivered.
Thus, a chasm is opening between the new conventional wisdom and the old ways of programming.
Government donor organizations, like US Agency for International Development (USAID), blame elected politicians for short time horizons and poor evaluation requirements. Donors know that programming is political, and therefore requires flexibility—but feel they must justify spending to legislatures with catchy numbers and zero risk of corruption. Therefore, they force implementers to stick to detailed activity-based budgets and forecast spending years in advance. They know that opponents regroup and push back, making sustainable reform non-linear and generational. But to meet politicians’ political cycles, they require proposals to state clearly defined measurable results within a few years.Donors know that development moves in paths that mimic sailboats, but they are forced to program for trains. Ironically, this is exactly the quandary developing countries face, suggests the crowd of researchers pushing for more political thinking.
Politicians are not the sole culprits. Donors’ overly-monolithic thinking about what programs are “best” has been at fault, too. Donors are now beginning to understand that questions of how to get something done are fundamentally different from what should be done. Yet in their search for rigor, they are still enamored of randomized control trials and other forms of evaluation that are poorly suited to path-dependent, timing-sensitive questions of how to make change in complex systems.
What Is to be Done?
How does the development community move forward? How do we match what we now know, to what we actually do? How can the development community create and evaluate programs in one- to five-year funding cycles that can contribute to political, local, non-linear reform that may take fifty years to show results?
First, the development community can move from delivering programs, to catalyzing change. To deal with the vastly different time scale of funding versus reform, they can change the frame from what programs donors can deliver, to what locals can do with donor support.
We know that change happens through local leadership, broad coalitions, and networks. The strength and health of these networks, and the position of graduates from “leadership” programs, are measurable. Thus, instead of quantifying worthless output measures, or wringing hands because impact can’t be measured in short time frames, donors can measure the strength, relative position, and resilience of coalitions and agents of change they are helping.
For instance, USAID ran an anti-corruption program in Georgia in the early 1990s. It failed. But the person who ran the program continued to dedicate his life to good governance, founded an influential nonprofit, brought many of the anti-corruption ideas into government after the Rose Revolution, and is now Speaker of the Parliament. What if the program had measured the human capital it had catalyzed, rather than the program it had unrealistically tried to deliver during an inhospitable political period?
Second, the development community can focus on changing incentive systems and other “rules of the game.” Most development programs are like throwing a rock in a pond—they make a big splash, but little change. Altering the rules that shape a system is like moving rocks along the water’s edge—move enough of them and the stream will follow a different course.
Among the most crucial ways to alter the rules of the game is to empower citizens. Growing numbers of governments around the world, from Algeria to Cambodia, are erecting legal and logistical barriers to their citizens’ free speech and organization, because civil society groups can be so effective at holding them to account. Tearfund, a Christian charity, has found a measureable, network-driven method to address this problem in difficult situations. It has catalyzed approximately 15,000 self-help groups in Ethiopia to bring together women to save money. But unlike most groups, they also help women problem-solve together. A pyramidal structure enables local leaders to meet at regional and national levels to brainstorm shared national problems. Under an authoritarian regime, such groups can constitute nascent elements of grassroots democracy, when a window for reform opens.
Third, donors in the development community, knowing that reform requires multiple battles, can budget to build on success instead of moving to the next issue of the moment. For instance, a group of Romanian NGOs implemented a well-executed program of accountability throughout the country, and managed to pull down a corrupt government and prevent a new crop of corrupt parliamentarians from taking office. But funding ended when the country entered the European Union. The corrupt officials regrouped, fired the Minister of Justice, and resumed business as usual. Creating a network is more difficult than maintaining one. Continued support could have enabled the civil society groups to continue holding their government accountable.
Fourth, the development community can target measurement to different audiences. One goal of evaluation is to reassure donors that their money is well spent. Another is to enable programs to learn and improve. These require different metrics, which most development agencies get backward.
Development experts assume that politicians want numbers. But raw figures mean little to non-experts, and often lead to useless output measurements. Instead, politicians need short, clear stories that succinctly describe theories of change and reasons for success or failure. Stories carry meaning. They also allow a development agency to illustrate why funds are better spent doubling down on success and fighting backlash, rather than tossing inadequate funds at politically sexy problems that it may be ill-suited to address. Oxfam, for example, has put such rigorous and thoughtful stories on everything from airport advertisements in the main terminals used by members of the US Congress for their weekly trips home, to its websites, and brings the domestic reformers to Congress to tell their stories directly.
While anecdotal stories are an important tool, too often they serve as good-enough “learning” in development agencies strapped for measurement and evaluation resources. Instead, more rigorous iterative measurement of outcomes should be used more frequently. In talking with practitioners worldwide, I’ve been struck by how many are confused by “outcome” measurement. Most need more training on how to set goals that can be clearly attributed to program activities, but can be unchanging and have intrinsic meaning towards the desired impact.
New thinking can make development agents more impactful. But altering action requires confronting the political economy of our own bureaucracies. Are we brave enough to turn the lens inward?
This article was originally published in the Stanford Social Innovation Review.