Governments, charities, and nongovernmental organizations (NGOs) spend trillions of dollars each year to combat poverty, but they often lack rigorous methods to evaluate the programs they fund. Now, one of the first randomized, controlled tests of a large-scale economic aid program has shown small but encouraging effects for people living on less than $1.25 per day. A one-time dose of cash, cows, goats, or other assets and services made families healthier and better off for at least a year, researchers found.
The study is a "capstone paper" for evaluating economic development projects, says Justin Sandefur, a development economist at the Center for Global Development in Washington, D.C., who was not involved in the work. "The literature has been building up to something like this.” Critics, however, say that the study's leaders asked the wrong questions when evaluating the program’s success.
What sets the study apart from most previous evaluations is its randomized, controlled design—a relatively new approach honed by lead author Abhijit Banerjee, an economist at the Massachusetts Institute of Technology in Cambridge, and his colleagues. Set up to mimic the design of a drug trial, the study randomly assigned aid to a subset of people within a development project called the “Graduation program.” Researchers then compared the outcomes with those of a matched control group, which did not receive any benefits from the program. But—in a way that could potentially confound the findings—members of the control group in some cases received aid from other organizations.
Ethical concerns have prevented many NGOs from conducting such trials because they require giving aid to some and not to others, Sandefur says. But, "the aid industry is becoming increasingly comfortable" with the approach as it proves its value, he says.
In the study, aid workers identified 11,000 of the poorest households in six impoverished regions of Ethiopia, Ghana, India, Pakistan, Peru, and Honduras, then divided them into treatment and control groups. Participants in the treatment group received food, health care, and training in basic financial skills, such as starting a savings account. They were also allowed to choose from a list of assets they could use to make a living, such as goats, cows, chickens, or beehives.
After collecting baseline information about household finances, health, and food supply, aid workers traveled from house to house over the next 2 years to track participants’ progress. One year after the 2-year program was concluded, workers returned to see how each household had fared.
Compared with the control group, people who received the aid package had increased their monthly spending by roughly 5%. They also saved about 95% more than controls, and they increased their monthly livestock revenues by about 40%, the team reports online today in Science. That pattern suggests that participants did not simply spend the money and eat the livestock they were given, but used them instead to generate more income, Banerjee says. "They don't fall back into the extreme poverty they were in."
More striking than the income gains was the program's return—how much extra each participant made for every dollar invested, Banerjee says. In India, for example, participants made a 433% return, or $4.33 for every dollar spent on the program. Based on results that showed a 260% return on investment in Ethiopia, the country is planning to expand the approach to 10 million people.
Not all countries had such success, however. After a charitable organization delivered thousands of chickens to 800 of the poorest households in Lempira, Honduras, disease killed off most of the birds. A year later, families in the program were no better off than they’d been before, and some were worse off—all in all, the Honduras project suffered a loss to the tune of—198% of its initial investment, based on loss of household assets.
Honduras's failure highlights the importance of carefully tailoring the model to specific regions and choosing more resilient assets, the authors say. But critics point to larger concerns with the study. Although the general approach and results are "positive and useful," says economist Jeffrey Sachs of Columbia University, "we don't learn very much about how levels of community development, government policies, local ecology, prevalent farm systems, and social organization influence the outcomes." He also notes that the 3-year program was too short to fully evaluate long-term impacts.
An alternative approach to evaluating aid effectiveness would ask: "What works for whom, in what ways, under what conditions?" says Michael Patton, a program consultant in the Minneapolis-St.Paul area, and former president of the American Evaluation Association, an association for professional evaluators. Although randomized control trials can tell researchers whether an intervention has worked on average, they can't explain why certain programs work better than others under different conditions, he says. "The supposed scientific allure and perceived rigor of randomized control trials actually inhibits more useful comparative designs and thoughtful, detailed examination of variation."
The “brute force” way to establish a program's validity is to reproduce findings in multiple diverse contexts, as Banerjee and colleagues did, Sandefur says. This year, a series of randomized control trials found that microcredit—the extension of small loans to the poor—brings little lasting benefit to loan recipients, a deflating finding that the international development community is "still trying to digest," he says.
In contrast to microcredit, the Graduation program does not ask for repayment; it provides both big asset transfers, such as stock and animals, and a cash transfer. "In essence, this is a welfare program, combining giveaways with a dose of training and encouragement," Sandefur says. The fact that the approach appears to work better than the microloan system "is a really optimistic finding," he says. “It will be hugely informative for the design of antipoverty programs."