NOTE: The views expressed here belong to the individual contributors and not to Princeton University or the Woodrow Wilson School of Public and International Affairs.

Friday, September 23, 2011

"Randomized control trials" on trial: Evaluating the efficacy of RCTs

Jake Velker, MPA


Are randomized trials the way to finally start making a dent in reducing poverty, after years of hopeful thinking and disappointing results? Does this tool for evidence-based policymaking hold the key for practitioners to determine which poverty reduction programs work and which don’t? These questions motivate two recent books published by researchers who are at the vanguard of the randomized control trials (RCT) movement: More Than Good Intentions by Dean Karlan and Jacob Appel of Innovations for Poverty Action (IPA) and Poor Economics by Abhijit Banerjee and Esther Duflo of the Abdul Latif Jameel Poverty Action Lab (J-PAL). They deliberately introduce randomization in the implementation of anti-poverty measures to provide confidence in the programs’ efficacy (or lack thereof).

The results thus far have been dramatic and not always intuitive: microfinance is less effective than we hoped [1]; free bed nets are used more often (and prevent more malaria) than those that cost money.[2] IPA and J-PAL are currently involved in dozens of trials and are generally credited with bringing an unprecedented level of rigor to the evaluation of development—a field normally dominated by grand theories and polemics.

Enough praise has been heaped on the “randomistas” that I feel confident I can focus on the criticisms of their methodology without sounding uncharitable.[3]

The first criticism—leveled forcefully by Princeton’s Angus Deaton—is that randomized trials do not help us in any systematic way to gain an understanding of why interventions work.[4] In this sense, IPA and J-PAL are part of a broader trend in economic research that eschews theory in favor of real-world applications and problem-solving. This is irksome for many economists, particularly those who believe that poverty cannot be solved without a broader accounting of the mechanisms that keep people trapped in poverty. Randomized trials are beginning to test theoretical frameworks more directly, but there is still much progress to be made.

The most relevant criticism, however, is political. IPA and J-PAL economists have been accused of ignoring the institutional constraints against which their interventions would inevitably contend if scaled up. Often, research from randomized trials offers a conclusion like “we find that intervention X lowers Y disease transmission by Z percent.” While it is extremely helpful to have confidence in the efficacy of a treatment, glaring questions remain. Does the program work when it is implemented by a weak bureaucracy, rather than university-trained researchers? At scale, who will be responsible for administering the recommended program? What are their incentives to perform? Where will the money come from? If the intervention is so successful, were there good reasons it wasn’t tried before?

A troubling case in point involves one of the most heralded studies from the RCT movement to date. Working in Kenya, economists Michael Kremer and Ted Miguel found that providing de-worming medicine to students boosted school attendance cost-effectively.[5] Spurred by their research, the Kenyan government committed to making de-worming medicine available to more than 3,000,000 of its primary school children in 2009. But the policy was recently discontinued due to a dispute between the Kenyan government and international donors over corruption and the administration of education funding.[6] It should go without saying that for the ultra-poor, these sorts of bureaucratic obstacles are the norm, rather than the exception.

If economics is just supply and demand, the work of IPA and J-PAL has focused thus far mostly on demand. It is difficult to quantitatively study governance—imagine what an RCT studying a poor country’s provincial governance, for example, might look like—and even harder to actually improve the quality of basic services in developing countries. So many of the interventions RCTs have found to be effective involve classic public goods, which by definition remain under-provisioned by private markets. But the bureaucracies of developing countries are generally ineffective, if not downright corrupt. This is where economics loses its relevance and institutions and leadership rear their ugly heads.

These problems have not been amenable to ever-more creative randomized trials. In fact, many of the most celebrated finds of the RCT movement are relative “no-brainers.” Who, after all, would argue against treating poor school children for intestinal worms? Esther Duflo and her colleagues have said that we do not know what works. Many would respond that we know perfectly well what works; but do not know how to do it. Perhaps the real questions start once an intervention has been proven to work.

Randomistas respond to this critique as follows. First, it was never their ambition to overhaul the political economy of the developing world. The fact that they have found real evidence of effective interventions is in itself a major accomplishment. They believe that their approach can improve lives even in discouraging political settings. They are not promising a sweeping social revolution, but rather a “quiet revolution” of incremental gains. And even critics will concede that though the modesty of this approach may be unsatisfying, it is nonetheless an improvement on the empty promises all too frequent in the development world.



------------------------
References

[1] Abhijit Banerjeey, Esther Duflo, Rachel Glennerster, and Cynthia Kinnan, “The miracle of microfinance? Evidence from a randomized evaluation,” Working Paper (unpublished), May 2009.


[2] Jessica Cohen and Pascaline Dupas, “Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment,” Quarterly Journal of Economics, Vol. 125:1, 2010.


[3] For examples of such praise, see: Ian Parker, “The Poverty Lab: Transforming development economics, one experiment at a time,” New Yorker, May 2010
; James Crabtree, “Attested Development,” Financial Times, April 2011; William Easterly, “Measuring How and Why Aid Works – or Doesn’t,” Wall Street Journal, April 2011; Ben Goldacre, “How can you tell if a policy is working? Run a trial,” The Guardian, May 2011; and Nicholas Kristof, “Getting Smart on Aid,” New York Times, May 2011.

[4] Angus Deaton, “Instruments, Randomization, and Learning about Development,” Journal of Economic Literature, Vol. 48:2, June 2010.


[5] Edward Miguel and Michael Kremer, “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities,” Econometrica, Vol. 72: 1, January 2004.

[6] Justin Sandefur, “Held Hostage: Funding for a Proven Success in Global Development on Hold in Kenya,” Global Development: Views from the Center blog, Center for Global Development, April 2011.

4 comments:

  1. Jake, regarding your point on the discontinued program in Kenya, yes, this is unfortunate, but the problem is one of scaling up, it is not the fault of RCT as a method. What is the counter-factual? Are you saying that the Kenyan government would have been better off funding this program without it going through RCT first? Certainly not. Rather, I think the point is that RCT is effective, but only at evaluating for that particular scale. That doesn’t mean that RCT is poor methodology; it only means that people shouldn’t draw the wrong conclusions from RCT-based studies.

    ReplyDelete
  2. During my time in the international aid sector, I've seen an increasing desperation to “know” what is inherently beyond logic and induction. It is time to examine economists' beliefs that there are technocratic, precise ways of measuring progress in order to make consequential judgments based on these measures. The increasing obsession with RCTs, abstract metrics and experimental design, stemming from a reductive, managerial approach in aid, is quite far from the intimate, difficult, and complex factors at play at the national and grassroots levels.
    You can read my review of "More Than Good Intentions" at: http://www.how-matters.org/2011/05/24/rcts-band-aid-on-deeper-issue/ as well as a compilation of articles on the RCT debate in international aid at: http://www.how-matters.org/2011/05/24/rcts-and-aid-effectiveness-compilatio/

    ReplyDelete
  3. Thank you Jake. This was a helpful read, and I appreciate the time that you put in!

    ReplyDelete
  4. RCT has been a new tool to work for the western economists in the poverty-stricken underdeveloped world- be that in Asia, Africa or any other part of the world. The concern and premise of poverty studies has in fact never been the interest or mainstay of economics in the so called western universities. All the economists who claim to have worked with poverty related RCT types of issues neither have ever experienced poverty nor belong to the poor's world. Therefore, anything related to poverty literature has barely been observational at the most and in no way participatory outcome. lets first bring debate out of MIT, Harvard or Cambridge to the place where poverty has sucked the human life, to Asia or Africa. The academic institutions of these area and then see what shape the entire debate take place.

    ReplyDelete