Empowering Evaluation: Looking beyond the numbers

Elizabeth Hoody, MPA

As a former grant-maker, I was glad to see that 14 Points talked about the challenge of evaluating anti-domestic violence work in a post by Payal Hathi last May. In my own experiences working in a women’s rights foundation, I saw just how difficult it can be to quantify the impact of women’s rights organizing in numerical terms. While numbers can tell an important story (such as how many young women receive sexual and reproductive health education), they often leave out what for me is most compelling about a group’s work. This might be the reflections of an individual young woman who now feels that she can talk to her partner about contraceptives or the story of a group of girls who decided to form their own anti-trafficking student organization after participating in a prevention workshop. So while there are many valid and pressing questions about how to “get the numbers right” in program evaluation, my bigger concern these days is how to evaluate impact beyond the numbers…and then again how to aggregate and share this type of evaluation in a way that donors, policymakers, and peer organizations can easily understand.

In the past year, I have come across several creative examples of evaluation strategies that do attempt to move beyond the numbers. Many of these strategies attempt to articulate, verbally or visually, the systemic impact of an organization’s work. One example is a map that was released by the Global Fund for Women just this past week, which captures the Fund’s impact around the world using bright spots. In the Global Fund’s words, this map “explores where a relationship between Global Fund for Women and grantee groups is more likely to yield a higher movement building impact.” While the map does rely on a series of numerical indicators, the visual analysis tells a bigger story about the collective impact of Global Fund for Women grants on women’s rights movements around the world.

The second example is the Gender At Work Framework, which “helps organizations see their work from new perspectives by combining best practices in organizational development with feminist thought.” One tool that the framework uses is a graph where civil society organizations can plot the different types of social changes they are addressing through their work. The graph places the continuum of “individual versus systemic” change on vertical axis and “formal vs. informal” changes on the horizontal axis, resulting in four quadrants of change:

• Women’s access to resources (quadrant I)
• Women’s and men’s consciousness (quadrant II)
• Informal cultural norms and exclusionary practices (quadrant III)
• Formal institutions, laws, and policies (quadrant IV).

Women’s organizations use the tool to visually represent the changes that they are trying to impact through their work. The graph is also a good indicator of what programs will be easier to quantitatively evaluate (such as programs that work for specific policy changes) and those that will be difficult to track through traditional metrics (how do you measure changes in men’s and women’s consciousness?).

What I like most about the Gender At Work Framework is that it encourages grassroots organizations to define evaluation framework themselves at the beginning of their planning processes. When this happens, evaluation shifts from being a chore for donors to being an effective way of tracking and reflecting on an organization’s progress towards its goals. In my personal, unquantifiable opinion, the resulting story more often than not contains richer analysis, more honest reflections, and genuine learning.

For the past year, I have been working on the Advisory Committee of FRIDA – The Young Feminist Fund, which is a newly formed foundation that is run for and by young feminist activists. As FRIDA prepares for its first grant-making round, these questions about evaluation are on my mind. How do we tell our story to our first funders and how do we empower the first FRIDA grantees to take ownership over evaluation of their work? Can we be accountable and collaborative in evaluation? As a starting point, we are planning to develop the first grant evaluation process in partnership with FRIDA’s first round of grantees. As we begin to hash out exactly what the process will look like, I am grateful for these new tools that push us towards more creative, dynamic ways of articulating our goals, visions, and impacts.

Saving Congress From Itself: Can the Independent Payment Advisory Board make Congress’s Medicare cost control problems go away?

David Mitchell, MPA

With the Joint Select Committee on Deficit Reduction – a.k.a., the Super Committee – missing a major deadline this week, the prospects for a debt deal before the next election seem bleaker than ever.

Eventually, Congress will have to act. The long-term deficit situation is truly unsustainable and the sequestration trigger agreed upon in August will begin sharply cutting Defense Department programs and Medicare provider payments in January 2013 (assuming Congress and the President allow it to stay in effect).

But for those hoping that responsible decision-making on the country’s entitlement and tax programs will materialize after next year’s election, prepare to be disappointed. In particular, Medicare – the public health insurance program for the aged and disabled, and by far the largest contributor to our long-term fiscal mess – has been subject to congressional mismanagement now for years.

As Wes Joines pointed out in a post earlier this month, the Medicare physician payment system is broken and has been so for more than a decade. Private insurance carriers that participate in the Medicare Advantage program have been overpaid since 2003, when the Republican-controlled Congress set artificially high payment rates as part of the same bill that expanded subsidized prescription drugs at the government's expense. And members of both parties have proven themselves unable to withstand the temptation of using the Medicare program to steer benefits to special interests. Whether it’s boosting payments to rural hospitals, delaying competitive bidding for durable medical equipment, or shielding beneficiaries from scheduled benefit cuts, there are many recent examples of costly Congressional micromanaging on both sides of the aisle.

Underlying this mismanagement is a simple political calculus: members of Congress believe that they must avoid being linked to any policy that will hurt the country’s 47 million Medicare beneficiaries (not to mention the tens of millions more about to join the program) or risk defeat at the polls. Cutting benefits is one obvious no-no, but cutting provider payments is also politically dangerous, since doctors and hospitals may then stop treating Medicare patients or otherwise incite seniors’ anger.

So what can be done? One hope is that legislators themselves may be looking for a way out of this Medicare cost-control political vortex, especially given the hard decisions that most political elites know will have to be made as part of an eventual debt reduction deal.

One sign of this thinking is Congress’s recent decision to establish an Independent Payment Advisory Board (IPAB) as part of last year’s health reform legislation.[1] The IPAB is designed to take Medicare payment policymaking out of the hands of Congress and put it into the hands of expert technocrats. Though the IPAB was not a major focus of the yearlong debate on health care legislation (Americans were otherwise obsessed with abortion, the public option, and death panels[2]), it may prove to be one of the most consequential provisions included in the ACA. In the words of former budget director Peter Orszag, IPAB represents “the largest yielding of sovereignty from the Congress since the creation of the Federal Reserve.”

Here’s how it works: A 15-member board appointed by the president and confirmed by the Senate will propose sharp cuts to Medicare payments if cost growth in the program continues on its current trajectory. What makes this Board different from past Medicare commissions – or even the current Super Committee – is that the recommended cuts will go into effect unless Congress finds equal savings elsewhere in the program or supermajorities in Congress vote to waive the new rules (and even then only if the president signs the resulting bill). There is concern that future Congresses will not allow themselves to be constrained by these parliamentary hurdles and will try to prevent the cuts by simply not confirming IPAB appointees or passing a new law revoking some or all of IPAB’s powers. But the IPAB provision includes rules to check these congressional urges, so there is reason to believe that IPAB will have teeth.

Beyond the technical details, one’s optimism about the Board depends in large part on what one believes is ailing the US health care system. If high prices are the culprit (as many on the left believe), IPAB could prove effective at withstanding political pressure from doctors, hospitals, and other providers and keeping prices low. If over-utilization is the main cost driver (as many on the right believe), IPAB’s usefulness will be limited. This is partly by design: currently, the Board can only recommend changes to provider payment rates, not benefits.

But many outside experts – including some who sat on the US Fiscal Commission last year – recommend expanding IPAB’s powers. And the president has urged Congress to lower IPAB’s cost growth rate target, making it more likely that recommendations will be triggered. Some have even speculated that IPAB could be the vehicle by which a new all-payer rate setting scheme could be implemented.

It’s unlikely that Congress will go along with any of the above proposals any time soon. Indeed, many in Congress have called for IPAB’s repeal. And it’s true that further empowering a board of unelected technocrats is not an easy sell to the American people (especially since the biggest problems facing Medicare – and the federal budget as a whole – require moral, not mathematical, answers). But as the extent of our long-term structural deficit becomes more apparent – and the situation grows more urgent – members of Congress may be tempted to delegate more and more tough decisions to IPAB. Some might view this as an undemocratic and irresponsible abdication of authority – in other words, the coward’s way out. But as Edgar Allan Poe once wrote: “That man is not truly brave who is afraid either to seem or to be, when it suits him, a coward.”

----------------------

Notes

[1] The Affordable Care Act also created the Center for Medicare and Medicaid Innovation (CMI), which has broad powers to experiment with new payment systems like accountable care organizations and bundling, and then apply the most successful models nationwide – all without further congressional action. This is a promising idea, but the Congressional Budget Office and others are skeptical of its cost-saving potential and so it will not figure as prominently as IPAB in debt reduction negotiations.

[2] Since the bill’s passage, some have used the “death panel” moniker to describe IPAB, but during the debate that phrase was used in reference to a provision that attempted to expand the use of living wills.

21st Century Slavery: The scourge of human trafficking and how we can fight it

Elina Sarkisova, MPA

Human trafficking, although not a new phenomenon, has experienced exceptional growth in the last quarter of the 20th century, becoming one of the fastest growing criminal enterprises in the world. According to the State Department, an estimated 600,0000 to 800,000 men, women, and children are trafficked across international borders each year, the majority being women and children. When accounting for internal trafficking, the number of victims jumps to 12.3 million. Most states have taken steps, both on an individual and collective level, to try to reverse this trend; most noticeably in the form of international legal instruments and national laws aimed at migration controls, crime prevention, and victims’ assistance programs. However, these efforts have proved largely ineffective and are even to blame, in some circumstances, for making matters worse.

Why human trafficking is on the rise
The competing and often conflicting forces of globalization and national sovereignty help explain the increase in the overall volume of human trafficking. While globalization has brought about unprecedented opportunities for economic growth and prosperity, the results of economic liberalization (a key feature of globalization) have been mixed for developing countries. The swift spread of free markets has led to increased levels of inequality both within and between states and widespread economic instability, thus fostering strong “push” factors in migrants’ countries of origin. Meanwhile, increased demand for cheap labor in developed countries (characterized by a growing ratio of elderly to working-age people), declining birth rates, and high cost of labor, fosters strong “pull” factors.

On the other hand, governments have stepped up efforts to limit the movement of people across their borders, even when such policies are highly inconsistent with national labor market demands. This is usually the result of strong societal resistance to immigration and/or political pressures. Such restrictive immigration policies have the effect of increasing undocumented migration flows. However, trafficking flourishes where an additional factor is present: few opportunities to overcome barriers to illegal migration without the help of a third party. Most trafficking cases start out as voluntary transactions between a potential migrant and a smuggler – an exchange that is usually initiated by the migrant. However, the situation quickly deteriorates upon arrival at the destination country, where the victim often finds himself in a situation of forced labor.

National responses to human trafficking and why they haven’t worked
States tend to perceive human trafficking through the lens of national security, best addressed through migration controls, crime prevention and victims’ assistance programs. The United States, for instance, has increased border control enforcement along its 2,000-mile border with Mexico over the last two decades. However, strong migratory pressures render such efforts useless at best and counterproductive at worst, contributing to an increasing death toll in the border area and driving increasing numbers of economic migrants to seek the services of unscrupulous migration intermediaries intent on exploiting the gap between labor supply and demand. Law enforcement (e.g. police raids) has also become a key tool in countries’ toolbox for combating human trafficking. However, such efforts simply drive the trade further underground, where victims become more and more isolated and vulnerable to abuse, especially in the case of prostitution. Lastly, while some countries have initiated victims’ assistance programs, efforts have largely been limited to “rescue” operations that offer little in the way of long-term solutions and conditional assistance programs that overlook underlying incentives.

A way forward
The human security framework can offer a new, more useful way of conceptualizing an old problem – namely, from the standpoint of the driving forces behind the trade, the mechanisms that facilitate the trade, and the threat that such a trade poses to society. Broadly, policy response should focus on prevention (targeting underlying causes of supply and demand specific to each country), prosecution (law enforcement), and protection (victim assistance programs). Policy options should include, at a minimum:

• Women, children, and minority empowerment programs: Past projects aimed at the economic empowerment of women have, for instance, tended to be components of broader economic development programs rather than anti-trafficking strategies. Efforts need to be streamlined and incorporated into a broader anti-trafficking strategy.

• Immigration reforms: Destination countries should create temporary guest worker programs, perhaps in a more targeted way than some countries have already done, as part of a broader anti-trafficking agenda.

• Anti-corruption programs among border patrol officials: Trafficking would not be as widespread were it not for the help of corrupt government officials, especially in border patrol areas, who are complicit in this trade.

In the end, improving the track record of international anti-trafficking programs will require both structural changes in the manner in which the issue is conceptualized by states and operational changes in the manner in which programs are implemented. Only then could we realistically attempt to end this scourge of the 21st century.

The Dragon has been Domesticated: A benign view on the rise of China

Ankit Panda, BA

As early as 1941, historian Henry Luce controversially called the 20th century the “American Century.” And today, just 11 years in, many are ready to bestow the 21st century honors to China. In retrospect, one may argue it makes sense to dub the previous century an “American” one because the United States did, in a sense, singlehandedly drive the normative alignment of modern international institutions and norms, albeit only after Luce’s declaration. So China’s impressive growth and greater face on the international stage should cause at least some change, right? Conventional realist wisdom would have us think so, but there is good evidence that the People’s Republic, and the Chinese Communist Party, have neither the interest nor the ability to do so.

Under any regime of international law (institutional or treaty-based), states have options to 1) comply with statutes and norms, 2) create statues and norms, and 3) “evade” statutes and norms. Starting with the leadership of Deng Xiaoping, the PRC has moved more from a rule-evading country towards one more interested in complying with existing institutions. Although conventional wisdom and the media might paint a very different picture of Chinese compliance, a quick look at China’s record at the World Trade Organization and the International Monetary Fund (IMF), in addition to its response to the 2008 financial crisis, provides strong evidence to believe otherwise.

At the IMF, the PRC, once chastised for its stalwart rule-evading (and frankly, rule-ignoring) behavior, has moved towards greater compliance. While the ongoing Article IV consultations on China’s controversial monetary policy and the depreciation of the renminbi remain a source of friction, notably, Chinese rhetoric in defense of this policy emphasizes mutual interests (between the West and China) and demonstrates a real stake in the continued “success” of the global financial system. Similarly, China’s reaction to the financial crisis in 2008 was swift and in line with systematic recommendations from the IMF and the US – a $580 billion asset-relief program similar to TARP in the US. Furthermore, Chinese support of the Eurozone during its sovereign debt woes demonstrates a further willingness to take a stake in the success of the status quo (although this final point may be more politically-motivated than I’m willing to admit in this short post).

The PRC’s short history at the WTO so far also demonstrates a mixed compliance record trending towards greater compliance over the years. Initially, Chinese accession encountered several obstacles, but it ultimately managed to convince the organization that without China, the WTO wasn’t truly a “world” trade organization. The WTO is struggling with China, which, as a mechanically-complex economy based on a particular set of normative principles, refuses in several cases to acquiesce to the exogenous and incompatible norms of the institution. On the other hand, China's victory over the European Union in a December 2010 case demonstrates an example of Chinese rule-taking and compliance with the WTO. Granted, this paints China as a rule-taker when the rules enforce its self-interest. Such victories come at the cost of leverage for the West, but bring with them greater Chinese stakes in the success of the rules-based order. The liberal international order is slowly beginning to accommodate China.

While experts like Robert Kaplan and C. Fred Bergsten may identify the many political and security threats that result from a stronger China, it’s important to consider the less-exciting but equally important economic perspective. It’s difficult to say definitively that there is one lens through which we should view China – frankly that would be an oversimplification – but overall, it’s important to recognize that China has been accommodated into the current system of international political economy and that this accommodation has made it an important stakeholder in the success of that system. Unfortunately for the US, even the liberal international order will move us away from the “American Century” towards a more complex and multi-polar global order.

Game Changer: Israel’s new calculus on attacking Iran

Jordan Reimer, MPA


After a long hiatus, there is renewed international focus on Iran’s nuclear program, from hyperventilating Israeli media reports last month on an imminent attack on Iran’s nuclear facilities to the recent release of an unprecedented International Atomic Energy Agency report pointing to Iranian activities “relevant to the development of a nuclear device.”

In the international debate on what to do about Iran’s nuclear ambitions, for those who advocate taking military action against the Islamic Republic the conventional wisdom has settled on Israel doing the world’s dirty work – as it did against Iraq in 1981 and Syria in 2007. Those who oppose Iran's nuclear activities but are still circumspect of this course of action advance a counterargument that falls essentially along two lines of thought: logistical infeasibility and geopolitical inadvisability.

The first contention details several obstacles: a) the vast geographic distance between Iran and Israel, b) the fact that Iranian nuclear sites are dispersed throughout the country (a lesson learned no doubt from Iraq’s vulnerability), c) the reality that such sites are situated in hardened bunkers located either underground or deep inside mountain ranges, and d) the possible existence of secret sites unknown to Israeli and American intelligence.

A successful action plan would therefore mandate continuous multi-sortie strikes over several parts of the country by Israeli fighter jets. (Israel does have a long-range missile program, but such a weapon is inappropriate for precision strikes.) However, an attack of this sort would soon fall susceptible to advanced Iranian anti-aircraft capabilities. Israeli airplanes would therefore be limited to execute only a quick once-over before returning home. And, in light of Israel’s deep concern for the lives of its soldiers, given the distance between Jerusalem and Tehran a viable plan would require mid-air refueling to guarantee the pilots’ safe return, a near impossibility considering the non-hospitable airspace Israel must traverse to reach Iran.

With all of these constraints, even if Israel did successfully pull off this type of limited mission, it would hardly improve Israel’s national security: experts assume Israel would set back Iranian efforts only a few years at the most. If anything, it would be merely a strategic victory, signaling the resolve behind Israel’s rhetoric and indicating its sincere desperate determination to prevent Iran from going further in its nuclear work.

For all of these reasons, an Israeli attack with potential to severely cripple Iranian nuclear ambitions was largely considered a fantasy.

No longer. On February 21st, 2010, Israel Aerospace Industries unveiled a new unmanned aerial vehicle (UAV), the Heron TP, nicknamed “Eitan.” Able to fly at medium-to-high altitudes for over 20 hours while carrying a variety of payloads, equipped with a silent engine ensuring stealth, and significantly, capable of reaching Iran, the drone relieves Israeli military logisticians of concerns regarding refueling and the safety of its airmen against anti-aircraft artillery. The drones could be mass-produced and then flown incessantly until all Iranian nuclear sites are obliterated. Even if the drones are shot down, it would come at a relatively modest cost (the price tag might be steep but it would be infinitely better than an Israeli soldier in Tehran’s hands). The UAVs could be equipped with self-detonation devices should they be captured to avoid Iran gaining insight into Israeli military technology.

Importantly, the second line of reasoning against an attack – geopolitical considerations – remains unchanged: bombing Iran is still a terrible idea. It will indefinitely delay the moribund Middle East peace process, inflame the Arab and Muslim world against the US at a time when it has a legitimate chance to turn over a new leaf, force a wedge between both US-Israel and US-Arab/US-Muslim relations, and almost certainly provoke Hezbollah and possibly Hamas to launch retaliatory attacks against the Israeli civilian populace. Not least, it could conceivably convince Iran to finally declare its intention for a nuclear weapon as a deterrent against future Israeli aggression, a stand which it has consistently refused to take.

To (mis)quote Rudy Giuliani, “The use of military force against Iran would be very dangerous. It would be provocative.” Yes. And it shouldn’t be done. (Though it’s outside the purview of this article, let me at least acknowledge that though there is no silver bullet, a healthy mix of sanctions, coalition-building, and containment should be sufficient to keep Iran at bay, should it develop the dreaded weapon.)

The revelation of the “Eitan,” then, is both a blessing and a curse. On the positive side it signals to Iran Israel’s credible conventional deterrent against development of a bomb (no doubt why the UAV’s unveiling was so public), which might induce the ayatollahs to rethink any potential attempts to acquire such a weapon. And yet, on the other hand, it removes the only barrier that Israel’s security hawks faced in their myopic and monomaniacal desire to carry out a strike on Iran. We can only hope that cooler heads prevail and the remaining reasons buttressing the argument not to attack remain foremost in the minds of Israeli policymakers and American interlocutors.

Non-Communicable Diseases: The Sword of the Damocles for developing countries

Siddharth Chatterjee, MPP ’11
Dr. Ayham Alomari


Thanks to the excellent work done by the United Nations Children’s Fund (UNICEF), the World Health Organization (WHO), the Global Alliance for Vaccines and Immunization (GAVI), the International Federation of Red Cross and Red Crescent Societies (IFRC), the Bill and Melinda Gates Foundation, governments, and other institutions to ramp up immunization for vaccine-preventable diseases, it is having the desired effect in reducing infant and child mortality. This important initiative has to be kept up and would contribute greatly to achieving the UN’s Millennium Development Goals by 2015.

But there is another threat that looms like the Sword of the Damocles: non-communicable diseases (NCDs) such as cardiovascular, cancer, diabetes, and lung diseases. To some extent, of course, the increasing prevalence of NCDs is an arithmetic consequence of reducing morbidity and mortality from communicable diseases. If people do not suffer and die from communicable diseases, they typically will die from an NCD. The tragedy, however, is that humans around the world suffer and die from NCDs prematurely, long before the natural limit to which modern medicine and non-medical interventions, including community-based efforts, can push the onset of suffering and the occurrence of death. In addition, the high burden of NCDs makes them a laden threat to health and development. NCDs are barriers to poverty reduction, health equity, economic stability, and human security.

A recent issue of the Economist states: “Indeed, of the 36m people killed by NCDs [annually], some 80% live in low- and middle-income countries. These diseases are associated with increased prosperity and longevity, and the results are costly. The World Economic Forum estimates that NCDs will cost low- and middle-income countries $7 trillion over the next 15 years.”

The most striking feature of NCDs is that, contrary to what most people think, it’s not just the rich that are feeling the damaging effects of physical inactivity, obesity, and poor diets. Far from it, the poor and middle class within the developing world are facing NCDs-related deaths exponentially. According to the statistics above, each day there are 100,000 deaths from NCDs, with 80% occurring in the world’s poorest countries. And unless we act collectively and with conviction the future looks ominous. According to WHO, NCD deaths are projected to increase by 15% globally between 2010 and 2020.

With the lives of 36 million people annually at stake, we all know what it takes to prevent NCDs – healthy lifestyles. NCDs could be preventable by eliminating shared risk factors such as tobacco use, unhealthy diet, physical inactivity, and harmful use of alcohol. Those risk factors are the main contributors to the magnitude and scale of NCDs worldwide. But many hurdles stand in our way, most stemming from a lack of urgency as well as political will to deal with this growing scourge.

To find ways to engage broader community involvement in NCDs, prevention through dialogue and concrete action were the topic of an event co-hosted by the IFRC and the International Federation of Pharmaceutical Manufacturers Association (IFPMA) at this year’s UN General Assembly in September. The panelists – who represented leaders among Red Cross Red Crescent National Societies, WHO, the private sector, and academia (including the active participation of two prominent members of Princeton University) – reinforced the call for integrated multi-sector and multi-partner NCD prevention initiatives.

Greg Vickery, President of the Australian Red Cross, shared his experience in working with Indigenous Australians, who are among the most vulnerable to NCDs. “Our response is threefold – through the ‘Save-a-Mate’ resuscitation and education program tackling the alcohol problem they face; breakfast clubs that teach school children healthy eating habits; and the ‘Food Cents Programme’ that shows families how to eat healthily on a tight budget, i.e. simple strategies to support healthy eating habits.” Professor Uwe Reinhardt, James Madison Professor of Political Economy at the Woodrow Wilson School, reminded the audience that part of this community-based effort must be to make the individual be both able and willing to play an active role in the management of his or her own health.

So basically, the message we want to stress is simple. Says IFRC’s Secretary General, Mr. Bekele Geleta, “Humanitarian organizations such as the Red Cross Red Crescent, whose staff and close to 13 million active volunteers world-wide work closely with local communities, play an enormous role in bringing about changes in behavior and attitudes towards health and lifestyle.” We at the IFRC strongly believe that including NCDs in our health programs is consistent with our Red Cross Red Crescent Strategy 2020 to enable healthy and safe living. Our Framework for NCDs focuses on prevention, innovation & research, monitoring & evaluation, partnership, and advocacy.

But we cannot do it in silos or on our own. To make a real difference for those who are not on a health services radar screen, it is critical that we forge robust partnerships not only with governments, the UN, and NGOs but also with important thought leaders and opinion leaders like Princeton University. This has to be a collective and well-orchestrated endeavour to prevent what the UN has called “a public health emergency in slow motion” from spreading. Simple lifestyle change is the key. Resources – intellectual and financial, as well as partners from the public and private sectors alike – need to come together.

The IFRC is keen to move forward on this important initiative. In the words of Professor Reinhardt, “One thinks of the Red Cross Red Crescent more in connection with earthquakes, tsunamis, and other spectacular and sudden natural disasters. I was surprised and encouraged to learn how much the organization is doing in response to another, albeit slowly developing, natural disaster, the growing burden of premature deaths due to NCDs, much of that burden the result of people’s inability or unwillingness, or both, to manage their own health better.”

The actions, then, appear to be simple, and the outcomes desirable. But it will take a significant investment of time and money to strengthen the ties that bind the Sword above us.



Siddharth Chatterjee is the Chief Diplomatic Officer and Head of International Relations at the IFRC. Dr. Ayham Alomari is a Senior Health Officer, Community Based Health and First Aid, NCDs at the IFRC.

Vying with Velker: RCTs reconsidered

Shawn Powers, MPA ’11


In “Randomized Controlled Trials on Trial,” Jake Velker proposes several reasons to be skeptical of randomized controlled trials (RCTs) as a method of program evaluation. While Jake makes some good points, as a “randomista” I think the picture he paints of the RCT movement is far too pessimistic. I will consider each of his arguments in turn.

1. External validity
Jake first takes up the critique, championed by Princeton’s Angus Deaton, that RCTs suffer from external validity problems—in other words, the results of an evaluation may not generalize well to other contexts. While the criticism is frequently leveled at RCTs, the question of external validity applies to all empirical work. In general, I find these discussions about the differences between Deaton and RCT proponents a bit overblown. Perhaps this is because everyone loves a good spat between high-profile intellectuals (see also: Sachs and Easterly). In reality, according to the profile of Esther Duflo in The New Yorker that Jake references, Deaton has described his criticisms “as more in the form of an amicus brief than an attack." The arguments raised by Deaton and others are having an impact, as Jake acknowledges, and RCTs increasingly are testing rich behavioral hypotheses. The incentives of academic publication are also moving RCTs toward greater theoretical sophistication. Gone are the days when a randomized design was a novel enough identification strategy that it could propel a study to publication in a top economics journal.

Considerations of theory aside, whether or not a particular result generalizes is itself a testable, empirical question. We cannot—and should not—test everything everywhere, but if a particular approach proves effective in multiple contexts, our confidence in its “generalizability” should increase accordingly. Replication studies can also test variations in the length or intensity of treatment, disentangle the impact of different components of a program, or test how a small-scale intervention performs as it is scaled up.

If the goal is to achieve certainty that intervention X will achieve result Y in context Z, we will never achieve it, with RCTs or any other method. However, considering evidence from even one rigorous evaluation is a big improvement over flying blind. As we consider multiple evaluations, together with insights from theory and other empirical work, the picture becomes that much clearer.

2. Institutional constraints
Jake’s main criticism is that economists conducting RCTs “have been accused of ignoring the institutional constraints against which their interventions would inevitably contend if scaled up.” He cites corrupt bureaucracies, weak institutions, a lack of (or perverse) performance incentives, and budgetary problems as barriers to successful replication of programs found to be effective. The underlying message seems to be that if the RCT movement wants to influence policy successfully, it cannot just publish research findings and hope for the best.

I could not agree more with this last point, but the RCT community is much farther along on this front than Jake suggests. Both J-PAL and our sister organization, Innovations for Poverty Action (IPA), have policy staff dedicated to bringing research findings to bear on the often-messy world of policymaking. While our academic affiliates are involved with policy outreach, non-academic policy staff help extend the reach of their research findings. This process is never easy and not always successful, for all the reasons Jake mentions, but we have found that it is possible to improve policy even in very constrained environments.

As an aside, Jake also suggests that governance in developing countries is not amenable to quantitative study. I would have thought the same before I started with J-PAL, but in fact, J-PAL affiliates currently have at least 42 completed or ongoing evaluations in political economy and governance, including many that address precisely the issue he raises of the incentives of government officials and service providers.

3. Do we already know what works?
Finally, Jake entertains the idea that perhaps we already know what works, since “many of the most celebrated finds of the RCT movement are relative ‘no-brainers’.” I find this argument troubling for two reasons. First, we have been following our intuition about what works in development for decades, with not a lot to show for it. What we have seen is succession of fads, with decidedly mixed results in terms of reducing poverty. There was a time when infrastructure was the “no brainer,” later it was basic needs, still later the focus turned to sustainable development, and today infrastructure seems back in vogue. To suggest that we already know what to do invites just this kind of intellectual drift.

Second, while it may be true that many RCTs report seemingly obvious findings, some of them surprise us—and we never know in advance which those will be. For example, a number of NGOs and opinion leaders have championed the idea that distributing sanitary products to adolescent girls will remove a barrier to female education. The underlying common-sense assumption is that menstruation causes many missed days of school. However, a randomized evaluation of a program that distributed an easy-to-use sanitary product in Nepal found no significant effect on school attendance (although the girls used, and liked, the product). As always, we should avoid over-generalizing from one study, but at minimum these findings suggest that proponents of this approach should adjust their expectations about what it can deliver. In other cases, RCTs have contributed clear evidence to debates where both camps have common-sense arguments on their side, such as the vexed issue of whether, and how much, to charge poor people for basic health and education products and services. Finally, even if the qualitative findings of an RCT seem to confirm common sense, policymakers may still want to know how an intervention stacks up quantitatively against other interventions with the same goal, in terms of both raw impact and cost-effectiveness.

RCTs are no more a panacea for development than anything that came before them. But as long as there is more ideology and wishful thinking in development policymaking than evidence, I believe that the continuing growth of the RCT movement is a welcome trend.

Shawn Powers is a Policy Manager at the Abdul Latif Jameel Poverty Action Lab (J-PAL). The opinions expressed here are his own.

New Year's Irresolution: Medicare’s sustainable growth rate and physician reimbursement

Wes Joines, MPA


Unless Congress acts between now and the end of 2011, at least one group will not be experiencing a happy New Year: physicians who provide services to Medicare patients. Under current law, starting in 2012, reimbursement for Medicare-provided services will be reduced by an estimated 30%. Why is this happening? It is all related to policies enacted nearly 15 years ago in an earlier iteration of debt reduction efforts.

The Balanced Budget Act of 1997 was signed into law on August 5, 1997 and was designed to balance the federal budget by 2002. Of its $160 billion in spending cuts during that time period, $112 billion was applicable to the Medicare program, which is the primary health coverage program for older and some disabled Americans. A key component of the cuts to Medicare included, for the first time, a budgetary restraint on Medicare’s total expenditures to maintain budget neutrality. Known as the sustainable growth rate (SGR), it is a major component of the current formula for determining annual updates to physician reimbursement. While Medicare payment rate increases since 1992 had been tied to trends in physician utilization (i.e. efficient use of medical tests and facilities by a doctor), in 1997, for the first time, the implementation of the SGR meant that Medicare reimbursement changes would be linked to four factors: 1) changes in input costs, 2) changes in Medicare fee-for-service enrollment, 3) changes in the volume of physician services relative to growth in the national economy, and 4) changes in expenditures due to changes in law and/or regulation.

The SGR resulted in annual increases to the Medicare fee schedule until 2002, when a 4.8% reduction took place. Since that time, rate reductions called for by the formula have been deferred, although Congress has not changed the underlying SGR formula or the cumulative spending targets. Because of vast increases in the volume and complexity of health care services for the Medicare population in recent years, especially when compared to the SGR designers’ projections, the formula specifies cuts in physician payments that become more severe with each passing year. In fact, at a cost of $19 billion, a last-minute December 2010 vote delayed a scheduled 25% reduction in the SGR that was to take place in January 2011.

So, here we are again, this time in late 2011, deciding whether or not reimbursement for Medicare providers will be cut. Even before the current debt reduction debate and increasing prevalence of political gridlock in Congress, policy movement regarding the SGR involved numerous short-term fixes. For example, from 2003 through 2010, Congress included provisions in 13 separate pieces of legislation to forestall reimbursement cuts. As a long-term fix for the SGR – e.g. replacing it with a current fee freeze – would be extremely costly to the taxpayer (some estimates currently peg it around $300 billion over 10 years), short-term fixes have generally proved to be an easier bargain (as much as they have irritated physicians and their respective trade associations).

At this point, anyone’s guess is as good as another’s regarding the level of reimbursement for Medicare services on January 1, 2012. Although the current political climate is not one that generally supports massive spending to doctors that would be required for a long-term fix, many believe that cuts of the magnitude prescribed by the SGR would not be conducive to ensuring beneficiary access to services. Therefore, another short-term fix might be in the works as a stop-gap measure. However, there is also a chance that the currently-convened deficit reduction “Super Committee” might address the SGR as part of its proceedings.

If compromise is within reach, within or outside of the Joint Select Committee on Deficit Reduction, it may be similar to a plan recently recommend by the Medicare Payment Advisory Commission (MedPAC), which ironically enough, was also established by the Balanced Budget Act of 1997 and serves as an independent advisor to Congress. MedPAC’s plan, which would cost $200 billion over 10 years (instead of the $300 billion of the fee freeze), would protect both primary care and specialty physicians from the deep cuts called for by SGR. Primary care physicians would see physician fees associated with Medicare services frozen for 10 years, while specialists would see smaller cuts (of 5.9% per year) over the first three years that would then remain frozen for the remaining seven years in the budget window.

Granted, MedPAC’s suggestion is not a panacea, but it is a good start. At the very least, it should focus us on attempting to resolve this looming crisis.

Skip Over Low-Hanging Fruit to Reach the Millennium Development Goals

Leslie Lai, MPA


In 2000, the UN established the Millennium Development Goals (MDGs) to improve the social and economic conditions of the world’s poorest countries by 2015. Of the eight MDGs, two specifically target the reduction of child mortality and the improvement of maternal health. Despite increased aid in the past decade from foundations, bilateral donors, and multilateral organizations, several countries in Sub-Saharan Africa will fail to reach these two goals in the next four years.

The main reason for this is the lack of an equity focus in the planning processes of national health ministries. For economic and political reasons, governments have focused on targeting “low hanging fruit,” or populations that are most easily reached, through the expansion of traditional delivery service mechanisms such as hospitals. While there has been significant progress in health outcomes due to increased international aid, improvements in national averages conceal widening disparities in poverty and worsening health outcomes for the most marginalized populations. Furthermore, the perceived difficulty and economic inefficiency in reaching the poorest hinders donors from targeting technical assistance to those who need the most help.

A practical solution to this challenge is to show evidence that integrating an equity focus into health planning can actually be cost-effective. Organizations such as the United Nations Children’s Fund (UNICEF) has proven this with a bottleneck analysis budgeting software created with the World Bank to assess the marginal costs of scaling up coverage of proven high-impact interventions. The tool incorporates over 186,000 input variables including the building of new facilities, vaccine transport costs, training of community health workers, etc. It also measures the potential number of lives saved per US $1 million invested depending on the mix of health services delivered and to whom.

Using health data from 68 developing countries, UNICEF used the software to show that each of these countries could not only provide essential health services to the most deprived in an economical fashion, but could also potentially achieve the health-related MDGs by 2015. Surprisingly, UNICEF’s analysis also showed that countries ignoring the hardest-to-reach would miss the MDG targets. So just what exactly does a country need to do to reach the MDGs and expand coverage to its hardest-to-reach beneficiaries?

Ethiopia is a successful example of a country that has successfully incorporated the bottleneck analysis tool into its national health plan to expand coverage to its underserved. Ethiopia’s key intervention is the training and deployment of Health Extension Workers (HEWs), young women who provide immunizations and maternal health services to children and women in rural areas. Based on marginal budgeting calculations, Ethiopia could potentially save 68.1 children’s lives per US $1 million spent with the HEW program compared to only 52.9 lives saved without. To achieve this, donors and technical assistance would need to enhance the HEW program or implement similar types of interventions.

Since Ethiopia has historically experienced serious health problems among women and children, it is an ideal model for Sub-Saharan African countries with similar issues. Unfortunately, political matters may obstruct successful implementation in other countries. And in the meantime, the clock is running out…