NOTE: The views expressed here belong to the individual contributors and not to Princeton University or the Woodrow Wilson School of Public and International Affairs.

Friday, December 16, 2011

Testing Treatments: Building a culture of evidence in public policy

Brett Keller, MPA

Back in September the New York Times reported on an unexpected finding from a clinical trial: “A promising but expensive device to prop open blocked arteries in the brain in the hope of preventing disabling or fatal strokes failed in a rigorous study.” Many promising medical innovations fall short when they finally reach clinical trials, but this story was unusual because the stents had already been approved by the FDA under a so-called humanitarian exemption. The FDA approved the stents to reduce the risk of stroke, but those who received it had twice as many strokes.

How did this happen? The Times chronicled experts’ puzzlement: “Researchers said the device seemed as if it should work.” And Joseph Broderick, a prominent neurologist, is quoted as saying “Quite frankly, the results were a surprise.” Researchers are delving into this case to discover why the stent failed, but policymakers from all fields should take it as a valuable lesson. This is one more argument for testing policies whenever possible: not only does expert opinion sometimes get things wrong, but without good data there is often no way to really know when they are right.

Similar lessons can be gleaned from the history of surgical response to breast cancer. In The Emperor of All Maladies (2010), a new history of cancer, oncologist Siddhartha Mukherjee chronicles the history of such failed interventions as the radical mastectomy. Over a period of decades this brutal procedure – removing the breasts, lymph nodes, and much of the chest muscles – became the tool of choice for surgeons treating breast cancer. In the 1970s rigorous trials comparing radical mastectomy to more limited procedures showed that this terribly disfiguring procedure did not in fact help patients live longer at all. Some surgeons refused to believe the evidence – to believe it would have required them to acknowledge the harm they had done. But eventually the radical mastectomy fell from favor; today it is quite rare. Many similar stories are included in a free e-book titled Testing Treatments (2011).

As a society we’ve come to accept that medical devices should be tested by the most rigorous and neutral means possible, because the stakes are life and death for all of us. Thousands of people faced with deadly illnesses volunteer for clinical trials every year. Some of them survive while others do not, but as a society we are better off when we know what actually works. For every downside, like the delay of a promising treatment until evidence is gathered properly, there is an upside – something we otherwise would have thought is a good idea is revealed not to be helpful at all.

Under normal circumstances most new drugs are weeded out as they face a gauntlet of tests for safety and efficacy required before FDA licensure. The stories of the humanitarian-exemption stent and the radical mastectomy are different because these procedures became more widely used before there was rigorous evidence that they helped, though in both cases there were plenty of anecdotes, case studies, and small or non-controlled studies that made it look like they did. This haphazard, post-hoc testing is analogous to how policy in many other fields, from welfare and education, is developed. Many public policy decisions have considerable impacts on our livelihoods, education, and health. Why are we note similarly outraged by poor standards of evidence that leads to poor outcomes in other fields?

A recent example from New York City helps illustrate how helpful good evidence can be in shaping policy. A few years ago Mayor Michael Bloomberg rolled out a massive program that seemed to make a lot of sense: pay teachers bonuses based on their students’ performance. The common sense proposal was hailed as “transcendent” and gained the support of the teachers’ union. It cost $75 million, and it didn’t work. How do we know? The program was designed from the beginning as a pilot where schools were randomly assigned to the program or to a control group, and the research showing that the program had no effect on outcomes was subsequently published. What would have happened if this policy had been put in place without an effective evaluation plan? In all likelihood New York officials would now be touting its success at conferences and urging other cites to implement similar programs. Instead it was quietly shelved. That this particular program did not have the intended effect is disappointing, but it is much better than if we believed it worked and continued on unaware.

The pros and cons of randomized trials have been discussed here on 14 Points before – see recent posts by Jake Velker and Shawn Powers. The cases I presented here are ones where the results were not “no-brainers” at all, and without systematic evaluation bad policies would have been or tragically were put in place. While good evidence does not have to come from randomized trials, there are still many areas where they are underused. In areas where they are feasible (i.e. not macroeconomics) such evidence should be the norm, and those who implement policies with great optimism but without planning for thoughtful evaluation should be panned. Even without random assignment of the treatment, the best policy evaluations should involve a serious attempt to estimate the counterfactual: what would have happened in the absence of the intervention. Moving beyond arguments over specific programs and whether they work, policymakers can move us towards better outcomes by creating a culture where strong evidence is valued. After all, the clinical trial as we know it in medicine is a 20th century innovation; it hasn’t always been this way.

1 comment:

  1. How to Get Your Ex boyfriend Back "I suggest you read this because I’m extremely happy that me and my bf living together again"‎ Am Mary 21yr from England, my boyfriend of a 4yr just broke up with me and am 30 weeks pregnant.I have cried my self to sleep most of the nights and don’t seem to concentrate during lectures sometimes I stay awake almost all night thinking about him and start to cry all over again.Because of this I end up not having energy for my next day’s classes ,my attendance has dropped and am always in uni and on time.Generally he is a very nice guy ,he ended it because he said we were arguing a lot and not getting along.He is right we’ve been arguing during the pregnancy a lot .After the break up I kept ringing him and telling him I will change.I am in love with this guy and he is the best guy I have ever been with.I’m still hurt and in disbelief when he said he didn’t have any romantic feelings towards me anymore that hurt me faster than a lethal syringe.He texts me now and then mainly to check up on how am doing with the pregnancy,he is supportive with it but it’s not fair on me, him texting me as I just want to grieve the pain and not have any stress due to the pregnancy.i was really upset and i needed help, so i searched for help online and I came across a website that suggested that Dr Unity can help solve marital problems, restore broken relationships and so on. So, I felt I should give him a try. I contacted him and he told me what to do and i did it then he did a spell for me. 28 hours later, my bf came to me and apologized for the wrongs he did and promise never to do it again. Ever since then, everything has returned back to normal. I and my bf are living together happily again.. All thanks to Dr Unity. If you have any problem contact Dr.Unity now and i guarantee you that he will help you.Email him at: ,you can also call him or add him on whats-app: +2348071622464 ,His website: .