What Do We Do With Data Soup?

Katherine DiSalvo, MPA


As policy professionals, we’re likely to encounter messes of contradictory findings more and more throughout our careers.

There is contradictory data on important issues like the real level of US poverty, whether moving people out of a neighborhood of concentrated poverty improves their chances in life, the success of charter schools, or the effectiveness of giving away free bed-nets to combat malaria.

Do you know what to do with data soup? At the Woodrow Wilson School, I don’t think we students learn this sufficiently.

According to R. Kent Weaver, in Ending Welfare as We Know It (Brookings, 2000), the 1980s and 1990s saw a “multiplication” of policy research with “differing assumptions and conclusions.” Simultaneously, interest groups were adopting social science techniques and creating “a welter of conflicting findings.” In a separate article Weaver and a colleague assert that this may result in the “devaluation of the currency” of policy research. Weaver argues that it may “cause legislators to simply dismiss all evidence that does not fit their personal or constituency preferences.”

Devaluation of policy research is becoming commonplace. Even in the era of “data-driven” education leadership, Brenda Welburn, the head of the National Association of State Boards of Education (NASBE), recently told a WWS workshop team researching “School Choice and Impacts on Cities” that State Boards of Education members don’t know whose data to trust. As board members attempt to make state education policy and funding decisions, sometimes they don’t know how to do it with facts. “We [at NASBE] are dealing with perceptions, often,” Welburn said.

I don’t think it’s easy to digest data soup, and I think the Woodrow Wilson School needs to do more to help its students develop this ability. You may scoff and tell me you know how to wade through the stew. You know statistics! You know what research methods matter!

I don’t think any policy professional can rely on statistical prowess alone. The statistics program at the Wilson School is strong, and its decision to expand statistics requirements was a good one. However, with our limited time we students (not to mention professionals) cannot dig into data sets, look at assumptions, and evaluate every conclusion we read for ourselves. While some such analysis might be possible before an important policy decision or publication, we consume too much information to scrutinize it all.

The best proof that policy students won’t always use technical skills to sort through conflicting data professionally is that Woodrow Wilson students don’t always do so here! When I encounter conflicting data in classes, I’m too often told we students should dig deeper and decide who’s right…later.

We can’t rely exclusively on the “the gold standard” professors teach us to love: data generated by randomized control trials (RCTs). This creates an easy top tier of information on too few topics. Additionally, all the emphasis we hear on the “gold standard” may lead us to trust in RCT-based research too easily. The best part of the WWS course on data-based decision making is hearing Professor Lorenzo Moreno talk about how complicated it can be to do the right thing in the evaluation field. All that glitters…

We policy students need more practice criticizing questionable research. We need more practice wading through data mess and taking and defending a stand – not on politics, as we do in the introductory 501 course, Politics and Public Policy, but a stand on what we think is the truth. We need more sophisticated conversations about what data to trust and about how to evaluate vendors of policy research when we cannot evaluate each product. We need more shorthand than one “gold” standard.

We also need to talk about making policy in a world where different “facts” are consumed by different constituencies, and the truth is always up for debate. It’s the world in which we live, and it’s likely to get worse. If the Woodrow Wilson School could prepare us to digest data soup and to help change these cooking trends, that would truly be in the nation’s service and in the service of all nations.