Field Notes: Does it Work? How evaluations shape policy and decision-making
Governments spend billions on programs. Ardyn Nordstrom’s research asks how evaluations shape — and sometimes fail to shape — policy decisions.
What are you focused on these days?
My research focuses on two related questions about how evidence on “what works” gets made and used by policymakers. The first is “what works to improve health and education outcomes in low-income countries, and how do we know this?” The second is “how does evidence get used once it exists in Canada?”
On the first question, I’ve spent several years working with data from the Girls’ Education Challenge (GEC), which is the largest-ever donor-funded initiative in girls’ education around the world. All of these projects underwent large-scale evaluations to promote learning across the portfolio, and most of my work has focused on evaluating single programs within this portfolio. But more recently, I’ve been wrapping up a SSHRC-funded project that looks at all thirty-seven GEC projects at once to see how the design of the evaluations of each project affected the decisions about how projects were improved over their lifecycle.
On the second question, I recently published a new paper that examines every evaluation report published in the last ten years by the ten largest federal departments in Canada. I looked at how evaluators define “effectiveness” and found that departments have been drifting towards a focus on how programs are being implemented rather than also focusing on the outcomes these programs achieve, even though Canada’s federal Policy on Results requires programs to focus on outcomes.
Why is this work important right now?
Governments and donor agencies are facing hard choices about what to fund, and knowing what works is essential to make good decisions here. At the same time, making sure evaluations that provide this kind of evidence are done in a way that is informative to policymakers and are ready when they’re needed is really difficult, especially when evaluation systems within organizations are complex. So I’m excited that all the areas of my work have been leading me to think more deeply about not just which programs work, but also the evidence-generating systems that allow us to figure that out. This is particularly relevant in Canada right now, with the national Policy on Results under review. This means the rules governing how every federal department evaluates its programs are being reviewed for the first time since 2016, giving us a rare window where empirical work on evaluation practice can truly shape evaluation policy rather than just describe it.
What is a question you hope to answer with your research?
In the broadest sense, I want to know “Does the kind of evidence an evaluation produces change what an organization actually does afterward?” If evaluations take too long, don’t answer the right questions, don’t create evidence that people trust, or are sycophantic, then evaluations become a performative check-box exercise that uses up valuable time and resources. We have a large literature on evaluation quality and a separate literature on evaluation use, and they rarely meet. I want to bridge that gap.
What is something people would be surprised to learn?
Lots of people have no idea that the federal government publishes reports on nearly every federally funded program in Canada. There are thousands of pages of findings about what works (and doesn’t work) in Canadian public programs that almost no one reads. A lot of my work lately has been focused on using new natural language processing methods to analyze these altogether to figure out what’s working within federal programs (and the evaluations themselves!) in Canada.
Another thing that continues to surprise me is how rarely federal evaluations in Canada try to establish causality in federal programs. While this is definitely not the right approach for every project, there have been huge advancements in methods that can be used to figure out whether programs cause a change in the outcomes they’re targeting. These are widely used by economists and other scholars evaluating programs for academic work, but are barely ever used within federal evaluations even when they could be tremendously informative, and I think this is often a missed opportunity.
What’s the biggest misconception about your research area?
Despite my last answer, a common misconception is that methodological complexity is always better. It’s vital to be accurate and rigorous in evaluations, but doing something more complicated like a randomized controlled trial isn’t always better if it answers a question that no one is asking, or will only be ready after answers are needed. Complexity comes with tradeoffs because more complicated evaluations cost more time and resources, so finding the right kind of evaluation design needs to consider the balance of these tradeoffs.
Any new projects you’re excited about?
Yes! I’m currently working on an exciting extension of the paper I published earlier this year on the Policy on Results, looking at what happens after the reports are written. I’m doing this by tying each evaluation report to the management response and action plan, where departments declare what they’re committing to doing in response to an evaluation’s findings. These are the closest thing we have to a record of evaluation use, so I’ve built a new dataset that tracks the relationship between the evaluation findings and the action plans. This will let me figure out what types of evaluation findings are most useful for decision makers, and what kinds of evaluation design choices helped make the findings useful.
What’s your favourite class to teach?
We have the best students at the School of Public Policy and Administration, so this is a really hard choice! I love teaching Evaluation Design for our Diploma in Program and Policy Evaluation students. This is where students get to apply all the theory they learn about evaluation design and bring it to a real-world project that they evaluate over the course of their 16 months in the program. Helping students navigate the very real challenges of doing an evaluation in practice and watching these turn into useful evaluations is such a joy. But I also really love teaching our Microeconomics for Public Policy Analysis in the School’s Master’s of Public Policy and Administration program. A lot of students come to the course feeling either worried about the math involved or very disinterested in all things related to economics. It’s an exciting challenge to get students engaged to see why using economic analysis can be so useful. Usually, by the end of the course, I can directly see the impact my class has had on how students see the world, which is incredibly rewarding.