Michael Lenczner is the founder and CEO of Ajah, which works with governments, academic institutions, nonprofits and funders to design strategies that increase the availability and utility of data. Lenczner is a leader in the area of data use for social impact and has been working in the area of public-interest technology since 1999. As part of PANL Perspectives’s series on Data, he spoke about issues and challenges related to data technology and training in the nonprofit sector in Canada. Sections have been edited for brevity.
Question: Why is “data” important in the nonprofit sector, and where are we currently?
Lenczner: What gets measured, gets managed. In our sector, we measure things to understand what we’re doing and how we’re doing them – and what their impacts are. A lot of organizations are still in a space of learning about data and trying to understand how to use it to help their clients. Ajah helps with quantitative data and operational data, and things that are less easy to guess at.
Lenczner: Anyone with a passing interest in statistics knows that things that seem obvious may not actually be so, and that it’s easy to mix up cause and effect — and it’s tricky to nail down some things, such as data-collection processes.
For example, I once spoke to an executive director of a social service agency outside of Ottawa, and it had a mandate to deliver multiple social services in different areas — and it was funded by multiple governments departments and used separate reporting systems. But it was complicated just to find out how many clients that agency saw each day, because separate reporting systems couldn’t talk to one another, because the separate IT systems didn’t work that way. It turned out that the key data the agency needed had to be collected using pen and paper by people at welcome desks.
Q: In terms of access to data in the nonprofit sector, how does Canada compare to the US and UK?
The US and UK are way ahead of Canada in terms of access to national data.
Lenczner: While our sector can certainly improve our ability to access information, I think in general, we have a very professional, very skillful sector that delivers social services and supports.
But availability of data in Canada isn’t as good as it is in the US, where they’re much better at this, partly because in the US, any data owned by the feds is owned by the public and is publicly accessible.
In Canada, anything owned by the feds is automatically Crown-owned and isn’t easily accessible; we have a more paternalistic government.
In the UK as well, data are more easily accessible, because they’re organized at a national level — and not divided at a provincial level, the way they are here. For example, in Canada, we have a federal carceral system and a federal justice system, and separate provincial and territorial carceral and justice systems on top of that as well. So, to get access to court filings, you have to go province to province, and, as a result, our data use isn’t as sophisticated as it would be in a larger, unitary system.
Q: Is there a lot of support for data- or evidence-based policy in our sector?
“Evidence-informed policy is policy that isn’t purely conjecture or isn’t rationally derived or designed without real-world experience or information. Lived experience is a great source of knowledge, as is information gathered through anthropology-type tools and methodologies, but having quantitative data is really helpful, too, because it gets you away from the idea of choosing anecdotes with which we’re overly familiar — gets you closer to seeing a population.” –Michael Lenczner
Lenczner: It’s important to increase the capacity of the nonprofit sector to understand what policies the sector needs, and to advocate for those policies. It’s pretty straightforward to say we need better funding, but many policy questions aren’t that simple and don’t have easy uptake in our sector. So, having not only quantitative data, but more evidence-informed policy, is a good thing, but I don’t think there’s a lot of support for that in our sector across Canada.
Q: What’s an example of evidence-informed policy — or where do we need it?
Canada Revenue Agency offices. The CRA changed its guidelines about who foundations can give money to. Lenczner says evidence-based policy would involve following up on the CRA change and be on the lookout for unintended consequences, such as fraud.
Lenczner: The Canada Revenue Agency (CRA) changed the definition of a non-qualified donee. Foundations are now more free to give money to nonprofits that were previously considered non-qualified donee. One of the reasons for the change related to groups being excluded from getting access to a charitable designation. Now, people who couldn’t previously access foundation money can — and there would be benefits flowing from that. That’s the hope.
The downside is that the CRA change deregulates foundations — and foundations are tax vehicles for people with resources, with money. There’s already fraud in our sector, and it’s not something we talk about very much, and I’m not sure how much fraud exists, but if you make fewer rules for people who have money, then there’s obviously the possibility of increasing fraud or increasing undesirable outcomes from that deregulation. So, evidence-based policy would involve following up on the CRA change and be on the lookout for unintended consequences. In the context of declining trust in Canadian institutions in general, the charitable sector is one of the few areas in which Canadians have maintained high trust, and we should certainly try to maintain that trust.
Q: What are limitations to data and ‘datafication’?
“We should be a lot more circumspect about applying an MBA, data-is-good approach to all organizations. We should sometimes get out of people’s way, let them do their work, and have faith in their organizational ability to deliver, versus asking them to prove with data that their interventions are having an impact.” –Michael Lenczner
Lenczner: It feels like there’s a tug of war between how much data can help us and how much it can hurt us, or, put another way, the counter-position spills into “We don’t like Facebook” or “We don’t like Elon Musk and Twitter,” so therefore, we shouldn’t use much technology. I get confused about how that position relates specifically to the delivery of social services. In any case, there are clear benefits to using data. For example, data or evidence helps organizations to better understand clients and the assumptions underlying services, similar to a doctor providing data about a life-or-death decision.
But ‘datafication’ is another thing to worry about. If you’re an organization seeing 30-40 people a week, and you have a staff of 10, it’s not exactly clear why you need quantitative methodologies — or why you need numbers to improve your outputs or impacts. If you’re looking at $100 million, you should be asking very different research questions than a program run with $1 million. I think the sector lacks some nuance around this conversation.
Q: Who are the big players in this part of the sector — and how do governments and funders fit in?
Lenczner: Why are so many organizations doing low-quality research? Is this cargo cult behaviour, where small groups mimic what the big guys do? If your evaluation isn’t statistically significant, then you can’t draw inferences from it. You can’t just say, “Take this with a grain of salt,” because the data and inferences could be very misleading. Statisticians get frustrated by this, especially when they’re brought in at the end of a project and someone drops a bunch of data and reports on the desktop and says, “Make something out of this.” Most statisticians say, “This is garbage. I can’t do anything with this.” We need to bring statisticians into the research process at the beginning, and design research protocols together.
Lenczner: I’ve been in the sector a long time, working mainly with small organizations, with budgets under $1 million each, and mainly in community involvement around transparency, democracy and engaging youth, but it was only after 15 years that I heard about a nonprofit organization called the Social Research and Demonstration Corporation. I looked at what they did in terms of data and evaluations, and their evaluations were done by teams of people with PhDs, and the budget for an evaluation was $1 million — and those are the evaluators that governments use when they want to change social programs. So, to some extent, there are two levels in the nonprofit sector.
On one level, small organizations are told to go off and make their own evidence and, by hook or by crook, come up with answers, but, at another level, when there are more dollars on the line, the rigour of the evidence changes, and governments use an entirely different set of tools and groups to generate data — and they allow select groups to access government databases.
For example, only select groups can access a government database of everyone who subscribes to disability supports, or everyone who gets unemployment insurance, or everyone who ended up back in jail. For recidivism, you can use data from entire populations (prison populations), and look at which interventions in jail and outside of jail led to recidivism. You get completely different answers with a $1 million evaluation using population-level data sets than you would from an $80,000 evaluation that you give to a nonprofit — to answer the exact same questions regarding what interventions work and what don’t.
Earlier in my career, I learned that small organizations had been playing in the kiddie pool — we didn’t have access to population-level databases. Now, foundations are out there telling everyone they need to determine their impact, need to be doing research, but if your organization sees only a limited number of people each month, then you can’t easily know your impact numerically. Data usually involves hard research questions, and you have to answer them with rigour, but instead, everyone is out there doing low-quality research on their own, and when you put that low-quality research together, you don’t get very much. So, why are organizations getting asked to do this in the first place? Sometimes I wonder if it’s just to make the funder happy.
Q: What can we expect from governments in terms of change, and what should our sector do in response?
There’s a lot happening around data use within government. Employment and Social Development Canada is collaborating more closely with Statistics Canada for example. The nonprofit sector needs to reach out more to the government.
Lenczner: At the federal level, Statistics Canada got $800 million in a recent budget to increase its capacity. It’s being funded well, and the sector needs to reach out to find out what the new methodologies are, find out about new tools, and find out how to use them.
In Ontario, the government created the provincial Data Authority, which is doing work to suck up data from major departments across Ontario and to do research and policy work with those data sets — and to make those data sets available to policymakers and researchers.
So, governments are moving forward with their abilities to use their own data and increase access to that data, and the nonprofit sector needs to engage with that. It takes two to tango. If we don’t show up and ask questions, then, of course, government is going to ignore us and our needs and priorities.
Q: Is there anything else we can do to improve the data situation in our sector?
“To improve our ability to know things and to use data for the charitable sector in Canada, one of the important things we need to do is familiarize researchers and policy makers with quantitative methodologies.” –Michael Lenczner
Lenczner: We’ve had strong capacity in the past; Imagine Canada has done a lot of work, as has the Caledon Institute, and there have been different initiatives over time, but I still think this is a relative weakness in our sector.
I connected with the MPNL program, at Carleton University, to mention that there are emerging methodological approaches, and that the sector probably needs to invest in those to take advantage of those new approaches. It’s easy to be overly influenced by anecdotes. This is why evidence can be used as a corrective to relying on traditional wisdom or traditional tropes. Carleton is doing work by having a focus on quantitative research and by focusing on a new generation of researchers who will have more experience with quantitative methodologies.
Meanwhile, government can increase ethical access to some of the administrative data that it possesses, that it’s already making available to the for-profit sector. There’s a tool that Statistics Canada is developing called the LFE, the Linked File Environment, which gives researchers a model of each corporation in Canada, every single corporation, that’s created from multiple data sources from government agencies. There’s no reason that the nonprofit section can’t benefit from such tools.
In the third article of our Data series, Shawn Bunsee, Vice President of Data & Analytics and Chief Privacy Officer at CanadaHelps until 2022, explains how the nonprofit sector needs to mature when it comes to data issues.
It’s like there’s been a missing connection between Stats Can and us about the power of some of these new tools they’re using — tools that could help the sector to better understand the populations and groups of people they’re working with. We need to change that.
Michael Lenczner is on LinkedIn. Photos are courtesy of James Wainscoat, Jean Gagnon, Markus Spiske, Andrea Lightfoot, Erik McLean and Brandon.
Wednesday, February 22, 2023 in Data Discoveries, For homepage, News & Events
Share: Twitter, Facebook
