{"id":5521,"date":"2026-03-19T15:05:47","date_gmt":"2026-03-19T19:05:47","guid":{"rendered":"https:\/\/carleton.ca\/cicp-pcpob\/?p=5521"},"modified":"2026-03-19T15:06:01","modified_gmt":"2026-03-19T19:06:01","slug":"evaluation-is-about-learning-not-just-compliance","status":"publish","type":"post","link":"https:\/\/carleton.ca\/cicp-pcpob\/2026\/evaluation-is-about-learning-not-just-compliance\/","title":{"rendered":"Measuring What Matters: Rethinking Impact Evaluation"},"content":{"rendered":"\n
\n
\n\n
\n
\n
\n

\n Measuring What Matters: Rethinking Impact Evaluation \n <\/h1>\n \n \n <\/header>\n\n <\/div>\n\n <\/div>\n\n <\/div>\n<\/section>\n\n\n\n

In Canada\u2019s charitable sector, \u201cimpact\u201d has become a buzzword. It is embedded in grant applications, strategic plans, and funder conversations as a signal of seriousness and accountability. Driven by well-intentioned demands from donors and funders, organizations are increasingly pressured to prove the profound, long-term change they create. However, data from the\u00a0Charity Insights Canada Project (CICP)\u00a0reveal a significant gap between what funders expect and what organizations can realistically deliver.<\/p>\n\n\n\n

Confusion in the Language of Evaluation<\/strong><\/p>\n\n\n\n

A significant hurdle is the inconsistent use of terminology. In evaluation practice, outputs, outcomes, and impact are distinct. Outputs <\/em>describe the direct product of activities, such as \u201cthe number of workshops delivered\u201d. Outcomes<\/em> capture the short-to-medium term changes for participants, such as \u201cimproved knowledge or skills\u201d. Impact<\/em>, by contrast, refers to broader and longer-term systemic change, for example \u201creduction in unemployment rates\u201d.<\/p>\n\n\n\n

In practice, however, these distinctions are often blurred. Reviewing evaluation reports in Canada and elsewhere, Susan Phillips and Victoria Carlan<\/a> found that nonprofits frequently equate \u201cimpact evaluation\u201d with simpler measures such as outputs or outcomes. As one CICP panellist expressed, \u201cNot sure how to measure impact and outcomes other than collecting numbers of participants and some qualitative feedback from participants.<\/em>\u201d (CICP 2.10.38<\/a>) Therefore, it is important to discern: are Canadian organizations evaluating impact or counting outputs.<\/p>\n\n\n\n

Findings from the CICP: Are Charities Truly Measuring \u201cImpact\u201d?<\/strong><\/p>\n\n\n\n

At its core, impact measurement or evaluation<\/em><\/a> <\/em>seeks to determine whether an intervention causes<\/em> a particular outcome. It relies on counterfactual or quasi-experimental designs that compare what happened with what would have happened without the intervention in order to isolate causality. \u00a0<\/p>\n\n\n\n

CICP findings show that most evaluation in the sector focuses on monitoring and learning<\/em>, not causal proof. While 81 % of charities say they measure impact in some form (CICP 2.10.38<\/a>), only 8% conduct evaluations using counterfactual methods – the methodological standard for establishing causation (CICP 1.11.44)<\/a>. Most charities rely on self-designed questionnaires (50%) and recorded outputs or outcomes (around 50%), rather than externally validated tools from government (10%) or researchers (8%) (CICP 2.10.38<\/a>).<\/p>\n\n\n\n

These patterns reveal that charities are indeed evaluating<\/em>, but primarily to learn, improve, and remain accountable to their missions, but not to prove causality. Similar observations have been reported in earlier sector studies, including the 2003 VSERP report on evaluation in the voluntary sector<\/em><\/a> and Imagine Canada\u2019s 2018 State of Evaluation<\/em> report<\/a>.<\/p>\n\n\n\n

The question, then, is whether the push for \u201cimpact\u201d has outpaced the sector\u2019s actual capacity, and even its purpose.<\/p>\n\n\n\n

A Cautionary Tale from the \u201cImpact Revolution\u201d in the US and the UK<\/strong><\/p>\n\n\n\n

Experiences in the United States and the United Kingdom offer a cautionary lesson. In the 2000s, charities were encouraged to measure and publish their own impact. Reflecting later on this \u201cimpact revolution,\u201d philanthropy advisors Caroline Fiennes and her colleague Ken Berger<\/a> acknowledged that this practice backfired because the system was stacked against producing reliable evidence.<\/p>\n\n\n\n

Fiennes argues that expecting charities to conduct rigorous impact evaluations is often unrealistic. Organizations may feel pressure to present flattering results, or what Ben McNamee from DARO<\/a> calls \u201cvanity metrics.\u201d Many charities also lack the methodological expertise, financial resources, and sample sizes needed for rigorous evaluation. In short, charities are experts in delivering services, not necessarily in conducting causal social science research.<\/p>\n\n\n\n

Rethinking What Counts as Evaluation<\/strong><\/p>\n\n\n\n

If rigorous experimental impact evaluations are not feasible for most charities, what does meaningful evaluation look like instead.<\/p>\n\n\n\n

Research by Phillips and Carlan suggests that evaluation may be more productive when centered on learning, reflection, and adaptation. Several approaches reflect this shift. For example, developmental and participatory evaluation<\/em> support innovation by helping organizations adapt their programs as conditions evolve. Developmental and participatory evaluation involve staff, communities, and sometimes funders in defining what success looks like. Collective approaches<\/em>, where funders and grantees jointly design learning frameworks, can better align expectations with the realities of organizational capacity. Utilization-focused evaluation<\/em>, meanwhile, prioritizes the practical use of findings so organizations can adjust programs and make decisions in real time.<\/p>\n\n\n\n

Concluding remarks<\/strong><\/p>\n\n\n\n

The bottom line is that evaluation should ultimately be about learning, not just compliance. For charities, this means listening closely to clients, making better use of their own data, and drawing on high-quality external research rather than trying to produce impact studies on their own. But meaningful evaluation also requires resources. Funders therefore have a critical role to play, not only by investing in independent evaluations that can serve as public goods, but also by supporting the learning-oriented evaluation activities that charities undertake as part of their everyday work.<\/p>\n\n\n\n

\n
\n\n \n\n\n
\n

\n Author\n <\/h2>\n\n <\/header>\n \n
\n \n\n \n
\n
\n \"\"\n <\/figure>\n \n
\n

\n Nguyen, Thi Kim Quy\n <\/h2>\n <\/header>\n\n
\n