Rethinking How We Define and Measure Impact
Two years ago, I met two civil society leaders who had spent more than a decade working with young people. One ran a STEM-focused programme. The other worked with teenagers on leadership and confidence. Both had watched young people move through their programmes and later go on to do remarkable things.
Yet when asked a familiar question, “What impact have you had?” both struggled to respond.
Not because nothing had changed, but because the change they were trying to influence was complex, deeply human and often visible years after the intervention ended.
This is a challenge I recognise from my own work across programme leadership, monitoring, evaluation and learning, and communications. Sometimes the problem is not that organisations have no impact. It is that our ways of defining, measuring and communicating it are too narrow.
The OECD DAC defines impact as the extent to which an intervention generates, or is expected to generate, “significant positive or negative, intended or unintended, higher-level effects.” Those four possibilities, positive, negative, intended and unintended, already tell us something important. Impact is bigger than whether we delivered what we planned. It requires us to ask: what changed, for whom, and why did it matter?
But that definition still has to be interpreted within the reality of the work. Meaningful change will not look the same for every organisation. For one organisation, it may be greater equity, stronger systems or more voice. For another, it may be confidence, agency, learning or expanded choices. This is why defining impact should not sit entirely outside the organisation, imposed by a funder, evaluator or reporting framework. The people doing the work, and particularly those experiencing the change, should have a meaningful role in deciding what progress looks like and what is worth paying attention to.
Once we start from there, some of our assumptions about impact also become worth questioning.
Assumption One: More people reached means more impact
One of the most common assumptions in impact measurement is that bigger numbers signal bigger impact. For many CSOs, this makes intuitive sense. We count how many people attended, how many communities were reached, how many young people were trained, or how many people received information. As those numbers grow, the intervention appears more successful.
But reach tells us how far an intervention travelled. It does not tell us what changed.
Reaching 50,000 people lightly is not automatically more meaningful than changing the trajectories of 500 people deeply. Breadth and depth tell us different things, and which matters more depends on what the intervention is trying to achieve. A public awareness campaign may reasonably prioritise reach, while an intervention focused on confidence, leadership or agency may need to pay much closer attention to the depth of change.
Assumption Two: Impact must always mean improvement
Many CSOs I have worked with tend to look for upward movement: more participation, higher scores, increased income, stronger confidence, better attendance. Our reporting systems reinforce this. We establish a baseline and then expect the numbers to move upwards.
Get Cynthia Bavo Harris’s stories in your inbox
Join Medium for free to get updates from this writer.
But impact can also mean preventing things from getting worse.
In difficult or deteriorating contexts, maintaining an existing outcome can itself be significant. If girls’ school attendance is under increasing pressure, for example, helping attendance remain stable when similar communities are experiencing a decline may represent an important result. The absence of visible improvement does not necessarily mean that nothing changed. Sometimes an intervention is holding the line against forces that would otherwise have pushed things backwards.
Assumption Three: Every organisation must prove that its intervention works from scratch
There is also a tendency, particularly when organisations are under pressure to demonstrate impact, to feel that they must establish the effectiveness of the entire intervention themselves.
They do not always have to.
If, for example, a meta-analysis study on young people found that informal and out-of-school STEM programmes had a significant positive effect on students’ STEM interests and attitudes, including self-efficacy and career interest, then a small grassroots organisation running a STEM programme does not need to establish from scratch that this kind of intervention can contribute to these outcomes.
The grassroot organisation can draw on the wider evidence base, demonstrate that its programme is delivering the kinds of experiences associated with those outcomes, and concentrate its limited measurement resources on understanding what is happening among the young people it actually serves. That might mean tracking changes in confidence, interest or aspirations before and after the programme, alongside conversations with participants about what changed, how it changed and what they believe contributed to it.
Assumption Four: If impact has happened, we should be able to see it now
Timing creates another problem. CSOs often have to demonstrate results within the same period in which they deliver an intervention: at the end of a workshop, a six-month project or a one-year grant. But the point at which we measure something can fundamentally shape the story we tell about whether it worked.
This was perhaps the biggest challenge facing the two civil society leaders I met.
The STEM organisation could reasonably measure changes in interest, confidence or aspirations shortly after the programme. But outcomes such as subject choices, university pathways or eventual careers may take years to become visible.
The leadership programme faced the same problem. Changes in confidence, voice or participation might emerge relatively quickly. The deeper significance may only become apparent much later, through the opportunities young people pursue, the decisions they make, the organisations they join or the leadership roles they eventually take on.
This means distinguishing between what we can reasonably expect to observe now and what may only become visible later.
Good impact measurement is not about demanding long-term outcomes too early. It is about identifying the signs of progress we would expect to see along the way, and being clear about what those signs, can and cannot yet, tell us.
The issue of timing brings us back to the two CSOs, and to the wider challenge of how we think about impact. Neither organisation was struggling because nothing was changing. The difficulty was that the changes they cared about did not always fit neatly into the assumptions that often shape impact measurement: that bigger numbers mean greater impact, that progress must always be visible and upward, that each organisation must prove effectiveness for itself, and that meaningful outcomes should emerge within a project cycle.
Looking at impact differently creates more room to recognise what is actually happening: depth as well as reach, stability as well as improvement, contribution as well as proof, and early signs of change alongside outcomes that may take years to fully emerge.
