Five Signs That Your Program Evaluation Could Better Integrate Diverse Perspectives

Co-written by Amanda Parriag (ParriagGroup) and Paul Chaulk (Atlantic Evaluation Group)

If there’s one thing we’ve learned (sometimes the hard way) over the last 25 years, it’s that it is entirely possible to carry out a thorough, well-resourced evaluation that still manages to miss the mark.

In our experience, good evaluation goes beyond methodology, honing in on whose stories are told, whose experiences are measured, and whose voices define “success.” And when these questions are overlooked, even unintentionally, it can leave out key perspectives.

Today, we’re focusing on some of the subtler ways that a program evaluation can improve and more effectively integrate diverse voices into the conversation. If you’re currently managing, contracting, or providing input into an evaluation, we have five indicators you can look out for:

1. Participation was more about consultation than co-design and does not meaningfully shift the evaluation focus.

Involving participants in evaluation is crucial. The harder question is: at what point in the process, and how much influence should they have? There’s a big difference between engaging community members after drafting an evaluation framework and actively involving them and centring their priorities from the start.

Oftentimes, this manifests in subtle ways like overlooking relational or cultural impacts, measuring satisfaction instead of safety, or using outcome metrics that focus on the funder’s aims rather than community definitions of progress.

The real question isn’t whether to involve participants, but whether you’re ready to let their input influence the evaluation design.

Ask yourself: If participants had complete control over the evaluation measures, would any of them look different?

2. Disaggregated data exists but isn’t meaningfully integrated.

Disaggregation of experiences by subgroup has become more common, which is a real step forward. However, the real challenge lies in understanding why findings differ by groups.

A program serving a diverse population may demonstrate strong aggregate outcomes while a specific subgroup faces a different reality. If such disparities aren’t reflected, the evaluation isn’t accurately reflecting the reality of program delivery.

The good news is that small subgroup sampling can be a design choice, not just a constraint; if the analysis is expected to be underpowered, this should shape the recruitment plan from the outset. Viewing it as a challenge instead of an opportunity can inadvertently prioritize convenience over analytical integrity.

Ask yourself: Are you highlighting subgroup experiences with the same emphasis as the overall results, or treating them as an afterthought?

3. The data collection process fostered safety for the evaluator over the participants.

Trauma-informed evaluation practices are increasingly cited, but they can be applied narrowly, focusing on phrasing and interview protocols while leaving the underlying structural dynamics untouched. Creating safety matters, which means considering who’s in the room; whether the facilitator shares linguistic, cultural, or lived experiences with the participant; and if responses are genuinely anonymous.

There’s also a temporal aspect that’s often overlooked. Data collection usually occurs quickly, but trust is (inherently) built slowly. When evaluators enter communities for short engagements, request sensitive information, and then leave, they are relying on a trust they haven’t earned.

Establishing the right structural conditions for meaningful participation needs to happen before data collection begins. It includes building relationships, fostering community involvement, and clearly communicating how findings will be used. Though these steps can feel daunting when you have limited time or budget, they make a world of difference.

Ask yourself: If participants were completely comfortable sharing their experiences, would your data look the same?

4. Qualitative findings are presented but not relied on as deeply as quantitative data.

There may be an implicit hierarchy in mixed-methods evaluations, with quantitative data representing findings, while qualitative data is treated as illustrative. Although not always explicit, this influences how the work is valued. A handful of quotes could be chosen to support the quantitative narrative rather than complicating or deepening it.

This is particularly significant in equity-driven evaluations, where issues like stigma, distrust, internalized barriers, and gaps between what people say in focus groups and what they experience privately are often not understood.

Fully integrating qualitative data and being willing to report when qualitative and quantitative findings diverge is critical.

Ask yourself: Where your qualitative and quantitative findings don’t quite line up, are you explaining that tension, or glossing over it?

5. The evaluation attributes outcomes to the program without considering what program participants brought to the table.

Attribution is one of the oldest problems in evaluation. But in equity-centred assessments, there’s a very specific attribution error that warrants greater attention: treating participants as blank slates and ascribing outcomes to program exposure. This systematically undervalues the resilience, resourcefulness, and agency of participants. It also devalues programs supporting clients facing barriers, as their outcomes may appear modest compared to the program’s overall impact.

The inverse also exists – when programs serving more privileged groups demonstrate strong results, those outcomes are credited to the program, even though they may largely reflect participants’ pre-existing assets. In the end, neither scenario provides an accurate assessment of true effectiveness.

Contextualizing outcomes requires acknowledging what the program was able and unable to achieve, without assuming too great a role for exposure to the program. That’s a difficult line to hold, and it’s where expertise and evaluative judgement really come into play.

Ask yourself: Does your evaluation design allow you to accurately assess the program’s contribution to participant success while accounting for unique participant and community context?

None of this is about bad intentions.

In our experience, evaluators and organizations conducting these assessments care deeply about getting it right but face competing pressures. Gaps often arise from resource constraints, inherited frameworks, and the subtle pressure to deliver findings that are more polished and less nuanced than the actual evidence supports.

Further, equity-centred evaluations challenge evaluators to embrace complexity, honestly report on uncertainties, and keep people at the centre of the evaluation. We recognize the nuance of that work but truly believe it’s what drives evaluation practice forward.

If any of these patterns resonate with you, we would love to hear from you.