Across interventions, Passages applied a systematic co-learning approach over a two-year evaluation period starting with building partnerships, developing a programme TOC that included norms-shifting pathways, answering key gaps in pathway evidence and evaluating intervention effects. Case examples of two norms-shifting interventions that underwent realist evaluations are described here and summarised in Table 1. Both interventions operate in rural communities to improve the normative environment supportive of behaviours leading to good reproductive health outcomes. The Husbands’ Schools intervention, which operates in several thousand communities throughout Niger, aims to increase women’s low use of reproductive health services. In many Nigerien communities, norms of unquestioned male authority, a lack of couple communication and joint decision-making and men and women’s distrust of antenatal, delivery and family planning services serve to limit the success of normative and behaviour change efforts. The Girls Holistic Development intervention, which operates in one southern region in Senegal, aims to improve adolescent girls’ health and school outcomes and reduce the practice of child marriage and related teen pregnancy. In many Senegalese communities, older women’s influential advisory and support role in child-raising has diminished compared with earlier times. The expectation that boys should attend school and prepare for breadwinner roles contrasts with the expectation that girls should help with chores and prepare for marriage. These community norms serve to check SBC efforts.

As NSI programme theories would predict, both interventions’ activities are less focused on building knowledge and more on engaging people in discussion to examine prevailing beliefs in light of current issues that lead to new ideas and possibilities of ‘being’. Project extension agents and trained volunteers (social change agents) use a mix of group reflective dialogues and role-modelling to break taboos, allow debate on community expectations of appropriate behaviours for women and girls and men and boys and discussion of a vision for the future. Over time, new change agents and influencers emerge to encourage others in the community to adopt new ideas and behaviours. Eventually, enough people are exposed to and internalise new ideas and desire change that a tipping point of agreement on a new norm or acceptable behaviour is reached.

The application of realist evaluations under the Passages project (Figure 1) followed the prescribed realist evaluation approach, emphasising an iterative and interactive process over two years. The participatory development of programme TOC was the starting point of the evaluation and related studies. The role of Passages’ staff was two-fold – as external evaluators guiding the different evaluation studies and as co-facilitators with NSI managers of the realist evaluation process that began with theory-building. In this section, a brief description is given of how TOCs became the organising construct with evaluators and NSI staff and stakeholders in the evaluation.

In the opening workshop, Passages evaluators worked alongside local stakeholders with operational knowledge of the NSIs – project implementors, monitoring and evaluation (M&E) staff, local health, education and religious stakeholders – to refine the project TOC. Building programme TOCs together created a discussion space that welcomed different perspectives and built theoretical consensus on how the NSI worked. Examining the causal pathways, the group identified evidence gaps, which led to subsequent questions to guide realist evaluation inquiry.

Over two years, Passages evaluators, consultants and NSI staff completed a series of jointly defined evaluation studies to answer different evaluation questions. The studies used a range of data types based on an established data rule: all types of data could be used for evidence, from project activity monitoring systems and existing studies to new data that would be collected in rapid and quasi-experimental evaluation studies.

Every six to eight months, realist-evaluation workshops allowed project staff, local stakeholders and Passages evaluators to review and analyse new findings in light of the TOC. In the meeting that closed the evaluations, a cumulative review with the initial stakeholder group of study questions and findings led to adjustments to TOCs. Dissemination meetings with national stakeholders created momentum to continue advancing the NSIs’ scale-up in Niger and Senegal, given the new evidence and deepened understanding of how the projects worked to foster social change that influenced behaviours.

This article followed all ethical standards for research without direct contact with human or animal subjects.

While there are many ways to visually depict TOCs, it was found to be crucial to work with the type of programme theory of change that shows change pathways from programme activities to their intermediate or mediating effects on programme outcomes. This depiction of programme theory helps to unpack the implementation box to show pathways and their interconnections leading to change. The decision was also made to group project activities into components of similar activities, instead of examining pathways of each activity separately within a TOC. This provided a way to engage programme stakeholders in thinking about how an intervention works without getting into programme or evaluation ‘weeds’. This programme-practical approach allowed realist evaluation to move from complex theory and mapping to a practical, conceptually grounded, collaborative evaluation approach.

The initial workshop focused on TOC building. The original TOCs indicated only behaviour change outcomes; the expected normative changes were not articulated or were vaguely defined. During the workshop, project staff and local stakeholders and evaluators added missing outcomes to TOCs. (See Figure 2 example from the Husbands’ Schools intervention, which added an enabling environment outcome during the TOC workshop.) Related workshop discussions on how normative shifts occurred because of project activities helped clarify how the change mechanisms, produced from activity implementation, led to community reactions and actions (intermediate or mediating effects) that facilitated norms shift. The pathways discussions led to new understandings, sometimes revelatory, by implementers, local stakeholders and evaluator-researchers.

Both Husbands’ Schools and Girls Holistic Development interventions had previously undergone mixed methods, nonexperimental evaluations focused on individual behaviour change outcomes. These evaluations showed they were effective in behaviour change, with some anecdotal evidence of normative change. Given this earlier evidence of effectiveness, the researchers decided not to test alternative hypotheses for each component as is prescribed in some realist evaluation guidance. Instead, participants identified evidence gaps in pathways, and several studies were developed to assess whether change was occurring along those paths as expected by programme theory. This allowed prioritisation of evaluation questions. The dashed boxes in Figure 2, for example, show evidence gaps that were not supported by stakeholders’ own experience-based knowledge or past studies. They guided evaluation research (learning) questions.

In sociology, a realist perspective accepts that social facts are just as real as physical facts and can be known by blending theory with observation, measurement and empirical studies. Accordingly, several ideas underlie realist evaluation. Social and structural factors affect how programmes work; multiple realities and truths exist; and social systems are porous and flexible. The approach recognises that change mechanisms may not be visible, but they can be understood as the combination of resources and how they change community reasoning. Finally, context influences how mechanisms operate (Westhorp 2014).

To achieve a realist perspective, the researchers’ evaluation–practice partnership intentionally applied these values throughout their tailored approach to realist evaluation: participatory, complexity, shared ownership, practice-oriented, and valuing all forms of data. The participatory value was visible in the structure whereby everyone’s voice was valued, all stayed present during meetings and space was given for all to speak. Reflecting the complexity of social change meant being flexible in working with the expected and unexpected pathway findings and working with stakeholder knowledge of social systems in which NSIs operated. Shared ownership included sharing decision-making across organisational hierarchies and geographies and involving crucial stakeholders in all evaluation stages (design, implementation, analysis and dissemination). Learnings that advanced the project or future performance of related projects were the highest priority to be practice oriented. Finally, all forms of data were valued, existing and new, experiential and empirical, repurposed and newly generated. These five values were not named in this way at the outset but were reflected in the processes.

In practice, this meant a participatory, co-learning approach over two years to welcome into an evaluation process the NSI staff and stakeholders who have a deep understanding of how NSI activities interact with the lived realities of their respective societies. Norms-shifting intervention staff and stakeholders included people selected for their involvement in and knowledge of the programme design and implementation, interaction with community members who participate in the programme and involvement in research. In meetings, facilitation was shared by project staff and Passages evaluators, with attention and emphasis on sharing and collaboratively learning, reinforcing the realism concept that multiple realities exist and thus no one person could hold the ‘right’ answer.

Different types of data and data sources represented a mosaic of information that could be used to answer evaluation questions. The ways in which the researchers worked with different data demonstrated the value that all forms of data count. For example, existing monitoring data might be repurposed to answer an evaluation question on implementation fidelity across regions. New studies could collect new empirical data on norms-shifting which had not yet been systematically collected or examined, or experiential data could be drawn from insider knowledge held by NSI staff, stakeholders and participants (it engages with the perceptions, experiences, reflections, and observations of staff, stakeholders and community members who facilitate or participate in interventions). Discussions of data and evidence led to sharing different interpretations and understandings of how programmes worked allowing for consensus and additional questions to emerge.

By incorporating realist values, it was easier to discern a project’s effects that can be hidden in the perceptions, opinions and implicit or explicit logics of the community actors which underlie their attitudes and actions vis-à-vis a projects’ resources. The embedding of values into the realist evaluation process allowed for an emic understanding of a programme operating within its social context (Lacouture et al. 2015; Pawson & Tilley 1997).

To close this section, the authors share three interrelated, at times surprising, effects of their realist evaluation experience that foster substantive evaluation of SBC, with examples to illustrate their points: (1) valuing insider knowledge and critical reflection of context increases programme and evaluation significance; (2)participatory TOC-building connects programmes, implementation context and evaluation research in profound ways that benefit programmes and evaluations; and (3) accepting a range of data types broadens the value of data to inform decision-making by more people.