The use of complexity methods in evaluation is still not widespread in practice (Barbrook-Johnson et al. 2021) and there are limitations to appropriate approaches to evaluate complex interventions (Picciotto 2016). Working with complexity remains difficult for evaluators who are more comfortable with depicting a linear progression of activities, outputs, outcomes, and impact (Junge, Cullen & Iacopini 2020). Indeed, there is a growing awareness in the evaluation community that traditional evaluation methods may not capture the complexity of interventions fully, possibly impeding the effectiveness of evaluations in achieving their objectives (Bamberger, Vaessen & Raimondo 2016).

The concept of complexity has been broadly discussed in the evaluation field over the past two decades (Barbrook-Johnson et al. 2021). In evaluation, complexity expresses the emergent, unpredictable, and non-linear nature of associations between actions and outcomes (Glouberman & Zimmerman 2004). From a systems perspective, complex interventions emphasise identifying uncertainties and the role of the environment in modelling the dynamic relationships between implementation, mechanisms, and context (Moore et al. 2019). Interventions functioning in complex social systems are shaped by interactions among various changing actors and factors (Keshavarz et al. 2010).

To address the challenges of evaluating complex interventions and the inherently social nature of policy development, Barbrook-Johnson et al. (2021) emphasise many strategies. They stress the importance of utilising frontier research. Also, they advocate for fostering early and open dialogue between commissioners and evaluators. Lastly, they recommend establishing a flexible evaluation contract to enable an evolving and adaptive research design (Barbrook-Johnson et al. 2021). Integrating stakeholders’ perspectives leads to a more comprehensive and nuanced understanding of the intervention’s effects and contributions (Patton 2012). The hypothetical causal relationships in an intervention can be well-founded and validated by the insights of the stakeholders who are directly engaged (Koleros & Mayne 2019). Therefore, the endorsement of a participatory approach is even more essential in the evaluation of complex interventions (Oliver, Lorenc & Tinkler 2020).

Despite the increasing incorporation of the participatory systems mapping (PSM) into evaluations and its integration in certain evaluation guidelines, particularly for those operating within complex contexts (Taylor et al. 2019), there is often a dearth of detailed guidance on its effective implementation or integration into the evaluation process (Wilkinson et al. 2021). This article seeks to provide insights and guidance on how the PSM method has been used to foster a shared understanding within the complex intervention evaluation. Through this systematic review, we contribute to the relatively limited body of case studies that employ PSM to evaluate complex programmes (Suno Wu, Barbrook-Johnson & Font 2021). The aim is to assess the potential of PSM in enhancing the understanding of mechanisms and factors underlying the achievement of results for evaluation purposes, while also shedding light on challenges and limitations associated with its application.

This article is structured into four sections. Firstly, it provides an overview of PSM. Secondly, it details the methodology used to conduct the systematic review. Thirdly, the article presents the study’s results, with a particular focus on the methods employed by the identified research, their findings, and limitations. Lastly, the conclusion emphasises whether PSM can be utilised in evaluating complex interventions, and discusses the potential and limitations of its use for evaluation purposes.

Participatory approaches are helpful in evaluating complex interventions. Within this category, PSM was recently introduced in the evaluation field as a method to understanding how interventions interact with their context to achieve desired outcomes. Before presenting the practical implementation of PSM, its theoretical underpinnings will be described. This involves presenting the foundational concepts of participatory research and systems thinking. According to Király et al. (2015), PSM is grounded at the intersection of participation and systems thinking.

The epistemological principle for participatory research was influenced by Skolimowski’s (1994) concept of the participatory mind, which emphasises active involvement and collaboration between researchers and participants (Heron 1996, as cited in Breu & Peppard 2003). This approach considers the outer world as objectively given but subjectively represented in the human mind (Reason 1998). A participatory approach overcomes epistemological limitations encountered in interpretivism and constructivist paradigms by integrating subjective experiences with objective inquiry (Denzin 1992; Hammersley 1989, as cited in Breu & Peppard 2003). The ultimate goals of participatory action research are empowerment and the generation of knowledge and action that directly benefit the participants involved, leading to the production of useful knowledge (Worren, Moore & Elliott 2002). Grounded in post-normal science, participation in research is a method for data gathering to gain a deeper understanding and more realistic descriptions of the system being studied (Setianto, Cameron & Gaughan 2014; Stave 2002). Post-normal science incorporates a plurality of perspectives, acknowledges the role of subjective values, and shifts from a logical positivist approach to a more pragmatic, pluralist methodology (Costanza 2001; Frame & Brown 2008). In addition, the theoretical assumptions of participatory research propose that participation seeks to enhance intersubjective knowledge, acknowledging that no single actor can fully comprehend the entire system (Meadows 2008). Similarly, Mccall et al. (2021) assert that PSM is based on the recognition that the knowledge of local people is both valid and valuable.

Furthermore, systems thinking, developed from cybernetics and engineering theory, offers a comprehensive framework for addressing complex societal issues by emphasising a holistic understanding of the interrelationships between system components (Senge 1990). It focuses on perceiving systems as wholes rather than isolated parts, allowing for a deeper comprehension of the dynamic behaviour of the system (Cabrera, Colosi & Lobdell 2008). The systems thinking approach acknowledges the purposive nature of systems. It transcends traditional subject and/or object boundaries by linking individuals, social systems, and the natural environment through pathways and feedback loops (Bateson 1972, as cited in Sedlako et al., 2014). A fundamental principle is that the interconnections between a system’s elements shape its behaviour and reveal its function or purpose over time (Meadows 2008; Sedlacko et al. 2014). Evaluation methods are shifting from a mechanistic, theory-based approach to a more systemic perspective (Martinuzzi & Kopp 2010). Consequently, systems-thinking approaches are viewed as more effective alternatives for evaluation, as traditional methods such as logic models have been criticised for their simplicity and linearity (Renger et al., 2019).

Participatory systems mapping is an explorative method that enhances group understanding through the visualisation of causal relationships and feedback loops and provides a neutral communication tool facilitating dialogue between parties with different types of knowledge (Király et al. 2015). It stands out as a flexible and participatory method for comprehending system dynamics for evaluation purposes (Barbrook-Johnson & Penn 2021). This method aids in transforming overwhelming complexity into more manageable and comprehensible systems (Suno Wu et al. 2021; Tourais & Videira 2021). Primarily reliant on involving key intervention stakeholders in the evaluation process (Suno Wu et al. 2021), PSM aims to generate alternative scenarios that offer insights into the multifaceted aspects of interconnected system elements, build or inform a theory of change, identify new exploratory questions, and highlight unforeseen programme aspects (Barbrook-Johnson & Penn 2021; Tourais & Videira 2021). Based primarily on focus group discussions, the PSM method produces a map (see Figure 1 for illustration) that describes how the intervention interacts with its environment (Barbrook-Johnson & Penn 2021).

Barbrook-Johnson and Penn (2021) provided the first detailed insights into how PSM is conducted for evaluation purposes. Previous work grounded in systems thinking has also utilised participatory mapping methods to generate representative community insights, subsequently informing policy recommendations and evaluations. A notable example is System Constellation, an approach that elucidates relationships within a system through spatial arrangement, effectively incorporating actors’ perspectives into theory-based evaluation (Martinuzzi & Kopp 2010).

Practically, the PSM method begins with preliminary meetings with stakeholders to identify and delineate the scope of the system under evaluation, establishing its boundaries (Suno Wu et al. 2021; Barbrook-Johnson & Penn 2021). Following this, relevant stakeholders are invited to participate in a focus group discussion. In this session, participants start by selecting key factors that represent the outcomes of the intervention.

Subsequently, participants brainstorm and list the factors that either influence or are influenced by these focal factors. They then collaboratively build and categorise the connections between these factors, establishing causal relationships and dependencies among them. After elucidating the interrelationships connecting the factors, the evaluator proceeds to gather additional information from participants to weigh the connections between factors, investigate how external or influential factors affect programme outcomes and determine whether they are strong or weak based on their significance in the system.

Following the focus group discussion, the evaluator reviews the map comprehensively to verify its accuracy, clarity, and completeness. The map is then shared with focus group participants to solicit their feedback. Thereafter, utilising network analysis, the evaluator identifies patterns, key nodes, and mediating factors, extracting insights to better understand the system’s dynamics. The network analysis and insights from stakeholders are afterwards integrated to foster a shared understanding of how the intervention operates (Barbrook-Johnson & Penn 2021).

The study employed a systematic literature review, utilising databases such as Scopus, EBSCO, and Google Scholar. Specialised journals, including the American Journal of Evaluation, Canadian Journal of Program Evaluation, Evaluation, Evaluation and Program Planning, and Journal of Multidisciplinary Evaluation were thoroughly examined. Keywords such as complexity, participatory systems mapping, and evaluation were employed, and Boolean operators were used to refine the search, resulting in an inventory of 53 articles. The main search equation was ‘participatory systems mapping’ AND ‘evaluation’. These articles underwent review for eligibility based on predetermined criteria, including being published in peer-reviewed journals from 2002 to 2023 and focusing on PSM and its utilisation in evaluation practice or research.

A strategic skimming approach was utilised, involving the initial reading of titles and abstracts. The findings and limitations sections of articles were further reviewed only if the content appeared relevant. Articles were excluded for the following reasons: not addressing the use of PSM in evaluation, not providing sufficient detail on methods or data analysis, and not being primary studies applying PSM in evaluation. Consequently, 11 articles focusing on the use of PSM in the evaluation were identified. Figure 2 presents the PRISMA flow diagram through the different stages of the review process, including identification, screening, eligibility, and inclusion of studies.

Participatory systems mapping is a stand-alone method used across various fields. This article delves into its integration within the evaluation field, scrutinising its application across 11 evaluations within the sample. The research results are organised into three pivotal domains: methodologies (detailing the utilisation of PSM across the 11 studies), findings (highlighting insights gleaned from employing PSM for evaluation purposes in each study), and identified limitations observed during these evaluations.

The results underscore the diverse applications and benefits of PSM, while also highlighting various limitations and areas for improvement. These results are summarised in Table 1, which provides detailed information on the article title, authors, subject area, method, findings, and limitations for each reviewed study. To ensure consistency and facilitate comparison across studies, a standardised data extraction form was utilised.

The utilisation of complexity-based methods in evaluation is not yet widely adopted, posing challenges for evaluators accustomed to linear progressions. Addressing the challenges of evaluating complex interventions requires a deep exploration of system complexity, incorporating various perspectives, and employing adaptive and flexible designs. By recognising the importance of stakeholder involvement, a participatory approach becomes crucial in evaluating complex interventions that depict social change and provide a more comprehensive and nuanced understanding of the intervention’s effects and contributions.

By addressing an existing gap in insights and guidance on how PSM is applied for evaluation purposes, this research, to the best of our knowledge, is the first systematic review conducted on the use of PSM in the evaluation field.

This research demonstrated that the PSM method facilitates an in-depth exploration of complexity by significantly enhancing the comprehension of the interaction between interventions and the entire system, thereby fostering a shared and deeper understanding of how the intervention leads to desired outcomes. However, PSM does not serve as a model illustrating the intervention’s functioning. Instead, it captures participants’ perspectives on how intervention components interact with diverse factors within the broader system. The resulting map represents just only one perception of the intervention within a system, not the absolute representation. While PSM helps to address the complexity of interventions, it does not inherently surpass other methods. Method selection for evaluating complex interventions should be guided by appropriateness to purpose and context (Suno Wu et al. 2021).

The findings underscore PSM as a helpful tool that employs a system-based approach to foster engagement and learning among stakeholders. The primary contribution of PSM to the evaluation process, as highlighted in the reviewed articles, is its ability to facilitate collaborative efforts, enabling stakeholders to achieve a comprehensive understanding of how interventions operate and interact within their environment.

The study also found that network analysis and stakeholder insights were the predominant approaches utilised within the PSM method. Furthermore, causal loop diagrams and mixed methods, which involve integrating PSM with other methods, were notably employed within PSM.

However, the research highlights limitations in using PSM for evaluation purposes. These limitations include the inadvertent exclusion of certain stakeholders during the construction of system maps, the time-consuming nature of the method, the potential for diverse stakeholder insights to complicate result interpretation, the necessity to combine PSM with other methods for greater robustness, and the challenges posed by the composition of PSM focus groups, particularly with heterogeneous groups characterised by varying levels of influence and knowledge capital.

By emphasising the importance of an inclusive, stakeholder-oriented approach in PSM, which considers timing, context, and stakeholders’ ontological differences, we highlight the pivotal role of the evaluator as a facilitator in addressing challenges within systems thinking debates. This facilitation is crucial for fully harnessing the potential of PSM. We contend that despite its valuable insights, PSM has limitations that underscore the need for careful consideration of contextual appropriateness and the composition of PSM focus groups. For instance, a series of meetings may be conducted with different stakeholder categories, taking into account their knowledge and influence on the intervention.

Furthermore, we advocate for the utilisation of PSM in evaluations where community insights are essential for enhancing the evaluator’s understanding of how interventions achieve desired outcomes, notably transformative social interventions. Moreover, we suggest that future research should explore the combination of PSM with other methods to achieve greater complementarity and validity.