Abstract
Background: Traditional evaluation methods suffer from some reductionism in dealing with complex interventions. To cope with this issue, researchers advocate for a more participatory approach to better understand how an intervention interacts with other factors within a broader system.
Objectives: Given the limited research incorporating Participatory Systems Mapping (PSM) into the evaluation process, this systematic review offers insights into how the PSM method can be applied to assess complex interventions.
Method: We conducted a systematic review of the PSM method’s application in the evaluation field. This review examined scientific publications in peer-reviewed journals dealing with the PSM and its use in evaluation.
Results: The research demonstrated that PSM has been a helpful tool for generating a comprehensive and profound understanding of how interventions function in complex settings. The findings indicate that PSM primarily utilises network analysis, integrated with stakeholders’ insights. However, several challenges have been identified: (1) PSM may fail to incorporate all stakeholders, resulting in an incomplete comprehension, (2) Addressing diverse stakeholder perceptions presents a significant hurdle and (3) Combining PSM with other methods is necessary to foster a more comprehensive approach.
Conclusion: This article provides insights into the application of PSM for evaluation purposes. It achieves this by summarising and analysing previous works, shedding light on the potential and limitations of its meaningful usage.
Contribution: This article addresses the lack of research employing PSM within the evaluation field, providing evaluators with valuable insights to leverage the use of this method in evaluating complex interventions.
Keywords: complexity; participatory systems mapping; systematic review; complexity; system thinking.
Introduction
The use of complexity methods in evaluation is still not widespread in practice (Barbrook-Johnson et al. 2021) and there are limitations to appropriate approaches to evaluate complex interventions (Picciotto 2016). Working with complexity remains difficult for evaluators who are more comfortable with depicting a linear progression of activities, outputs, outcomes, and impact (Junge, Cullen & Iacopini 2020). Indeed, there is a growing awareness in the evaluation community that traditional evaluation methods may not capture the complexity of interventions fully, possibly impeding the effectiveness of evaluations in achieving their objectives (Bamberger, Vaessen & Raimondo 2016).
The concept of complexity has been broadly discussed in the evaluation field over the past two decades (Barbrook-Johnson et al. 2021). In evaluation, complexity expresses the emergent, unpredictable, and non-linear nature of associations between actions and outcomes (Glouberman & Zimmerman 2004). From a systems perspective, complex interventions emphasise identifying uncertainties and the role of the environment in modelling the dynamic relationships between implementation, mechanisms, and context (Moore et al. 2019). Interventions functioning in complex social systems are shaped by interactions among various changing actors and factors (Keshavarz et al. 2010).
To address the challenges of evaluating complex interventions and the inherently social nature of policy development, Barbrook-Johnson et al. (2021) emphasise many strategies. They stress the importance of utilising frontier research. Also, they advocate for fostering early and open dialogue between commissioners and evaluators. Lastly, they recommend establishing a flexible evaluation contract to enable an evolving and adaptive research design (Barbrook-Johnson et al. 2021). Integrating stakeholders’ perspectives leads to a more comprehensive and nuanced understanding of the intervention’s effects and contributions (Patton 2012). The hypothetical causal relationships in an intervention can be well-founded and validated by the insights of the stakeholders who are directly engaged (Koleros & Mayne 2019). Therefore, the endorsement of a participatory approach is even more essential in the evaluation of complex interventions (Oliver, Lorenc & Tinkler 2020).
Despite the increasing incorporation of the participatory systems mapping (PSM) into evaluations and its integration in certain evaluation guidelines, particularly for those operating within complex contexts (Taylor et al. 2019), there is often a dearth of detailed guidance on its effective implementation or integration into the evaluation process (Wilkinson et al. 2021). This article seeks to provide insights and guidance on how the PSM method has been used to foster a shared understanding within the complex intervention evaluation. Through this systematic review, we contribute to the relatively limited body of case studies that employ PSM to evaluate complex programmes (Suno Wu, Barbrook-Johnson & Font 2021). The aim is to assess the potential of PSM in enhancing the understanding of mechanisms and factors underlying the achievement of results for evaluation purposes, while also shedding light on challenges and limitations associated with its application.
This article is structured into four sections. Firstly, it provides an overview of PSM. Secondly, it details the methodology used to conduct the systematic review. Thirdly, the article presents the study’s results, with a particular focus on the methods employed by the identified research, their findings, and limitations. Lastly, the conclusion emphasises whether PSM can be utilised in evaluating complex interventions, and discusses the potential and limitations of its use for evaluation purposes.
An overview of participatory systems mapping
Participatory approaches are helpful in evaluating complex interventions. Within this category, PSM was recently introduced in the evaluation field as a method to understanding how interventions interact with their context to achieve desired outcomes. Before presenting the practical implementation of PSM, its theoretical underpinnings will be described. This involves presenting the foundational concepts of participatory research and systems thinking. According to Király et al. (2015), PSM is grounded at the intersection of participation and systems thinking.
The epistemological principle for participatory research was influenced by Skolimowski’s (1994) concept of the participatory mind, which emphasises active involvement and collaboration between researchers and participants (Heron 1996, as cited in Breu & Peppard 2003). This approach considers the outer world as objectively given but subjectively represented in the human mind (Reason 1998). A participatory approach overcomes epistemological limitations encountered in interpretivism and constructivist paradigms by integrating subjective experiences with objective inquiry (Denzin 1992; Hammersley 1989, as cited in Breu & Peppard 2003). The ultimate goals of participatory action research are empowerment and the generation of knowledge and action that directly benefit the participants involved, leading to the production of useful knowledge (Worren, Moore & Elliott 2002). Grounded in post-normal science, participation in research is a method for data gathering to gain a deeper understanding and more realistic descriptions of the system being studied (Setianto, Cameron & Gaughan 2014; Stave 2002). Post-normal science incorporates a plurality of perspectives, acknowledges the role of subjective values, and shifts from a logical positivist approach to a more pragmatic, pluralist methodology (Costanza 2001; Frame & Brown 2008). In addition, the theoretical assumptions of participatory research propose that participation seeks to enhance intersubjective knowledge, acknowledging that no single actor can fully comprehend the entire system (Meadows 2008). Similarly, Mccall et al. (2021) assert that PSM is based on the recognition that the knowledge of local people is both valid and valuable.
Furthermore, systems thinking, developed from cybernetics and engineering theory, offers a comprehensive framework for addressing complex societal issues by emphasising a holistic understanding of the interrelationships between system components (Senge 1990). It focuses on perceiving systems as wholes rather than isolated parts, allowing for a deeper comprehension of the dynamic behaviour of the system (Cabrera, Colosi & Lobdell 2008). The systems thinking approach acknowledges the purposive nature of systems. It transcends traditional subject and/or object boundaries by linking individuals, social systems, and the natural environment through pathways and feedback loops (Bateson 1972, as cited in Sedlako et al., 2014). A fundamental principle is that the interconnections between a system’s elements shape its behaviour and reveal its function or purpose over time (Meadows 2008; Sedlacko et al. 2014). Evaluation methods are shifting from a mechanistic, theory-based approach to a more systemic perspective (Martinuzzi & Kopp 2010). Consequently, systems-thinking approaches are viewed as more effective alternatives for evaluation, as traditional methods such as logic models have been criticised for their simplicity and linearity (Renger et al., 2019).
Participatory systems mapping is an explorative method that enhances group understanding through the visualisation of causal relationships and feedback loops and provides a neutral communication tool facilitating dialogue between parties with different types of knowledge (Király et al. 2015). It stands out as a flexible and participatory method for comprehending system dynamics for evaluation purposes (Barbrook-Johnson & Penn 2021). This method aids in transforming overwhelming complexity into more manageable and comprehensible systems (Suno Wu et al. 2021; Tourais & Videira 2021). Primarily reliant on involving key intervention stakeholders in the evaluation process (Suno Wu et al. 2021), PSM aims to generate alternative scenarios that offer insights into the multifaceted aspects of interconnected system elements, build or inform a theory of change, identify new exploratory questions, and highlight unforeseen programme aspects (Barbrook-Johnson & Penn 2021; Tourais & Videira 2021). Based primarily on focus group discussions, the PSM method produces a map (see Figure 1 for illustration) that describes how the intervention interacts with its environment (Barbrook-Johnson & Penn 2021).
 |
FIGURE 1: An example of participatory systems maps. |
|
Barbrook-Johnson and Penn (2021) provided the first detailed insights into how PSM is conducted for evaluation purposes. Previous work grounded in systems thinking has also utilised participatory mapping methods to generate representative community insights, subsequently informing policy recommendations and evaluations. A notable example is System Constellation, an approach that elucidates relationships within a system through spatial arrangement, effectively incorporating actors’ perspectives into theory-based evaluation (Martinuzzi & Kopp 2010).
Practically, the PSM method begins with preliminary meetings with stakeholders to identify and delineate the scope of the system under evaluation, establishing its boundaries (Suno Wu et al. 2021; Barbrook-Johnson & Penn 2021). Following this, relevant stakeholders are invited to participate in a focus group discussion. In this session, participants start by selecting key factors that represent the outcomes of the intervention.
Subsequently, participants brainstorm and list the factors that either influence or are influenced by these focal factors. They then collaboratively build and categorise the connections between these factors, establishing causal relationships and dependencies among them. After elucidating the interrelationships connecting the factors, the evaluator proceeds to gather additional information from participants to weigh the connections between factors, investigate how external or influential factors affect programme outcomes and determine whether they are strong or weak based on their significance in the system.
Following the focus group discussion, the evaluator reviews the map comprehensively to verify its accuracy, clarity, and completeness. The map is then shared with focus group participants to solicit their feedback. Thereafter, utilising network analysis, the evaluator identifies patterns, key nodes, and mediating factors, extracting insights to better understand the system’s dynamics. The network analysis and insights from stakeholders are afterwards integrated to foster a shared understanding of how the intervention operates (Barbrook-Johnson & Penn 2021).
Research methods and design
The study employed a systematic literature review, utilising databases such as Scopus, EBSCO, and Google Scholar. Specialised journals, including the American Journal of Evaluation, Canadian Journal of Program Evaluation, Evaluation, Evaluation and Program Planning, and Journal of Multidisciplinary Evaluation were thoroughly examined. Keywords such as complexity, participatory systems mapping, and evaluation were employed, and Boolean operators were used to refine the search, resulting in an inventory of 53 articles. The main search equation was ‘participatory systems mapping’ AND ‘evaluation’. These articles underwent review for eligibility based on predetermined criteria, including being published in peer-reviewed journals from 2002 to 2023 and focusing on PSM and its utilisation in evaluation practice or research.
A strategic skimming approach was utilised, involving the initial reading of titles and abstracts. The findings and limitations sections of articles were further reviewed only if the content appeared relevant. Articles were excluded for the following reasons: not addressing the use of PSM in evaluation, not providing sufficient detail on methods or data analysis, and not being primary studies applying PSM in evaluation. Consequently, 11 articles focusing on the use of PSM in the evaluation were identified. Figure 2 presents the PRISMA flow diagram through the different stages of the review process, including identification, screening, eligibility, and inclusion of studies.
 |
FIGURE 2: Flowchart of the literature review. |
|
Ethical considerations
This article followed all ethical standards for research without direct contact with human or animal subjects.
Results
Participatory systems mapping is a stand-alone method used across various fields. This article delves into its integration within the evaluation field, scrutinising its application across 11 evaluations within the sample. The research results are organised into three pivotal domains: methodologies (detailing the utilisation of PSM across the 11 studies), findings (highlighting insights gleaned from employing PSM for evaluation purposes in each study), and identified limitations observed during these evaluations.
The results underscore the diverse applications and benefits of PSM, while also highlighting various limitations and areas for improvement. These results are summarised in Table 1, which provides detailed information on the article title, authors, subject area, method, findings, and limitations for each reviewed study. To ensure consistency and facilitate comparison across studies, a standardised data extraction form was utilised.
| TABLE 1: An overview of the use of participatory systems mapping in the evaluations. |
Methods utilised
The analysis of the 11 studies revealed four primary methodological approaches used by evaluations employing PSM: network analysis, the construction of causal loop diagrams, mixed methods combining PSM with other approaches, and the knowledge brokerage approach. Notably, some studies are included in multiple categories.
Most reviewed evaluations (n = 6, 55%) employed network analysis and stakeholders’ insights to develop shared system views. Network analysis was used to examine connections between individual nodes or factors in a system, along with emergent patterns and structures. Stakeholder input helped to understand how influence flows within an intervention system.
Additionally, the development of causal loop diagrams (n = 4), drawing upon information from various participants in the PSM focus group, provided a holistic view of an intervention system by capturing the interconnectedness of its components. This facilitated comprehension of the underlying factors influencing the intervention’s dynamics as a whole.
Furthermore, PSM was combined with other methods and tools (n = 5). It was integrated into a randomised evaluation to control for political variables (Reyes-García et al. 2012). Mixed methods were also employed, incorporating qualitative interviews and diverse mapping techniques alongside PSM (Brown et al. 2017).
Lastly, one evaluation (n = 1) emphasised the facilitation of knowledge brokerage through participatory interactions during the PSM focus group, highlighting the practical contributions of these methods (Sedlacko et al. 2014). The knowledge brokerage approach aimed to facilitate the production of the system map by ensuring the effective sharing of insights from all participants in the PSM focus group.
The research revealed that network analysis and stakeholder insights are the most commonly used approaches within PSM. In addition, causal loop diagrams and mixed methods, where PSM is combined with other methodologies, are also frequently employed.
Findings
The review highlighted that PSM emerged as a helpful tool for coping with complexity (Sedlacko et al. 2014; Suno Wu et al. 2021; Wilkinson et al. 2021), facilitating collaboration (Király et al. 2015; Lopes & Videira 2017, Tourais & Videira 2021), fostering a more comprehensive understanding of the contexts and interactions of intervention components (Barbrook-Johnson & Penn 2021; Brown & Eckold 2019; Suno Wu et al. 2021), and initiating cultural change (Barbrook-Johnson & Penn 2021) and social learning (García-Nieto et al. 2019) among stakeholders.
The PSM provides a comprehensive understanding of intervention contexts and interactions (Suno Wu et al. 2021). It facilitates a practical exploration of complexity, integrating well-documented and transparent processes (Barbrook-Johnson & Penn 2021). Furthermore, it complements existing evaluation approaches and methods, particularly when coupled with the System-based Theory of Change diagrams, allowing for the development of practical theories considering feedback, broader context, and potential adverse outcomes. For instance, PSM was successfully used during a fuel poverty intervention evaluation by mapping the complexity of the context and making it more familiar to build the theory of change (Wilkinson et al. 2021).
Moreover, research by Tourais and Videira (2021) found that PSM functioned well as a collaborative approach for generating a shared understanding of their intervention (Tourais & Videira 2021). Drawing from PSM, participants in the Portuguese tourism sector collaborated to develop a unified perspective on the transition towards tourism sustainability, facilitating the establishment of a comprehensive roadmap for desired transition pathways (Tourais & Videira 2021).
Furthermore, the approach generates accurate and representative community information (Brown & Eckold 2019) and supports social learning in groups with diverse stakeholders (García-Nieto et al. 2019). The effects attributed to participatory mapping are not solely from mapping activities but are intertwined with broader processes influencing community engagement (Reyes-García et al. 2012). By mapping the intervention components and their interaction with the context, PSM also contributes to broader processes that shape stakeholder engagement and learning. The PSM embodies a collaborative effort in learning and map-building (Barbrook-Johnson & Penn 2021). In doing so, the participants’ perspectives on the system become more explicit (Moon et al. 2019).
Employing mixed methods, particularly through the integration of PSM in a convergent parallel design, proves beneficial for planning decision support, contingent on demonstrating convergence in construct meaning, spatial location, and consistency in values (Brown et al. 2017). The collaborative creation of causal system maps with stakeholders enhances understanding of cause-effect mechanisms and interrelationships between system components (Lopes & Videira 2017). The study conducted by Lopes and Videira (2017) concluded that PSM elucidated diverse value dimensions of ecosystem services management. By fostering a collaborative exchange of insights, PSM supported the identification of interrelationships among various ecosystem services.
In summary, the review of the 11 evaluations’ perspectives on using PSM for evaluation purposes revealed that PSM is a practical tool for navigating complexity. It fosters collaboration, deepens comprehension of intervention contexts, and stimulates interactions and learning among stakeholders.
Limitations
The systematic review identified several challenges associated with using PSM for evaluation purposes. Firstly, the construction of system maps may exclude some stakeholders, resulting in incomplete representations of factors impacting the intended outcomes (Wilkinson et al. 2021). As PSM focuses on the system within which the intervention will act rather than solely on the intervention itself, integrating a wide range of stakeholders is crucial for identifying mechanisms behind causal links, especially in complex systems. Omitting certain stakeholders may result in missing pathways from the intervention to the desired outcome (Tourais & Videira 2021). Secondly, additional time for in-depth debate is necessary for a comprehensive understanding during the PSM focus group (Tourais & Videira 2021). For practical purposes, Tourais and Videira (2021) advocate conducting PSM in sub-groups and sharing the findings with the entire group to cross-check the structure and completeness of the system map.
Thirdly, some researchers have emphasised that the diverse perceptions, expectations, and priorities of stakeholders regarding ecosystem components complicate the interpretation of results (García-Nieto et al. 2019). The PSM may unveil ontological differences that would otherwise remain hidden, presenting a challenge in systems thinking debates (Sedlacko et al. 2014). Brown et al. (2017) stressed that the perception within the population of interest necessitates disaggregation and analysis of spatial data by sampling group (Brown et al. 2017).
Fourthly, the composition of PSM focus groups presented challenges regarding the influence of stakeholders and their degree of knowledge. The applicability of PSM focus groups is significantly different when participants are grouped uniformly based on their level of influence on the intervention, compared to when participants with varying levels of influence are mixed, leading to different system maps (García-Nieto et al. 2019). These researchers conducted two PSM focus groups in the Nacimiento Watershed, Spain, with different group compositions. In the first focus group, participants were grouped uniformly based on their influence on land management, while in the second one, mixed groups were formed with participants having varying levels of influence. The study found that the composition of the participants affected the outputs of participatory mapping, leading to different resulting maps (García-Nieto et al. 2019). Fifthly, the potential utilisation of PSM may be limited for groups with lower levels of knowledge, indicating constraints on its applicability (Király et al. 2015). Certainly, the authors acknowledge its potential limitations in an experiment applying the PSM method to the issue of sustainable consumption. They argued that the participatory potential of systems mapping might be constrained for groups with lower levels of knowledge, as it can be demanding in terms of required expertise or understanding (Király et al. 2015).
Lastly, PSM should be reinforced by existing data collection methods and analysis (Suno Wu et al. 2021). By using PSM method in the evaluation of a sustainability programme in Barcelona, these authors concluded that employing a complex systems approach, facilitated by PSM, can offer a more comprehensive understanding of the contexts and interactions within tourism policy (Suno Wu et al. 2021).
The research identified several challenges associated with using PSM for evaluation purposes. These challenges include the potential exclusion of stakeholders, the time-consuming nature of in-depth debates during focus groups, and complications arising from diverse stakeholder perceptions and priorities. Furthermore, the composition and knowledge level of focus group participants significantly impact the outputs of PSM. Finally, to enhance the effectiveness of PSM, it should be supplemented with existing data collection and analysis methods.
Conclusion
The utilisation of complexity-based methods in evaluation is not yet widely adopted, posing challenges for evaluators accustomed to linear progressions. Addressing the challenges of evaluating complex interventions requires a deep exploration of system complexity, incorporating various perspectives, and employing adaptive and flexible designs. By recognising the importance of stakeholder involvement, a participatory approach becomes crucial in evaluating complex interventions that depict social change and provide a more comprehensive and nuanced understanding of the intervention’s effects and contributions.
By addressing an existing gap in insights and guidance on how PSM is applied for evaluation purposes, this research, to the best of our knowledge, is the first systematic review conducted on the use of PSM in the evaluation field.
This research demonstrated that the PSM method facilitates an in-depth exploration of complexity by significantly enhancing the comprehension of the interaction between interventions and the entire system, thereby fostering a shared and deeper understanding of how the intervention leads to desired outcomes. However, PSM does not serve as a model illustrating the intervention’s functioning. Instead, it captures participants’ perspectives on how intervention components interact with diverse factors within the broader system. The resulting map represents just only one perception of the intervention within a system, not the absolute representation. While PSM helps to address the complexity of interventions, it does not inherently surpass other methods. Method selection for evaluating complex interventions should be guided by appropriateness to purpose and context (Suno Wu et al. 2021).
The findings underscore PSM as a helpful tool that employs a system-based approach to foster engagement and learning among stakeholders. The primary contribution of PSM to the evaluation process, as highlighted in the reviewed articles, is its ability to facilitate collaborative efforts, enabling stakeholders to achieve a comprehensive understanding of how interventions operate and interact within their environment.
The study also found that network analysis and stakeholder insights were the predominant approaches utilised within the PSM method. Furthermore, causal loop diagrams and mixed methods, which involve integrating PSM with other methods, were notably employed within PSM.
However, the research highlights limitations in using PSM for evaluation purposes. These limitations include the inadvertent exclusion of certain stakeholders during the construction of system maps, the time-consuming nature of the method, the potential for diverse stakeholder insights to complicate result interpretation, the necessity to combine PSM with other methods for greater robustness, and the challenges posed by the composition of PSM focus groups, particularly with heterogeneous groups characterised by varying levels of influence and knowledge capital.
By emphasising the importance of an inclusive, stakeholder-oriented approach in PSM, which considers timing, context, and stakeholders’ ontological differences, we highlight the pivotal role of the evaluator as a facilitator in addressing challenges within systems thinking debates. This facilitation is crucial for fully harnessing the potential of PSM. We contend that despite its valuable insights, PSM has limitations that underscore the need for careful consideration of contextual appropriateness and the composition of PSM focus groups. For instance, a series of meetings may be conducted with different stakeholder categories, taking into account their knowledge and influence on the intervention.
Furthermore, we advocate for the utilisation of PSM in evaluations where community insights are essential for enhancing the evaluator’s understanding of how interventions achieve desired outcomes, notably transformative social interventions. Moreover, we suggest that future research should explore the combination of PSM with other methods to achieve greater complementarity and validity.
Acknowledgements
Competing interests
The author declares that he has no financial or personal relationship that may have inappropriately influenced him in writing this article.
Author’s contributions
S.E.B. is the sole author of this research article.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
The data that support the findings of this study are available from the corresponding author, S.E.B., upon reasonable request.
Disclaimer
The views and opinions expressed in this article are those of the author and are the product of professional research. It does not necessarily reflect the official policy or position of any affiliated institution, funder, agency, or that of the publisher. The author is responsible for this article’s results, findings, and content.
References
Bamberger, M., Vaessen, J. & Raimondo, E., 2016, Dealing with complexity in development evaluation: A practical approach, 1st edn., Sage Publications, Inc. Thousand Oaks, CA. https://doi.org/10.4135/9781483399935
Barbrook-Johnson, P. & Penn, A., 2021, ‘Participatory systems mapping for complex energy policy evaluation’, Evaluation 27(1), 57–79. https://doi.org/10.1177/1356389020976153
Barbrook-Johnson, P., Castellani, B., Hills, D., Penn, A. & Gilbert, N., 2021, ‘Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus’, Evaluation 27(1), 4–17. https://doi.org/10.1177/1356389020976491
Breu, K. & Peppard, J., 2003, ‘Useful knowledge for information systems practice: The contribution of the participatory paradigm’, Journal of Information Technology 18(3), 177–193. https://doi.org/10.1080/0268396032000122141
Brown, G. & Eckold, H., 2019, ‘An evaluation of public participation information for land use decisions: Public comment, surveys, and participatory mapping’, Local Environment 25(2), 85–100. https://doi.org/10.1080/13549839.2019.1703660
Brown, G., Strickland-Munro, J., Kobryn, H. & Moore, S.A., 2017, ‘Mixed methods participatory GIS: An evaluation of the validity of qualitative and quantitative mapping methods’, Applied Geography 79, 153–166. https://doi.org/10.1016/j.apgeog.2016.12.015
Cabrera, D., Colosi, L. & Lobdell, C., 2008, ‘Systems thinking’, Evaluation and Program Planning 31(3), 299–310. https://doi.org/10.1016/j.evalprogplan.2007.12.001
Costanza, R., 2001, ‘Visions, values, valuation, and the need for an ecological economics’, BioScience 51(6), 459–468. https://doi.org/10.1641/0006-3568(2001)051[0459:VVVATN]2.0.CO;2
Frame, B. & Brown, J., 2008, ‘Developing post-normal technologies for sustainability’, Ecological Economics 65(2), 225–241. https://doi.org/10.1016/j.ecolecon.2007.11.010
García-Nieto, A.P., Huland, E., Quintas-Soriano, C., Iniesta-Arandia, I., García-Llorente, M., Palomo, I. et al., 2019, ‘Evaluating social learning in participatory mapping of ecosystem services’, Ecosystems and People 15(1), 257–268. https://doi.org/10.1080/26395916.2019.1667875
Glouberman, S. & Zimmerman, B., 2004, ‘Complicated and complex systems: What would successful reform of medicare look like? in P.G. Forest, G. Marchildon & T. McIntosh (eds.), Changing Health Care in Canada, Issue 2, pp. 21–53, University of Toronto Press, Toronto.
Junge, K., Cullen, J. & Iacopini, G., 2020, ‘Using contribution analysis to evaluate large-scale, transformation change processes’, Evaluation 26(2), 227–245. https://doi.org/10.1177/1356389020912270
Keshavarz, N., Nutbeam, D., Rowling, L. & Khavarpour, F., 2010, ‘Social science & medicine schools as social complex adaptive systems : A new way to understand the challenges of introducing the health promoting schools concept’, Social Science & Medicine 70(10), 1467–1474. https://doi.org/10.1016/j.socscimed.2010.01.034
Király, G., Köves, A., Pataki, G. & Kiss, G., 2015, ‘Assessing the participatory potential of systems mapping’, Systems Research and Behavioral Science 33(4), 496–514. https://doi.org/10.1002/sres.2374
Koleros, A. & Mayne, J., 2019, ‘Using actor-based theories of change to conduct robust evaluation in complex settings’, Canadian Journal of Program Evaluation 33(3), 292–315. https://doi.org/10.3138/cjpe.52946
Lopes, R. & Videira, N., 2017, ‘Modelling feedback processes underpinning management of ecosystem services: The role of participatory systems mapping’, Ecosystem Services 28, 28–42. https://doi.org/10.1016/j.ecoser.2017.09.012
Martinuzzi, A. & Kopp, U., 2010, ‘Systemic constellations in theory-based evaluation – Tools and experiences’, in 6th European conference on Evaluation of Cohesion Policy on ‘New Methods for Cohesion Policy Evaluation: Promoting Accountability and Learning’, Warsaw, 30 November–01 December 2010, pp. 1–19.
Mccall, M.K., Investigaciones, C.D., Nacional, U. & México, A.D., 2021, ‘Participatory mapping and PGIS : Secerning facts and calues, representation and representativity’, International Journal of E-Planning Research 10(3), 105–123. https://doi.org/10.4018/IJEPR.20210701.oa7
Meadows, D., 2008, Thinking in systems: A primer, Chelsea Green Publishing, Chelsea.
Moon, K., Guerrero, A.M., Adams, V.M., Biggs, D., Blackman, D.A., Craven, L. et al., 2019, ‘Mental models for conservation research and practice’, Conservation Letters 12(3), 1–11. https://doi.org/10.1111/conl.12642
Moore, G.F., Evans, R.E., Hawkins, J., Littlecott, H., Melendez-Torres, G.J., Bonell, C. et al., 2019, ‘From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation’, Evaluation 25(1), 23–45. https://doi.org/10.1177/1356389018803219
Oliver, K., Lorenc, T. & Tinkler, J., 2020, ‘Evaluating unintended consequences: New insights into solving practical, ethical and political challenges of evaluation’, Evaluation 26(1), 61–75. https://doi.org/10.1177/1356389019850847
Patton, M.Q., 2012, ‘A utilization-focused approach to contribution analysis’, Evaluation 18(3), 364–377. https://doi.org/10.1177/1356389012449523
Penn, A.S., Knight, C.J.K., Lloyd, D.J.B., Avitabile, D., Kok, K., Schiller, F. et al., 2013, ‘Participatory development and analysis of a fuzzy cognitive map of the establishment of a bio-based economy in the Humber region’, PLoS ONE 8(11). https://doi.org/10.1371/journal.pone.0078319
Picciotto, R., 2016, ‘Evaluation and bureaucracy: The tricky rectangle’, Evaluation 22(4), 424–434. https://doi.org/10.1177/1356389016657934
Reason, P., 1998, ‘A participatory world’, Resurgence 186, 4–43.
Reyes-García, V., Orta-Martínez, M., Gueze, M., Luz, A.C., Paneque-Gálvez, J., Macía, M.J. et al., 2012, ‘Does participatory mapping increase conflicts? A randomized evaluation in the Bolivian Amazon’, Applied Geography 34, 650–658. https://doi.org/10.1016/j.apgeog.2012.04.007
Sedlacko, M., Martinuzzi, A., Røpke, I., Videira, N. & Antunes, P., 2014, ‘Participatory systems mapping for sustainable consumption : Discussion of a method promoting systemic insights’, Ecological Economics 106, 33–43. https://doi.org/10.1016/j.ecolecon.2014.07.002
Senge, P., 1990, The fifth discipline: The art and practice of the learning organization, Doubleday, New York, NY.
Setianto, N.A., Cameron, D. & Gaughan, J.B., 2014, ‘Identifying archetypes of an enhanced system dynamics causal loop diagram in pursuit of strategies to improve smallholder beef farming in Java, Indonesia’, Systems Research and Behavioral Science 31(5), 642–654. https://doi.org/10.1002/sres.2312
Skolimowski, H., 1994, The participatory mind: a new theory of knowledge and of the universe, Arkana/Penguin.
Stave, K.A., 2002, ‘Using system dynamics to improve public participation in environmental decisions’, System Dynamics Review 18(2), 139–167. https://doi.org/10.1002/sdr.237
Suno Wu, J., Barbrook-Johnson, P. & Font, X., 2021, ‘Participatory complexity in tourism policy: Understanding sustainability programmes with participatory systems mapping’, Annals of Tourism Research 90, 103269. https://doi.org/10.1016/j.annals.2021.103269
Taylor, S., Drayton, E., Mcbride, T., Ford, J., Eif, I., Better, J. et al., 2019, Evaluating early help: A guide to evaluation of complex local early help systems, Early intervention Foundation, viewed n.d., from https://www.eif.org.uk/files/pdf/evaluating-early-help.pdf.
Tourais, P. & Videira, N., 2021, ‘A participatory systems mapping approach for sustainability transitions: Insights from an experience in the tourism sector in Portugal’, Environmental Innovation and Societal Transitions 38, 153–168. https://doi.org/10.1016/j.eist.2021.01.002
Wilkinson, H., Hills, D., Penn, A. & Barbrook-Johnson, P., 2021, ‘Building a system-based theory of change using participatory systems mapping’, Evaluation 27(1), 80–101. https://doi.org/10.1177/1356389020980493
Worren, N., Moore, K. & Elliott, R., 2002, ‘When theories become tools: Toward a framework for pragmatic validity’, Human Relations 55(10), 1227–1250. https://doi.org/10.1177/0018726702055010082
|
Email this article 
Hide
Show all
