Original Research

Operationalising the transformative equity and the climate and ecosystems health criteria in evaluations: Lessons from pilot implementations

Sinenhlanhla Tsekiso, Ian Goldman, Jennifer Norins, Lungiswa Zibi, Sibongile Sithole

Received: 31 Aug. 2025; Accepted: 19 Nov. 2025; Published: 10 Apr. 2026

Copyright: © 2026. The Authors. Licensee: AOSIS.
This work is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license (https://creativecommons.org/licenses/by/4.0/).

Abstract

Background: The world is experiencing overlapping crises of climate breakdown, ecosystem degradation, rising inequality, and conflict. Traditional evaluation criteria inadequately address equity and environmental sustainability. To bridge this gap, key South African evaluation system stakeholders developed two new criteria: Transformative Equity (TE) and Climate and Ecosystems Health (CEH).

Objectives: This article shares lessons from the application of the TE and CEH criteria in three specific use cases. It examines how the criteria shape evaluation practice, highlights successes and challenges from pilot evaluations, and explores implications for institutionalisation.

Method: The article draws on authors’ direct involvement in the development and piloting of the criteria, key informant interviews with representatives from piloting institutions, and literature reviews conducted by Young Emerging Evaluators (YEEs) to map potential indicators and resources.

Results: Pilots revealed that the criteria (TE and CEH) encouraged evaluators to consider equity, climate, and ecosystem health more systematically. Positive outcomes included revisions of evaluation guidelines, inclusion of beneficiary perspectives, and stronger alignment with just transition policies. However, challenges included data gaps, limited climate literacy, methodological ambiguities, and institutional delays.

Conclusion: Integrating TE and CEH into evaluations requires adoption at design stage; practical tools; stronger data systems; and targeted capacity development. Institutional buy-in and leadership support are essential for mainstreaming these criteria.

Contribution: This article contributes to the growing body of knowledge on how evaluation can advance equity and environmental justice. It demonstrates how criteria reshape evaluation practice, strengthen accountability, and support transformative policy and institutional change.

Keywords: equity; evaluation; transformative change; inclusion; indicator mapping; institutional transformation.

Introduction

South Africa has been on a journey of transformation since the dawn of its democracy in the mid-1990s. As a country deeply scarred by colonialism and apartheid, the government aims for a transformative development agenda to achieve a ‘just, fair, prosperous and equitable’ society by 2030 (National Planning Commission 2012:61). However, in reality, the country continues to be fractured by grossly disparate social infrastructure, spatial divides, limited job growth and persistent poverty (Hausmann et al. 2023). While the country attempts to redress these critical social challenges, South Africa has also committed to responding to the threats of climate change by transitioning away from coal-based energy to more renewable energy sources, in a socially just manner (Presidential Climate Commission 2022). Although the country is recognised as a pioneer in just-transition planning for low- and middle-income countries (Strambo, Patel & Malmele 2024), the move to a just-transition also carries the weight of ensuring justice and addressing livelihood concerns, among the already disenfranchised communities surrounding the decommissioned mines, as well as others directly affected by the coal value chain (Munnik & Hallowes 2024; Strambo et al. 2024).

In 2021, the South African Monitoring and Evaluation Association (SAMEA), the South African Department of Planning, Monitoring and Evaluation (DPME), and other interested departments and individual evaluators collaborated on the development of two evaluation criteria that aimed to facilitate these desired social and ecological transformations. The working groups engaged in this process believed strongly that evaluation should do more than just measure programme performance; evaluation should actively contribute to meaningful transformation. The groups agreed that the widely used Organisation for Economic Co-operation and Development - Development Assistance Committee (OECD-DAC) criteria¹ do not sufficiently or explicitly address issues of equity or environmental sustainability, as was noted during public consultation in 2019 on the updated criteria (Kennedy-Chouane 2021) and further echoed by Patton (2020), Chaplowe and Hejnowicz (2021), Ofir (2021) and others. The groups adopted the hypothesis that if we want issues of a just-transition to be properly considered and redressed (1) programme designers, evaluation commissioners and evaluators need to be intentional in ensuring interventions promote transformative equity (TE), and a healthy planet, and (2) that the application of these two evaluation criteria in all evaluations undertaken by government would contribute to stronger policies and programming that promotes justice, equity and regeneration of our ecosystems.

The Transformative Equity (TE) and Climate and Ecosystems Health (CEH) evaluation criteria and guidelines (DPME 2023a; 2023b) were officially published by the DPME in 2023 (the guidelines can be found on SAMEA’s Knowledge Hub²). Over the 3 years since the criteria were developed, the working group has advocated for the adoption of the criteria by evaluation teams across different sectors, providing sensitisation and capacity-building opportunities and involving young emerging evaluators in a process of identifying complementary resources to support application. In addition, the working group has engaged with organisations interested in piloting the criteria in their upcoming evaluations.

The development process of the criteria and their conceptual frameworks has been described elsewhere (see Hazell et al. 2024; Norins et al. 2023). Following a brief introduction to the two criteria, the article focuses on the lessons emanating from three use cases where the criteria have been applied in evaluations, along with reflections from DPME as a key evaluation champion. To supplement the description of the criteria, the initial work conducted by young emerging evaluators on mapping of equity and CEH criteria is also presented. It draws on participant experience as the authors were involved in the development of the criteria and guidelines, and their application in the pilots. To understand the emerging lessons in more depth, semi-scripted key informant interviews were conducted with evaluation managers or evaluation leads from each agency that piloted the evaluations to better understand how the criteria were being applied, their experiences in the application of the criteria, the extent to which the organisations were institutionalising the criteria into their evaluation frameworks, and what support they perceive would help with adoption. In addition, evaluation plans and reports were reviewed where guidelines were influenced by these criteria, for example, for the New Development Bank (NDB) and the Department of Science and Innovation (DSTI).

After describing each use case, the article discusses the key challenges, successes and lessons learned from the experiences and puts forward recommendations for integrating equity more deeply into evaluation frameworks. Finally, the paper explores the broader policy and research implications, emphasising how the guideline can support institutional transformation and strengthen accountability in evaluation processes.

Development and evolution of the transformative equity and climate and ecosystems health criteria and guidelines

In 2021, during the coronavirus disease 2019 (COVID-19) pandemic, SAMEA facilitated a 2-week hackathon as an alternative to the in-person conference that SAMEA normally holds. The hackathon provided groups of participants from different sectors and levels of experience, including representatives from the public sector, private sector, non-profit organisations (NPOs), independent evaluators and emerging evaluators, the opportunity to work in a co-creation process on specific tasks to address monitoring and evaluation (M&E) priorities (for more on the SAMEA Hackathon, see Hazell et al. 2024). Two of the nine hackathon groups focused on conceptualising and developing two new evaluation criteria aimed at supporting evaluations’ role in addressing the interlinked crises of inequality and the growing climate and ecosystems crisis being felt in South Africa and globally. The finalisation of the criteria and accompanying guidelines took place in 2022.

One group developed the CEH criterion (DPME, 2023a), which examines an intervention’s interaction with and contribution to climate change and the viability or health of an ecosystem. The dimensions of this criterion include; firstly, how interventions can affect the environment and contribute to climate change and ecosystems breakdown (or the converse); secondly, the potential implications of climate and ecosystems breakdown on interventions (e.g. how they could be affected by potential disasters) and thirdly, the extent to which adaptive capability is being introduced to interventions, that is their resilience (See Figure 1). The CEH guideline recognises that climate change hits vulnerable communities hardest, often deepening existing social and economic divides. It encourages evaluators to explore how environmental risks intersect with social injustices and to assess whether interventions promote outcomes that are both fair and sustainable.

FIGURE 1: Climate and ecosystems health dimensions.

The second group developed the TE criterion (DPME, 2023b), which assesses whether an intervention’s objectives, design, implementation and impact contribute to, or do not contribute to, addressing systemic inequities and promoting a more inclusive society. In the conceptualisation phase, the TE group spent several sessions deliberating on terminology and what dimensions of equity were relevant for the specific context of South Africa (see Norins et al. 2023). Given South Africa’s history of apartheid and based on literature considering the systemic enablers of inequity (Bitar 2021; Makgetla 2020), the group proposed to use five dimensions (i.e. population, cause and effect, space, content and intention, and timing), which serve as an analytical lens for surfacing inequities that may be perpetuated through interventions, knowingly or unknowingly (Figure 2). Further, the TE guidelines provide a set of guiding values for evaluation teams to adopt to explicitly consider systemic injustice and issues of power within interventions and evaluations, including the need to be intentional about the inclusion of marginalised voices in evaluation processes (Norins et al. 2023).

FIGURE 2: Transformative equity dimensions.

Although the CEH and TE criteria were developed as distinct frameworks, the interconnection between climate and ecosystem-health and socio-economic justice is both evident and critical, and the underlying systemic challenges are causing both. Together, these criteria offer evaluators a powerful way to assess interventions through a lens that embraces social, economic and environmental justice, helping to steer programmes towards truly transformative impact.

For each criterion, a guideline was also developed to support the integration of the criterion into evaluation terms of reference (ToRs), based on the DPME’s guideline on ToRs, and includes examples of key principles to guide evaluation commissioners and evaluation teams in considering each criterion in the design, implementation and follow-up of the evaluation. Given the working groups’ intention that the new criteria be integrated into all evaluations and given the evident and critical interconnection between climate and ecosystem-health and socio-economic justice (Peek 2025), SAMEA and DPME facilitated training workshops and sensitisation meetings on both criteria, and served on evaluation steering committees to support the piloting of the criteria.

To further expand the criteria’s use, SAMEA engaged three young emerging evaluators (YEEs) for a period of 6 months to compile existing resources, references and indicator sets that could serve as additional guidance to evaluators as they incorporate the criteria into evaluations they conduct. This process entailed an in-depth review of existing literature and documentation related to the different dimensions of the TE and CEH criteria. The YEEs systematically selected indicators that could potentially be applicable to the South African context and the extent to which they have been recognised and adopted regionally and internationally.

The process resulted in two key outputs: a comprehensive spreadsheet compiling indicators aligned with the different dimensions of the TE and CEH criteria, and an annotated bibliography summarising the relevance and rationale for the inclusion of each indicator. The YEEs conducted a literature review, engaging with articles and policy documents from South Africa and abroad, including the MER framework of the South African Department of Forestry, Fisheries and the Environment, the Presidential Climate Commission, SA Stats, OECD Education Equity Dashboard and the World Health Organization (WHO) health data repository.

The YEEs working on both criteria noted the challenges in finding appropriate indicators and resources for the specific dimensions and South Africa’s specific historical, structural and economic context. For the TE criterion, there were many indicators focusing on issues of population-based equity (TE Dimension 1), leading to several sub-categories for health, education, income, gender, etc. The other dimensions, particularly the dimensions for intention and time, presented more challenges in identifying appropriate proxies for measurement. For the CEH criterion, two YEEs focused on the potential resource use and pollution effects of the intervention (to assess CEH Dimension 1: extent to which an intervention contributes to degeneration), and then indicators to measure adaptation and mitigation (to assess CEH Dimension 2: an intervention’s regenerative contribution). The YEEs noted better availability of indicators for issues of resource use and pollution (CEH Dimension 1) but less availability of indicators to cover an intervention’s regenerative contribution, perhaps because of the recency of focus on regenerative practices (Shackelford & McDougall 2023) compared to the longer history of environmental impact assessments (Burdge 1991).

These indicator lists and annotated bibliography are still in draft stages, requiring further refinement and peer review. The indicators may be more useful for programme planning and monitoring purposes, while also giving evaluators a starting point for what to consider in their assessment of issues of equity and CEH. One of the next steps for this work is to develop checklists or rubrics that evaluation teams can use to help them in considering the potential relevance and influence of each dimension, particularly in cases where there is no pre-existing data. The overarching aim is to develop a consolidated toolkit that can be used in conjunction with existing guidelines to support evaluations that actively promote or critically engage with TE.

The next sections describe the examples of organisations that agreed to integrate the criteria into planned evaluations, as well as reflections from the YEEs in their mapping process.

How the criteria were applied in pilot evaluations

Finding suitable pilots

Since the initial drafting of the guidelines in 2022, SAMEA and DPME have followed multiple steps to promote them by evaluation units and consultancies across South Africa. Piloting the criteria in real-time evaluations formed a critical process in promoting use, starting with a piloting process in 2022 involving three purposefully selected evaluation teams, and continuing with sensitisation and recruitment of willing organisations. South African Monitoring and Evaluation Association and DPME aimed to use the pilot process to test the relevance and feasibility of the guidelines across sectors and to generate practical insights on how evaluators and commissioners could embed equity, climate and ecosystem-health considerations into their evaluations. With the publishing of the guidelines by DPME in 2023, SAMEA and DPME have collaborated on other webinars, conference sessions and capacity-building trainings to promote further use. In addition, DPME continues to sensitise officials on the guidelines and will be including the criteria in the next national evaluation policy framework, currently under revision.

Since 2022, SAMEA has been aware of seven evaluations that have used one or both criteria to guide evaluation questions. For five of these evaluations, SAMEA provided varied technical support as requested, including guidance discussions and input on the development of ToRs to ensure that the criteria were integrated into evaluation designs from the outset. South African Monitoring and Evaluation Association also served on evaluation steering committees for two of the evaluations and as a peer reviewer for others. This collaborative, practice-based approach is key in building the capacity of both evaluators and commissioners to apply the criteria meaningfully within their institutional contexts.

Documenting the application process and lessons learned from the application of the TE and CEH criteria has been a critical objective of the working group since the beginning of the piloting process in 2022. The authors reached out to three organisations that have successfully applied the criteria to their evaluation ToRs and processes (i.e. the DSTI, the Independent Evaluation Office of the National Development Bank [IEO-NDB], and the Development Bank of Southern Africa [DBSA]) to understand their successes and challenges in their process. The authors also engaged a representative from the evaluation unit of DPME as a champion of evaluation broadly and a co-developer of the two guidelines. Further detail on each case is presented below, before an analysis of the findings.

Implementation evaluation of the water research roadmap

The DSTI was an early adopter, participating in the initial training workshop provided in 2022, to apply the criteria to their Implementation Evaluation of the Water Research Roadmap. During this training, the participating programme manager, steering committee lead and evaluation manager worked with workshop facilitators to modify the existing ToR for an implementation evaluation of the Water Research Roadmap. The evaluation team committed itself to bringing the revised ToR to the bid committee and ensuring that the commissioned evaluation team would apply the two guidelines to the evaluation. The South African Monitoring and Evaluation Association served on the steering committee over the 2 years of commissioning, implementation and finalisation of the report, providing suggestions on the process and participating in the learning. Figure 3 provides an example of how the two criteria were incorporated into the questions of this evaluation.

FIGURE 3: Graph from the Implementation Evaluation of South Africa’s Water RDI Roadmap, Pretoria.

The South African Monitoring and Evaluation Association facilitated a ToR-refinement meeting with participants from DSTI’s evaluation and programme units. As a result, the DSTI evaluation guideline was rewritten to include detailed instructions on how to assess ecosystem services alongside equity considerations. The revised guideline established clear entry points for TE and CEH at three stages: planning, where teams draw up stakeholder lists to ensure inclusion of marginalised voices; implementation, where fieldwork protocols must capture environmental indicators, such as water quality and biodiversity health; and follow-up, where evaluation conclusions are assessed for alignment with the national just-transition framework. The review of the updated M&E manual confirmed the inclusion of flow charts and checklists that explicitly reference these criteria.

The DSTI shared key lessons from piloting the TE and CEH criteria. Firstly, integrating these criteria from the start of a programme is essential. It enables better data collection and more meaningful evaluations. Secondly, support from DPME and SAMEA was instrumental in overcoming challenges and ensuring the effective application of the guidelines. A major challenge was the lack of practical guidance in existing literature and programme documents, making it difficult to apply the criteria effectively, especially around equity and CEH in water access. The adjustments made to strengthen the instruments and include equity and CEH considerations resulted in equity and CEH-informed recommendations being included in the final report. Consequently, the DSTI has now formally adopted the criteria in its updated ‘Guidelines and Standards for Evaluation’.

Evaluations of the implementation of renewable energy by the independent evaluation office of new development bank

The New Development Bank is a demand-driven bank, committed to cooperative collaboration with participating countries in their individual and collective efforts of advancing environmental sustainability and resilience, inclusive social development and technological integration. The Independent Evaluation Office (IEO) of the NDB enhances the bank’s development effectiveness by independently assessing its operations, policies and strategies, while maintaining full autonomy from management and reporting directly to the Board of Directors.³ Since 2023, the IEO-NDB has conducted several evaluations on their investments in renewable energy projects in South Africa. The evaluation office was first introduced to the criteria in 2023, when SAMEA provided a peer review of their performance evaluation of the Greenhouse Gas Emissions Reduction and Renewable Energy Sector Development Project. The IEO included a reference to the two new criteria, recognising the potential role they have in evaluations of this nature. For subsequent evaluations, the evaluation teams have engaged eagerly with SAMEA and DPME to consider how to more deliberately integrate the two criteria in their evaluation questions and evaluation matrices.

The different evaluation teams have relied on the guidelines as well as discussions with SAMEA and DPME, both in the scoping and review phases of the evaluations, and to peer review the evaluations. However, follow-up interviews highlighted that although CEH principles appeared in guidance documents, NDB staff treated these measures as optional add-ons.

Of interest, the first edition of the IEO’s evaluation manual (New Development Bank, 2024:17) uses the DPME/SAMEA criteria as an example of the IEO’s approach to cooperative collaboration and being driven by countries’ contexts and priorities.

Development bank of South Africa

The DBSA is a leading Development Financial Institution (DFI) mandated to promote inclusive, sustainable economic growth and regional integration by supporting and financing projects and programmes in South Africa, the Southern African Development Community (SADC), and the wider African continent. The Bank regularly conducts evaluations to assess, among others, the social, financial, economic and environmental outcomes of funded projects or programmes, using international best practices such as OECD/DAC criteria.

As a key player in renewable energy projects and other infrastructure projects, the DBSA is a more recent adopter of the criteria, following an introductory discussion with the evaluation unit and sharing of the guidelines in 2024. For their recent ‘Impact and Implementation Evaluation of the Inter-Bursary Support Programme (IBS)’, the evaluation office only incorporated the TE criterion because the team found it easier to integrate into the evaluation questions as a first pilot attempt, particularly given that the evaluation team did not engage in specific training on the two criteria. Furthermore, the TE criterion aligns with DBSA’s requirement to appraise the ‘Living experience of recipients’ to better understand and appreciate what difference (if any) the projects have had on communities and people’s lives. Both criteria examine the development outcomes of projects with particular emphasis on the recipient’s gender, age, employment and geographic location. Therefore, the DBSA has embraced the ‘Transformative Equity’ criterion and has committed to incorporating it into all evaluations and subsequently ensure the criterion informs the design of the data collection instruments.

Institutionalising criteria in the national evaluation system

The Department of Planning, Monitoring and Evaluation (DPME) has been a key collaborator and champion in the development of the two criteria. South African Monitoring and Evaluation Association and DPME have actively advocated for their use through webinars and trainings, and DPME is including the two criteria as key components in the updated national evaluation policy framework. Despite these sensitisation efforts, an interview with a representative from the DPME evaluation unit revealed that DPME staff appreciated the conceptual framework but lacked the practical tools to apply the criteria. They requested sample rubrics, indicator banks and hands-on training sessions. However, without senior-level endorsement and clear integration into their approval workflows, DPME members reported that the guidelines were not being implemented effectively.

Findings across the cases

To learn from these case organisations on the relevance of and feasibility of using the guidelines across sectors, the authors interviewed seven representatives from the evaluation teams from the different organisations (i.e. two from DSTI, one from DBSA and four from IEO/NDB). The authors also interviewed a representative from DPME to understand how the guidelines are being taken forward in DPME evaluations.

Table 1 summarises some key themes emerging, notably some good practices, institutional adoption strategies, operationalisation challenges, unanticipated benefits and synergies with existing evaluation frameworks.

Ethical considerations

This article followed all ethical standards for research without direct contact with human or animal subjects.

Results

Overall, there was significant institutionalisation arising. Respondents acknowledged that evaluation teams began asking sharper questions about who benefits and who is excluded, produced richer analyses of equity dimensions (e.g. inclusion of the ‘Living experience of recipients’ in DBSA evaluations and projects), and incorporated new measures of ecosystem-health. However, capacity constraints and unresolved methodological questions, especially around the climate and ecosystems’ health criteria, limited full integration in some instances.

Institutionalisation varied across the four organisations. At the DBSA, following a presentation to the evaluation team on the two guidelines, senior management issued a formal directive requiring all evaluation teams to include a ‘Beneficiary Impact’ section in their reports, which led to a redesign of data collection instruments, with teams adding questions about marginalised groups to survey questionnaires and focus-group protocols. The NDB also embraced the equity criterion, integrating it into appraisal tools for its renewable energy portfolio. The DSTI rewrote its evaluation guideline to include detailed instructions on how to assess ecosystem services alongside equity considerations, with entry points in evaluation planning, implementation, where fieldwork protocols must capture environmental indicators, such as water quality and biodiversity health and follow-up, where evaluation conclusions are assessed for alignment with the national just-transition framework. By contrast, DPME had formally incorporated both guidelines into its national evaluation system guidelines, yet struggled to move beyond the pilot phase, but continues to champion for the criteria’s use and is including them in the revision of the national evaluation policy framework. There is interest in using the criteria and guidelines; however, there is a need for hands-on training on how to use the criteria. This will enable greater use of the criteria within the national evaluation system by the departments that the DPME supports. Therefore, in three of the cases, there was a significant impact from the guidelines on institutionalisation, but the DPME case points to challenges.

Challenges in operationalising the guidelines

Data availability and quality emerged as the most pervasive barrier across the three pilots. At DBSA, evaluation teams recognised that their management information system did not capture disaggregated demographic fields needed to report on gender, income level and geographic location. As a result, analysts could not quantify how programme outcomes differed across vulnerable sub-populations (TE Dimension 1). Similar challenges surfaced at NDB, where appraisal databases lacked markers for women-led or rural community investments, forcing teams to deploy manual surveys and add-on data collection to fill the gaps. At DSTI, there was no structured system for tracking ecosystem-health indicators, such as water quality or biodiversity status; evaluators resorted to proxy measures drawn from technical reports and community focus-group feedback.

In reflecting on incorporating the equity criteria in the evaluation processes, the NDB highlighted challenges around data collection for these criteria, which are more implicit than traditional impact and sustainability measures, and the difficulty of evaluating them when not incorporated into project design. They agreed that while these criteria could overlap with existing materials, they should be treated as separate, stand-alone criteria for trial purposes, with a recommendation to emphasise them as a distinct subtopic under impact in future evaluations. Although the NDB has incorporated community voices through stakeholder engagements and interviews, while also addressing climate and ecosystem-health considerations in their evaluation framework, there is still a need to improve questions and monitoring frameworks to better assess interlinkages between criteria, with a focus on adaptive management and resilience. Lastly, there is a need for more sensitisation and training to help people understand and use this framework. This will strengthen its application and integration.

Institutional delays, notably around procurement, further impeded timely adoption. At DSTI, it took 3 months to finalise the revised ToRs because successive steering committee meetings and supply-chain approvals pushed back the pilot’s start date. By the time the guidelines were formally approved, evaluation instruments were already in use, necessitating amendments mid-stream that weakened methodological coherence.

Capacity gaps were consistently reported across all institutions. At DBSA, evaluators experienced difficulty operationalising the CEH criterion because of limited climate literacy within the evaluation teams. At DBSA, evaluators experienced difficulty operationalising the CEH criterion because of limited climate literacy within the evaluation teams, particularly regarding climate’s relevance to evaluation. This highlights the need for targeted capacity development to strengthen evaluators’ understanding of the linkages between climate, ecosystem health and programme performance. Similarly, at DPME, while there was reportedly a high level of awareness of the TE criterion, only a few evaluators applied it systematically in their work.

The responses reflect that institutions faced challenges in incorporating the TE and CEH guidelines into mid- or endline evaluations, particularly because of gaps in appropriate data and the expected learning that would take place during a pilot. Respondents noted that while the guidelines were helpful in framing evaluation questions and raising awareness of issues, such as equity, more support and training are needed to apply them consistently. Institutions emphasised the importance of embedding these criteria early in programme design, building team capacity and improving tools for data collection and analysis. There was also a call for clearer indicators and examples to guide evaluations in integrating CEH and equity concerns effectively.

Across all sites, most noticeably at DPME, evaluators reported uncertainty about translating high-level criteria into concrete protocols, especially with the CEH guideline, and identified the need for practical guidance not sufficiently included in the guidelines. In the absence of practical rubrics, indicator banks or worked examples, staff felt unequipped to apply the guidelines systematically. The DPME respondent also observed pushback against perceived ‘added complexity’ of including these two criteria in an evaluation scope, especially from departments prioritising rapid evaluations because of financial constraints. The DPME representative suggested more hands-on training sessions and prototype checklists to bridge the gap between conceptual understanding and day-to-day evaluation practice. The work being developed by the YEEs aims to address this gap.

These operational challenges highlight the urgency of strengthening data systems, streamlining approval processes and equipping evaluators with user-friendly tools before further roll-out of the TE and CEH guidelines.

Enabling factors and outcomes

The pilots revealed several enabling factors that supported the integration of the guidelines: issuing of the guidelines by DPME gave credibility, senior management commitment at DBSA and DSTI drove institutional reforms; revisions of internal evaluation manuals and ToRs to operationalise the guidelines; the involvement of peer reviewers with just-transition expertise provided technical depth. These factors contributed to greater alignment between evaluations and national policy priorities, particularly the just-transition and inclusion frameworks.

Senior management buy-in set the foundation for revising key instruments, while the engagement of technical experts ensured alignment with just-transition and equity goals. These pathways together strengthened the inclusion of equity and CEH considerations across the evaluation cycle.

These challenges reinforce the need for stronger guidance, clearer indicator frameworks and enhanced evaluator training to support the meaningful integration of TE and CEH principles into evaluations. They also point to the importance of embedding these criteria at the programme design stage so that relevant data can be collected systematically from the outset.

Emerging impacts illustrating the potential power of these approaches

Piloting the guidelines generated a range of impacts that we could not have foreseen, illustrating the potential transformative power of taking these issues seriously. At DBSA, teams observed strengthened stakeholder trust when beneficiary voices featured prominently in closing presentations. Executives were so moved by recorded testimonials that they approved additional equity allocations for follow-up initiatives. Embedding the equity criterion in its appraisal tools at NDB has begun to foster greater awareness and dialogue on gender and equity considerations within renewable energy projects. While challenges remain in data collection and applying these criteria systematically, the process has helped highlight gaps and opportunities, laying the groundwork for more inclusive and responsive approaches in future funding cycles. At DSTI, the introduction of indigenous-knowledge focus groups in the Water RDI Roadmap evaluation uncovered community-driven water-management practices. These insights led to the co-development of new agroecology modules in subsequent capacity-building workshops. Even at DPME, where formal uptake stalled, the joint development of rubrics and indicator banks sparked unprecedented collaboration between evaluation and policy teams.

Synergies with existing frameworks

Alongside these impacts, pilots revealed clear synergies between the new criteria and existing evaluation frameworks. DBSA aligned the TE criterion with the Made in Africa Evaluation principles by merging indicators for rural programmes. The DBSA aligned the TE criterion with the Made in Africa Evaluation principles by merging indicators for rural programmes, because both frameworks emphasise marginalised voices. The IEO-NDB integrated the CEH criterion into its Green Climate Fund results framework, thereby streamlining reporting requirements and avoiding parallel data collection efforts, and ensured CEH measures formed part of routine funding-appraisal cycles. These alignments not only reduced administrative burden but also reinforced the strategic value of TE and CEH measures within established institutional processes.

These synergies demonstrate that, when thoughtfully embedded, TE and ecosystem-health measures can reinforce, and be reinforced by, established institutional policies.

Lessons

Overall, we see that the guidelines were adopted by all three of the case organisations and are beginning to influence institutional processes, thus having an impact in terms of the organisation’s work. The dimensions introduced in the guidelines, such as population, space, timing, cause and effect, and intention, proved useful in helping teams move beyond generic commitments to equity, enabling sharper and more context-specific analysis. The resulting evaluations show marked progress towards social and environmental justice objectives.

Piloting the guidelines revealed that institutional change is possible when senior leadership champions the new criteria and evaluation teams possess sufficient technical capacity. In settings where directors issued clear mandates and revised internal manuals, teams adapted work plans, data collection instruments and reporting templates to reflect equity and ecosystem-health considerations. These pilots demonstrated that embedding high-level principles into day-to-day practice can improve the focus on marginalised groups, surface environmental impacts and build stakeholder trust. These criteria can also be combined with other related frameworks, such as MAE evaluations, to increase the space for marginalised voices to be heard.

Nevertheless, persistent obstacles remain. First of all, the lack of interoperable data systems and disaggregated indicators continues to limit rigorous assessment of who benefits and which ecosystems thrive. Evaluators need routine access to disaggregated demographic and environmental data that populate their indicator banks and reporting templates. Methodological ambiguities, especially around intersectionality and ecosystem metrics, highlight the need for clearer frameworks and worked examples. Embedding the criteria at the project design stage will increase the likelihood that data are collected systematically from the outset, rather than patched in retrospect.

In the case of DSTI, procedural delays in finalising ToRs and securing procurement approvals force teams to adopt guidelines mid-stream, compromising methodological coherence.

The operationalisation of equity and climate criteria requires practical tools, accessible guidance and dedicated capacity-building. Evaluators need more than theoretical frameworks; they need methods that are participatory, inclusive and capable of uncovering the differential impacts of policies and programmes. Applied training and peer-learning forums will help evaluators move from conceptual understanding to confident application.

Conclusion

Drawing on insights from both the pilot evaluations and the indicator mapping process, several key implications emerge for how evaluation criteria and frameworks can more effectively address issues of equity and environmental justice in real-world settings.

Firstly, integrating equity into evaluation practice is not simply a matter of adding new indicators. It requires a fundamental shift in how evaluation is conceptualised and applied. Traditional approaches often prioritise efficiency, effectiveness and narrow interpretations of sustainability. While these remain important, they tend to overlook deeper questions related to fairness, structural inequality and the resilience of natural systems. Our experience suggests that equity and ecological concerns must be treated as core pillars of evaluation frameworks if they are to meaningfully influence programme design, implementation and impact. It also demonstrates the potential power when this shift happens.

Secondly, when equity and climate considerations are intentionally prioritised, evaluations can surface critical information about who benefits, who is excluded and how interventions interact with broader social and environmental systems. This holistic perspective allows for a more nuanced understanding of both intended and unintended outcomes. It also strengthens the evaluative value of the work by ensuring that marginalised voices and environmental risks are not treated as peripheral, but are integral to the analysis.

Thirdly, integrating these dimensions equips decision-makers with more robust evidence to guide resource allocation. Evaluations that consider the intersections between social inequalities and environmental vulnerabilities can help shift attention towards interventions that are not only technically sound but also just and future-oriented. Embedding these principles in evaluation contributes to broader development goals, including inclusive economic transformation, ecological regeneration and the advancement of a just transition.

Finally, integrating equity and environmental sustainability into evaluation practice is not merely a technical enhancement, but it is a value-based and political act. It reflects a conscious choice by evaluation commissioners, practitioners and professional bodies to use evaluation as a tool for transformation. This orientation challenges evaluators to consider how their work contributes to, or undermines, broader goals of justice, inclusion and sustainability. The implications extend beyond individual evaluations to the norms, standards and systems that govern evaluation practice at national and global levels.

Acknowledgements

This article is based on a conference paper originally presented at the 11th African Evaluation Association (AfrEA) Conference, held in Kigali, Rwanda, on 18–22 March 2024. The conference paper, titled ‘New evaluation criteria to support transitions to more just and environmentally healthy societies’, was subsequently expanded and revised for this journal publication. This republication is done with permission from the conference organisers.

The authors would like to acknowledge the participation of SAMEA, DPME, DSTI, IEO-NDB, DBSA, Tracey Bailey, Makgorometja Chepape, Amahle Nciweni and Lelethu Bodlani.

The authors disclose professional affiliations with SAMEA, the International Evaluation Academy, and the Government of South Africa. They have disclosed these affiliations fully and confirm that it has not influenced the design, conduct or reporting of the research presented in this article.

The authors also disclosed receipt of financial support from The Rockefeller Foundation through a grant to Genesis Analytics. The authors declares that they have no other financial or personal relationships that may have inappropriately influenced them in writing this article.

Sinenhlanhla Tsekiso: Conceptualisation, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. Ian Goldman: Conceptualisation, Data curation, Formal analysis, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing. Jennifer Norins: Conceptualisation, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. Lungiswa Zibi: Conceptualisation, Formal analysis, Methodology, Writing – original draft. Sibongile Sithole: Conceptualisation, Data curation, Formal analysis, Investigation, Methodology, Resources, Writing – original draft, Writing – review & editing. All authors reviewed the article, contributed to the discussion of results, approved the final version for submission and publication, and take responsibility for the integrity of its findings.

Support for the AfrEA Conference climate strand, where some of this work was initially presented, was provided by The Rockefeller Foundation through a grant to Genesis Analytics.

Data supporting the findings cannot be made publicly available because of confidentiality restrictions. However, the authors are available for discussions on the methodology and data management, upon request.

The views and opinions expressed in this article are those of the authors and are the product of professional research. They do not necessarily reflect the official policy or position of any affiliated institution, funder, agency or that of the publisher. The authors are responsible for this article’s results, findings and content.

References

Footnotes

1. https://www.oecd.org/en/topics/sub-issues/development-co-operation-evaluation-and-effectiveness/evaluation-criteria.html.

2. https://www.samea.org.za/category/knowledge-hub/guidelines-resources/.

3. https://www.ndb.int/governance/independent-evaluation/