Abstract
Background: The world is facing rapidly declining health of the climate and ecosystems on which all species depend, with wealth accumulating in the hands of a few, a result of unsustainable economic systems. Evaluation has the potential for a significant role in learning from the past and helping to guide a regenerative future, but for this, the approach to evaluation and the systems that produce them must be transformative and take on more holistic approaches to society and the planet.
Objectives: This study aims to explore how cases of African national evaluation systems (NESs) apply elements of a decolonised social-ecological model and how this could be strengthened.
Method: This study involves a constructive critical analysis of the South African and Benin NESs, drawing on literature on decolonising evaluation and a new institutionalism lens to the formation of post-colonial bureaucracies, tested in a webinar conversation around decolonising evaluation in November 2023.
Results: The African NESs have embedded learning, exhibit both machine-based and ecological-based elements, and experience some decolonised aspects. A key limitation is the lack of involvement of communities in the systems.
Conclusion: This study argues for: (1) allowing NESs to break from historical forms of bureaucratic functioning; (2) developing a systems-based approach as the basis for new thinking around NESs, strengthening their ecological aspects; (3) embracing the learning approaches we see in both countries; (4) embracing principles of participatory democracy and co-production by strengthening the voice of non-state actors, particularly citizens, in the formation of NESs and (5) changing power dynamics, in NESs and evaluations.
Contribution: This article is contributing to a debate on how evaluation systems can be decolonised and power relations changed.
Keywords: decolonisation; evaluation; institutionalism; national evaluation systems; eco-just world; social-ecological systems; machine-based systems; just transition.
Introduction
In the evaluation community, much attention is now being focussed on transforming evaluation to address the profound social-ecological challenges of our time. The Prague Declaration of 2019 on Evaluation for Transformational Change1 stated that:
We commit to evaluations that help us learn, understand and support the transformational and systemic changes needed in our countries and the world, as agreed upon in the 2030 Agenda for Sustainable Development. (p. 1)
New approaches to ‘development’ and to evaluation are needed everywhere. A sustainable balance between the social, economic and environmental domains is crucial in light of the existential threats of the climate crisis, mass extinction of species, growing local and global inequity, and the unsustainable use of the resources of the planet. All too often, evaluations support neo-liberal bureaucratic models that maintain a dysfunctional status quo, perpetuate the exploitation of people and planet, and could lead to the collapse of civilisation as we know it (Parsons & Winters 2023). We are privileged to be part of this special edition of the African Evaluation Journal honouring Sulley Gariba, who questioned established approaches to evaluation and was an early advocate of Made in Africa approaches.2 We follow in that tradition seeking to make evaluation practice and evaluation systems contribute to the regeneration humanity and the planet need.
The International Evaluation Academy was established following the 2019 Prague Conference specifically to look at the transformation of evaluation, and one of its initial activities has been the publication, with the Journal of Multi-Disciplinary Evaluation, of a special edition on ‘Decolonising evaluation: Towards a fifth paradigm’.3 This article builds on this work to explore how national evaluation systems (NESs) should be rethought to help address the transformational challenges that humanity faces in this epoch.
As Chilisa and Bowman (2023) suggest in the editorial of the special edition, there is a:
[N]eed for each evaluator to question the philosophical foundations of their beliefs, where they come from, and the ways these beliefs influence how they go about doing evaluation and how they position themselves in the context of the history and social theory of evaluation. (p. 3)
The same applies to evaluation systems.
We provide an analytical framework around decolonisation and social-ecological systems from a ‘new institutionalism’ perspective and apply this to two African NESs – Benin and South Africa. We use decolonisation not just in the sense of moving away from Western systems of thinking, valuing and knowing, but moving away from an externally defined and controlled neo-liberal economic system, which is causing both climate and ecosystem breakdown and as well as extreme inequality. Bendell (2023: Kindle location 597–599) refers to ‘Imperial Modernity, the interlocking set of political, economic and cultural systems that shape our everyday lives to favour the accumulation of power by elites’, some of the most extreme versions of which are found in the Global South.
We analyse these NESs against the framework and suggest ways that NESs could be adapted to be more responsive to the social-ecological system changes needed by humanity. Since its inception, the Academy has been exploring decolonisation and related system science issues, and this article and the accompanying webinar conversation seek to focus this analysis and ongoing debate on decolonising NESs. The UN Declaration on the Rights of Indigenous People (UNDRIP) recognises First Nations and Indigenous peoples as sovereign nations. While the analysis in this article focusses on African nation states and national evaluation systems, the analysis can be applied to evaluation systems that Indigenous nations may seek to apply. In such a case, they are freer to apply a fully decolonised lens and develop systems based on Indigenous ways of knowing and being4.
The need for change
The stability and resilience of the Earth system and human well-being are inseparably linked. Rockström et al. (2023) warn that we are exceeding seven of eight globally quantified safe and just planetary or earth system boundaries in over half of the global land area. Current global policies would lead to 2.4–2.7 degrees centigrade of warming by 2100,5 which would cause severe harm to 1.5 billion people (Rockström et al. 2023). In addition, we are destroying many of the species on the planet, on which our food supply, a safe environment and ultimately humanity’s own survival depend.
This devastation is a consequence of the dominant values of competition, exploitation of people and nature, and the logic of continued exponential growth, treating the environment as an externality and valuing profit over the well-being of the population. As Picciotto (2023) states:
[T]he neoliberal establishment shaped development cooperation programmes that successfully propagated a rising-tide-lifts-all narrative, which concealed a rapid deterioration of the natural environment and tolerated blatant social disparities that left billions of poor people behind, while vastly increasing the wealth of one percent of the global citizenry. (p. 134)
Post-colonial governments have often perpetuated the distance of urbanised, Westernised elites from rural populations practising deep indigenous traditions. The purpose of ‘development’ has often been seen in economic terms as creating wealth, albeit often for a few, and Westernised social services such as health systems, education, etc., which do not build on traditional knowledge. These services favour urban rather than rural people, distance people from their connection to nature and seek to reframe people as employees and consumers rather than active protagonists managing their lands and their societies.
So if states have often perpetuated an exploitative, elite-driven development model, how has evaluation fared? While evaluation developed initially in response to welfarist programmes notably in the United States (US), in the global South, evaluation has usually been driven by development cooperation, imposed for accountability purposes to report back to home governments and parliaments. It has not sought to support locally driven evaluation processes that would support learning and improvement in the global South (UNEG 2022).
Since the 1990s, some Southern governments have sought to develop centralised evaluation systems, starting with Colombia in 1994, Mexico in the mid-2000s and in Africa, Benin in 2007 and South Africa and Uganda from 2011. However, it is striking how few countries in Latin America, Africa and Asia have sought to establish such NESs (Goldman et al. 2023).6
All too often where evaluation is practised, governments have not properly integrated it into the public management process where it can inform planning, budgeting and implementation (Twende Mbele & CABRI 2022). Even where evaluations are becoming established, they may become routinised and conducted as part of the machinery of government, without focussing on transformative outcomes, especially for those disadvantaged and disempowered by the current system.
In this article, we ask how we can redesign NESs such that they help to build the just, restorative and regenerative world we all need.
Analytical framework on decolonisation
In this article, we apply the frame of new institutionalism to: (1) understand colonial-derived systems of bureaucracy which promote neo-liberal values and (2) efforts to decolonise NESs. Understanding the call for the decolonisation of (state-led) NESs requires an examination of how these systems are formed within the historical context of bureaucratic institution-building. This provides the groundwork for framing a critique of the current processes and architecture, a critique that forms the template for establishing, strengthening and institutionalising NESs. This also helps in understanding the challenges of NESs, such as the routinisation of evaluations, their accountability and compliance orientation, and the difficulties of integrating evidence from evaluations into budget and policy decision-making systems.
The theory of institutionalism broadly (including the expanded concepts of old and new institutionalism, among myriad scholarly variations of the same) helps to explain how colonial systems of bureaucracy continue to shape many of our modern democracies in Africa and the global South and provides a valuable way of understanding the current form and function of national evaluation systems. Old institutionalism, in particular, became popular in the study of states and public administration, as its appeal is the notion of the predictability of institutions, how they form and how they behave (Hira & Hira 2000). This desired predictability is common to the oft-critiqued Weberian-type bureaucracies that dominate modern democracies globally. Elements typically include an elite and hierarchical bureaucratic machinery, meritocratic recruitment and little involvement of non-state actors in bureaucratic functioning. This machine-like model has created the seemingly immutable iron cage image of bureaucratic institutions. In this view of the immutability of institutions, there is ‘no challenge to hegemony’ (Hira & Hira 2000:269), which leaves no room for a transformative and decolonial approach to the establishment of new institutions. Old institutionalism may explain why and how certain characteristics of bureaucratic institutions have developed over time, but the theory is limited in that it ignores the dynamic (and unpredictable) interaction of variables such as identity, culture and the politics of interests working together in the decision-making process (Hira & Hira 2000). New institutionalism provides a more open paradigm, where institution-building is more dynamic and a combination of exogenous and endogenous factors. Therefore, institutions are not merely subject to what came before, to normative rules that dictate form and function, to the rational choices of individual political actors, to policy making and resource allocation or to carefully calculated self-interests (March & Olsen 2009).
The dominance of the modernisation paradigm and orthodox conceptualisations of growth-led development continue to influence the bureaucratic machinery of today. They influenced the formation of post-1960s independent African states (Badie 1992), coupled with the structural adjustment programmes of the 1980s with their stringent requirements for accountability through monitoring and reporting. The dominant market-oriented ideology led in the 1980s to New Public Management approaches driving public sector reforms, which has led to decades of institutional isomorphism, with bureaucracies copying a pattern acceptable to the global order in their building of governance mechanisms. Institutionalism suggests that bureaucracies maintain their perceived legitimacy in the global order by incorporating formal practices and procedures as a process of assimilation and compliance, a ‘public service bargain’ (Barzelay & Gallego 2006:534). It brings into question whether NESs in Africa are based on this assimilation of accepted, dominant and neo-liberal practices, and to what extent these need to be transformed in order to integrate a more locally adapted, indigenous and context-relevant governance architecture.
African bureaucracies were heavily influenced by colonial models of state formation which favoured colonial elites and tended to build institutions that reappropriated a form of the Weberian state, essentially creating a hybrid model (Bayart 1989, 1996). These systems often do not have sufficiently localised systems, practices and institutions designed to deliver to the bulk of the population. However, the modern African state operates on the basis of rules, norms and beliefs selected from the African and/or local culture by the elite (Bayart 1989, 1996), wherein a unique combination of characteristics set it apart from being either strictly Western or African in form. The result is a hybrid state that is top-down, hierarchical, parochial and exclusive, and perpetuates the colonial pattern of existing to serve the elites. Decoloniality means that the dysfunctional attributes of the hybrid institutional form must be dismantled. In terms of NESs, it means that evaluation systems should be aligned to local needs and contexts, rather than to donor-led agendas for accountability. They need to contribute to the transformational changes needed in society and the economy that humanity needs to survive and thrive. Morkel and Sibanda (2022) posit that the design and development of NESs should include the strengthening of participatory governance principles as part of the efforts at transformation: ‘every effort must be made to strengthen … democratic principles of inclusion in the very systems that produce evaluations …’ (Morkel & Sibanda 2022:5). For example, evaluation systems could factor in citizen and civil society participation in evaluation steering committees, and new ways of validating, utilising or overseeing the implementation of evaluation findings by communities and civil society.
In NESs, the policies, guidelines, institutional architecture and accountability structures may follow a set of international norms and rules that have been established by the application of donor-based needs for upward accountability. One example is the Organisation for Economic Cooperation and Development’s Development Assistance Committee (OECD DAC) evaluation criteria. Such systems may never be questioned by the institutions applying them, as they are believed to be an unchangeable international standard that must be upheld.
March and Olsen (2009:4) examine the reasons why contemporary bureaucratic institutions often function according to patterns and rules that may not serve their current needs and context. The reasons include the relatively enduring and unchanging nature of rules and ways of organising practices in institutions, regardless of individual preferences or circumstances; the ‘behavioural codes’ according to which appropriate behaviours are assigned; and the enrolling of third parties in a process of enforcing and sanctioning rules and compliance (or non-compliance) (March & Olsen 2009:4). However, in African states (in the interstices between colonial and post-colonial state formation) what we often see are parallel rules of the game operating to the formal bureaucratic system, for example, deployment of political cadres instead of supposedly merit-based recruitment systems or subverting formal procurement process.
Some of the principles of institutionalism, broadly, can be compared to the ‘patterns of organization of machine-based and ecology-based systems’, which are offered as ways of framing a decolonised approach to evaluation (Parsons & Winters 2023:30). Contemporary neo-colonial, functionalist bureaucratic systems have characteristics of machine-based systems, that is hierarchical structures with top-down control, layers of nested hierarchical units, relatively closed boundaries, grounding in rules focussed on homogeneity, resilience through stability, predictability, linearity and control (adapted from Parsons & Winters 2023:30). Ecological systems present a transformative perspective and offer the following: in terms of structure, the characteristics are distributed controls; a nested web-like network; flexible, relatively open boundaries; the emergence of complex features and self-organising with context. In the case of processes, systems are grounded in principles of ‘symbiosis and/or partnering; resilience through flexibility and diversity; cyclicity; interdependence; and expectation of unpredictability’ (Parsons & Winters 2023:30).
These elements are on a spectrum rather than a simple binary, and both elements are needed, in balance, as government systems do require a degree of predictability – for example, users expect similar minimum standards of service. However, this balance is being tipped by the drastic changes underway to climate and ecosystems, which require more flexible and adaptive responses. Parsons and Winters (2023) refer to the yin:yang balance between these systems.
In designing new, decolonised NESs, it is useful to note Parsons and Winters’ (2023) further work contrasting key inquiry design features in machine-based and ecology-based systems. According to the authors, machine-based systems have well-defined boundaries, are hierarchical, and are controlled and stable, certain aspects of which bureaucracies need (such as predictability). Machine-based systems have competing interests, benefits and values (which have to be negotiated), and the dynamics of the system are linear. Ecology-based systems are seen as ‘open to the flow of energy and matter; networked; and evolving, emergent, self-organizing’ (boundaries and interrelationships), and so helpful in times of complex change (Parsons & Winters 2023:38). Ecology-based systems are also generative and cooperative. Lastly, the dynamics of static, untransformed systems is that they are linear and ‘servomechanistic’ (Parsons & Winters 2023:38), while ecology-based systems are generative, cooperative, connected, evolving, self-organising and non-linear. A decolonised approach to NESs would need to lead towards intentionally creating the conditions, processes and buy-in for strengthening the ecology-based aspects of institution-building. Table 1 looks at the case studies using this approach.
| TABLE 1: How the national evaluation systems display social-ecological characteristics. |
Hassnain (2023:150) provides a useful framing of decolonised evaluation, which contributes to the argument for social-ecological principles to prevail in NESs. The author proposes that decolonisation involves moving away from characteristics such as either/or thinking, fear of open conflict, perfectionism and ‘worship of the written word’ (Hassnain 2023), all of which have commonalities with the outdated, Weberian-esque characteristics of ‘effective’ bureaucracies. In contrast, decolonised characteristics include (among others) ‘both/and’ thinking, viewing conflict as healthy, valuing high quality as opposed to perfectionism, seeing progress as more than economic growth, but manifesting in well-being, increasing humanity’s well-being and being comfortable engaging in discomfort (Hassnain 2023:150). Table 2 applies this model to the cases. Applying both/and thinking, certain aspects of colonised systems may be useful (e.g. predictable bureaucracies), while others are very dangerous (such as exploitative approaches).
| TABLE 2: Application of decolonisation characteristics. |
So as government systems, there are essential elements of machine-based systems needed for an institution to be predictable with some degree of standardisation of processes based on policies, guidelines and standards. However, the potential to evolve in response to emerging challenges and opportunities is critical, a feature of an ecology-based system.
The next two sections consider two NESs on the African continent, in South Africa and Benin. They analyse how they fare in terms of their allegiance to, and alignment with, the ‘baggage’ of colonial systems of organising bureaucracies, how far they are decolonised and to what extent they show elements of ecological systems.
The South African national evaluation system
The establishment of the South African National Evaluation System
The development of an NES in South Africa was initiated in 2011 by a newly established Department of Performance (later Planning), Monitoring and Evaluation (DPME). From its establishment, DPME sought not only to learn from others but also to adapt the models they saw to the local context (Phillips et al. 2014). Work began on a National Evaluation Policy Framework (NEPF) in 2011 by reflecting on South Africa’s prior experience in implementing evaluations, followed by a study tour to middle income countries that were pioneering NESs, notably Mexico and Colombia. After the visit, the team involved co-wrote the NEPF in a writeshop on their return, consulted with wider stakeholders, notably the South African M&E Association (SAMEA), and it was approved by Cabinet.
In parallel, work started on piloting an evaluation. As elements were piloted, guidelines were written. Goldman et al. (2015) describe the learning-by-doing approach, eventually leading to an evaluation system supported by standards, guidelines, training, quality assessment system, etc., led by an evaluation unit of 15 staff, which was well established by 2014. At the regional and continental scale, from 2012 the country developed a deep collaboration with Benin and Uganda, formalised in 2016 as Twende Mbele,7 and later extended to Niger, Kenya and Ghana (Goldman et al. 2018).
The system was based on certain principles or approaches:
- A utilisation and learning (rather than accountability) focus, so systems were set up to maximise the likelihood of ownership and learning by the government departments involved, and the likelihood that they would implement the findings.
- Definition of types of evaluation, to develop a common language, but agnostic about methodologies (qualitative, quantitative, etc.).
- Stakeholders needed to own the evaluation. Stakeholders were not only defined initially as the national departments involved but also included any identified key beneficiary groups (e.g. small farmer associations in an agricultural evaluation), and often provinces were included as stakeholders.
- There needed to be collaboration across the state to make the system work, that is, it was not just DPME’s work, but the work of the whole government.
- Evaluations had to have credibility and integrity, and DPME’s role in national evaluations was to provide quality control and ensure legitimacy, and to broker relationships, where in some cases attitudes differed strongly.8
While some attempts were made to involve beneficiary groups in evaluations (such as small farmer associations, representatives of small and medium-sized enterprises, and community health worker associations), the focus was primarily on government, on trying to ensure the quality of the evaluations and to get government ownership and commitment to implement the recommendations that emerged.9
Factors affecting possible evolution to a more ecological approach
After a period of growth and then stability from 2011 to 2017, the NES has been vulnerable to leadership transitions in DPME as well as the impact of fiscal austerity in government. This has led to cuts in key staff in DPME and a dramatic cut in the budget for evaluations.10
A fundamental issue for the system to evolve in a more decolonised and social-ecological way is changing the power dynamics so that public services and the evaluation system are responsive to the views and needs of users of government services and communities affected by interventions, rather than being dominated by senior managers in government. The evaluation of the NES conducted in 2016/2017 reported that the roles and relationships between stakeholders were unclear (Goldman et al. 2019), and that:
[T]he DPME needs to broaden and deepen its view of the operations of the governance system, how policy is made and how bias and interests impact on the use of evidence … how the evidence being generated by the NES is used in the policymaking and public domains [and] that the role of civil society and of think tanks in particular be clarified. (p. 10)
Another factor is the ability of the system to evolve in a relational way from exploitative approaches to promoting a more equitable approach that promotes ecological health. South Africa has promoted a growth-led model based on jobs in the formal sector but unemployment levels are extremely high and South Africa is the most unequal country in the world with a Gini coefficient of 0.67.11 There is a long way to go in developing an inclusive, more equal society with widespread levels of well-being.
The key for the government to be able to operate in a more learning and participatory approach implied by a more ecological approach are the leadership dynamics and how far these promote an authoritarian or a facilitative co-creation culture and evolution – both within the lead champion, DPME, and in how it operates with stakeholders.
Benin’s evaluation system
The establishment of the national evaluation system
Under pressure from international institutions, ‘good governance’ was a dominant paradigm for public management in the early 2000s in the West African Economic and Monetary Union area, including Benin. The development of an NES in the country was driven by the belief that evaluation linked to planning and programming could help improve performance at all levels of governance. The new President in 2007 was a technocrat and aware of the potential of evaluation for improving accountability and performance. A decision was taken to create a Ministry of State in charge of the Development and Evaluation of Public Policy.
To design the NES, the country undertook a national evaluation capacity assessment in 2010 which recommended options for enhancing evaluation at the national scale (Davies & Houinsa 2010). A National Evaluation Policy (NEP 2012–2021) was developed in 2011, to ensure country ownership over evaluation and change the situation where evaluations were predominantly undertaken by donors and external consultants (Davies & Houinsa 2010; Republique du Bénin 2012). The policy sets up six principles to guide the evaluation process, including the principle of ‘plurality’, which recommends that evaluation should consider all legitimate views expressed on the intervention being evaluated. A Bureau for the Evaluation of Public Policies (BEPP) was set up as the main body for the implementation of the NEP to develop the instruments for its implementation. The country also developed a National Evaluation Guide. The Guidelines recommend involving marginalised populations in undertaking evaluations and providing templates and guidance (Secretariat General de la Présidence 2017). The process seems to combine the ecological system-based principle of openness with a framework of stable rules and regulations that reflect a machine-based system.
The mandate of the office in charge of the coordination of evaluation (BEPP, later BEPPAG) was largely inspired by the approaches of the French Evaluation Society and the Canadian Evaluation Society (CES) (Secretariat General de la Présidence 2017:54). For the quality assessment of evaluations, BEPP adopted the quality criteria set out in international standards and endorsed by the African Evaluation Association (AfrEA). The evaluation criteria themselves are derived from the OECD-DAC criteria to which the country added some complementary criteria such as utility, governance, information system and ownership. Through the ‘ownership’ criteria, the country aims to anchor evaluations in local reality by defining their purpose and scope according to national priorities and by using national experts to plan and undertake evaluations.
As part of the country’s endeavour to build a locally grounded evaluation system that is also well aligned with international best practices, ‘benchmarking’ and ‘networking’ are set as core principles, with ideas of collaboration and integrity incorporated in the design. Collaboration is promoted at national, regional, continental and global scales. At the national level, the system aims to include all national stakeholders. An Institutional Framework for the Evaluation of Public Policy (CIEPP)12 clarifies relationships between the various stakeholders. A National Evaluation Council (CNE) was launched as the steering and advisory body, with representatives of civil organisations and decentralised state’s entities. At the regional and continental scale, from 2012 the country developed a deep collaboration with South Africa and Uganda, through the Twende Mbele programme, as already mentioned. To foster collaboration within the WAEMU member states, it also launched the West African Capacity Building for Impact Evaluation programme (WACIE), partnering with the International Initiative for Impact Evaluation (BEPP, n.d.). Hence, collaboration at regional and continental levels has been through South–South cooperation mechanisms, while at a global scale, it is through North–South Cooperation, for example, with the French Evaluation Society and with CES.
Factors affecting possible evolution to a more ecological approach
The NES is threatened by the bureaucratic hierarchy and political dynamics, dependent upon politicians’ decisions as there is no law embedding evaluation. Since its inception, the BEPP has been a department in the Prime Minister’s Office, then in the Presidency and now a General Directorate within a Ministry. These differing positions have affected its bargaining power and freedom of manoeuvre in enforcing the evaluation system as well as bringing instability to the system.
Being a unit within the government bureaucracy, the NES serves the dominant growth-led development paradigm from which the national development plan (Vision Bénin-2025 Alafia) was designed. Evaluation thinking as well as evaluators’ and policy planners’ mindsets are too fascinated by ‘growth’ to bring about the profound changes that are needed. For example, although improving accountability towards citizens is one of the purposes of the NES, the evaluation of the NEP reports no impact of evaluation on the quality of accountability. Reporting to the public is predominantly on implementation processes, outputs and expenditure, with little light shed on significant changes needed or underway (Secretariat Général de la Présidence 2018; Secretariat Général de la Présidence 2019:50). The system is still more procedural than transformative.
The evaluation of the NEP also reported a weak collaboration between actors, and that the CNE has failed to engage stakeholders. A gender perspective was absent from the initial NEP, and collaboration and dialogue remain a challenge for the NES (Secretariat Général de la Présidence 2019). It is essentially a central government function, with only 2% of the evaluations commissioned by local government, versus 80% by the central government, in the period from 2007 to 2017 (Hounliho 2021:17).
Similarly, the evaluative practice promoted by the current NES is largely dominated by Western epistemology. Evaluations barely draw on indigenous practices and knowledge, especially relevant in the field of environment and climate change. The potential contribution of such knowledge for improving public policies could be significant, as the country is known for its long tradition of Voodoo spirituality and culture, where human beings seek harmony and connection with nature.13 Unfortunately, such approaches tapping into indigenous ontology are still very scarce in the evaluation field. Indigenous knowledge is still largely seen by the bureaucracy as archaic and non-scientific, and an opportunity lost to tap into other forms of epistemology and ontology than the Western one inherited from colonisation. The Journal of MultiDisciplinary Evaluation (JMDE) special edition provides several examples of how this can be carried out (e.g. Ahrens et al. 2023).
Findings and lessons
How far do the national evaluation systems see elements of ecological systems emerging
Table 1 summarises how far the two systems show ecological characteristics versus machine-based system characteristics, bearing in mind a necessary balance between these.
Both evaluation systems are focussed on government stakeholders, emphasising learning rather than holding government departments accountable for performance, the latter a punitive approach that induces gaming behaviour and resistance to evaluation. Both countries have shown the ability to evolve and learn from others, applying learning principles rather than isomorphic mimicry. The Twende Mbele partnership demonstrates a co-creation approach that has helped the partners take forward new areas, such as rapid evaluation.
South Africa shows a more distributed structure, with a range of champions, which has helped the system survive beyond a low point between 2018 and 2020. Both have developed a coalition to support the system, although in Benin this is not operating at present (CNE).
While we see elements of a more ecological approach, to date these have not fundamentally questioned the growth-led paradigm, which a system more responsive to non-government stakeholders might show. In South Africa’s case since late 2021, there is evidence of a move to taking on systemic challenges of inequity and addressing climate and ecosystems health, through a co-creation evaluation hackathon process with SAMEA that developed new guidelines to be applied in all evaluations. Benin also is increasing its emphasis on social justice and climate.
How far have the national evaluation systems designs been ‘decolonised’?
Table 2 applies the characterisation of decolonisation developed by Benoit and adapted by Hassnain (2023). As with the social-ecological systems framework, there are elements of Western systems which are relevant, and many elements are on a spectrum rather than a simple binary, decolonised or not. We see elements of decolonisation in a comfort in dealing with conflict, an emphasis on qualitative rather than quantitative approaches, a focus on learning, increasing efforts to address social justice, equity and not just growth, and inclusion of all stakeholders. However, the inclusion of community and end users in the system is weak,14 in decision-making such as support for the NES, or in steering committees. Communication of results to end users is also negligible, let alone in accessible ways, such as using local languages, or mechanisms that are culturally adapted (storytelling, games, theatre, proverbs, etc.). This is not unique to evaluation but reflects wider top-down approaches in the public sector. A question arises whether and how NESs could incorporate much more community and/or user engagement, and whether this could be influential more widely in public management, for example, promoting user and community involvement in monitoring, or design and implementation. Much more work is needed on these aspects and there are a number of good examples in the JMDE special edition, such as in Chilisa and Bowman (2023).
How can new institutionalism help us moving forward?
Understanding how institutionalism can frame the way NESs have been formed in the past can help identify levers to: (1) change how we think about systems (from predominantly machine-based to more ecological models) and (2) provide new ways of thinking about building strong and effective state-led evaluation systems. Different definitions of institutionalism and its variants are useful for understanding as well as improving political and/or bureaucratic systems. From one viewpoint, the state can be seen as static and following the patterns and rules of the old order. Alternatively, state institutions can be seen from a ‘cultural community perspective which sees political life as organised by shared values and world-views in a community of common culture, experience, and vision’ (March & Olsen 2009:5). However, in these cases, there is a constant tension between more ‘machine-based systems’ and more ‘ecological models’ (Parsons & Winters 2023), and these modes of thinking and organising in these systems are always in tension. Attempts to decolonise political and bureaucratic institutions to establish more effective evaluation functions will have to include the art of balancing the two in a way that matches the local context but responds to the need for transformative change.
McConnell (2008:330) provides a reminder that room must be found for the ‘long-term fixity’ of states, their institutions and procedures, as well as the ‘fluidity of the culture in which they marinate’, and of course how these adapt to change. Peters (2019:31) further helpfully reminds us that individuals are not ‘automata, responding only to their socialization that the inhabitants of behavioural theories of politics have sometimes appeared to be. Rather, individual actors in normative institutionalism must pick and choose among influences and interpret the meaning of their institutional commitments’.
A key practical way in which NESs can balance the two is by embarking on a mass effort to onboard citizens and non-state actors in the workings of the NES. This would be ideal from the initiation of the system, but where the NES is not new (such as in South Africa), this could include an evaluation of the system from the perspective of citizens and non-state actors. This would create a fundamental shift in NES being viewed as a distant, elite part of the bureaucratic machinery, to a commonly owned, shared system where room can be created for a balanced emphasis on evaluation for learning, as well as accountability (and not the domination of the latter). This could open up spaces for the involvement of citizens and non-state actors in evaluation design, decision-making and the expansion of the communication of evaluation findings to the public (and in ways that allow for public engagement and influence on recommendations). This could have knock-on effects on encouraging greater citizen involvement in development planning and the design of service delivery interventions, helping to address the desire for a more people-centred approach to development shared by most democratic governments.
How could a more regenerative system be developed?
In both countries, case studies show how the evaluation systems emerged through learning by doing processes. A current evolution is in more explicitly focussing on promoting a just transition. South Africa has produced two evaluation criteria and guidelines on transformative equity and climate and ecosystem health to ensure that all evaluations mainstream thinking about a just transition to greater equity and climate and ecosystem health. This has been helped by a close relationship between DPME and SAMEA, the voluntary organisation for professional evaluation (VOPE), which also stresses a co-creation approach. Major collaborative efforts between SAMEA and DPME need to continue to make sure these changes actually happen in evaluation practice and have a transformative influence on their interventions being evaluated. The South African M&E Association is continuing with this as a core theme of work, in collaboration with DPME, and through support from the Just Energy Transition Project Management Unit in the Presidency (Presidency 2023). Benin has also incorporated issues of equity and climate in their Guide.
In both countries, the evaluation system is currently concentrated in the hands of the executive, which makes it vulnerable to leadership transitions, as happened in South Africa 2017–2020. There is a need to revitalise the institutional framework by increasing participation: widening participation across government; widening participation of VOPEs, think tanks and other civil society organisations; strengthening links to Parliament and strengthening engagement with the media, to increase the permeability to external pressures to make the system more relevant.
Another area is to move away from a focus on individual programmes or policies, to the deeper systems that underlie them (Parsons & Winters 2023). South Africa, for example, has started doing syntheses of individual evaluations to see the overall operation of a sector, such as support for small-scale farming. In the Just Energy Transition Implementation Programme, there is an overall theory of change (TOC) for this hugely ambitious programme and then for the portfolios within it. Evaluations are being planned at both levels.
What is key is how to change the power dynamics so that evaluations embrace the views of those affected by interventions, and they have a role in ensuring that evaluations address their concerns and needs, and so influence how interventions are strengthened/redesigned. Possible ways to involve stakeholders and community members in much stronger ways include:
- Moving accountability to the centre to accountability to users and those affected, ensuring users and those affected are involved from the onset in conceptualising and overseeing an evaluation through representatives of community groups affected and users having a strong role in evaluation steering committees. Hassnain (2023) stresses building in time in the budget to pivot evaluations based on community feedback and to close the evaluation learning loop with participants. Power differentials need to be addressed so users also have a strong voice in intervention planning and evaluation.
- Involving those affected by the intervention in data analysis, interpretation and reporting. Local participation can be increased by working together with indigenous evaluators, community members and religious leaders in gathering, understanding and reporting the data, and in identifying outcomes as perceived by them, as well as intervention managers. This can help in using relational approaches to understand the deeper systems in play (Mokgolodi 2023).
- Making more effort to communicate in an inclusive and participatory way to share evaluation knowledge and the outcomes of an evaluation, including recommendations. This would make results more useful to local people and hence used by those to whom ‘change’ matters the most (Hassnain 2023). Because of budget constraints, not all evaluation reports can be translated into all languages, but simple communicative materials could be produced such as infographics in local languages. Evaluation findings can be communicated in more inclusive ways such as orally through video, podcasts, graphical mediums, presentations, performances and radio. This requires a much greater investment in communication than at present and a greater willingness to open evaluations to a wider audience.
- Verifying the findings with respondents in small gatherings. This is currently not happening except for validation workshops, which often include steering committee members and other stakeholders, but not those that data were collected from.
Ahrens et al. (2023) provide a very nice example of applying these in practice in the USA with Hispanic people.
Conclusions
The need for more ecological system approaches
What is clear from the analysis is that state-led national evaluation systems currently operate within a hybridity of colonial and post-colonial systems, with structures, norms and standards that lean towards machine-based modalities, while being affected by group norms that may subvert routine functioning. This is understandable, owing to the significant influence of colonial systems on the countries in these case studies, and most others across the African continent, as well as the weakness of many government institutions. Some degree of routine and predictability is needed, but in a way that responds to the needs of users rather than bureaucrats. However, we need to allow NESs to break from historical forms of bureaucratic functioning and take on systems-based approaches as the basis for new thinking around NESs, strengthening their ecological aspects.
Embracing learning approaches
Both systems have been designed to fit local contexts, building on endogenous and indigenous needs and contexts, and using peer learning to support agility, learning and movement, all of which lean towards ecological models of systems-building. The South African state’s openness to working with non-state actors (such as SAMEA) in co-creating significant elements of the NES is further testament to this. Additionally, in both cases, there is an awareness of, and deliberate action, to be transformative, by embracing more diversified approaches to evaluation and research paradigms, for example, not exhorting single approaches or methods such as randomised control trials.
Embracing principles of participatory democracy and co-production by strengthening the voice of non-state actors, particularly citizens, in the formation of national evaluation systems
Some unique features of the South African system (such as its deliberate coalition building and distributed structure) have led to its resilience even through challenging periods. Benin developed a coalition system, which provided a similar buffer effect to its national evaluation system. A key area of improvement is to ensure a more participatory approach to evaluation system strengthening, with examples given in the sections on the South African and Benin cases. The weak inclusion of citizens in government evaluations is an area that creates a leaning towards machine-based systems, where citizens are not considered in bureaucratic affairs, and to predefined outcomes set by outsiders, rather than those perceived and desired by those affected by interventions. The balance between rule-focussed, machine-based systems and more open, ecology-based systems is one of the most important concerns for national evaluation systems to respond to the ever more uncertain worlds of today and tomorrow.
Changing power dynamics
Key to decolonisation is changing the power dynamics between centres of power and citizens. This includes doing what is necessary to challenge the existing power dynamics inherent in the evaluator (and/or donor)/evaluand relationship so that evaluations are context-specific, sensitive to felt needs and help to shift power from above, to power from below, and contributing to a change of leadership culture which is less autocratic, more inclusive and empowering, and more learning focussed.
We argue that the confluence of these actions will transform NESs in such a way as to circumvent the limitations of the routinisation of evaluation production and move systems to include wider voices, evolve more effectively and address the systemic challenges facing these countries.
Further action research and action learning are needed in the area of institution building and institutional change, to understand how NESs are established, how to produce patterns of behaviour that are decolonised, with more ecological systems of organising and how to bring in these more participatory models in practice. The focus must be on determining pathways that will bring about transformation in the systems and practices that will renew, regenerate and shift people and the planet away from the brink of collapse to a more resilient future.
Acknowledgements
The authors would like to thank Bagele Chilisa and Nicky Bowman for their amazing work getting the special edition of the Journal of Multidisciplinary Evaluation on Decolonising Evaluation published on which this article has built and Judy Scott-Goldman, Nicky Bowman and Beverly Parsons for their editing and comments on the article. They would also like to acknowledge the participants in the webinar who brought their insight to bear on the topic.
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Authors’ contributions
I.G. is the President of the International Evaluation Academy (IEAc), led the South African NES from inception to mid-2018 and contributed to the development of this article and overall editing. C.M. is the Director of CLEAR Anglophone Africa and developed the analytical framework and conclusions. E.D.A. is the Director of CLEAR Francophone Africa and wrote the Benin case study. T.G.M. led the South African evaluation system and wrote the South African case study. All contributed to the lessons learned.
Ethical considerations
This article followed all ethical standards for research without direct contact with human or animal subjects.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
No data were used for this article – It builds on the experience of the authors as well as existing literature.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Afoutou, E., 2023, ‘La place des plantes dans l’alliance cosmique et le medium de l’argent dans la théerapeutique ajatado’, in C. Byl, A. Cappe, R. Laugrand, N. Loodts & L. Simon (eds.), Anthropologie du végétal: l’ontologie flottante du végétal et la plante ambassadrice, pp. 23–42, Academia, Louvain-La-Neuve.
Ahrens, V., Cruz, E., Pasturczak, M., Bakken, L.L. & Moore, T.R., 2023, ‘Between funding requirements and community priorities: Centro Hispano of Dane County’s transformative approach to program evaluation’, Journal of MultiDisciplinary Evaluation 19(44), 2023. https://doi.org/10.56645/jmde.v19i44.787
Amisi, M., Buthelezi, T., Magangoe, S., Taylor, S., 2020, ‘Use of evidence in a complex social programme: Case of an evaluation of the state’s response to violence against women and children in South Africa’, in I. Goldman & Pabari, M. (eds.), Using evidence in policy and practice: Lessons from Africa, Routledge, Abingdon.
Badie, B., 1992, L’Etat importé. L’occidentalisation de l’ordre politique, Fayard, Paris.
Barzelay, M. & Gallego, R., 2006, ‘From “new institutionalism” to “institutional processualism”: Advancing knowledge about public management policy change’, Governance 19(4), 531–557. https://doi.org/10.1111/j.1468-0491.2006.00339.x
Bayart, J.-F., 1989, L’Etat en Afrique. La politique du ventre, Fayard, Paris.
Bayart, J.-F., 1996, La greffe de l’Etat, Karthala, Paris.
Bendell, J., 2023, Breaking together, Schumacher Institute, Bristol.
BEPP, n.d., Programme de renforcement des capacités et d’évaluation d’impact en Afrique de l’Ouest, viewed 20 March 2024, from https://evaluation.gouv.bj/articles/24.
Chilisa, B. & Bowman, N.R., 2023, ‘The why and how of the decolonization discourse’, Journal of MultiDisciplinary Evaluation 19(44), 2–10. https://doi.org/10.56645/jmde.v19i44.919
Davies, I.C. & Houinsa, D.G., 2010, Etudes diagnostique des capacités nationales d’évaluation au Bénin, Ministère de la Prospective, du Développement, de l’Evaluation des politiques publiques et de la Coordination de l’action gouvernementale & PNUD, Cotonou.
Goldman, I., Byamugisha, A., Gounou, A., Smith, L.R., Ntakumba, S., Lubanga, T. et al., 2018, ‘The emergence of government evaluation systems in Africa: The case of Benin, Uganda and South Africa’, African Evaluation Journal 6(1), a253. https://doi.org/10.4102/aej.v6i1.253
Goldman, I., De la Garza Navarrete, T., Kalugampitiya, A., De Erice Dominguez, A.M., Agbojan, E.D., Chirau, T. et al., 2023, ‘The emergence of evaluation systems in low- and middle- income countries’, in F. Varone, S. Jacob & P. Bundi (eds.), Handbook of public policy evaluation, Edward Elgar Publishing, viewed 19 October 2023, from https://www.e-elgar.com/shop/usd/handbook-of-public-policy-evaluation-9781800884885.html.
Goldman, I., Deliwe, C.N., Taylor, S., Ishmail, Z., Smith, L., Masangu, T. et al., 2019, ‘Evaluation2 – Evaluating the national evaluation system in South Africa: What has been achieved in the first 5 years?’, African Evaluation Journal 7(1), a400. https://doi.org/10.4102/aej.v7i1.400
Goldman, I., Mathe, J.E., Jacob, C., Hercules, A., Amisi, M., Buthelezi, T. et al., 2015, ‘Developing South Africa’s national evaluation policy and system: First lessons learned’, African Evaluation Journal 3(1), 107. https://doi.org/10.4102/aej.v3i1.107
Hassnain, H., 2023, ‘Decolonizing evaluation: Truth, power, and the global evaluation knowledge base’, Journal of MultiDisciplinary Evaluation 19(44), 142–155. https://doi.org/10.56645/jmde.v19i44.803
Hira, A. & Hira, R., 2000, ‘The new institutionalism: Contradictory notions of change’, American Journal of Economics and Sociology 59(2), 267–282. https://doi.org/10.1111/1536-7150.00026
Hounliho, B., 2021, Meta-evaluation des evaluations de politiques publiques conduites au Benin de 2007 à 2019, Ministère du developpement et de la coordination de l’action gouvernementale/CLEAR FA, Cotonou.
March, J.G. & Olsen, J.P., 2009, ‘Elaborating the “new institutionalism”’, in S.A. Binder, R.A.W. Rhodes & B.A. Rockman (eds.), The Oxford handbook of political institutions, pp. 3–20, Oxford University Press, Oxford.
McConnell, S., 2008, ‘The old institutionalism and the new’, Polity 40(3), 326–331. https://doi.org/10.1057/pol.2008.5
Mokgolodi, H., 2023, ‘Decolonizing evaluation of Indigenous guidance and counseling approaches: A review of selected evaluated programs’, Journal of MultiDisciplinary Evaluation 19(44), 231–246. https://doi.org/10.56645/jmde.v19i44.779
Morkel, C. & Sibanda, A., 2022, ‘Country-led monitoring and evaluation systems through the lens of participatory governance and co-production: Implications for a Made in Africa evaluation approach’, in African Evaluation Journal 10(1), a622. https://doi.org/10.4102/aej.v10i1.622
Parsons, B.A. & Winters, K., 2023, Paradigm-based evaluation for eco-just systems transformation’, Journal of MultiDisciplinary Evaluation 19(44), 24–44. https://doi.org/10.56645/jmde.v19i44.799
Peters, B.G., 2019, Institutional theory in political science: The new institutionalism, Edward Elgar Publishing, Cheltenham and Camberly, UK.
Phillips, S., Goldman, I., Gasa, N., Akhalwaya, I. & Leon, B., 2014, ‘A focus on M&E of results: An example from the Presidency, South Africa’, Journal of Development Effectiveness 6(4), 392–406. https://doi.org/10.1080/19439342.2014.966453
Picciotto, R., 2023, ‘Evaluation transformation implies its decolonization’, Journal of MultiDisciplinary Evaluation 19(44), 131–141. https://doi.org/10.56645/jmde.v19i44.797
Presidency, 2023, Just energy transition implementation plan, Government of South Africa, Pretoria.
Republique du Bénin, 2012, Politique nationale d’évaluation (2012–2021), Gouvernement de la Republique du Benin, Cotonou.
Rockström, J., Gupta, J.D, Qin, D., Lade, S.J., Abrams, J.F., Andersen, L.S. et al., 2023, ‘Safe and just Earth system boundaries’, Nature 619, 102–111. https://doi.org/10.1038/s41586-023-06083-8
Secretariat General de la Présidence, 2017, Guide méthodologique national d’evaluation, UNICEF.
Secretariat Général de la Présidence, 2018, Rapport de suivi de l’utilisation des résultats de l’évaluation, Gouvernement de la Republique du Benin, Cotonou.
Secretariat Général de la Présidence, 2019, Evaluation de la mise en œuvre de la politique nationale d’évaluation 2012–2023 et élaboration de son plan d’action 2020–2021, Gouvernement de la Republique du Benin, Cotonou.
Twende Mbele/CABRI, 2022, Using M&E to Link Budgets and Planning in National Governments, Discussion Paper, viewed 19 October 2023, from https://twendembele.org/reports/using-me-to-link-budgets-and-planning-in-national-governments/.
UNEG, 2022, United Nations contributions to national evaluation capacity development and the evolution of national evaluation systems: An overview of implementation of General Assembly Resolution 9/237, UNEG, viewed 19 October 2023, from http://www.unevaluation.org/document/detail/3053.
Footnotes
1. See https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwi1l77nguSBAxXMNuwKHcr2AzEQFnoECA8QAQ&url=https%3A%2F%2Fideas-global.org%2Fwp-content%2Fuploads%2F2019%2F10%2FPrague-Declaration-4-October-2019.pdf&usg=AOvVaw2bFCs607KJAlvQr2f3wBYi&opi=89978449.
2. See https://zendaofir.com/made-africa-evaluation-part-1/.
3. See https://journals.sfu.ca/jmde/index.php/jmde_1/issue/view/77.
4. This is an area of work that can be explored going forward.
5. See https://climateactiontracker.org/global/temperatures/.
6. The main countries in the global South with national evaluation systems actually functioning are Colombia, Mexico, Chile, Costa Rica, South Africa, Benin, Uganda and Philippines. There are some systems at subnational level, such as some states or provinces in Mexico, Brazil, South Africa, India or some ministries.
7. Twende Mbele is a partnership of African governments seeking to use monitoring and evaluation as tool for enhancing development impact. See www.twendembele.org.
8. For example, in an evaluation of gender-based violence, with strong differences in paradigms between the police and Department of Social Development (Amisi et al. 2020).
9. Based on the experience of the lead author, who led the South African national evaluation system at that time.
10. The current budget for evaluations is R2 million, compared to R11 million at its peak. DPME, personal communication.
11. See https://theconversation.com/south-africa-cant-crack-the-inequality-curse-why-and-what-can-be-done-213132.
12. Cadre Institutionnel de l’Evaluation des Politiques Publiques, CIEPP.
13. For instance, in a recent article, anthropologist Afoutou (2023) explored the cultural universe of the Ajatado (a cultural group who live across the Benin and Togo border) investigating the relationships humans have with plants. It emphasises the ontological fluidity between plants and humans, categories entangled in a universe conceived as a cosmic harmony in which deities, ancestors and other elements of the cosmos are interrelated.
14. For instance, out of the 77 people who were interviewed during the evaluation of the National Evaluation Policy in Benin, only seven came from civil society.
|
Email this article 
Hide
Show all
