Abstract
Background: Made in Africa Evaluation (MAE) has gained traction in the last decade, mainly through the agenda of decolonising knowledge and promoting Africa’s epistemic identity through promoting African grounded epistemologies, African indigenous knowledge systems and African grounded evaluation methodologies. While emphasis has been given to theorising MAE and possible methodological implications, limited attention has been given to implications of MAE for development evaluation praxis.
Objectives: This article aimed to explore the praxis implications of MAE to development evaluation practice.
Method: An exploratory research design was adopted, guided by theoretical constructs from critical systems heuristics (CSH). The assessment is guided by existing evaluation frameworks, practice guidelines (including the African Evaluation Guidelines – Standards and Norms, and the African Evaluation Principles) and theoretical and methodological guidelines. Data were collected through secondary reviews, expert and experiential knowledge regarding development evaluation practice.
Results: The study findings show that the critical practice components for MAE include appreciating sources of motivation as guiding principles of developing modalities for evaluation practice; understanding and integrating sources of power and politics of value judgements in development evaluation practice; developing sources of knowledge; and appreciating sources of legitimation (defining the beneficiaries of MAE and implications for practice).
Conclusion: In practice, MAE evaluation should adopt methodological approaches that borrow from African-rooted paradigms, including relational approaches and tools grounded in African institutional frameworks, social systems and values. Made in Africa evaluation should mainstream an empowerment evaluation approach that aims at contributing towards positive social change and promoting epistemic freedom of African evaluators.
Keywords: Made in Africa Evaluation; evaluation practice; critical systems heuristics; African Evaluation Association; knowledge.
Background
Since the early 2000s, African evaluation experts, scholars and voluntary organisations for professional evaluation have worked towards developing evaluation approaches based on African indigenous philosophies (Uwizeyimana 2020). This type of approach to development evaluation has been dubbed ‘Africa-rooted evaluation’ or ‘Made in Africa Evaluation’ (MAE). The MAE agenda was made central to the aspirations of the African Evaluation Association (AfrEA) at its inception in 1999, culminating in MAE as a pillar of its 2018–2021 strategy (IDEV 2019). Since the early 2000s, the concept of MAE has gained traction, mainly through the agenda of decolonising knowledge and promoting Africa’s epistemic identity. Chilisa and Mertens (2021:245) define MAE as ‘an approach that embraces African resistance to blind borrowing of Western values and standards to evaluate programs in Africa’. In broad terms, the approach provides impetus for African evaluation experts and scholars to deliberately adapt evaluation tools, instruments, strategies and models towards developing evaluation practice, theory and methodologies grounded in local cultures, indigenous knowledge systems and African philosophies. However, although the approach is rooted in African worldviews, it embraces other knowledge systems (Chilisa 2020). Therefore, a fundamental issue is in the grounding of development evaluations within African epistemologies, African indigenous knowledge systems and African-rooted evaluation methodologies.
The MAE agenda is in line with the contemporary discourse on decolonising knowledge for epistemic freedom in Africa. In this context, Chilisa et al. (2016) assert that decolonisation of evaluation entails restructuring of power relations in the current global architecture of evaluation knowledge generation. This will allow African evaluators to play a meaningful role in the conceptualisation of what is evaluated, when it is evaluated, by whom and with what methodologies. In this regard, emphasis is placed on the need for appreciating context, culture, history and beliefs to shape the nature of evaluations, in recognition of diverse, dynamic and complex African socio-economic development realities. Such an approach will enhance the relevance and applicability of knowledge and evidence from evaluations. In fact, Mbava (2017:141) argues that current recommendations from most evaluations do not pay sufficient attention to variations in local contexts where programmes are implemented. Additionally, Cloete (2016:1) argues that currently, evaluation practice in Africa is based on external values and contexts. It is highly donor-driven, with accountability mechanisms skewed towards aid recipients rather than an accountability and learning mechanism for both aid recipients and funders.
There is consensus by proponents of MAE on the fact that current evaluation principles, assumptions and practices which are based on Eurocentric epistemologies are frequently found to be unsuitable in the African context (Chilisa & Malunga 2012; Uwizeyimana 2020). According to Uwizeyimana (2020), these current approaches focus more on efficiency, effectiveness and economic development, instead of addressing issues of communal well-being, empathy and indigenous cultural values and traditions – which are more relevant to the African context. In addition, it is believed by these proponents that African rooted evaluation will bring much deeper understanding on the macro–micro disconnects, attribution, ethics and values, and power relations. In this regard, an MAE approach will likely contribute towards achieving some of the sustainable development goals (SDGs) through their inherent capacity to raise issues about culture, ethics, values and paradigmatic stances in development work (Chilisa & Mertens 2021). In this context, Chilisa and Malunga (2012) suggest two critical transformations in the current evaluation design and practices that will facilitate the transition towards MAE. These are adaptation and domestication of the Western evaluation theory and practices to ensure relevance to the African context and development of relational evaluation paradigms based on the African indigenous philosophy of ubuntu.1 Such a transformation, through the MAE approach, will likely improve the relevance of evaluation to African contexts and enhance effective utilisation of evaluation results for social change.
Although efforts to make evaluation culturally relevant have become central to evaluation discourses globally, global attempts at culturally responsive practice have not succeeded in incorporating African voices (Chilisa et al. 2016). Evaluation scholars, academics and practitioners must overcome several hurdles if the MAE is to have meaningful success. According to Uwizeyimana (2020), some of the critical issues include the need by indigenous African evaluation and nonindigenous African scholars and experts to agree on the African values, practices and traditions on which the MAE approach should be based. Additionally, the indigenous African philosophy of ubuntu has its own downsides which have serious implications for MAE. It is therefore important for some of these shortcomings to be made explicit before making ubuntu as the foundation of MAE. Conceptually, key challenges include the proposal for adopting and indigenising the existing Eurocentric evaluation theories and practices without fully exploring and having consensus on the key tenets of MAE (which include mainstreaming African epistemologies in evaluations).
Much of the discourse on MAE to date has focused on the conceptual aspects (IDEV 2019). Several grey areas remain on how MAE can be put into practice. Omosa et al. (2021), for example, advocate for reviewing the AfrEA guidelines and considering evolving definitions of MAE. Although a review of literature points to publications that outline some components for rolling out MAE in practice (e.g. Chilisa, Major & Khudu-Petersen 2017; Cram, Tibbetts & LaFrance 2018), there is a need for a more systemic way of looking at the practice implications of MAE. This article aims at filling this research gap through exploring the praxis implications of MAE. The article addresses the following research question: what are the critical components for MAE development evaluation practice? An exploratory research design is adopted, guided by theoretical constructs from critical systems heuristics (CSH). The assessment is guided by existing evaluation frameworks, practice guidelines (including the African Evaluation Guidelines – Standards and Norms) and theoretical and methodological guidelines. Data collection was based on secondary literature reviews and expert and experiential knowledge regarding development evaluation practice. It is envisaged that the research findings will contribute towards strengthening the MAE agenda.
Conceptual framework
Development evaluations aim at providing results, evidence and knowledge that informs policies, programmes and projects aimed at inducing change in social systems. In this regard, evaluations are commissioned as interventions to transform society or social systems. This invariably calls for a conceptual framework that evaluates design and implementation or results of interventions for transforming social systems. This research therefore adopts a conceptual framework based on CSH, which provides a framework for a critical reflective process on the design of social interventions. Critical systems heuristics has foundations in the seminal work of West Churchman in the 1960s focusing on transformation of social systems through interventions (Midgley 2000). It offers an approach for socially rationalising plans, interventions and development practice (Luckett 2006) and interrogation of social systems design or boundaries and assessment of their social implications (Flood & Jackson 1991). Critical systems heuristics is a strand of critical systems thinking, within the field of applied systems thinking, concerned with the normative core of professional practice and how professionals can (and should) reflect on, critique and justify the values and assumptions that influence their work (Ulrich 2012). The central argument advanced here is that CSH can support reflections on the design of approaches to development evaluation practice through being critically reflective. Being critically reflective means identifying, questioning and justifying the values and assumptions that influence and should influence evaluation practice.
Critical systems heuristics is concerned with discursive acts, with decisions made by multiple parties with varying goals, epistemic frameworks and ways of describing the world and the systems being designed. Rather than seeking to classify the component elements of an assumed system or provide a model of their relations, CSH focuses on the reflexive consideration of a designed system’s purpose or goals, and how these are justified by a ‘reference system’ of assumptions and judgements (McCord & Becker 2019). Therefore, in this research, CSH is utilised to assess how evaluation practice within the context of MAE can be designed as a social intervention. Critical systems heuristics as a framework for reflective professional practice is organised around the central tool of boundary critique (Ulrich & Reynolds 2010). Boundaries are what we socially construct in designing and evaluating any human activity system of interest. Boundary judgements determine which empirical observations and value considerations count as relevant and which others are left out or are considered less important. Because they condition both ‘facts’ and ‘values’, boundary judgements play an essential role when it comes to assessing the meaning and merits of a claim.
The primary boundary of any human activity system is defined by the ‘purpose’ that it is expected to serve. Therefore, CSH is utilised to surface, elaborate and critically consider boundary judgements, that is, the ways in which people or groups decide what is relevant to the system of interest (any situation of concern). The CSH questions can be applied to any purposeful system of interest, that is, any area or situation of concern that might be associated with human purpose, whether it is of individual or collective concern, that is, evaluating the actual purposes and implications of purposeful activity with relevant stakeholder groups (Reynolds 2007).
In this research, the assessment of components that define the practice of MAE is premised on the assertion that the practice of MAE is designed based on defining system boundaries that determine what is regarded as pertinent for driving social transformation. The process of defining system boundaries is judgemental based on an interaction between knowledge, morals, values and beliefs (Romero 2002:45–46). In this regard, assessing the design or conceptualisation of MAE in practice therefore implies critiquing boundary decisions. The process encompasses an assessment of the socio-economic, sociocultural and ethical implications of boundaries as a way of rationalising MAE in practice.
Critical systems heuristics is viewed as ‘critical’ in the sense of unearthing values, world views, assumptions and beliefs for social transformation incorporated in defining boundaries. Ulrich (1983) brought clarity to this assertion through the following quotation:
In the context of applied social inquiry and planning, being critical therefore means to make transparent to oneself and to others the value assumptions underlying practical judgement, rather than concealing them behind a veil of objectivity. (p. 20)
The approach is ‘heuristic’ in its ability to discover or expose ‘objectivist’ deceptions through critical reflections. In professional practice, heuristic procedures serve to identify and explore relevant problem aspects, assumptions, questions or solution strategies. At best, CSH can seek to reveal and problematise the normative assumptions informing a plan, making clear the contingent nature of these boundaries and making them the subject of deliberation. In the context of the practice of MAE, this entails that the design should not be taken as the only objective designs, but rather it should be reflected on through dialogue to incorporate multistakeholder values and world views. It is ‘systemic’ in the sense of reflecting the lack of a priori comprehensiveness or holism in the design of the practice of the MAE approach in terms of the totality of relevant conditions (ethical, political, cultural, ideological, etc.), under which rational decision-making and action take place (Flood & Jackson 1991; Luckett 2006; Ulrich 1983:21). Critical systems heuristics unearth power dynamics and recognise that power can distort purposeful systems through unquestioned decisions and assumptions.
Ulrich (1983) developed a set of boundary questions to unearth and critique assumptions on system boundaries. These interrogate what the system ‘is’ (normative mode) and what the system ‘ought’ to be (objective mode), and the answers are contrasted. In this research, the normative mode for the MAE is assumed to be informed by the African evaluation principles (AEPs) forming the reference system. The principles make cognisance of African knowledge systems and values; they are very much in sync with the MAE agenda. They reflect the demand for evaluation that is ‘Made in Africa’ – proudly tailored by and for African contexts, needs and knowledge systems, yet informed by international good practice insights, theories and practices. The current evaluation practice is used as the ‘reference’ system, identifying first the ideal purpose of the system of interest being evaluated in the ‘ought’ mode. The AEPs (AfrEA 2021) provide five key principles that provide a framework for evaluation practice conducted in and for Africa. These are further disaggregated into 22 implementation principles outlined in Table 1.
| TABLE 1: Made in Africa Evaluation implementation principles. Summary of the African evaluation principles 2021. |
The 12 CSH boundary questions are premised on the assumption that designing of purposeful systems is based on four sources of influence, which are motivation for the design, control and power dynamics, expertise required for the system, and legitimating and collective mobilisation (see Figure 1).
In this research, rather than utilising these questions as templates, the questions were utilised in developing a set of questions deemed appropriate for the assessment of MAE in practice. This is in line with debates on application of CSH (e.g. Flood & Jackson 1991:301; Midgley 2000:225) that argue for flexibility and creativity in application of CSH. The key boundary questions applied in this research based on Ulrich’s four areas of influence are illustrated in Table 2.
| TABLE 2: Key research questions based on critical systems heuristics domains. |
Unfolding the key tenets for Made in Africa Evaluation praxis
Sources of motivation
According to the AfrEA African Evaluation Principles (AEPs) (AfrEA 2021), development evaluations should ideally serve the people of Africa (Afrocentric) with the aim of advancing the development and empowering African societies. Such empowerment should be driven through effective accountability mechanisms, new knowledge and insights, and ownership of both evaluation and the associated social change (AfrEA 2021). In practice, attaining these ‘minimum requirements’ entails utilisation of evaluation approaches rooted in African indigenous paradigms. This involves adoption of ontological, knowledge (epistemological) and value systems rooted in African cultures, histories and philosophies (Chilisa et al. 2017:327; Cram et al. 2018). In practice, evaluators need to adapt participatory and indigenous evaluation methodologies to enhance greater inclusion of communities in deciding on and benefitting from the evaluation outcomes. Such participatory processes can be implemented, for example, through existing community dialogue processes or forums, ‘before’ and ‘after’ maps and identified significant change stories where people or evaluands identify their own indicators of success.
According to Keane, Khupe and Seehawer (2017), knowledge generated in MAE approaches should relate back to the lives of those who contributed to the evaluation process. In addition, there is a need to incorporate existing knowledge, value systems and perspectives in the evaluation process and designs to allow MAE approaches to reflect the lived realities of all evaluation stakeholders and beneficiaries (Mbava 2019). In addition to appropriate knowledge generation, evaluation findings should be disseminated back to the people in culturally appropriate ways and in a language that can be understood (Chilisa & Malunga 2012). In practice, this should be implemented in practical ways that allow respectful and active engagement of communities to ensure that knowledge and processes arising from evaluation are shared and disseminated to benefit the beneficiaries of the interventions being evaluated. Evaluations should be contextualised and culturally appropriate in line with African worldviews, values and culture. In practice, for example, Chilisa et al. (2016) propose the use of African relational-based evaluation approaches. These are grounded on philosophies, worldviews and paradigms that perceive reality and knowledge through relational ways. Their basis is on African value systems of togetherness and relational epistemological assumption based on the assertion that no single person holds knowledge, but rather knowledge is co-created. Thus, development evaluation rooted in MAE should be framed around networks of existing community relationships and institutional arrangements (Chilisa & Mertens 2021). Emphasis should focus on how the values of belonging, togetherness, interdependence, relationships, collectiveness and love are utilised in development evaluations. At the community level, elements of community strength and community relationships should inform evaluation practice. In this regard, Chilisa et al. (2016) further propose several genres of evaluation approaches which are relevant for MAE practice, including ethno-philosophy and proverb-based evaluation approaches. The former focuses on unified knowledge from the collective and diverse worldviews of Africans. The later capitalises on language, proverbs, metaphors, folklores, stories, songs, artefacts and oral traditions as sources of knowledge. In addition, it utilises African oral literature, concepts and theories to inform development evaluation practice.
According to a study by Muwanga-Zake (2009), these relational approaches need to be rooted in the Afrocentric world view and ubuntu philosophy. Ubuntu also encompasses the concept of wellness and the concept of ‘I am because we are’ – which recognises that, for an indigenous African individual and his or her community, communalism is what characterises being African (Nzimakwe 2014:30). Muwanga-Zake (2009) further illustrates some potential participatory approaches, grounded in a relational paradigm that can be utilised for various phases of the evaluation process. For example, local community voices should play a more prominent role in the development of programme theory, instead of testing programme theories developed elsewhere. In this context, evaluators can utilise lived experiences extracted through proverbs, metaphors, folklores, stories, songs, artefacts and oral traditions to formulate programme theory. Dialogic approaches through collective deliberation and communal decision-making can also be utilised (Mbava 2019:19) to enhance active participation of key stakeholders in the construction of what is evaluated, when, by whom and how (Mbava & Chapman 2020). According to Chilisa (2015:18), there are existing community-level dialogic systems that can be utilised in most parts of sub-Saharan Africa. The lekgotla (South Africa) or dare (Zimbabwe) is a democratic public meeting where members of a specific group convene to deliberate issues. It forms a rich conduit for knowledge generation through in-depth interrogation of identified issues, critical thinking and consensus building.
According to Mertens (2018), MAE should practise methodological pluralism. It should utilise a transformative mixed methods approach based on dialogic methods that integrate local knowledge systems through community conversations. Such an approach is argued to embrace the complexity and cultural diversity of African societies. In addition, an empowerment dimension is important for increasing social justice, and researchers should design, plan and implement evaluations in collaboration with communities as active stakeholders. For example, planning an evaluation in accordance with guidelines on how communities normally interface with development partners or evaluators is important in establishing relationships, developing rapport and ensuring evaluations are culturally responsive or relevant. Transformative methodologies allow evaluators to firstly utilise qualitative data collection to fully understand the context and lived experiences of community members and establish trust and relationships, followed up with quantitative data to substantiate evaluation findings (Mertens 2012).
According to Bowman, Francis and Tyndall (2015) and Hood, Hopson and Kirkhard (2015), MAE in practice needs to be guided by culturally responsive and indigenous evaluations frameworks. This allows evaluations to be embedded in culturally defined ethics, norms, values, beliefs and traditions. Evaluators must make intentional and explicit focus on culture during design and implementation for ethical, high-quality and relevant MAEs. This allows responsiveness to African contexts, enhances validity and improves utility of evaluation findings. Further, evaluators must prioritise inclusiveness through participatory, community-driven and empowering methodologies (Clarke et al. 2021). Some of the culturally sensitive evaluation tools include oral storytelling and poetic inquiry (LaFrance, Nichols & Kirkhart 2012).
Sources of power and politics of value judgements in development evaluation practice
As outlined in the preceding sections, a fundamental shift in the promotion of MAE is the need to decolonise evaluation and shift from a Eurocentricity-driven evaluation enterprise towards Afrocentric practice. Development evaluation has inherent power dynamics through its role as a tool in the development of contemporary social policy. In addition, the conventional understanding of evaluation as a value-free activity and neutral technocratic tool brings arguments on the practicalities of value-free evaluation in practice, as it operates within social systems that have inherent social meanings and power relations (Taylor & Balloch 2005). In this regard, the practice of development evaluation has implicit issues of power regarding the legitimacy of knowledge, as evaluation has value judgements regarding setting standards around what should be considered as real programme outcomes, what knowledge measures that reality and what values support the evaluation. Chilisa et al. (2016), for example, describe the current trends in development evaluation as ‘the worst instrument of epistemological imperialism’. Evalutaion therefore speaks to power dynamics around who makes decisions and judgements about the realities and relevance of measures to improve interventions.
Currently, African evaluations2 are viewed as maginalised, with no power to determine approaches for theorising, designing and ultimately controlling evaluation findings and reports (Uwizeyimana 2020). In this regard, Afrocentric evaluation practice requires shifts in the decision-making environments and building capacity for African evaluators. The principle of ‘powerful for Africans’ in the draft AEPs further elaborates the need for shift in epistemic power dynamics. The principle further outlines other power-related issues to include the need for appropriate, empowering processes, encouraging reciprocity, allowing mutual accountability, enabling learning for useful insights and valuing and strengthening local capacities.
Despite development evaluation being generally viewed as a scientific knowledge generation-based inquiry, it is an inherently political activity (Azzam 2010; Brandon & Fukunaga 2014; Eckhard & Jankauskas 2019). It is therefore critical to understand the roles of various stakeholders in influencing evaluation processes and how such dynamics affect the validity of evaluation results. Additionally, evaluation practice should integrate mechanisms that minimise impacts of stakeholder interests on evaluation design and implementation, as well as the positionality of evaluators to enhance integrity of evaluation results. For example, Morris and Clark (2013:66) point to situations where evaluators are pressurised to misrepresent findings as common occurrences in evaluations. In addition, Azzam (2010:45) outlines situations where stakeholders can dramatically affect how an evaluation is designed and implemented. In line with these power dynamics, Eckhard and Jankauskas (2019) propose incorporation of ‘evaluation stakeholder influence potential’ as a key component in evaluation practice. Evaluation of stakeholder influence is important for the MAE approach for epistemic reasons (the need to ensure appreciation of MAE principles by stakeholders or changes in evaluation culture) and instrumental reasons (to ensure evaluation designs are in line with MAE paradigms). The AEPs call for the evaluation process to include appropriate empowering processes and encourage reciprocity and mutual accountability among various stakeholders involved in the evaluation process.
Evaluation stakeholder influence potential is based on four political resources for influence, which are power for agenda-setting, staff and budgetary resources, access to evaluation results and access to evaluators. In practice, different stakeholders have varying levels of influence for each of these resources; thus, stakeholders have varying capacity to influence evaluation results. According to March and Olsen (1998), an additional source of influence is informal political power that may have influence on evaluation results.
The power dynamics in evaluation invariability touch on issues of positionality. This hinges on African evaluators’ worldviews and the positions they adopt during the evaluation in line with social and political contexts (Foote & Bartell 2011). According to Merriam et al. (2001:411), one’s positionality is understood as ‘where one stands in relation to “the other” in research’. Thus, the positionality of evaluators in the African context will be grounded by their ontological, epistemological beliefs and assumptions about human nature and agency (Grix 2019). This positionality is framed by values and beliefs in line with factors including political allegiance, religious faith, gender, sexuality, historical and geographical location, ethnicity, race, social class and status. A key challenge faced by African evaluators in practising MAE will likely be the need of ‘unlearning’ Western paradigms and having the desire to adapt Western approaches to African contexts. According to McCorkel and Myers (2003:228), a possible strategy to overcome this challenge might be pursuing a strategy that involves ‘a recognition and analysis of how the evaluator’s positionality facilitates specific forms of understanding and impedes others’. This will involve continuous reflections by African evaluators.
Gawlewicz (2014) further proposes the need to incorporate issues of positionality into evaluation practice guidelines. According to D’Silva et al. (2016), possible integration of positionality in enhancing MAE might be achieved through ensuring that evaluation teams are diverse in terms of nationality, gender and other dimensions and that evaluators are conscious of their positions and continuously reflective in practice. This will enhance effective engagement with communities to ensure effective grounding of the MAE within specific contexts. The need for a diverse team is illustrated in the following quotation (D’Silva et al. 2016):
[A]n outsider, not as familiar with a community, might see what the familiar eye might miss. However, ‘outsiders’ who do not reflect on how their own social position might affect the research process and the theoretical lens used in analysis can miss intricacies that are not normally part of that researcher’s experience. (p. 104)
Sources of knowledge
The creation of a knowledge base to support the practice of development evaluation rooted in the MAE paradigm requires a strong indigenous African evaluation capacity. This invariably requires attention towards evaluation capacity development (ECD), that is, unleashing, strengthening, creating, adapting and maintaining evaluation capacities over time (OECD 2006). Effective ECD is therefore a primary source for supplying competent African evaluators for evaluation knowledge generation. According to Tarsilla (2014), ECD in Africa is currently ‘donor-centric’, conducted in an unsystematic manner towards fulfilling the needs of donor-driven projects. This approach to ECD has been identified as being blind in promoting context-relevant learning. Morkel and Mangwiro (2019) further argue that, despite efforts to strengthening ECD in Africa, the generative mechanisms through which ECD strengthens evaluation capacity remain unclear. Another challenge is the lack of a common agreed-upon framework for essential evaluator competencies for ECD in Africa (Morkel & Ramasobama 2017).
In general, most of the practising, prospective and emerging evaluators rely on the content and processes made available at tertiary training institutions. However, there is generally a lack of a harmonised and systematic approach to formal graduate education in evaluation in Africa (Tirivanhu et al. 2020). This scenario has resulted in a situation where evaluation practice is, to a large extent, influenced by social science research approaches. This is mainly because of the research background of most practising evaluators in Africa. The lack of a clear and systematic ECD framework poses challenges to developing the required critical mass of African evaluators to develop the required knowledge repositories for MAE. In addition, deviations are needed from the functional (training to conduct evaluations following the donor-driven evaluation) towards a transformative ECD strategy. The transformative ECD agenda will focus on decolonising and indigenising evaluation practice, coupled with more representation of Africa-based scholars in scholarly knowledge production on MAE. This will address existing epistemic injustices (Chilisa & Malunga 2012).
Sources of legitimation: Made in Africa Evaluation for whom? And what are the implications for practice?
According to the draft AEPs, evaluation practice should be powerful for Africans, and it should be Afrocentric. These principles invariably call for the practice of an empowerment and emancipatory evaluation agenda that empowers African society through supporting people to improve their lives and enhances epistemic freedom. Empowerment evaluation encompasses the application of ‘evaluation concepts, techniques, and findings to foster improvement and self-determination’ (Fetterman 1994, 2001:3). In this regard, an empowerment agenda for MAE will revolve around strengthening the roles of individuals and communities and ensuring their influence and control over decisions that affect their livelihoods (Coombe 2002). Therefore, an empowerment evaluation process should allow evaluands to oversee their environment (physical, economic, social, cultural and psychological) using available resources (Zimmerman 2000). Empowerment evaluation therefore shifts political and decision-making dynamics through placing decision-making in the hands of community or evaluands. A key principle of empowerment evaluation is the need for promoting collective insights, wisdom and experiences to inform evaluations (Fetterman & Wandersman 2005). In practice, evaluands should be engaged throughout the evaluation phases (conceptual design, data collection, analysis and reporting), and this enhances knowledge utilisation (Fetterman 2001).
The practice of an empowerment approach for MAE should ideally be informed by the 10 principles of empowerment evaluation (Fetterman & Wandersman 2005). These principles are improvement, community ownership, inclusion, democratic participation, social justice, community knowledge, evidence-based strategies, capacity-building, organisational learning and accountability. The principle of improvement should be guided by the desire for African-rooted evaluations to contribute towards social change through a desire to contribute towards positive societal results. The community ownership principle is based on the assertion that evaluands have a right to make decisions about actions that affect their lives. In this regard, evaluators are guided by the fact that programme improvement occurs when the community or evaluands are empowered to exercise decision-making through legitimate authority that drives the evaluation process. The inclusion principle ensures that evaluation encompasses inclusive community ownership with direct participation of key stakeholders in decision-making. The democratic participation principle recognises the capacity of stakeholders for intelligent judgement and action when equipped with appropriate information and conditions, and this enables stakeholder buy-in. According to Fetterman and Wandersman (2005), the practice of democratic participation requires the involvement of stakeholders in decision-making to ensure that evaluation fits the needs and values of evaluands.
The principle of social justice aims at ensuring fair, equitable allocation of resources, opportunities, obligations and bargaining power (Dalton, Elias & Wandersman 2001). Thus, evaluators must take cognisance of social inequities and aim at integrating them into the evaluation design and implementation process to ensure positive social change. The principle of community knowledge ensures incorporation of community-based knowledge, values and wisdom into evaluations. This is very much in line with the thrust of MAE evaluation of promoting epistemic freedom. Empowerment evaluation should embrace local knowledge systems (including tacit knowledge) and appreciate the capacity of evaluands to generate their own solutions. Through the principle of evidence-based strategies, empowerment evaluation should value scientific approaches rooted in African epistemologies as a valid evidence base. This principle is in line with the draft African Evaluation Guidelines’ principle of ‘connected with the world’ (see Table 1). Made in Africa evaluation should appreciate the role of Western knowledge in providing good practices to avoid the risk of totally trying to reinvent the wheel and to build from existing literature or practice – keeping in mind the need for adjusting to African contexts.
The principle of capacity-building is linked with issues of ECD outlined in the preceding section. According to Fetterman and Wandersman (2005), empowerment evaluators should be guided by the belief that allowing stakeholders to learn the basic evaluation steps and skills enhances their capacity to shape and improve their livelihoods. The principle of organisational or community learning must allow evaluators to embrace the importance of continuous learning. African-rooted evaluations should encourage learning and integrate learning in evaluation design and implementation. This may include integrating after learning reflection sessions to bring out learning points from evaluations. Finally, the principle of accountability should provide a basis for evaluation as an innovative accountability mechanism through generation of valid results.
Conclusion and recommendations
This article explored the practical implications for MAE. The key research question was the following: what are the critical components for MAE development evaluation practice? An exploratory research design was adopted, guided by theoretical constructs from CSH, existing evaluation frameworks, practice guidelines (including the African Evaluation Guidelines and African Evaluation Principles – Standards and Norms), and relevant theoretical and methodological guidelines. Data were collected through secondary literature reviews, as well as expert and experiential knowledge regarding development evaluation practice. The results indicate the critical practice components for MAE to (1) include appreciating sources of motivation as guiding principles of developing modalities for evaluation practice, (2) understanding and integrating sources of power and politics of value judgements in development evaluation practice, (3) developing sources of knowledge and (4) appreciating sources of legitimation (MAE for whom and implications for practice). Methodologically, the research recommends adoption of African-rooted paradigms, including relational approaches and tools grounded in African worldviews, social systems and values. Made in Africa evaluation should mainstream an empowerment evaluation approach that aims at contributing towards positive social change and promoting epistemic freedom of African evaluators.
Acknowledgements
The author would like to thank the Centre for Learning on Evaluation and Results - Anglophone Africa (CLEAR-AA) for funding the research.
Competing interests
The author declares that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Author’s contributions
P.T. is the sole author of this research article.
Ethical considerations
This article followed all ethical standards for research without direct contact with human or animal subjects.
Funding information
This research was funded by the Centre for Learning on Evaluation and Results – Anglophone Africa (CLEAR-AA).
Data availability
Data that support the findings of this study are available from the author upon reasonable request.
Disclaimer
The views and opinions expressed in this article are those of the author and do not necessarily reflect the official policy or position of any affiliated agency of the author.
References
African Evaluation Association (AfrEA), 2021, The African evaluation principles, viewed 18 June 2022, from https://afrea.org/call4proposals/The-African-Evaluation-Principles.pdf.
Azzam, T., 2010, ‘Evaluator responsiveness to stakeholders’, American Journal of Evaluation 31(1), 45–65. https://doi.org/10.1177/1098214009354917
Bowman, N.R., Francis, C.D. & Tyndall, M., 2015, ‘Culturally responsive indigenous evaluation: A practical approach for evaluating indigenous projects in tribal reservation contexts’, in S. Hood, R. Hopson & H. Frierson (eds.), Continuing the journey to reposition culture and cultural context in evaluation theory and practice, pp. 335–359, Information Age Publishing, Charlotte, NC.
Brandon, P.R. & Fukunaga, L.L., 2014, ‘The state of the empirical research literature on stakeholder involvement in program evaluation’, American Journal of Evaluation 35(1), 26–44. https://doi.org/10.1177/1098214013503699
Chilisa, B., 2015, A synthesis paper on the made in Africa evaluation concept: Commissioned by African Evaluation Association (AfrEA), viewed 15 November 2021, from https://afrea.org/wp-content/uploads/2018/06/MAE2-Final-31st-august.pdf.
Chilisa, B., 2020, Indigenous research methodologies, 2nd edn., Sage, Los Angeles, CA.
Chilisa, B., Major, T.E., Gaotlhobogwe, M. & Mokgolodi, H., 2016, ‘Decolonizing and indigenizing evaluation practice in Africa: Toward African relational evaluation approaches’, Canadian Journal of Program Evaluation/La Revue canadienne d’évaluation de programme 30(3), 313–328. https://doi.org/10.3138/cjpe.30.3.05
Chilisa, B., Major, T.E. & Khudu-Petersen, K., 2017, ‘Community engagement with a postcolonial, African-based relational paradigm’, Qualitative Research 17(3), 326–339. https://doi.org/10.1177/1468794117696176
Chilisa, B. & Malunga, C., 2012, ‘Made in Africa evaluation: Uncovering African roots in evaluation theory and practice’, in Bellagio Centre (eds.), Report of the African Thought-Leaders Forum on Evaluation for Development: Expanding Thought Leadership in Africa, 14–17 November 2012, pp. 32–38, Bellagio, viewed 15 November 2021, from https://idev.afdb.org/sites/default/files/Evaluations/2020-03/article%204%20Towards%20defining%20and%20advancing%20%E2%80%9CMade%20in%20Africa%20Evaluation.pdf
Chilisa, B. & Mertens, D.M., 2021, ‘Indigenous made in Africa evaluation frameworks: Addressing epistemic violence and contributing to social transformation’, American Journal of Evaluation 42(2), 241–253. https://doi.org/10.1177/1098214020948601
Clarke, G.S., Douglas, E.B., House, M.J., Hudgins, K.E.G., Campos, S. & Vaughn, E.E., 2021, ‘Empowering Indigenous communities through a participatory, culturally responsive evaluation of a Federal Program for Older Americans’, American Journal of Evaluation 1–20. https://doi.org/10.1177/10982140211030557
Cloete, F., 2016, ‘Developing an Africa-rooted programme evaluation approach’, African Journal of Public Affairs 9(4), 55–70.
Coombe, C., 2002, ‘Participatory evaluation: Building community while assessing change’, in M. Minkler & N. Wallerstein (eds.), Community-based participatory research for health, pp. 1–60, Jossey-Bass, San Francisco, CA.
Cram, F., Tibbetts, K.A. & LaFrance, J., 2018, ‘‘Editors’ notes: A stepping stone in indigenous evaluation’, Evaluation 2018(159), 7–16. https://doi.org/10.1002/ev.20327
Dalton, J., Elias, M. & Wandersman, A., 2001, Community psychology: Linking individuals and communities, Wadsworth, Belmont, CA.
D’Silva, M.U., Smith, S.E., Della, L.J., Potter, D.A., Talley, T.A. & Best, L., 2016, ‘Reflexivity and positionality in researching African-American Communities: Lessons from the field’, Intercultural Communication Studies XXV(1), 94–109.
Eckhard, S. & Jankauskas, V., 2019, ‘The politics of evaluation in international organizations: A comparative study of stakeholder influence potential’, Evaluation 25(1), 62–79. https://doi.org/10.1177/1356389018803967
Fetterman, D.M., 1994, ‘Empowerment evaluation’, Evaluation Practice 15(1), 1–15. https://doi.org/10.1177/109821409401500101
Fetterman, D.M., 2001, Foundations of empowerment evaluation, Sage, Thousand Oaks, CA.
Fetterman, D.M. & Wandersman, A., 2005, Empowerment evaluation principles in practice, The Guilford Press, New York, NY.
Flood, R. & Jackson, M.C., 1991, ‘Critical systems heuristics: Application of an emancipatory approach for police strategy toward the carrying of offensive eeapons’, Systems Practice 4(4), 283–302. https://doi.org/10.1007/BF01062006
Foote, M.Q. & Bartell, T.G., 2011, ‘Pathways to equity in mathematics education: How life experiences impact researcher positionality’, Educational Studies in Mathematics 78, 45–68. https://doi.org/10.1007/s10649-011-9309-2
Gawlewicz, A., 2014, ‘Language and translation strategies in researching migrant experience of difference from the position of migrant researcher’, Qualitative Research 16(1), 27–42. https://doi.org/10.1177/1468794114557992
Grix, J., 2019, The foundations of research, Macmillan International, New York, NY.
Hood, S., Hopson, R.K. & Kirkhard, K.E., 2015, ‘Culturally responsive evaluation: Theory, practice, and future implications’, in K.E. Newcomer, H.P. Hatry & J.S. Wholey (eds.), Handbook of practical program evaluation, 4th edn., pp. 281–317, Jossey-Bass, San Francisco, CA.
Independent Development Evaluation (IDEV), 2019, Made in Africa evaluation, Volume 2: Practical applications, viewed 17 November 2021, from https://idev.afdb.org/en/document/made-africa-evaluations-volume-2-practical-applications.
Keane, M., Khupe, C. & Seehawer, M., 2017, ‘Decolonising methodology: Who benefits from Indigenous knowledge research?’, Educational Research for Social Change 6(1), 12–24. https://doi.org/10.17159/2221-4070/2017/v6i1a2
LaFrance, J., Nichols, R. & Kirkhart, K.E., 2012, ‘Culture writes the script: On the centrality of context in indigenous evaluation’, in D.J. Rog, J.L. Fitzpatrick & R.F. Conner (eds.), Context: A framework for its influence on evaluation practice. New directions for evaluation, vol. 135, pp. 59–74, Jossey-Bass, San Francisco, CA.
Luckett, K., 2006, ‘An assessment of the application of “Critical Systems Heuristics” to a policy development process’, System Practice Action Reserach 19, 503–521. https://doi.org/10.1007/s11213-006-9040-6
March, J.G. & Olsen, J.P., 1998, ‘The institutional dynamics of international political orders’, International Organization 52(4), 943–969. https://doi.org/10.1162/002081898550699
Mbava, N.P., 2017, ‘The potential value of the realist evaluation method in programme impact evaluations in South Africa’, Unpublished PhD dissertation, Stellenbosch University.
Mbava, N.P., 2019, ‘Shifting the status quo: Africa influencing global evaluation’, in J. Musumba (ed.), Made in Africa Evaluations. Volume 1: Theoretical approaches, pp. 12–21, African Development Bank Evaluation Matters, Quarter 3, Abidjan.
Mbava, N.P. & Chapman, S., 2020, ‘Adapting realist evaluation for Made in Africa evaluation criteria’, African Evaluation Journal 8(1), a508. https://doi.org/10.4102/aej.v8i1.508
McCord, C. & Becker, C., 2019, Sidewalk and Toronto: Critical systems heuristics and the smart city, viewed 03 December 2021, from https://arxiv.org/pdf/1906.02266.pdf.
McCorkel, J.A. & Myers, K., 2003, ‘What difference does difference make? Position and privilege in the field’, Qualitative Sociology 26(2), 199–231. https://doi.org/10.1023/A:1022967012774
Merriam, S.B., Johnson-Bailey, J., Lee, M.Y., Kee, Y., Ntseane, G. & Muhamad, M., 2001, ‘Power and positionality: Negotiating insider/outsider status within and across cultures’, International Journal of Lifelong Education 20, 405–416. https://doi.org/10.1080/02601370120490
Mertens, D.M., 2012, ‘Transformative mixed methods: Addressing inequities’, American Behavioral Scientist 56(6), 802–813. https://doi.org/10.1177/0002764211433797
Mertens, D.M., 2018, Mixed methods design in evaluation, Sage, Los Angeles.
Midgley, G., 2000, Systemic intervention: Philosophy, methodology, and practice, Kluwer/Plenum, New York, NY.
Morkel, C. & Mangwiro, N., 2019, ‘Implications of evaluation trends on capacity development’, in C.B. Mapitsa, P. Tirivanhu & N. Pophiwa (eds.), Evaluation landscape in Africa – Context, methods and capacity, pp. 191–216, African Sun Media, Stellenbosch.
Morkel, C. & Ramasobama, M., 2017, ‘Measuring the effect of evaluation capacity building initiatives in Africa: A review’, African Evaluation Journal 5(1), a187. https://doi.org/10.4102/aej.v5i1.187
Morris, M. & Clark, B., 2013, ‘You want me to do what? Evaluators and the pressure to misrepresent findings’, American Journal of Evaluation 34(1), 57–70. https://doi.org/10.1177/1098214012457237
Muwanga-Zake, J., 2009, ‘Building bridges across knowledge systems: Ubuntu and participative research paradigms in Bantu communities’, Discourse Studies in the Cultural Politics of Education 30(4), 413–426. https://doi.org/10.1080/01596300903237198
Nzimakwe, T.I., 2014, ‘Practising Ubuntu and leadership for good governance: The South African and continental dialogue’, African Journal of Public Affairs 7(4), 30–41.
Omosa, O., Archibald, T., Niewolny, K., Stephenson, M. & Anderson, J., 2021, ‘Towards defining and advancing “Made in Africa Evaluation”’, African Evaluation Journal 9(1), a564. https://doi.org/10.4102/aej.v9i1.564
Organisation for Economic Co-operation and Development (OECD), 2006, The challenge of capacity development: Working towards good practice, OECD, Paris.
Reynolds, M., 2007, ‘Evaluation based on critical systems heuristics’, in B. Williams & I. Imam (eds.), Using systems concepts in evaluation: An expert anthology, pp. 101–122, Point EdgePress, Reyes, CA.
Romero, R., 2002, ‘The relevance of Ulrich’s critical systems heuristics for social planning and evaluation’, Cuadernos de Administracion 15(24), 41–71.
Tambulasi, R. & Kayuni, H., 2005, ‘Can African feet divorce western shoes? The case of “Ubuntu” and democratic good governance in Malawi’, Nordic Journal of African Studies 14(2), 147–161.
Tarsilla, M., 2014, ‘Evaluation capacity development in Africa: Current landscape of international partners’ initiatives, lessons learned and the way forward’, African Evaluation Journal 2(1), a89. https://doi.org/10.4102/aej.v2i1.89
Taylor, D. & Balloch, S., 2005, The politics of evaluation: Participation and policy implementation, The Policy Press, Bristol.
Tirivanhu, P., Robertson, H., Waller, C. & Chirau, T., 2020, ‘Assessing evaluation education in African Tertiary education institutions: Opportunities and reflections’, South African Journal of Higher Education 32(4), 229–244. https://doi.org/10.20853/32-4-2527.
Ulrich, W., 1983, Critical heuristics of social planning, John Wiley & Sons, Oxford.
Ulrich, W., 2000, ‘Reflective practice in the civil society: The contribution of critically systemic thinking’, Reflective Practice 1(2), 247–268. https://doi.org/10.1080/713693151
Ulrich, W., 2012, CST’s two ways: A concise account of critical systems thinking, viewed 29 November 2021, from http://wulrich.com/downloads.html.
Ulrich, W. & Reynolds, M., 2010, ‘Critical systems heuristics’, in M. Reynolds & S. Holwell (eds.), Systems approaches to managing change: A practical guide, pp. 243–292, Springer, London.
Uwizeyimana, D., 2020, ‘UBUNTU and the challenges of Africa-rooted public policy evaluation approach’, Journal of African Foreign Affairs (JoAFA) 7(3), 113–129. https://doi.org/10.31920/2056-5658/2020/v7n3a6
Zimmerman, M.A., 2000, ‘Empowerment theory: Psychological, organizational, and community levels of analysis’, in J. Rappaport & E. Seidman (eds.), Handbook of community psychology, pp. 43–63, Kluwer Academic/Plenum, New York, NY.
Footnotes
1. Ubuntu expresses African people’s ‘interconnectedness, their common humanity, and the responsibility to each other that flows from African people’s deeply felt connection and communal ways of life’ (Tambulasi & Kayuni 2005:147).
2. Those evaluations conducted in Africa and often claiming use of participatory approaches.
|
Hide
Show all
Email this article (Login required)
