Abstract
Background: The Made in Africa Evaluation (MAE) agenda is an effort at decolonising and indigenising evaluation practice in Africa. This involves developing new evaluation practices, theories, approaches and methodologies originating from African cultures, worldviews, knowledge systems, philosophies and African paradigms.
Objectives: This study aims to explore achievements in the development of an African evaluation paradigm or approach and identifying the gaps.
Method: An evidence gap map (EGM) was utilised because it entails a systematic search of literature with the intention of presenting the existing evidence on a specific topic, identifying gaps in knowledge and determining future research needs.
Results: Five main paradigms that fall within the MAE were identified, including the Afrocentric paradigm, the postcolonial indigenous paradigm, the African relational evaluation paradigm, the transformative evaluation paradigm and the culturally competent evaluation paradigm. In addition, five key categories of MAE approaches were identified, including (1) the least indigenised approach; (2) the adaptive or integrative evaluation approach; (3) the African relational-based evaluation approach; (4) the predominantly indigenous approach and (5) the third space (geocentric) evaluation approach.
Conclusion: African contexts, cultures and people have been discussed in five ways in MAE literature including collaborative, relational, complex, previously colonised and disenfranchised. The EGM shows that these paradigms and approaches have been well established theoretically and conceptually; however, the application of these in evaluations has mostly been found in the adaptive or integrative approach, which integrates Western methodologies. This suggests a lack of practical guidelines to using the paradigms, approaches and methods originating from Africa, including the African relational-based evaluation approach.
Keywords: Made in Africa Evaluation; Africa-rooted; Africa-led; evaluation approaches; paradigms.
Introduction
The Made in Africa Evaluation (MAE) agenda is an effort at decolonising and indigenising monitoring and evaluation (M&E) practice on the African continent (Chilisa 2015). The decolonisation of evaluation is viewed as a multifaceted process that involves the refusal to continue to blindly adopt Western theories, values and standards to the evaluation of programmes implemented in Africa. It also involves the adaptation of evaluation models, instruments and tools to ensure contextual and cultural relevance to Africa. Furthermore, decolonisation involves ‘the development of novel evaluation practices, theories and methodologies that emanate from local cultures, indigenous knowledge systems, African philosophies and African paradigms’ (Chilisa et al. 2016:316). The process of decolonisation of evaluation should be coupled with the indigenisation of evaluation to make evaluation practice truly appropriate for Africa contexts (Chilisa et al. 2016).
Chilisa et al. (2016:316) defined indigenisation as ‘the blending of an imported discipline with the generation of new concepts and approaches from within a culture’. Therefore, it is posited that the indigenisation of evaluation in Africa can be measured by the extent to which the evaluation process is centred around African people; the extent to which evaluation outcomes, standards, tools and methodologies originate from African realities and worldviews; and the extent to which the ways of knowing and associated values are considered valuable by African stakeholders and beneficiaries (Chilisa et al. 2016). ‘A decolonised MAE approach is thus African-people centred, values culturally relevant and indigenised evaluation processes and methodologies predominantly informed by African worldviews and paradigms’ (Chilisa 2015).
Paradigms inform the approaches and methods, research processes, priorities, choices, actions and dissemination of evaluation findings (Chilisa & Mertens 2021). Scholars argue that there is an urgent need for African evaluators to engage with the ontological, epistemological and axiological assumptions that inform their evaluation methodologies. This is because an evaluation approach removed from its overarching paradigm cannot address the epistemic violence and decolonisation of African evaluation practice from Western theories, etc. (Chilisa & Mertens 2021). Efforts have been made to articulate what the MAE paradigm and approach is (Chilisa 2015; Chilisa et al. 2016; Cloete 2016; Cloete & Auriacombe 2019; Mbava & Chapman 2020). This body of work is intended to create an evidence base of paradigms and approaches that fall within the spectrum of MAE. This is meant to contribute to scholarship and understanding of ‘what MAE is, can be and can do’ (Chilisa 2015:5).
Evidence gap maps (EGMs) are a systematic evidence synthesis methodology that present the available evidence relevant to a specific research question or a particular topic. In addition, EGMs are used to identify gaps that need to be filled with new evidence and collections of studies and increase the discoverability and use of studies by decision-makers, research commissioners and researchers (White et al. 2020). According to Chilisa (2015):
[T]here is also no comprehensive review of what MAE has meant to those who have applied it, nor is there documentation of practical examples of how the concept has been applied or what it means in practice. (p. 5)
Using an EGM methodology, this study aims to explore achievements in the development of an African evaluation paradigm or approach and identify where the gaps are in decolonising and indigenising evaluation practice in Africa.
Research methods and design
Study design
An EGM is a systematic search of a wide array of literature in a particular field with the intention of presenting the existing evidence on a specific topic, identifying gaps in knowledge and determining future research needs (Miake-Lye et al. 2016; Snilstveit et al. 2016; White et al. 2020). Evidence gap maps are the collection of thematic evidence, structured around a conceptual framework, aimed at identifying patterns and visually presenting it in a user friendly format, often a diagram or graph or a database (Snilstveit et al. 2016). Furthermore, EGMs can be utilised to map research evidence that addresses topics such as methodological scope and quality of existing evidence (Winters, Langer & Geniets 2017). Thus, making it perfect design for this research study.
For the purposes of this study, discourse analysis was utilised as the analytical framework for assessing relevant literature included in the evidence map and identifying gaps. Discourse analysis was perceived as the most appropriate and useful technique, given its emphasis on the analysis of language within its social and culture context. This analytical method is useful if the aim of the study entails understanding culture. It can assist with identifying how culture has an effect on the way concepts are spoken about (Warren 2020). This is an important analytical framework when reviewing the repertoire of MAE paradigms and approaches.
The evidence gap mapping process
This section provides the step-by-step process taken to developing the EGM.
Step 1: Clarifying the study scope and research question
The initial step of the study entailed setting the scope of the EGM by defining the mapping framework. The mapping framework is a matrix that generally represents two or more variables that are important to the particular field or topic under exploration. Most EGMs plot evidence on interventions and outcomes in a mapping framework. Given that the focus of this study was the methodological scope of MAE approaches, the columns and rows of the mapping framework (see Figure 3) were adapted to reflect key paradigms underlying evaluation approaches. The rows of the matrix cover all the paradigms identified within MAE literature, while the columns of the matrix cover the approaches within the MAE school of thought. The mapping framework was developed based on a review of key academic and grey literature. Therefore, extensive reading of journal articles and grey literature focusing on MAE was carried out. The MAE concept is variably referred to in literature as Africa-centric, Afrocentric, Africa-rooted, Africa-led or decolonised evaluation. Nevertheless, the overarching objective of MAE is to drive an evaluation agenda that is led and owned by African people (Mbava 2019). The latter assisted with identifying relevant literature for the EGM.
In addition, the mapping framework was used to identify a set of substantive inclusion criteria for the EGM. The focus of the EGM generally requires additional inclusion criteria that will help set the limits of the scope. The most commonly used model to define the inclusion criteria for systematic reviews including evidence maps, is the population, intervention, comparison and outcomes (PICO). However, the PICO model was not adequate because it is typically used for quantitative and empirical research, with mapping frameworks focused on intervention or outcome configurations (Snilstveit et al. 2013). This is the reason scholars such as Cooke, Smith and Booth (2012) developed a new model to help determine substantive inclusion criteria for qualitative and mixed methods evidence maps: the sample, phenomenon of interest, design, evaluation and research type (SPIDER) model. This model was more appropriate for this study. This model is referred to as the search strategy tool in the EGM methodology. In this study, the following SPIDER search strategy criteria were included.
Sample: The first criterion for the literature search was the sample. The sample included peer-reviewed journal articles, book chapters, systematic reviews, organisational reports, blogs, evaluation magazines and newsletters from evaluation organisations such as voluntary organisations for professional evaluation (VOPEs); higher education institutions; evaluation consultancies; nongovernmental organisations (NGOs) and civil society organisations (CSOs), etc.; M&E practice briefs; evaluation conference articles; and PhD dissertations that theoretically discuss paradigms that fall within the scope of MAE and MAE approaches, from any university, in any discipline. Only a few of the documents reported on research studies or trainings, etc., that used the approaches. The decolonisation subject has been highlighted in both evaluation practice and research. Therefore, a few of the documents reviewed refer more to research than evaluation.
These documents were published between 2007 and 2021, as the discourse on the decolonisation of evaluation and research predates the Bellagio Conference in 2012. While the concept of MAE is focused on the African continent, publications on the topic have been produced by evaluators, researchers, scholars and academics, students, evaluation and research organisations, VOPEs, etc., across the globe. Therefore, there were no limits in terms of the geographic scope of the literature search.
Phenomenon of interest: The second criterion for the literature search was the phenomenon of interest. What was of interest in this EGM was the paradigms within the MAE scope and the associated evaluation approaches.
Design: The third criterion for the literature search was the study design. In this EGM, published literature theoretically discussing or utilising MAE paradigms and approaches, describing any evaluation or research design, was selected for inclusion.
Evaluation: According to Cooke et al. (2012:1438), in the PICO model, the ‘O’ refers to Outcomes. In a qualitative study, the outcomes may be unobservable and subjective constructs, and thus, the search strategy term ‘evaluation’ was deemed more appropriate. In light of the fact that this study is focused on paradigms and evaluation approaches, the fourth search criteria also included the following constructs: paradigm, approach, method, philosophy, knowledge systems, value systems, worldviews and perspectives, culture and context were important.
Research type: According to Cooke et al. (2012:1438), the search criteria can include three research types: qualitative, quantitative and mixed methods. Therefore, the fifth search criterion included all MAE literature theoretically discussing or utilising paradigms and approaches that fall within the scope of MAE, using all research types including quantitative, qualitative or mixed methods data collection and analysis techniques.
Step 2: Systematic search for evidence
In this study, the search for evidence was conducted in a systematic manner. The search to identify relevant literature to include in the EGM was targeted at all relevant academic databases and journals, as well as grey literature sources. These sources are listed in Table 1. The final stage of the search strategy involved snowballing, meaning checking the reference lists of all the academic journal articles collected to see if there is any additional literature on MAE approaches and methods.
| TABLE 1: List of academic databases, journals and organisational websites searched. |
The search terms used when navigating these databases, online journals and websites were ‘MAE’, ‘indigenous evaluation methods’, ‘African approaches to evaluation’, ‘Africa-led evaluation’, ‘Africa-centric evaluation’, ‘Africa-rooted evaluation’, ‘African philosophy and evaluation’ and ‘evaluation in Africa’. The search process was conducted between 02 November 2021 until 18 November 2021, taking approximately two and a half weeks.
Step 3: Screening for inclusion
The search showed that the literature on MAE is extremely limited, as the search identified only 95 citations or documents (50 journal articles and 45 grey literature documents). The screening of the literature entailed reviewing all the abstracts and examining them against the inclusion criteria. Once potentially relevant documents were identified, the screening of the full texts took place. In the first screening of the journal article abstracts, 26 articles were identified as relevant, while 24 peer-reviewed articles did not make it into the evidence map. In the screening of the grey literature, 21 documents were identified as relevant, whereas 24 documents were not included.
Step 4: Data extraction and critical appraisal
Evidence gap maps do not entail summarising the findings of the selected literature or studies, but instead they are more focused on providing descriptive information on various key aspects of that literature. The data extraction process is carried out by using a structured coding tool (see Figure 1), which includes all the selection criteria identified as relevant for the EGM, particularly all the data to be included in the map.
As a result of the limited scope of the MAE literature, therefore only one reviewer utilised the coding tool to transparently and systematically explore what has been achieved in the development of MAE approaches and methodologies. The researcher read all the included journal articles and grey literature and extracted the data for the rows and columns of the EGM, which are ‘paradigm’ and ‘approach’. In addition, the two key constructs of the EGM were broken down further to facilitate use of the SPIDER model. For example, the review of the literature also involved assessing information on the ontologies and epistemologies of the paradigms deemed fit for MAE.
Step 4 also involved critical appraisal of the selected literature. This process also involved the use of a tool. The appraisal process in this study focused on assessing the relevance of the selected literature to the research question. In examining the relevance of the selected literature, the aim was to determine the relevance of the literature to the EGM’s objective and research question. The results section will provide information the extent of the sample and the literature remaining following the inclusion and exclusion process.
Step 5: Visualisation and analysis
This step of developing the EGM involved organising, presenting and analysing the mapped evidence base. In this study, the EGM was created on Microsoft Excel. The visualisation ensures the enhanced use of the evidence base. The EGM was populated with data extracted from the accepted and included literature. There were previous versions of the EGM that included the various methods that fall within the various paradigms and MAE approaches. The final version, however, only indicates the number of documents that discuss the evaluation paradigms and MAE approaches (see Figure 3).
Ethical considerations
This article followed all ethical standards of research without direct contact with human or animal subjects.
 |
FIGURE 2: Overview of searching and screening. |
|
Results
Search results
The final EGM presents the results that were extracted from 47 documents, some of which reported on more than one paradigm or approach (see the EGM in Figure 3). The literature that was excluded was more focused on discussing the conceptualisation of MAE, explaining the historical events that led to the call for a MAE agenda, discussing current discourse around MAE and simply advocating for a paradigm shift in evaluation, etc.; however, they do not discuss and describe any methodological aspects of MAE. The majority of the included literature only theoretically discusses the MAE approaches and the underlying paradigms; only a few documents report on research studies or trainings, etc., that utilised the MAE approaches. The other reasons that led to the inclusion of literature included reference to and discussion of key terms relating to decolonising evaluation, such as indigenous knowledge and evaluation, including not only decolonisation of evaluation in Africa but also in other non-Western contexts, because these methodologies are deemed applicable by several scholars. Thus, the EGM has a total of 64 findings.
 |
FIGURE 3: Visualisation of the evidence gap map. |
|
Description of the selected literature
Of the total number of the peer-reviewed literature that was included in the evidence map (n = 27), 25 documents were peer-reviewed articles published by the following journals: African Evaluation Journal (6), Sustainability Science (1), Qualitative Research (1), Educational Research for Social Change (1), Canadian Journal of Program Evaluation (2), American Journal of Evaluation (4), Journals (1), Journal of Mixed Methods Research (1), African Journal of Development Studies (1), Discourse: Studies in the Cultural Politics of Education (1), The Qualitative Report (1), the Journal of African Foreign Affairs (1), and the Journal of Black Studies (1). In addition, 4 of the peer-reviewed documents are book chapters (see Table 2).
| TABLE 2: Peer-reviewed literature sources. |
Of the total number of grey literature documents that have been found relevant for the evidence map, 8 magazine articles were published in the Evaluation Matters Magazine of the African Development Bank (AfDB), 2 are organisational reports commissioned by the African Evaluation Association (AfrEA) and the UN Food and Agriculture Office (FAO) Regional Office in Nairobi, 3 blogs were taken from the Evaluation from Transformation website and 3 blogs taken from the Evaluation for Africa website, 1 is an ethical protocol published by EvalIndigenous found on the EvalPartners website, 1 is a learning brief published by Twende Mbele, 1 is a PhD dissertation, 1 is a conference paper presented at the American Evaluation Association, and 1 is African Evaluation Guidelines developed by AfrEA (see Table 3).
The evidence gap map
Table 3 presents the EGM. The columns are the paradigms identified in the sampled literature, the rows are the categorised MAE approaches and within the cells are the specific methods that are either stated as falling within the MAE or identified by the researcher as suited to a specific approach. While the extent of the use of MAE approaches in evaluation practice is beyond the scope of this study, the findings of this study show the range of evaluation approaches that are deemed either applicable to MAE practice or developed for MAE practice. The sampled literature reviewed for this study suggests five paradigms that have either been developed in other non-Western contexts and are viewed as more appropriate for evaluation practice in complex contexts such as Africa; paradigms originating from the West but when adapted are viewed as applicable to the African context; or paradigms with ontologies, epistemologies and axiologies derived from African philosophies and world views. These include the ‘Afro-centric paradigm’, the ‘postcolonial indigenous paradigm’, the ‘African relational evaluation paradigm’, the ‘transformative evaluation paradigm’ and lastly, the ‘culturally competent evaluation paradigm’.
The reviewed literature also identifies several categories of evaluation approaches that fall within the MAE scope. While the literature suggests that the conceptualisation of these categories was developed by Chilisa (2015, 2017), the use of the categories is pervasive within the MAE literature. It is also important to note that where methods were not classified, the researcher utilised their discretion based on an understanding of both the method and the approaches. The literature identifies five such categories of African evaluation approaches in this MAE agenda. These include: (1) the ‘least indigenised approach’; (2) the ‘adaptive or integrative evaluation approach’; (3) the ‘African relational-based evaluation approach’; (4) the ‘predominantly indigenous approach’; and (5) the ‘third space (geocentric) evaluation’.
A review of the literature in the EGM shows that these paradigms and approaches have been theoretically and conceptually well established; however, the applications of these in evaluations have mostly been found in the adaptive or integrative approach. This is an important finding because, firstly, it suggests that there is a lack of practical steps and guidelines to using these paradigms, approaches and methods. Secondly, this suggests a lack of buy-in in predominantly and completely African-rooted evaluation paradigms, approaches and methods.
Discussion
Discourses on the African context and cultures
The analysis of the relevant literature reveals that there are five ways in which African contexts, cultures and people have been discussed in the MAE literature reviewed:
- African cultures (including worldviews, philosophies, values, practices, etc.) are collaborative (Apahou 2019; Easton 2012).
- African cultures (including worldviews, philosophies, values, practices, etc.) are relational (Chilisa 2015; Chilisa et al. 2016).
- African contexts are complex (Chouinard & Cousins 2013; Mbava 2017; Mbava & Chapman 2020).
- African contexts were previously colonised (Cloete & Auriacombe 2019; Gaotlhobogwe et al. 2018).
- African people have been disenfranchised (Chilisa & Mertens 2021; Mertens 2008).
The paradigms and approaches identified within the reviewed literature are aligned to these views of African contexts, cultures and people.
Made in Africa Evaluation paradigms
During the review of the sampled literature, five paradigms emerged as relevant to MAE practice.
Afrocentric paradigm
The first paradigm identified in the literature reviewed stems from various AfrEA conferences between 2007 and 2013 is the ‘Afrocentric paradigm’, which puts the African view of reality, African ways of knowing and value systems on the same level as mainstream and other scholarly inquiry (Chilisa, 2015). However, it recognises the impossibility of completely removing evaluation practice from its Western roots. The underlying philosophy in the Afrocentric paradigm is Afrocentricity developed by Molefi Kete Asante, which involves placing African people at the centre of their own stories. In terms of its axiology, Asante also argues that in order to understand the narratives of African history, culture and social institutions, people must allow Africans to see themselves as actors, rather than on the margins of Europe or any other cultural group (Edmonds 2021). In terms of ontology, Asante (1983:3) argued that ‘African culture makes the self the centre of the world’. Therefore, reality itself and every African experience is personal (Asante 1983). In terms of its epistemology, Afrocentric scholarship indicates that both Afrocentric methods and Afrocentrically generated knowledge should be reflective of the primacy of the spiritual, the relationship between the physical and the spiritual and the interconnectedness of all things. Lastly, Afrocentric methodology is used to generate knowledge that facilitates the freedom and empowerment of African people (Mazama 2001).
Postcolonial indigenous paradigm
The second paradigm identified in the MAE literature is the ‘postcolonial indigenous paradigm’. It emphasises decolonisation intention as its driving axiology. Thus, it is highly critical of the dominance of Euro-Western language and thought, cultural and academic imperialism (Chilisa, Major & Khudu-Petersen 2017). The postcolonial indigenous paradigm, whose ontology, epistemology, knowledge and value systems stem from the cultures, philosophies and histories of people and countries marginalised by colonialism. In terms its ontology, the aim is to deconstruct the ‘truths’ and ‘norms’ that have been presented as normal and natural by presenting them as both politically and socially biased. In terms of epistemology:
[D]isregarding indigenous knowledge would not only alienate the native population, but weakens development strategies. As such, greater emphasis must be placed on indigenous knowledge, not just framing development policies and strategies but in a way that challenges traditional western development… it would allow for re-conceptualization of development theory and would also equally allow indigenous populations to take ownership of the development project. (Omosa 2019:28)
African relational evaluation paradigm
The third paradigm found in the MAE literature is ‘African relational evaluation paradigm’. In this paradigm, the African ontology, epistemology and axiology are relational. The relational axiology places an emphasis on values founded on cooperation, collective responsibilities, respect, interdependence and interpersonal relationships among people as the highest value. ‘African ontology is based on multiple realities. Africans believe in the living and non-living’ (Gaotlhobogwe et al. 2018:51). Therefore, evaluators must understand that African people have a relationship with the universe and this connectedness must be incorporated into evaluation concepts and methodologies. In addition, the relational ontology is heavily informed by the ubuntu philosophy, which argues that ‘I am because we are’, representing the collectivist nature of the African people. The relational epistemology asserts that knowing is socially constructed by people who have relationships and connections with each other, with the environment, with the spirits of the ancestors, with both the living and the nonliving (Chilisa 2015). According to Gaotlhobogwe et al. (2018), relational methodologies are informed by indigenous African knowledge, histories, languages, metaphors, worldviews, philosophies and experiences. One example of such a methodology is the ‘talking circles’ method that values togetherness (Gaotlhobogwe et al. 2018).
Transformative evaluation paradigm
The fourth paradigm identified is the ‘transformative evaluation paradigm’. While not uniquely African or of African origin, it has key characteristics that can be applied to the African context. This paradigm is deeply rooted in the human rights agenda. Therefore, the ethical implications include the conscious inclusion of people generally excluded from mainstream society, making it applicable to culturally complex communities. This paradigm also pays attention to power issues in the research or evaluation process, including when designing and planning, implementing and using the research or evaluation findings. The axiological assumption relates to having respect for communities that have been pushed to the margins and recognition of the resilience within community members (Chilisa & Mertens 2021; Mertens & Musyoka 2007). In terms of its ontology, the transformative paradigm rejects cultural relativism, it recognises various versions of reality that emanate from different social positionalities and lastly, there is a conscious recognition of the consequences of privileging one version of reality over another. In terms of its epistemology, the transformative paradigm views knowledge as socially and historically located. In terms of methodology, the transformative paradigm endorses cyclical mixed methods designs, a consideration of cultural and contextual factors affecting the intervention and a focus on building respectful relationships during the evaluation process.
Culturally competent evaluation paradigm
The fifth paradigm is the ‘culturally competent evaluation paradigm’. Scholars define culturally competent evaluation:
[A]s a systematic, responsive inquiry that is actively cognizant, understanding and appreciative of the cultural context in which evaluation takes place; that frames and articulates the epistemology of the methodology; and that uses stakeholder-generated, interpretive means to arrive at the results and further use of the findings. (Botcheva, Shih & Huffman 2009:177)
Again, while having not emerged from the African continent, it does fit the brief of the MAE agenda that is in search of culturally sensitive or culturally competent programme evaluation approaches (Cloete & Auriacombe 2019; Uwizeyimana 2021). In terms of its methodology, firstly, cultural competence is important in evaluation because it ensures that the evaluation team are ethically responsible to be culturally competent in order to produce work that is honest, accurate, respectful of stakeholders and considerate of the general public welfare. Secondly, cultural competence is important in evaluation because it supports validity by ensuring that diverse voices and perspectives are honestly and fairly represented, which in turn helps to make valid inferences and interpretations. Lastly, evaluation practice is embedded in theories that are in themselves shaped by cultural values and perspectives. Therefore, it is important for evaluators to scrutinise these theories in order to understand how they describe societal issues and how to address them (AfrEA 2020).
Made in Africa Evaluation approaches
In addition, the literature reviewed highlights five categories of African evaluation approaches in this MAE framework. Made in Africa Evaluation is pluralistic because on the one hand, it advocates for the adaptation of Western evaluation designs, methods, models, tools and strategies to make them relevant to African contexts, and on the other hand, it advocates for the development of a completely new evaluation practice, theory and methodologies derived from local African cultures, indigenous knowledge systems, African philosophies, African worldviews, African values, etc. (Chilisa & Mertens 2021). The five main approaches identified within the MAE literature reviewed include the following.
Least indigenised approach: This evaluation approach is dominated by Western evaluation theory and practice, and only minimum changes are made to contextualise them. This means that evaluation practice in African communities is driven by Western theories and models. The main effort made at indigenising and contextualising evaluation practice is by simple, cosmetic changes such as translating data collection tools into the local African languages so that participants understand the questions (Chilisa 2017; Chilisa et al. 2016). African realities, values, epistemologies and worldviews are not taken into consideration. The least indigenous evaluation approach is predominantly Western and often lacks cultural relevance and validity of the evaluation results is questioned. African evaluation scholars assert that:
[T]his approach is not suitable for use in Africa as the community has no input in it. The community does not benefit anything from the research; instead they are just being exploited by the researchers. (Gaotlhobogwe et al. 2018:54)
Adaptive or integrative evaluation approach: This integrative evaluation approach involves a process in which evaluators choose Western methodologies and integrate them with the African evaluation methodologies. This approach places equal value on both Western and African worldviews. In this approach, African cultures and communities are critical. The African communities play a significant role in the evaluation and are not viewed as simple participants in the evaluation process but active evaluators. In addition, evaluators are also evaluated. Such an approach is said to the validity and usefulness of the evaluation outcomes (Chilisa 2015; Gaotlhobogwe et al. 2018; Omosa 2019). The proponents of this approach argue that it is not possible to completely remove evaluation practice from its Western roots, but that rather what is needed is a more relevant, refocused, pragmatic, resilient and integrative approach to address the challenge of decolonisation in order to improve the potential impacts of evaluation on societal change (Cloete & Auriacombe 2019). Scholars assert that the best example of this approach is the African peer review mechanism (APRM) (Chilisa et al. 2016).
African relational-based evaluation approach: This evaluation approach values the use of a new evaluation practice that is completely based on indigenised African evaluation conceptual frameworks, which acknowledge and include African cultures, languages, values, African knowledge systems, etc., in the evaluation process. In this approach, African evaluation frameworks must be rooted on African philosophies, African-relational ontologies, African-relational epistemologies, African-relational axiologies and uniquely African-relational methodologies. One of the key philosophies informing this approach is the ubuntu philosophy. These philosophies and worldviews have in common relational ways of viewing reality, relational ways of knowing and relational value systems that sum up an African relational paradigm (Chilisa 2015; Chilisa et al. 2016; Chilisa et al. 2017; Chilisa & Tsheko 2014; Twende Mbele 2021; Gaotlhobogwe et al. 2018; Uwizeyimana 2021). One example of a methodology in this approach is proverb-based evaluation (Chilisa & Malunga 2012; Easton 2012).
Predominantly indigenous approach: This evaluation approach requires that evaluations should be informed by different ways of knowing and value systems. It must incorporate other ways of knowing, such as Afrikology (Gaotlhobogwe et al. 2018). This approach did not emerge from Africa; however, it comes from non-Western contexts. Afrikology recognises:
[A]ll sources of knowledge as valid within their historical, cultural or social contexts and seeks to engage them into a dialogue that can lead to better knowledge for all. It recognises peoples’ traditions as a fundamental pillar in the creation of such cross-cultural understandings in which the Africans can stand out as having been the fore-bearers of much of what is called Greek or European. (Gaotlhobogwe et al. 2018:55)
The framework guiding this approach states that where the subject of inquiry is a local or indigenous phenomenon, then methodologies derived from African epistemologies and worldviews should be utilised. It argues that indigenous and local knowledge (ILK) should not be mainstreamed into conventional knowledge but should rather be allowed to co-exist with Western science, because they are two distinctly different systems. In the predominantly indigenous research and evaluation approach, evaluators are empowered by philosophical systems with a decolonisation intent, such as Afrikanisation and the African renaissance and draw, for example, from ethno-philosophy to inform the entire evaluation process, starting from the conceptualisation of the study to the reporting the findings (Chilisa 2017).
Third space evaluation approach: African evaluation scholars have also suggested the inclusion of an evaluation framework that is informed by a geocentric approach. Scholars state that a geocentric approach is a cultural interactive framework that is based on the contribution of local and cultural applications and adaptations rather than one that is dependent on imported elements of other cultures (Gaotlhobogwe et al. 2018). Although not originating from the African continent or African practitioners, this approach is viewed as relevant and applicable to African contexts. This approach recognises the Western, Chinese and African ways of knowing and value systems. Another approach that falls within this framework is participatory evaluation approach (Matsiliza 2012).
Made in Africa Evaluation methodologies
The EGM in Figure 3 shows examples of methods that the reviewed literature classifies into the five approaches highlighted in the MAE framework. Firstly, while the least indigenous approach predominantly involves the translation of data collection instruments into the local African languages, some evaluators also do make an effort at indigenising the methods of data collection. One study cited in the literature utilised this approach by incorporating participatory data collection tools such as dream mapping, community meetings, storytelling, campfires, etc., in order to solicit information from the community about the kind of intervention they needed (Omosa 2019). Secondly, many of the methodologies identified in the reviewed literature are classified within the adaptive or integrative evaluation approach. The review suggests the various ways in which techniques such as realist evaluation, participatory evaluation, theory-based evaluation, etc., can be adapted to better suit the African contexts in which they are utilised. Other methods viewed as adaptive include indigenous knowledge research, model client ECB approach, the APRM, community talk to reach consensus, the African evaluation guidelines, contextual analysis, orality approach, empowerment evaluation approach, developmental evaluation approach, the Africa-rooted evaluation tree and African feminist research. Some cited studies have adapted the participatory evaluation method in various ways, including by combining it with ubuntu-centred methods or participatory mixed methods evaluation. Thirdly, emancipatory research and human rights-based evaluation have also been adapted to fit the African contexts in which the evaluations are conducted. Fourthly, the African relational-based evaluation approach is uniquely African, and a significant number of evaluation methods have also been identified under this approach by some of reviewed literature. Examples of the methods under this approach include the relational ethics approach to evaluation, ubuntu-based research, ethno-philosophy, proverb-based evaluation, African-relational based evaluation method, Africa-rooted public policy evaluation approach, affect–symbolic–imagery method, mixed methods evaluation and transformative participatory mixed methods. The latter two methods do not originate from Africa; however, they suit the philosophy of the approach.
Fifthly, the EGM shows examples of methods that are classified under the predominantly indigenous approach. These include the Afrocentric method; indigenous knowledge systems research; the citizen-based approach; barazas; the African ethical protocol; the four Rs framework to ECD: rebirth, restoration, reclamation and responsibilities of the evaluation function; the Mo Ebrahim African governance index; the people-first impact method (P-FiM); the African sustainability barometer; Afrikology; the lekgotla; the African health research framework; the ideal community development evaluation framework; and the participatory rural appraisal (PRA).
Lastly, there are some methodologies that are classified under the third space (geocentric) evaluation approach, including the NICE – navigating the intersection of culture and evaluation framework; the T–R–T ECB method (technical–relational–transformational evaluation capacity-building, imparting skills needed for evaluation practice in complex contexts such as Africa); and the action training model (ATM).
Gaps
Firstly, while what has emerged and is being achieved is commendable, one of the biggest gaps relates to the richness of African cultures and languages. Literature indicates that there exist over 2000 cultures and 3000 languages in Africa (Apahou 2019; Michelitsch 2019); however, the description and understanding of the ways of knowing and views of reality are overly limited and cannot be representative of the entire continent. It may be suggested that further research is required to gain better understanding of Africa’s cultures, languages, histories and people, perhaps through ethnographic studies or the review of previous ethnographic literature on Africa. Secondly, it is understood that MAE relates to more than simply the development of new ontologies and epistemologies; however, the ways in which data are collected are Africa-led. This means either having evaluations led by African evaluators or ensuring the inclusion of beneficiaries in the evaluation process. It is the latter that has meant that the inclusion of other non-Western contexts approaches and paradigms, because of their applicability, has been deemed fit to be included in the MAE narrative. However, the issue is that if these do not originate from Africa, then should they be included. Thirdly, it is also important to question the inclusion of Western approaches and methodologies in the MAE framework, which has as its purpose the decolonisation of evaluative thinking and practice. The latter may reflect a lack of confidence in the buy-in for the use of MAE paradigms and approaches alone.
Very few studies in this literature use only MAE approaches and methodologies. Finally, another major critique emerging from this study is that within the African relational-based evaluation approach especially, a lot of the focus is on explaining the various concepts and philosophies, etc., underlying the evaluation methodologies created; however, it is unlikely that most of these can be replicated because there are no clear, practical steps given for each of these newly developed African methods. This is a serious limitation of some of the MAE approaches. This should be rectified, if MAE is expected to become more than simply a discourse or discussion. Currently, much of what is discussed on MAE paradigm and approaches is more theoretical, and better efforts need to be made to provide practical steps and guides on how MAE can be practiced in order to ensure its inclusion in curricula and trainings. It is only then that MAE will be perceived seriously by practitioners and governments on the African continent. It is only when it is laid out practically that we truly see whether MAE is truly able to achieve the effectiveness and results expected from evaluation in Africa.
Conclusion
This review shows that much has been performed to define MAE in terms of the various paradigms and their associated ontologies, epistemologies and axiologies. Much has also been carried out to categorise the different understandings of what constitutes MAE approaches, including those that are dominated by Western practice, theories and methodologies; those that are a combination of both African and Western thought and frameworks; those that are uniquely African in thought, philosophy, ontology, epistemology and axiology; those that are not only predominately Africa-rooted and Africa-led but also have some elements of Western practice; and finally, those that are more geographic in nature, referring to the third world and complex contexts such as Africa but not exclusive to this continent. There are also a vast number of methodologies that evaluators, researchers and ECB trainers have used within each of these approaches, each within a different MAE paradigm. While much has been achieved in terms of the ways in which MAE can be performed, the EGM shows that there are gaps that still need to be filled.
Acknowledgements
The author acknowledges the Centre for Learning on Evaluation and Results - Anglophone Africa (CLEAR-AA) staff who contributed to the editing and reviewing the draft manuscript.
Competing interests
The author declares that he has no financial or personal relationships that may have inappropriately influenced him in writing this article.
Authors’ contributions
T.F. is the sole author for this article.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
This study only utilised published journal articles, book chapters and grey literature. All secondary data used in this article are available upon request.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
AfrEA, 2020, Made in Africa evaluation, viewed 31 January 2022, from https://afrea.org/made-in-africa-evaluation/.
Apahou, Y.R.M., 2019, Orality and participatory approach for a ‘Made in Africa’ evaluation, eVALUation Matters Fourth Quarter 2019, viewed from http://idev.afdb.org/sites/default/files/Evaluations/2020-03/article-3%20Orality%20and%20participatory%20approach%20for%20a%20Made%20in%20Africa%20evaluation.pdf.
Asante, M.K., 1983, ‘The ideological significance of Afrocentricity in intercultural communication’, Journal of black studies 14(1), 3–19. https://doi.org/10.1177/002193478301400101
Botcheva, L., Shih, J. & Huffman, L.C., 2009, ‘Emphasizing cultural competence in evaluation: A process-oriented approach’, American Journal of Evaluation 30(2), 176–188. https://doi.org/10.1177/1098214009334363
Chilisa, B. & Malunga, C., 2012, ‘Made in Africa evaluation: Uncovering African roots in evaluation theory and practice’, in African Thought Leaders Forum on Evaluation for Development Expanding Thought Leadership in Africa, Paper presentation, The Bellagio Centre, Bellagio, November 14–17, 2012.
Chilisa, B., 2015, A synthesis paper on the Made in Africa evaluation concept commissioned by African Evaluation Association (AfrEA), viewed 26 January 2022, from https://afrea.org/wp-content/uploads/2018/02/MAE-Chilisa-paper-2015-docx.pdf.
Chilisa, B., 2017, ‘Decolonising transdisciplinary research approaches: an African perspective for enhancing knowledge integration in sustainability science’, Sustainability Science 12(5), 813–827. https://doi.org/10.1007/s11625-017-0461-1
Chilisa, B., Major, T.E., Gaotlhobogwe, M. & Mokgolodi, H., 2015, ‘Decolonizing and indigenizing evaluation practice in Africa – Toward African relational evaluation approaches’, Canadian Journal of Program Evaluation 30(3), 313–328. https://doi.org/10.3138/cjpe.30.3.05
Chilisa, B., Major, T.E. & Khudu-Petersen, K., 2017, ‘Community engagement with a postcolonial, African-based relational paradigm’, Qualitative Research 17(3), 326–339. https://doi.org/10.1177/1468794117696176
Chilisa, B. & Mertens, D.M., 2021, ‘Indigenous Made in Africa evaluation frameworks: Addressing epistemic violence and contributing to social transformation’, American Journal of Evaluation 42(2), 241–253. https://doi.org/10.1177/1098214020948601
Chilisa, B. & Tsheko, G.N., 2014, ‘Mixed methods in indigenous research: Building relationships for sustainable intervention outcomes’, Journal of Mixed Methods Research 8(3), 222–233. https://doi.org/10.1177/1558689814527878
Chouinard, J.A. & Cousins, J.B., 2013, ‘Participatory evaluation for development: Examining research-based knowledge from within the African context’, African Evaluation Journal 1(1), 9. https://doi.org/10.4102/aej.v1i1.43
Cloete, F., 2016, Developing an Africa-rooted programme evaluation approach, viewed 09 October 2021, from https://repository.up.ac.za/bitstream/handle/2263/59022/Cloete_Developing_2016.pdf?sequence=1.
Cloete, F. & Auriacombe, C., 2019, ‘Revisiting decoloniality for more effective research and evaluation’, African Evaluation Journal 7(1), 1–10, viewed 21 January 2022, from https://journals.co.za/doi/pdf/10.4102/aej.v7i1.363?casa_token=l0D6WQ4S36AAAAAA%3AM0JvUvEaKufCUxLD4MoUo8BSxgUKmufFWiJZgbyfwNE4aqsxwdPeTr3REVZNhhFN8QSFGeTTHmdSdNiK.
Cooke, A., Smith, D. & Booth, A., 2012, ‘Beyond PICO: The SPIDER tool for qualitative evidence synthesis’, Qualitative Health Research 22(10), 1435–1443. https://doi.org/10.1177/1049732312452938
Easton, P.B., 2012, ‘Identifying the evaluative impulse in local culture: Insights from West African proverbs’, American Journal of Evaluation 33(4), 515–531, viewed 08 October 2021, from https://journals.sagepub.com/doi/pdf/10.1177/1098214012447581?casa_token=E-1-pz4i5T0AAAAA:4GrAdE1wMB40FWXFcmOiBqo5XyAtr5EEcXzcbELW6yWHRIRDE8o58cebuWB1lGlYyPaP8jcv1kPKCT4.
Edmonds, D., 2021, Social science bites: Molefi Kete Asante on Afrocentrism, viewed 26 June 2022, from https://socialsciencebites.libsyn.com/molefi-kete-asante-on-afrocentrism.
Gaotlhobogwe, M., Major, T.E., Koloi-Keaikitse, S. & Chilisa, B., 2018, ‘Conceptualizing evaluation in African contexts’, Indigenous Evaluation: New Directions for Evaluation 2018(159), 47–62. https://doi.org/10.1002/ev.20332
Matsiliza, N., 2012, ‘Participatory monitoring and evaluation: Reviewing an inclusive approach in the south africa’s government wide monitoring and evaluation’, Africa’s Public Service Delivery & Performance Review 1(2), 67–83. https://doi.org/10.4102/apsdpr.v1i2.31
Mbava, P.N., 2017, ‘The potential value of the realist evaluation method in programme impact evaluations in South Africa’, Doctoral dissertation, Stellenbosch University, Stellenbosch.
Mbava, N.P., 2019, Shifting the status quo: Africa influencing global evaluation practice, eVALUation Matters Third Quarter 2019, viewed 16 November 2021, from http://idev.afdb.org/sites/default/files/Evaluations/2020-03/article%202%20%20Africa%20influencing%20global%20evaluation.pdf.
Mbava, N.P. & Chapman, S., 2020, ‘Adapting realist evaluation for Made in Africa evaluation criteria’, African Evaluation Journal 8(1), 11. https://doi.org/10.4102/aej.v8i1.508
Mertens, D.M., 2008, Transformative research and evaluation, Guilford Press, New York, NY.
Miake-Lye, I.M., Hempel, S., Shanman, R. & Shekelle, P.G., 2016, ‘What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products’, Systematic Reviews 5(1), 1–21. https://doi.org/10.1186/s13643-016-0204-x
Michelitsch, R., 2019, ‘From the Evaluator General’s Desk’, Evaluation matters: Aquarterly knowledge publication on development evaluation fourth quarter 2019. Made in africa evaluations volume 2: Practical applications, viewed 16 November 2021, from http://idev.afdb.org/sites/default/files/Evaluations/2020-03/EVALUation%20Matters%20Q4-2019-EN.pdf.
Mertens, D.M. & Musyoka, M., 2007, ‘Making research and evaluation our own: ethical implications for strengthening the foundations for Africa-rooted and Africa-led inquiry’, Mosenodi 15(1), 5–16, viewed 08 October 2021, from https://journals.co.za/doi/pdf/10.10520/AJA1021559X_17.
Mazama, A., 2001, ‘The Afrocentric paradigm: Contours and definitions’, Journal of black studies 31(4), 387–405. https://doi.org/10.1177/002193470103100401
Omosa, O.O., 2019, ‘Towards defining “Made in Africa evaluation”’, Doctoral dissertation, Virginia Tech, viewed 03 November 2021, from https://vtechworks.lib.vt.edu/bitstream/handle/10919/88834/Omosa_OO_D_2019.pdf?sequence=1.
Snilstveit, B., Vojtkova, M., Bhavsar, A. & Gaarder, M., 2013, Evidence gap maps-a tool for promoting evidence-informed policy and prioritizing future research, World Bank Policy Research Working Paper, (6725).
Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J. & Gaarder, M., 2016, ‘Evidence & gap maps: A tool for promoting evidence informed policy and strategic research agendas’, Journal of Clinical Epidemiology 79, 120–129. https://doi.org/10.1016/j.jclinepi.2016.05.015
Twende Mbele, 2021, What is meant by transforming evaluations for Africa?, viewed 18 November 2021, from https://twendembele.org/reports/what-is-meant-by-transforming-evaluations-for-africa/.
Uwizeyimana, D.E., 2021, ‘The need and feasibility of a separate Africa-rooted programme evaluation approach’, African Journal of Development Studies (formerly AFFRIKA Journal of Politics, Economics and Society 11(3), 101–120.
Warren, K., 2020, Qualitative data analysis methods 101: The “big 6” methods + examples, viewed 22 February 2022, from https://gradcoach.com/qualitative-data-analysis-methods.
White, H., Albers, B., Gaarder, M., Kornør, H., Littell, J., Marshall, Z. et al., 2020, ‘Guidance for producing a Campbell evidence and gap map’, Campbell Systematic Reviews 16(4), e1125. https://doi.org/10.1002/cl2.1125
Winters, N., Langer, L. & Geniets, A., 2017, ‘Physical, psychological, sexual, and systemic abuse of children with disabilities in East Africa: Mapping the evidence’, PLoS One 12(9), e0184541. https://doi.org/10.1371/journal.pone.0184541
|
Hide
Show all
Email this article (Login required)
