The Made in Africa Evaluation (MAE) agenda is an effort at decolonising and indigenising evaluation practice in Africa. This involves developing new evaluation practices, theories, approaches and methodologies originating from African cultures, worldviews, knowledge systems, philosophies and African paradigms.
This study aims to explore achievements in the development of an African evaluation paradigm or approach and identifying the gaps.
An evidence gap map (EGM) was utilised because it entails a systematic search of literature with the intention of presenting the existing evidence on a specific topic, identifying gaps in knowledge and determining future research needs.
Five main paradigms that fall within the MAE were identified, including the Afrocentric paradigm, the postcolonial indigenous paradigm, the African relational evaluation paradigm, the transformative evaluation paradigm and the culturally competent evaluation paradigm. In addition, five key categories of MAE approaches were identified, including (1) the least indigenised approach; (2) the adaptive or integrative evaluation approach; (3) the African relational-based evaluation approach; (4) the predominantly indigenous approach and (5) the third space (geocentric) evaluation approach.
African contexts, cultures and people have been discussed in five ways in MAE literature including collaborative, relational, complex, previously colonised and disenfranchised. The EGM shows that these paradigms and approaches have been well established theoretically and conceptually; however, the application of these in evaluations has mostly been found in the adaptive or integrative approach, which integrates Western methodologies. This suggests a lack of practical guidelines to using the paradigms, approaches and methods originating from Africa, including the African relational-based evaluation approach.
The Made in Africa Evaluation (MAE) agenda is an effort at decolonising and indigenising monitoring and evaluation (M&E) practice on the African continent (Chilisa
Chilisa et al. (2016:316) defined indigenisation as ‘the blending of an imported discipline with the generation of new concepts and approaches from within a culture’. Therefore, it is posited that the indigenisation of evaluation in Africa can be measured by the extent to which the evaluation process is centred around African people; the extent to which evaluation outcomes, standards, tools and methodologies originate from African realities and worldviews; and the extent to which the ways of knowing and associated values are considered valuable by African stakeholders and beneficiaries (Chilisa et al. 2016). ‘A decolonised MAE approach is thus African-people centred, values culturally relevant and indigenised evaluation processes and methodologies predominantly informed by African worldviews and paradigms’ (Chilisa
Paradigms inform the approaches and methods, research processes, priorities, choices, actions and dissemination of evaluation findings (Chilisa & Mertens
Evidence gap maps (EGMs) are a systematic evidence synthesis methodology that present the available evidence relevant to a specific research question or a particular topic. In addition, EGMs are used to identify gaps that need to be filled with new evidence and collections of studies and increase the discoverability and use of studies by decision-makers, research commissioners and researchers (White et al.
[
Using an EGM methodology, this study aims to explore achievements in the development of an African evaluation paradigm or approach and identify where the gaps are in decolonising and indigenising evaluation practice in Africa.
An EGM is a systematic search of a wide array of literature in a particular field with the intention of presenting the existing evidence on a specific topic, identifying gaps in knowledge and determining future research needs (Miake-Lye et al.
For the purposes of this study, discourse analysis was utilised as the analytical framework for assessing relevant literature included in the evidence map and identifying gaps. Discourse analysis was perceived as the most appropriate and useful technique, given its emphasis on the analysis of language within its social and culture context. This analytical method is useful if the aim of the study entails understanding culture. It can assist with identifying how culture has an effect on the way concepts are spoken about (Warren
This section provides the step-by-step process taken to developing the EGM.
The initial step of the study entailed setting the scope of the EGM by defining the mapping framework. The mapping framework is a matrix that generally represents two or more variables that are important to the particular field or topic under exploration. Most EGMs plot evidence on interventions and outcomes in a mapping framework. Given that the focus of this study was the methodological scope of MAE approaches, the columns and rows of the mapping framework (see
In addition, the mapping framework was used to identify a set of substantive inclusion criteria for the EGM. The focus of the EGM generally requires additional inclusion criteria that will help set the limits of the scope. The most commonly used model to define the inclusion criteria for systematic reviews including evidence maps, is the population, intervention, comparison and outcomes (PICO). However, the PICO model was not adequate because it is typically used for quantitative and empirical research, with mapping frameworks focused on intervention or outcome configurations (Snilstveit et al.
These documents were published between 2007 and 2021, as the discourse on the decolonisation of evaluation and research predates the Bellagio Conference in 2012. While the concept of MAE is focused on the African continent, publications on the topic have been produced by evaluators, researchers, scholars and academics, students, evaluation and research organisations, VOPEs, etc., across the globe. Therefore, there were no limits in terms of the geographic scope of the literature search.
In this study, the search for evidence was conducted in a systematic manner. The search to identify relevant literature to include in the EGM was targeted at all relevant academic databases and journals, as well as grey literature sources. These sources are listed in
List of academic databases, journals and organisational websites searched.
Academic databases and journals | Organisational websites |
---|---|
Web of Science (social science citation index, emerging sources citation index) | African evaluation database (AfrED) |
ERIC, Academic Search Complete, AfricaWide, Education Research Complete (EBSCO) | African Evaluation Association (AfrEA) and all the African nationals whose websites are provided |
Psychinfo | Independent Development Evaluation (IDEV) on the African Development Bank (AFDB) website |
Sabinet (ISAP, SA ePublication, theses and dissertations) | Centers for Learning on Evaluation and Results (CLEAR) Initiative website |
Proquest (social science, education) | Twende Mbele website |
Taylor & Francis | BetterEvaluation website |
Sage | University of the Witwatersrand website |
ResearchGate | University of Johannesburg website |
The Directory of Open Access Journals | Evaluation for Africa |
African Evaluation Journal | Khulisa website |
American Journal of Evaluation | 3ie website |
Canadian Journal of Program Evaluation | Evaluation for Transformation website |
Evaluation and Program Planning | Education Partnerships Group website |
African Journal of Development Studies | EvalPartners website |
Qualitative Research Journal | International Organization for Cooperation in Evaluation (IOCE) website |
African Journal of Public Affairs | UNDP website |
Discourse: Studies in the Cultural Politics of Education | Google Scholar and Google |
Administratio Publica | - |
EBSCO, Elton B. Stephens Company; ERIC, Education Resources Information Center; UNDP, United Nations Development Program; ISAP, Index to South African Periodicals.
The search terms used when navigating these databases, online journals and websites were ‘MAE’, ‘indigenous evaluation methods’, ‘African approaches to evaluation’, ‘Africa-led evaluation’, ‘Africa-centric evaluation’, ‘Africa-rooted evaluation’, ‘African philosophy and evaluation’ and ‘evaluation in Africa’. The search process was conducted between 02 November 2021 until 18 November 2021, taking approximately two and a half weeks.
The search showed that the literature on MAE is extremely limited, as the search identified only 95 citations or documents (50 journal articles and 45 grey literature documents). The screening of the literature entailed reviewing all the abstracts and examining them against the inclusion criteria. Once potentially relevant documents were identified, the screening of the full texts took place. In the first screening of the journal article abstracts, 26 articles were identified as relevant, while 24 peer-reviewed articles did not make it into the evidence map. In the screening of the grey literature, 21 documents were identified as relevant, whereas 24 documents were not included.
Evidence gap maps do not entail summarising the findings of the selected literature or studies, but instead they are more focused on providing descriptive information on various key aspects of that literature. The data extraction process is carried out by using a structured coding tool (see
Coding tool.
As a result of the limited scope of the MAE literature, therefore only one reviewer utilised the coding tool to transparently and systematically explore what has been achieved in the development of MAE approaches and methodologies. The researcher read all the included journal articles and grey literature and extracted the data for the rows and columns of the EGM, which are ‘paradigm’ and ‘approach’. In addition, the two key constructs of the EGM were broken down further to facilitate use of the SPIDER model. For example, the review of the literature also involved assessing information on the ontologies and epistemologies of the paradigms deemed fit for MAE.
Step 4 also involved critical appraisal of the selected literature. This process also involved the use of a tool. The appraisal process in this study focused on assessing the relevance of the selected literature to the research question. In examining the relevance of the selected literature, the aim was to determine the relevance of the literature to the EGM’s objective and research question. The results section will provide information the extent of the sample and the literature remaining following the inclusion and exclusion process.
This step of developing the EGM involved organising, presenting and analysing the mapped evidence base. In this study, the EGM was created on Microsoft Excel. The visualisation ensures the enhanced use of the evidence base. The EGM was populated with data extracted from the accepted and included literature. There were previous versions of the EGM that included the various methods that fall within the various paradigms and MAE approaches. The final version, however, only indicates the number of documents that discuss the evaluation paradigms and MAE approaches (see
This article followed all ethical standards of research without direct contact with human or animal subjects.
Overview of searching and screening.
The final EGM presents the results that were extracted from 47 documents, some of which reported on more than one paradigm or approach (see the EGM in
Visualisation of the evidence gap map.
Of the total number of the peer-reviewed literature that was included in the evidence map (
Peer-reviewed literature sources.
Source | Number of documents accessed |
---|---|
Online Academic Journals | - |
African Evaluation Journal | 6 |
American Journal of Evaluation | 4 |
Canadian Journal of Program Evaluation | 2 |
Sustainability Science | 1 |
Qualitative Research | 1 |
Educational Research for Social Change | 1 |
Journals | 1 |
Journal of Mixed Methods Research | 1 |
African Journal of Development Studies | 1 |
Discourse: Studies in the Cultural Politics of Education | 1 |
The Qualitative Report | 1 |
Journal of African Foreign Affairs | 1 |
Journal of Black Studies | 1 |
EBSCO Publishing: eBook Collection (EBSCOhost) | 1 |
Wiley Online Library | 1 |
Springer Publishing Company | 1 |
UNESCO Institute for Education | 1 |
EBSCO, Elton B. Stephens Company; UNESCO, United Nations Educational, Scientific and Cultural Organization.
Of the total number of grey literature documents that have been found relevant for the evidence map, 8 magazine articles were published in the Evaluation Matters Magazine of the African Development Bank (AfDB), 2 are organisational reports commissioned by the African Evaluation Association (AfrEA) and the UN Food and Agriculture Office (FAO) Regional Office in Nairobi, 3 blogs were taken from the Evaluation from Transformation website and 3 blogs taken from the Evaluation for Africa website, 1 is an ethical protocol published by EvalIndigenous found on the EvalPartners website, 1 is a learning brief published by Twende Mbele, 1 is a PhD dissertation, 1 is a conference paper presented at the American Evaluation Association, and 1 is African Evaluation Guidelines developed by AfrEA (see
Grey literature sources.
Type of document | Source | Number |
---|---|---|
Magazine articles | Evaluation Matters Magazine of the African Development Bank (AfDB) | 8 |
Organisational reports | African Evaluation Association (AfrEA) and the UN Food and Agriculture Office (FAO) Regional Office in Nairobi | 2 |
Blogs | Evaluation from Transformation and Evaluation for Africa | 5 |
Ethical protocol | EvalIndigenous found on the EvalPartners website | 1 |
Learning brief | Twende Mbele | 1 |
PhD dissertation | Stellenbosch University and Virginia Polytechnic Institute and State University | 2 |
Conference paper | American Evaluation Association | 1 |
African evaluation guidelines | AfrEA | 1 |
PhD, Doctor of Philosophy.
The reviewed literature also identifies several categories of evaluation approaches that fall within the MAE scope. While the literature suggests that the conceptualisation of these categories was developed by Chilisa (
A review of the literature in the EGM shows that these paradigms and approaches have been theoretically and conceptually well established; however, the applications of these in evaluations have mostly been found in the adaptive or integrative approach. This is an important finding because, firstly, it suggests that there is a lack of practical steps and guidelines to using these paradigms, approaches and methods. Secondly, this suggests a lack of buy-in in predominantly and completely African-rooted evaluation paradigms, approaches and methods.
The analysis of the relevant literature reveals that there are five ways in which African contexts, cultures and people have been discussed in the MAE literature reviewed:
African cultures (including worldviews, philosophies, values, practices, etc.) are
African cultures (including worldviews, philosophies, values, practices, etc.) are
African contexts are
African contexts were previously
African people have been
The paradigms and approaches identified within the reviewed literature are aligned to these views of African contexts, cultures and people.
During the review of the sampled literature, five paradigms emerged as relevant to MAE practice.
The first paradigm identified in the literature reviewed stems from various AfrEA conferences between 2007 and 2013 is the ‘Afrocentric paradigm’, which puts the African view of reality, African ways of knowing and value systems on the same level as mainstream and other scholarly inquiry (Chilisa,
The second paradigm identified in the MAE literature is the ‘postcolonial indigenous paradigm’. It emphasises decolonisation intention as its driving axiology. Thus, it is highly critical of the dominance of Euro-Western language and thought, cultural and academic imperialism (Chilisa, Major & Khudu-Petersen
[
The third paradigm found in the MAE literature is ‘African relational evaluation paradigm’. In this paradigm, the African ontology, epistemology and axiology are relational. The relational axiology places an emphasis on values founded on cooperation, collective responsibilities, respect, interdependence and interpersonal relationships among people as the highest value. ‘African ontology is based on multiple realities. Africans believe in the living and non-living’ (Gaotlhobogwe et al.
The fourth paradigm identified is the ‘transformative evaluation paradigm’. While not uniquely African or of African origin, it has key characteristics that can be applied to the African context. This paradigm is deeply rooted in the human rights agenda. Therefore, the ethical implications include the conscious inclusion of people generally excluded from mainstream society, making it applicable to culturally complex communities. This paradigm also pays attention to power issues in the research or evaluation process, including when designing and planning, implementing and using the research or evaluation findings. The axiological assumption relates to having respect for communities that have been pushed to the margins and recognition of the resilience within community members (Chilisa & Mertens
The fifth paradigm is the ‘culturally competent evaluation paradigm’. Scholars define culturally competent evaluation:
[
Again, while having not emerged from the African continent, it does fit the brief of the MAE agenda that is in search of culturally sensitive or culturally competent programme evaluation approaches (Cloete & Auriacombe
In addition, the literature reviewed highlights five categories of African evaluation approaches in this MAE framework. Made in Africa Evaluation is pluralistic because on the one hand, it advocates for the adaptation of Western evaluation designs, methods, models, tools and strategies to make them relevant to African contexts, and on the other hand, it advocates for the development of a completely new evaluation practice, theory and methodologies derived from local African cultures, indigenous knowledge systems, African philosophies, African worldviews, African values, etc. (Chilisa & Mertens
[
[
The framework guiding this approach states that where the subject of inquiry is a local or indigenous phenomenon, then methodologies derived from African epistemologies and worldviews should be utilised. It argues that indigenous and local knowledge (ILK) should not be mainstreamed into conventional knowledge but should rather be allowed to co-exist with Western science, because they are two distinctly different systems. In the predominantly indigenous research and evaluation approach, evaluators are empowered by philosophical systems with a decolonisation intent, such as Afrikanisation and the African renaissance and draw, for example, from ethno-philosophy to inform the entire evaluation process, starting from the conceptualisation of the study to the reporting the findings (Chilisa
The EGM in
Fifthly, the EGM shows examples of methods that are classified under the predominantly indigenous approach. These include the Afrocentric method; indigenous knowledge systems research; the citizen-based approach; barazas; the African ethical protocol; the four Rs framework to ECD: rebirth, restoration, reclamation and responsibilities of the evaluation function; the Mo Ebrahim African governance index; the people-first impact method (P-FiM); the African sustainability barometer; Afrikology; the lekgotla; the African health research framework; the ideal community development evaluation framework; and the participatory rural appraisal (PRA).
Lastly, there are some methodologies that are classified under the third space (geocentric) evaluation approach, including the NICE – navigating the intersection of culture and evaluation framework; the T–R–T ECB method (technical–relational–transformational evaluation capacity-building, imparting skills needed for evaluation practice in complex contexts such as Africa); and the action training model (ATM).
Firstly, while what has emerged and is being achieved is commendable, one of the biggest gaps relates to the richness of African cultures and languages. Literature indicates that there exist over 2000 cultures and 3000 languages in Africa (Apahou
Very few studies in this literature use only MAE approaches and methodologies. Finally, another major critique emerging from this study is that within the African relational-based evaluation approach especially, a lot of the focus is on explaining the various concepts and philosophies, etc., underlying the evaluation methodologies created; however, it is unlikely that most of these can be replicated because there are no clear, practical steps given for each of these newly developed African methods. This is a serious limitation of some of the MAE approaches. This should be rectified, if MAE is expected to become more than simply a discourse or discussion. Currently, much of what is discussed on MAE paradigm and approaches is more theoretical, and better efforts need to be made to provide practical steps and guides on how MAE can be practiced in order to ensure its inclusion in curricula and trainings. It is only then that MAE will be perceived seriously by practitioners and governments on the African continent. It is only when it is laid out practically that we truly see whether MAE is truly able to achieve the effectiveness and results expected from evaluation in Africa.
This review shows that much has been performed to define MAE in terms of the various paradigms and their associated ontologies, epistemologies and axiologies. Much has also been carried out to categorise the different understandings of what constitutes MAE approaches, including those that are dominated by Western practice, theories and methodologies; those that are a combination of both African and Western thought and frameworks; those that are uniquely African in thought, philosophy, ontology, epistemology and axiology; those that are not only predominately Africa-rooted and Africa-led but also have some elements of Western practice; and finally, those that are more geographic in nature, referring to the third world and complex contexts such as Africa but not exclusive to this continent. There are also a vast number of methodologies that evaluators, researchers and ECB trainers have used within each of these approaches, each within a different MAE paradigm. While much has been achieved in terms of the ways in which MAE can be performed, the EGM shows that there are gaps that still need to be filled.
The author acknowledges the Centre for Learning on Evaluation and Results - Anglophone Africa (CLEAR-AA) staff who contributed to the editing and reviewing the draft manuscript.
The author declares that he has no financial or personal relationships that may have inappropriately influenced him in writing this article.
T.F. is the sole author for this article.
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
This study only utilised published journal articles, book chapters and grey literature. All secondary data used in this article are available upon request.
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.