This multiple methods study employed a Delphi technique, interviews, prioritisation of actionable items by the same panel of expert used for the Delphi technique, and document analysis of evaluation reports. The Delphi approach involved two rounds of online surveys and analysis of participants’ statements (Hsu & Sanford 2007). In addition to the Delphi process, our Delphi participants completed an online questionnaire to garner additional data on topics such as participants’ perception of the needed next steps in the process of developing the MAE concept that extended beyond the data collection format used in the Delphi portion of the study. Then, we interviewed two additional experts. These in-depth interviews with two additional evaluation experts strengthened our Delphi-related results. Finally, we reviewed evaluation guidance documents and reports to seek evidence of whether and how the aspects of MAE were operationalised therein. Figure 1 is a schematic diagram of the various methods used in this study and how they relate to each other.

Multiple methods generally strengthen research designs because specific strategies have both strengths and weaknesses (Brewer & Hunter 2006). Mertens (2008) argued that using multiple methods helps in developing credible and accurate measurements and can increase validity. It achieves this by triangulating sources and capitalising on the strengths of each method employed (Creamer 2017).

The Delphi technique is an iterative survey method, developed by the RAND Corporation to systematically solicit informed opinions from participants within their domain of expertise and knowledge base (Helmer-Hirschberg 1967; Hsu & Sanford 2007). More specifically, according to Hsu and Sanford, the Delphi technique ‘is a widely used and accepted method for achieving convergence of opinions concerning real-world knowledge solicited from experts within certain topic areas’ Hsu and Sanford (2007:1). To implement the Delphi method, multiple rounds of questions based on a list of statements about the topic at hand are sent to an expert panel, who rate and add to the statements. The researchers then incorporate and synthesise the first round of expert panel responses to yield new statements and initial analyses which are then returned to the experts for further input.

We selected a Delphi analysis to address our first research question for several reasons. Firstly, the iterations embedded in use of the Delphi technique made it possible to build consensus or dissensus (Hsu & Sanford 2007) amongst those we surveyed concerning the MAE concept. The method’s feedback process provided an opportunity for the experts involved in our Delphi process to reassess their initial judgements. Secondly, the approach is well-suited to gather detailed data from experts in a way that promotes their broad participation because expert respondents could be located anywhere geographically. Finally, the use of the Delphi technique as a temporally flexible approach allowed participants time for reflection concerning their responses and therefore helped to reduce pressure on them (Dalkey 1972; Hsu & Sanford 2007).

We purposively selected 17 prospective participants. We reached out to those individuals using publicly available email addresses, and seven of the 17 agreed to participate in the Delphi portion of this study. Two additional individuals agreed to an in-depth interview; their comments and insights added validity to our findings. For both the Delphi phase and the interviews, we selected from amongst a group of potential participants who met the following criteria: (1) Top management bureaucrats, who are evaluators or evaluation commissioners in African governments, multilateral intergovernmental organisations (e.g. United Nations International Children Emergency Fund [UNICEF]), NGOs, and bilateral development entities in Africa; (2) African evaluation thought leaders, based on their work with AfrEA and previous championing of MAE; or (3) Have conducted evaluation research and have written explicitly or indirectly about MAE in their publications. Additionally, we required that invited participants have had at least 10 years’ experience in evaluation research or practice.

We used email to invite and subsequently communicate with respondents for this online study. A follow-up email was sent at the end of five business days to remind individuals who had not replied to their initial invitation to respond. We sent that reminder twice, using a modified approach to survey reminders as proposed by Dillman, Smyth and Christian (2014). When we had recruited seven participants, enough to form our panel of experts, we sent an introduction letter, the first round of a web-based Qualtrics survey and a consent form to all who had indicated interest. We sent a reminder email at the end of five business days urging participants to complete the survey. We gave participants 10 business days to complete that effort, after which we began to analyse the data provided by the surveys. We obtained participants’ email addresses through publicly available databases, such as the Voluntary Organization for Professional Evaluators (VOPEs), other published materials, and the AfrEA website.

The first round Delphi survey provided the expert panel a list of 10 statements describing MAE. To construct those statements, we identified prominent and common concepts that previous authors had employed in the salient literature to describe MAE. We list those statements, shown as S₁ to S₁₀, and their sources in Table 1. We sought in this round for participants to rate the relative importance of the derived MAE descriptors on a scale of one (least important) to six (highly important).

In addition to completing their importance rankings for each descriptive statement, we asked each respondent to provide up to five additional descriptors that, in their view, described MAE, but that were not captured in the original 10 depictions. These additional statements were then included in subsequent rounds (indicated as B₁, B₂, etc.) to also be rated by the rest of the expert panel. The statements rated in the second round are a combination of statements where there were dissensus and additional statements suggested in round one. The second round followed the same procedure for round one.

To triangulate and augment the validity of the findings from the Delphi portion of our analysis, we interviewed two additional African evaluation experts. These participants agreed to an individual interview to share their perspective on the MAE concept, also because they were not available to participate in the full-fledged Delphi study. We conducted a semi-structured online (via video Skype) interview with each individual. Whilst a larger sample of interviewees would have added still further nuance to the study, we appreciate the triangulation and thick description provided by even these two in-depth interviews.

We transcribed the two audio recordings in Microsoft Word 2010. Following Corbin and Strauss (2015) and Glaser and Strauss (1967), we undertook whole text analysis of our transcripts. Our participants expressed emotions and we paid attention to their tone of voice and emphasised phrases. We read each transcript twice and identified text relevant to our research questions by highlighting and noting it in the margin. Each segment of text comprised an excerpt, and excerpts consisted of one or more sentences or paragraphs. A full sentence was the smallest unit of analysis. In the data corpus, whenever two or more excerpts communicated the same information, we included only one in our analysis.

We scrutinised each excerpt and assigned one or more codes to capture its meaning. We compared and contrasted each code with the others we had assigned to identify distinctive properties. We then organised the resulting codes into a list and sought to develop categories that could encompass more than one code. Finally, we examined the contents of each category for coherence. We continued this process until we were satisfied that each of our categories was unique. The results of the interview process are presented in the Results and Discussion section, alongside the rest of the broader study’s results.

We asked our Delphi survey respondents to suggest links to reports they had written or of which they were aware that employed the MAE concept to address our second research question. This method was used to help address Research Question 2, How are MAE principles operationalised and presented in evaluation reports? However, when our participants did not suggest any reports, we purposively selected six evaluations reports from the databases of recognised evaluation funders and commissioners that potentially provided evidence of applying the MAE concept. We examined evaluation reports from the archives of the United States Agency for International Development (USAID) and the UNICEF. We used concept mapping by selecting reports that incorporated culturally responsive evaluation (CRE) and MAE concepts like indigenous evaluation methods, participatory evaluation, localised knowledge, challenging western views, and power relations. We next present and examine those reports in the light of our new working definition. For example, we selected a USAID report, ‘Evaluation of the TCE program in Mpumalanga and Limpopo’, and UNICEF report, ‘Real-time Evaluation of UNICEF Somalia Country Office (SCO) Humanitarian Response to the Pre-Famine Crisis in Somalia’ as part of the six evaluation reports.

Using a document analysis (concept mapping) approach (eds. Canas et al. 2008), we pilot tested our newly-developed MAE definition to analyse the six selected evaluation reports. We expected that some reports might address more than one central element of our working definition. We included reports as evidencing salient elements of the definition if they contained concrete examples of those issues. We highlighted specific sentences in reports that addressed elements of our draft MAE definition.

More specifically, using concept mapping, we read through each document at least two times, looking for evidence of the central themes of the MAE definition we had derived from our Delphi participants. We also took note of each report’s overall evaluation design/methods approach. For example, a central concept in the MAE definition is the promotion of African values. We placed any evaluation that suggested that it aimed to promote African values and that offered specific examples concerning how it had sought to do so, under the category ‘African Values’. The results of the document analysis process are presented in the Results and Discussion section, alongside the rest of the broader study’s results.

To address our third research question, in addition to asking our study participants to rate MAE related statements in order to enable us to develop a working definition of the construct, in a separate survey, we also asked our experts to evaluate the importance and feasibility of 12 actionable items to further develop and promote the MAE concept, as enumerated by Chilisa (2015). Because Chilisa presented these steps originally to chart a possible path forward for MAE, we used our empirical study as a way to build on and extend her 2015 work. Chilisa’s action steps are represented in Table 2 by statements W₁ to W₁₂. Note that this is not part of the Delphi technique, but rather was a separate survey administered to the same sample or experts who participated in the Delphi portion of the study.

We used Microsoft Excel to calculate the means for each of the statements concerning the criteria we asked our respondents to employ to evaluate each. We created a slope graph, which is presented in the Results and Discussion section, depicting the relationship between the assigned mean scores for the level of importance and the level of feasibility of the 12 actionable items.

Panellists rated the importance level of a total of 15 statements as a part of the Delphi process that addressed a range of issues linked to the MAE concept. In the end, the panellists ranked four statements (S₅ S_7, S_8, and B₃) as most important, as shown in Table 3. As a reminder, the ‘S’ statements were from the original Delphi round’s prompt, whilst ‘B’ statements were generated by expert participants themselves and then included in Round 2. Because our first research question was to define the MAE concept more effectively, we derived the central theme of each of the four statements and used those descriptors as codes for each (Corbin & Strauss 2015; Glaser & Strauss 1967).

Each idea presented above is considered central to each of the statements. Taken together, these animating ideas as represented in the central ideas/codes form a working definition of MAE:

Evaluation that is conducted based on AfrEA standards, using localised methods or approaches with the aim of aligning all evaluations to the lifestyles and needs of affected African peoples while also promoting African values.

This new definition aligns with Cloete and Auriacombe (2019) in their critique of Africa-rooted evaluation as a good example of decoloniality. They portrayed that Africa-rooted evaluation as an evaluation approach that promotes African practices and aligns with the African identities and conditions. Furthermore, one aspect of our new definition of MAE is using localised African approaches with the aim of aligning all evaluations to the lifestyles of the African people. This aspect further strengthens the argument for the ubuntu concept as the main fabric of MAE as elucidated by Chilisa and Malunga (2012), Cloete and Auriacombe (2019), and Uwizeyimana (2020). These authors presented ubuntu as epitomising the sense of community, collectiveness, and love amongst the African people, which is consistent with our new definition that places premium on aligning evaluation to the lifestyle and needs of African people whilst promoting African values.

Seven central themes emerged in our interviews with the two evaluation experts: (1) The importance of guidelines when conducting MAEs; (2) Importance of research concerning MAE; (3) Cultural competency and MAE; (4) Further research on localised methods; (5) The integration of international practice and AfrEA standards in MAE; (6) The relevance of CRE in MAE; and (7) The way forward for the MAE concept. For example, when asked if satisfied by the consensus definition of MAE we developed on the basis of our Delphi process respondents’ views, one interviewee emphasised the need to probe further on what constitutes ‘localised knowledge’. In the interviewee’s view, such knowledge must be incorporated in ways beyond citing and using examples of localised methods:

In short, this interviewee contended that there is a need to understand further the philosophy that undergirds localised knowledge to deepen awareness of its implications for MAE.

Furthermore, it is important to highlight that these seven themes align with the themes the Delphi panellist rated as important as discussed above. For example, one of the central themes from the interviewees is the integration of AfrEA standards in MAE. This theme aligns with conducting evaluation studies that are consistent with evaluation standards developed and used by the AfrEA. Also, the central theme that is focused on the relevance of CRE in MAE aligns with adapting evaluation work to the lifestyle and needs of the African communities, where evaluation is conducted. This alignment shows a convergence between the central themes from the interviewees and the statements rated as important to the Delphi panel.

The second research question of this study was to explore how MAE principles are operationalised and presented in evaluation reports. To illustrate the presence and distribution of each theme in each report we used a concept map, which appears as in Figure 3. The Figure suggests that the six evaluation reports align with AfrEA’s evaluation standards and guidelines, by showing evidence of being realistic, prudent, serving the needs of the intended stakeholders, and conducted legally and ethically. These reports also align with the needs of the African people by showing evidence that they are meeting the needs of the African people. Furthermore, African values were evident and promoted in reports 2, 3, 5, and 6, whilst evaluators employed localised methods in numbers 2, 3, and 6. For example, report 2 showed the evidence of employing localised methods by using focus groups that comprised traditional leaders/Indunas; traditional healers; youths, and others employed as part of the methodology. Additionally, report 2 showed evidence of promoting African values by targeting traditional leaders/Indunas and healers as part of the methodology for the evaluation.

For the third research question, the panellists considered Chilisa’s 2015 12 actionable items (represented by W_{1 -}W₁₂), which she presented as way posts for refinement of the MAE concept. However, only statements W₄ (fund research on MAE and evaluation that may be used as a test case for MAE) and W₁₀ (review AfrEA guidelines in the light of the MAE approach) stood out for our respondents. These two statements have high mean scores for both their levels of importance and feasibility. These findings appear in Table 4 and Figure 4. Statement W₄ has a mean score of 4.43 and 4.00 for the level of importance and of feasibility, respectively, whilst statement W₁₀ has the same mean score of 4.29 for both its perceived level of importance and feasibility.

Figure 4 is a slope graph that depicts the difference in mean scores that our respondents assigned for importance and feasibility for each of Chilisa’s 12 action items. In this slope graph, both the mean scores for the level of importance and the level of feasibility of statement W₁₀ are the same, with a mean of 4.29 for both variables. This is represented in the straight orange horizontal line and this high mean score shows that the panellists consider the statement as important and feasible. In statement W4, we can see a difference of 0.43 between the mean scores of the level of importance and the level of feasibility. These high mean scores, 4.43 and 4.00 for the level of importance and feasibility, respectively, show that the panellists also see the statement as important and feasible. This is represented in the yellow line (sloping down from left to right).

As shown in our new definition of MAE introduced earlier resulting from our empirical study presented in this manuscript, the recognition of AfrEA and other relevant Volunteer Organizations for Professional Evaluation (VOPE) guidelines, the use of localised knowledge and approaches, the increased consideration of the lifestyles of populations of interest, and the promotion of African values are central to the concept of MAE. Previous efforts have sought to expand the field by teaching evaluation competencies to ensure that would-be evaluators possess necessary technical skill-sets (Thomas & Madison 2010). However, beyond acquiring such competencies, our findings illuminate the need for African evaluators to become deeply aware of African philosophies and values, as revealed across the continent. These thoughts were also alluded to elsewhere. Cram (2018) opined that as a way of becoming responsive and adapting their practice to African cultures and values, evaluators must seek to acquire adequate knowledge and become more aware of African values (Cram 2018). Cram (2018) encourages partnership between evaluators and the tribal members. Cram argues that evaluators should seek advice and feedback from tribal members in the evaluation process, which will further deepen their knowledge of the African cultures and values.

Another example is the philosophy of ubuntu introduced earlier (‘I am because we are’). This philosophy is woven through the fabric of many African cultures and communities. In such communities, no single person can claim to speak for the entire community (Cloete & Auriacombe 2019; Uwizeyimana 2020). A similar philosophy is ingrained in West African culture. For example, in the Yoruba culture amongst the people of Nigeria and other West African countries, they embrace ajose and ajobi, a view that prizes collectivism and not individualism (Omosa 2016). Whilst there are many more examples, paying attention to the promotion of African values and the use of localised knowledge as illuminated by our findings appears vital in the training of young and emerging evaluators (YEEs) in Africa if the evaluators are to realise the aims of MAE.

Evaluation practitioners in Africa should be trained to prioritise the promotion of African values and increased consideration for the lifestyles of the African people in their understanding of the theory and practice of evaluation. If this type of training is encouraged, it will reduce the influence of the Eurocentric models of evaluation which continue to deny the important place of Africa’s rich history, context, and philosophy in evaluation.

Our finding from the third research question which is to review AfrEA guidelines in light of the MAE approach corroborates the need, expressed elsewhere, to review current AfrEA guidelines in the light of evolving definitions of MAE. This can potentially enhance MAE and, ultimately, yield better evaluation practice in Africa (Chilisa 2015). For continuous growth and development in any field, there is a need to revisit foundations and guidelines that constitute the field of practice and improve on them continually. The governing board of AfrEA might consider reframing AfrEA guidelines to align them with the current thinking on MAE.

As with every good nascent and emerging concept, the MAE will continue to be enriched. It will continually be shaped and framed by different perspectives and thinking so that we can start seeing changes in practice. One key finding from this study is the need for further research to operationalise localised methods and approaches. For example, what are specific examples of localised methods or approaches? What are the implications of methods involving storytelling, local courts, campfires, and proverbs? Also, what are the ways to actively represent and recognise these approaches in evaluation reports? Chilisa (2012, 2015) and Cloete (2016) have contributed much to the exploration of these terms and approaches. However, the need remains for further research along these lines.

Delphi methodology conveys important advantages, but it also has its limitations. Questions are often raised about the accepted sample size for a good Delphi study. Also, because the Delphi methodology is iterative and sequential as a result of the layered feedback process integral to the concept and use of it, some uncertainty can arise about the process when the sample size drops during the study because of participant attrition. Notably, in this study, because of personal and other issues beyond their control, two panellists had to be excused during the second round of the survey, and this reduced the number of panellists from seven to five.

However, it has been empirically established that the sample size has minimal impact on the quality of data during a Delphi study. What is most important in a Delphi study is the level of training and knowledge of panellists about the subject matter. In particular, Akins, Tolson and Cole (2005) established that response characteristics are stable for a small expert panel. In other words, there is stability in response characteristics irrespective of the sample size. One final methodological quandary related to our use of the Delphi method is that it itself is not a Made in Africa approach. It is established by Western epistemological ontological, and methodological assumptions. Yet, whilst some may find it ironic to study African methodologies using a non-African method, we maintain that the tool was appropriate for this type of study, that is, helping to arrive at expert-based consensus. Also, further support of this view is evident in the argument of Cloete (2019) in his critique of coloniality. He argued that to totally reject Eurocentric research and evaluation approaches is rigid and totally misplaced. Instead, African evaluators must acknowledge the importance and validity of western research and evaluation approaches whilst using them as a supplement to indigenous evaluation approaches.

In addition to the established findings discussed above, this study included interviews with two other stakeholders who champion MAE. These interviewees were initially scheduled to be part of the Delphi panellists but opted out because of their busy schedules. These interviews provided additional perspective on the findings from the Delphi. The participants interviewed did not only offer their understanding and definition of the concept, but they also offered a critical viewpoint of the consensus definition developed from the Delphi.

Additionally, six reports were sampled to address the second research question, which are not a comprehensive reflection of all evaluations on the continent. As such, claims about the mainstreaming of MAE concept in Africa may not be robust. However, it is sufficient to address the question because the main thrust of the question is to test-run the developed consensus definition of MAE and explore some illustrative ways in which evaluation in African aligns with the principles of MAE.