About the Author(s)


Yvonne Erasmus Email symbol
Africa Centre for Evidence, Faculty of Humanities, University of Johannesburg, Johannesburg, South Africa

Sunet Jordaan symbol
Africa Centre for Evidence, Faculty of Humanities, University of Johannesburg, Johannesburg, South Africa

Ruth Stewart symbol
Africa Centre for Evidence, Faculty of Humanities, University of Johannesburg, Johannesburg, South Africa

Citation


Erasmus, Y., Jordaan, S. & Stewart, R., 2020, ‘Scoping the impact evaluation capacity in sub-Saharan Africa’, African Evaluation Journal 8(1), a473. https://doi.org/10.4102/aej.v8i1.473

Note: Special Collection: SAMEA 7th Biennial Conference 2019.

Original Research

Scoping the impact evaluation capacity in sub-Saharan Africa

Yvonne Erasmus, Sunet Jordaan, Ruth Stewart

Received: 24 Feb. 2020; Accepted: 25 June 2020; Published: 23 Oct. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: There has long been an assumption that Africa has low levels of impact evaluation capacity and that when impact evaluations are conducted in the region, they need to be led and conducted by researchers from the North. The Africa Centre for Evidence at the University of Johannesburg conducted a scoping study on impact evaluation capacity in sub-Saharan Africa to test this assumption.

Methodology: We used a multicomponent design, which included a systematic author search, desk review, online survey (with 353 respondents) and key informant discussions.

Results: Contrary to previous assumptions, we found a large number of researchers with impact evaluation capacity across sub-Saharan Africa. We identified 490 impact evaluation publications, to which 1520 unique African researchers from 34 countries had contributed. South Africa had the most impact evaluation researchers who had published, followed by Kenya and Uganda, illustrating a concentration of capacity in Southern and Eastern Africa. Authors largely resided within schools of public health and health science faculties at universities. The study showed that modules and elements of impact evaluation training had been offered in 32 countries, indicating more training opportunities than anticipated, although formal, accredited training in impact evaluation was mostly presented outside Africa.

Conclusion: Contrary to previous assumptions, widespread capacity to conduct impact evaluations exists in sub-Saharan Africa, reducing the need for researcher capacity from the Global North to deliver impact evaluations in the region. However, our evidence suggests that capacity gaps exist in non-health sectors, creating an opportunity for further capacity support in these areas.

Keywords: Impact evaluation capacity; Sub-Saharan Africa; Evidence-informed decision making; Impact evaluation training; Impact evaluation publication; Survey; Research methods.

Introduction

As the demand for impact evaluation evidence increases across Africa, so do calls for greater capacity to undertake these technically complex evaluations. However, the discourse surrounding local African capacity in this field is mired in post-colonial deficit models of the Global South and appears to omit any consideration of ‘Made in Africa’ approaches (see Crawley 2017; Goldman, Byamugisha & Gounou et al. 2018). There is a widespread assumption that a lack of human and institutional capacity exists in sub-Saharan Africa to conduct impact evaluations (Althsuler & Staats 2019; Van den Berg, Naidoo & Tamondong 2017) and a suggestion that because much of the financial support, and therefore evaluation capacity, to conduct impact evaluations comes from the Global North,1 local capacity is not being developed (Goldman et al. 2018; Manning, Goldman & Licona 2020; World Bank 2009). Furthermore, frameworks for assessing monitoring and evaluation (M&E) capacity have largely been driven by Northern approaches and have not always been adapted sufficiently for Southern application (Blaser-Mapitsa & Khumalo 2018).

We know that there is a rise in the number of, and importance of, impact evaluations in the region (Altshuler & Staats 2019; Goldman et al. 2018). We also know that when research is conducted by ‘outsiders’, it risks being irrelevant, missing the nuances of the local context and marginalising both the local research community and those whom we hope will make use of that research in decision-making (Kalinga 2019). The literature acknowledges an increased demand for, and use of, impact evaluation results in sub-Saharan Africa, including by governments (Goldman et al. 2018; Morkel & Ramasobama 2017). Altshuler and Staats (2019) as well as Goldman et al. (2018) explain the importance of impact evaluations and specifically the use of the results of these studies for the sub-Saharan Africa policy context. These can assist with evidence-informed policy-making and other policy decisions. They also highlight that governments are reluctant to make policy decisions based on evaluations and research done by researchers from outside sub-Saharan Africa. This stresses the importance of understanding local impact evaluation capacity.

Contrary to the assumption about the absence of capacity (Altshuler & Staats 2019), the literature also highlights high-profile examples of impact evaluations conducted in the region. Examples include the Mwanza study in Tanzania that tested the link between improving treatment of sexually transmitted infections and reductions in human immunodeficiency virus (HIV) incidence (Grosskurth et al. 1995) and the REVAMP HIV resistance clinical trial (Siedner et al. 2017). In addition, the Abdul Latif Jameel Poverty Action Lab (J-PAL) has conducted over 300 randomised evaluations across Africa, resulting in substantial developmental and policy impact. This includes their randomised control trials and impact evaluations on the Teaching at the Right Level (TaRL) in India, which led to scaling this work to various schools in Africa (J-PAL 2017).

Based on our previous work on mapping evidence capacities in Africa (Stewart, Erasmus & the UJ-BCURE team 2017a; Stewart, Nduku & Langer 2017b), we suspected that many of the assumptions surrounding the discourse of impact evaluation capacity were unfounded. This includes the assumption of a deficit in impact evaluation capacity in sub-Saharan Africa (Altshuler & Staats 2019). Like many others in the Global South, we were keen to challenge the deficit models that continue to assume we are merely the beneficiaries of investments and that these investments will develop our capacity (Stewart et al. 2019). However, we wanted to ensure that our assertions were based on more than assumptions. We therefore set out to research the scope and depth of capacity to produce impact evaluations in sub-Saharan Africa.

In this particular study, we defined impact evaluation as (Altshuler & Staats 2019):

A type of evaluation design that assesses the changes that can be attributed to a particular intervention. It is based on models of cause and effect and requires a credible counterfactual (sometimes referred to as a control group or comparison group) to control for factors other than the intervention that might account for the observed change. (p. 1)

This article reports the findings of our stock taking of impact evaluation capacity in sub-Saharan Africa. Specifically, our research project focused on answering four questions: (1) Who in sub-Saharan Africa has published impact evaluations? (2) In addition to those who have published, who else might have capacity to conduct impact evaluations? (3) Where do African researchers receive training on impact evaluations? (4) What do these findings tell us about where there appears to be the most evidence of impact evaluation capacity?

Research methods and design

The project was supported by an advisory group who gave input into the scope and methodology of the project, as well as specific elements of the study, as requested.

In line with the objectives of the study, namely to scope the field, we drew on a variety of methods to assess the impact evaluation capacity from different angles. For example, we drew on key informant interviews, a desktop review, an online survey and an academic search to identify authors who have published impact evaluations. Each method in turn informed and complemented the next; for example the author search of who had published impact evaluations and the desktop search on who had provided impact evaluation training in part contributed to the distribution list for the online survey. The following subsection discusses each of these methods in turn.

Data collection methods

Data collection took place between July 2018 and May 2019.

Semi-structured key stakeholder discussions

A sampling frame was drawn up in preparation for this stage. We aimed to have discussions with individuals at organisations conducting impact evaluations both inside and outside of Africa, as well as different types of institutions (e.g. universities, research institutes and networks). We used a combination of purposive sampling (where we identified key stakeholders in the field beforehand) and a snowballing approach (where we followed leads suggested by those contacted) (Chambliss & Schutt 2006). In 2018, we conducted 12 semi-structured discussions over email or Skype with 14 key stakeholders about the current state of impact evaluation in Africa to (1) understand where current capacity to produce impact evaluations in Africa sits, what the gaps in capacity are and what the challenges are in filling those gaps and (2) identify individuals and organisations to include in other research activities that formed part of this project.

Desk research on available impact evaluation training courses

At the end of 2018 and early in 2019, we conducted desk research to collect data on training courses, conferences and workshops relevant to the field of impact evaluation. This included online training, formal qualifications, short courses, workshops and conferences. The purpose was to identify what training courses and providers are available in addition to where those surveyed had received training. This could help to match the demand for capacity building with the supply of courses and material that is already available. The team also captured what additional support is given to trainees, such as mentorship and coaching, where this information was available. However, a limitation of a desk review is that it did not capture training opportunities that were in the process of being developed and that institutional websites were not always up to date.

Online survey

Early in 2019, we conducted an online survey to capture information about existing African impact evaluations, training and capacity. We drew on various networks in the distribution of the survey, such as the Africa Evidence Network (AEN), Partnership for Economic Policy (PEP) Africa mailing list, International Development Research Centre (IDRC) think tank list, Network of Impact Evaluation Researchers in Africa (NIERA)/Center for Effective Global Action (CEGA), East Africa Social Science Translation (EASST) network, PEP Policy Impact Evaluation Research Initiative (PIERI) Africa researchers, International Initiative for Impact Evaluation (3ie) researchers who opted in to participate, NIERA/CEGA alumni and the South African Monitoring and Evaluation Association (SAMEA) Listserv. In addition, we also drew on other stages of the research process, such as the author search and training desk review discussed below, to identify additional individuals and organisations to distribute the survey to.

Follow-up interviews

Six follow-up interviews were conducted after completion of the online survey. These were with a subsample of survey participants and other organisations identified by the funder as priority from different stages of the research process. During this process, we aimed to better understand (1) what helped build existing impact evaluation capacity and (2) what opportunities exist to strengthen capacity. All but one of these interviews were with individuals at senior management level at university departments, research institutes or think tanks in East Africa, including in Uganda, Ethiopia and Kenya. An equivalent interview was conducted with a research institute in Benin, West Africa. All interviews were conducted in English.

Impact evaluation author search

We aimed to identify authors of published impact evaluations who were African citizens and currently residing in Africa. This served as an additional channel through which to identify existing capacity that might not be picked up in other phases of the project. In order to identify authors, at the end of 2019 we searched the 3ie Impact Evaluation Repository that consists of published development impact evaluations that were identified through systematic search and screening processes of 35 databases, search engines and websites.2 The filter for publications from sub-Saharan Africa was used. We excluded articles if there were no African authors with African affiliations. In order to identify relevant authors, we used the following process and selection criteria, although we also acknowledge some subjectivity in these determinations. We first excluded authors based on institutional affiliation if the institution listed was not based in Africa. International organisations with offices in Africa were included if the author was affiliated to the local office. We then conducted additional searching (e.g. Google and LinkedIn profiles) to obtain more information on the authors. Authors were included if they were citizens of an African country and had a local affiliation. We excluded non-African authors who had dual affiliations, one of which was to an African institution, and African authors who only listed affiliations with institutions not based in Africa.

Data analysis

In accordance with the study objectives, namely to ‘scope the field’, our analysis methods were largely descriptive in nature. For example, we used thematic analysis for all our qualitative interview data – our initial semi-structured interviews as well as follow-up interviews from the survey (Ritchie, Spencer & O’Connor 2003) – looking for common themes in response to our semi-structured interview schedule.

For the academic author search, we extracted the following data from each study into an Excel spreadsheet: author name, contact details (email), institution, country, title of the article, publication date, resource type and sector. Through insights from our key informant interviews, we decided to extract information regarding author placement post hoc (i.e. was the researcher first author or non-first author). Many authors’ institutions and contact details were not present in the article publication, requiring extensive desktop searching to identify institution affiliations. We used descriptive statistics to present our quantitative findings on the authors of impact evaluations.

Similarly, the survey, conducted through a Google Form, was exported into an Excel spreadsheet where data such as country and impact evaluation experience could be easily filtered and represented as descriptive statistics. Similarly, the desktop research on training resources was analysed based on specific parameters set such as region, length of course and whether the course was accredited or not. These data were collated in an Excel spreadsheet and are publically available on the Africa Centre for Evidence website.

Ethical considerations

The study was awarded ethical clearance by the University of Johannesburg’s Faculty of Humanities Research Ethics Committee (clearance number REC-01-014-2019).

Results

The results of our study are presented to answer four questions: (1) Who in sub-Saharan Africa has published impact evaluations? (2) In addition to those who have published, who else might have the capacity to conduct impact evaluations? (3) Where do African researchers receive training on impact evaluations? (4) What do these findings tell us about where there appears to be the most evidence of impact evaluation capacity?

Capacity to author impact evaluations

The interviewees were of the view that practitioners of impact evaluations in Africa are not always incentivised to publish their results as academic papers. This view was expressed at different points in the research process, such as the initial key informant interviews and follow-up interviews from the survey. The reasons given by the interviewees included that practitioners are not necessarily based at universities and therefore are not required to publish academically for their career progression. However, the interviewees told us that the writing of academic papers is emerging as a funding requirement, which is resulting in an increased number of publications.

Our search of the 3ie Impact Evaluation Repository yielded a total of 1718 impact evaluations published on sub-Saharan Africa between 1990 and 2015. From the total, 1228 articles (71%) did not have any African authors with African affiliations. However, the remaining 490 impact evaluations yielded 1520 unique African researchers with African affiliations. Only 210 (14%) of these 1520 were first authors. From the 490 publications in question, for 68 articles (14%) all the authors were African and had African affiliations, indicating nascent local capacity.

Of the 490 publications identified, 449 (91%) were journal articles, whilst the remainder included project reports and published working papers.

The following descriptions are based on the 490 articles and 1520 researchers identified.

Geographic distribution

The 1520 authors found were based in 34 different countries, as set out in Table 1.

TABLE 1: Distribution of authors by country, including World Bank 2019 country classification by income level.

Interestingly, when the countries with the largest number of impact evaluation authors are cross-referenced with the World Bank Country classification by income level, all but South Africa are low-income or lower-middle-income countries, suggesting that the capacity to publish impact evaluations does not necessarily correlate with the income status of a country, as evidenced in Table 1.

Based on data from the 3ie Impact Evaluation Repository, South Africa has the most impact evaluation researchers (n = 307), followed by Kenya (n = 210), Uganda (n = 161), Tanzania (n = 111) and Zambia (n = 104).

As mentioned earlier, only 210 (13%) of the 1520 African authors with African affiliations identified were first authors. This illustrates the existence of capacity to lead in the design and implementation of impact evaluations and writing up the findings for academic publication. When the geographic distribution of capacity was compared to the first authors on publications, we found that East Africa (n = 89) had more first authors on publications than Southern Africa (n = 72), although the latter region had more authors on publications overall. And whilst South Africa had the most impact evaluation authors of all countries in sub-Saharan Africa (n = 307), it also had the most first authors on publications (n = 54). Whereas Uganda had the third most authors overall (n = 161), it had the second highest number of first authors (n = 31). 3

A number of researchers from West Africa were also first authors on publications (Ghana, n = 16; Nigeria, n = 14; Burkina Faso, n = 6; the Gambia, n = 4; Côte d’Ivoire, n = 1; Guinea-Bissau, n = 1; Mali, n = 1; Senegal, n = 1).

A breakdown of African regions in Table 2 shows that 75% of the authors are based in Southern Africa (n = 591) and East Africa (n = 555).

TABLE 2: Distribution of authors by region.
Capacity by institution

Our search for African authors with African affiliations yielded various African organisations that appear to have the greatest impact evaluation capacity. Table 3 shows the list of organisations that came up the most in our database search, sorted by the number of author affiliations.

TABLE 3: Breakdown of key impact evaluation organisations.

Our results indicate a large amount of research capacity resides within schools of public health and health science faculties at university institutions. Many of the university research programmes were also linked with international institutions, for example, the London School of Hygiene and Tropical Medicine (LSHTM) and Johns Hopkins University.

In particular, Makerere University in Uganda had the greatest number of researchers (n = 66) at an institution. Specifically, the research units that conducted the impact evaluations were the School of Public Health (n = 17), the School of Medicine (n = 8) and the Infectious Diseases Institute (n = 5). Similarly, the University of Zimbabwe (n = 53) also had various units with research capacity in the College of Health Sciences (n = 13), Department of Community Medicine (n = 16) and Department of Obstetrics and Gynaecology (n = 4).

Additionally, the results show that 4 of the top 10 institutions with the greatest number of researchers were from South Africa. The following units in the University of Witwatersrand produce impact evaluations: the Faculty of Health Sciences (n = 19), the Department of Science and Technology (n = 8) and the School of Public Health (n = 7). Moreover, we found that the University of Cape Town’s School of Public Health/Family Medicine and the School of Child and Adolescent Health each had seven impact evaluation researchers.

Capacity by sector

Drawing on the sector classifications in the 3ie Impact Evaluation Repository, the health, nutrition and population sector dominated with 377 articles, making up 77% of the total studies identified. The 23% remaining articles are shared amongst 10 other sectors. Agriculture and rural development came second with 49 impact evaluations. The education sector is third highest on publication numbers, recording 37 studies.

Our search of the 3ie Impact Evaluation Repository showed that there is substantial impact evaluation production capacity across the African continent – with a large concentration in the East Africa region (n = 555). Interestingly, Southern Africa’s capacity is concentrated within five countries, whilst West Africa’s production capacity is spread across 14 countries. This finding represents a robust research capacity within South Africa (n = 307) and the potential to grow researcher capability in West Africa where there is evidence of existing publication capacity.

Although South Africa had the overall largest number of impact evaluation researchers in a country, Makerere University in Uganda had the largest number of researchers at a single organisation (n = 66), denoting strong institutional capacity. Moreover, Zimbabwe had the sixth largest number of researchers country-wide, but the University of Zimbabwe had the third largest number of researchers. This finding indicates another pocket of capacity: that generally research infrastructure is clustered in universities, in particular in faculties of health, and within those, in schools of public health.

Furthermore, over 75% of all impact evaluations found were in the health, nutrition and population sector. This finding demonstrates a disproportionate amount of inquiry in this field. As evident from Figure 1, many sectors are underdeveloped in generating impact evaluations, as a number of sectors had three or fewer publications. However, this could also be linked to the amount of funding that different sectors have received to conduct impact evaluations, which is not something that this study was able to investigate.

FIGURE 1: Distribution of publications by sector.

We conducted a supplementary academic search from 2016 onwards to supplement the findings from the 3ie Impact Evaluation Repository. This search confirmed an upward trend in African authors with African affiliations being authors on impact evaluation publications. The 3ie search yielded 490 publications with 1520 unique authors between 1990 and 2015, with a steady increase per year from 2011 onwards, leading to n = 69 in 2014 and n = 72 in 2015. Our supplementary search confirmed that this steady increase continued after 2015.

Beyond analysing publications – Who else might have the capacity to conduct impact evaluations?

In addition to those who have published impact evaluations, we sought to explore who else might have the capacity to conduct impact evaluations. For this, we drew in particular on our online survey. We received 353 responses to the survey; 63% of respondents were male and 37% female. A total of 93.5% of respondents are currently based in Africa, and 92.9% are citizens of an African country. More than half (55%) are based in the academic sector, 24% work for government and 32% work for think tanks and non-governmental organisations (NGOs). Of the respondents based in Africa, 36% were from Western Africa, 33% from Eastern Africa, 23% from Southern African and 7% from Central Africa. A total of 77.3% of respondents have been involved in conducting impact evaluations, providing a substantial basis for us to draw from.

Respondents to our survey were asked to name African organisations that they think have capacity to conduct impact evaluations. A large number of organisations were mentioned by survey respondents, and although there was some overlap and triangulation with other stages in our research, it has not been possible to verify all responses to ensure that they are accurate. Table 4 summarises the responses by country. The researchers only included organisations that were mentioned by at least two respondents.

TABLE 4: Distribution of African organisations with impact evaluation capacity based on survey responses.

A total of 77.3% of respondents to the survey reported having themselves been involved in conducting impact evaluations. Of these respondents, 11% reported that they had been involved in more than 10 impact evaluations, had citizenship in an African country and worked in that country. These institutional affiliations are included in Table 5.

TABLE 5: Institutional affiliations of survey respondents who had conducted 10 or more impact evaluations.

The survey results suggest that select government departments also have experience in conducting impact evaluations. Of the survey respondents, 24% were from government departments, and 13.5% of survey respondents who conducted impact evaluations were from government departments. Respondents from the following countries’ governments reported having experience in impact evaluations: Benin, Ethiopia, Kenya, Ghana, Madagascar, Nigeria, Somalia, South Africa, Sudan, Swaziland, Tanzania, Uganda and Zimbabwe.

Various international organisations have worked with African impact evaluation practitioners based in Africa. This included organisations such as 3ie, Innovations for Poverty Action (IPA), IPE, J-PAL, RTI International, Oxford Policy Management, ITAD, FHI 360, Population Services International, PEP, the United States Agency for International Development (USAID), Department of International Development, Centre for Effective Action (Berkeley), One Acre Fund, International Food Policy Research Institute, the World Bank group, various American and European universities, Clinton Health Access Initiative, IDinsight, Consultative Group for International Agricultural Research (CGIAR) centres and the Overseas Development Institute.

Who provides training on impact evaluations?

Our survey yielded a number of important findings about where African researchers receive training, the organisations providing impact evaluation training and the nature of the training. Of the 353 respondents who completed the survey, 67.6% (n = 238) indicated that they had received impact evaluation training in the last 10 years. The majority of this training was short attendance courses4 (57.3%). Almost 40% of respondents indicated that they had received impact evaluation training at universities outside Africa, at European universities in the Netherlands, Germany, France, the United States, Canada and others. Survey respondents from Eastern Africa received most of their impact evaluation training from institutions outside Africa, and some from institutions within East Africa. Only one respondent received training in West Africa, and two in Southern Africa. In Central Africa, an equal number of respondents received training outside Africa, and within Central Africa. Two respondents received training from institutions in Eastern Africa, and three from institutions in West Africa. Within Western Africa, the trends are the same as in East Africa. The majority of respondents reported that they had received training from institutions outside Africa, limited from within West Africa, six respondents from Southern African institutions and four from East Africa. Lastly, in Southern Africa, the respondents reported equal training from institutions outside Africa and within Southern Africa, and only three respondents received training from institutions in East Africa.

Only 33% (n = 109) of survey respondents reported that their own organisations had provided impact evaluation training during the past 10 years. The majority of this was in the form of short courses. Only 18.6% (n = 62) of respondents were aware of additional support provided to practitioners of impact evaluations. These support mechanisms were limited to mentorship programmes (mostly provided by PEP, 3ie and CEGA), some financial support for conferences and workshops (such as that provided by 3ie) and network support (e.g. the Africa Evidence Network).

Our survey was supplemented by a desk review of impact evaluation training resources and yielded similar results for East, West and Southern Africa. Most accredited courses (linked to a formal diploma or degree programme) focussed on M&E in general, and not impact evaluation or impact evaluation methodology specifically. Most courses were found to have at least one module on impact evaluation. In East and West Africa, there is anecdotal evidence of master’s programmes on impact evaluation that are being developed at Sokoine University of Agriculture in Tanzania and Gaston Berger University–Saint Louis in Senegal. The M&E courses are mostly based in departments of public administration, development studies, agriculture, and health. Workshops and non-accredited short courses are presented by academic institutions, international organisations and various NGOs. Courses ranged from 2 to 10 days and are conducted at national or regional levels. A sizeable number of workshops and short courses focus specifically on the value and use of impact evaluation as well as on training in impact evaluation methodology. Few of these training opportunities are externally funded. Funders of these short courses include the Hewlett Foundation and the Bill and Melinda Gates Foundation. Some conferences feature impact evaluation training components. However, the focus is either on M&E more broadly or on link impact analysis to evidence use. M&E training courses that are donor funded are geographically focussed in Western Africa, showing that capacity building in impact evaluation is centred in West Africa. The desk review identified several training and capacity-building opportunities in West Africa that were only available in French. This includes the CLEAR–CESAG (Centre for Learning on Evaluation and Results - Centre Africain d’Etudes Supérieures en Gestion) courses such as Le Séminaire en Evaluation d’Impact en Afrique Francophone.

Overall, in two-thirds of African countries, we found evidence that impact evaluation training had taken place in some form or another, indicating more training opportunities than anticipated. The majority of impact evaluation training is presented by university departments, centres or institutions based at universities. Other major providers of impact evaluation training include AERC (African Economic Research Consortium), the EASST network, the Partnership for African Social and Governance Research (PASGR), ESRF (Economic and Social Research Foundation), and CLEAR and CESAG. A large number of international organisations provide impact evaluation training in African countries, such as IPA, J-PAL Africa, 3ie, Measure Evaluation (in association with GEM-Net Health [Global Evaluation and Monitoring Network for Health]) and PEP in conjunction with the University of Laval, amongst others. These findings are supported by the work of Manning et al. (2020).

When reflecting on the impact evaluation training offered, the interviewees were of the view that the level at which training is pitched is often a challenge. People are keen to learn but have varied degrees of knowledge and experience in M&E, which makes it difficult to determine the level at which training opportunities should be offered. Training opportunities should therefore be well targeted and differentiated and efforts should be made to enable people to attend, as training is often in a capital city, which makes transport and accommodation prohibiting factors.

A total of 13% of survey respondents expressed the need for training opportunities, capacity building and mentorship opportunities to increase their impact evaluation capacity.

Understanding capacity across different African regions

Looking across all of our data, the following discussion summarises where we found evidence of the greatest impact evaluation capacity per region, as visually represented in the map in Figure 2.

FIGURE 2: Overall evidence of greatest capacity by region.

Eastern Africa

The greatest capacity in the Eastern African region appears in four countries: Uganda, Ethiopia, Tanzania and Kenya. Institutions from these four countries featured prominently in the findings from the author search, as discussed in the previous section. These four countries also have the most confirmed impact evaluation training opportunities in Eastern Africa. For Ethiopia, four institutions appear to have the most evidence of impact evaluation production capacity: Jimma University, Haramaya University, Mekelle University and the Addis Continental Institute of Public Health. In addition, there are examples where all authors on publications from these institutions are local to these areas, potentially indicating nascent local capacity.

The Ethiopian Evaluation Association, the Ethiopian Economics Association, the Ethiopian Economic Policy Research Institute and EDRI (Ethiopian Development Research Institute) were also mentioned as organisations that have the capacity to conduct impact evaluations, although they did not feature prominently in the author searches. These institutions are think tanks that have been funded under the IDRC TTI (Think Tank Initiative).

For Kenya, five institutions appear to have the most impact evaluation publications: Kenya Medical Research Institute, University of Nairobi, Moi University, APHRC (African Population and Health Research Center)and Kenyatta University. Other institutions that were mentioned as having the capacity to conduct impact evaluations but that did not feature prominently in the author searches included the AERC and the Kenya Institute of Public Research.

Makerere University in Uganda topped the overall publication search for institutions through which authors have published impact evaluations. Other institutions in Uganda through which impact evaluations have been published include the Uganda Virus Research Institute and the Infectious Diseases Research Collaboration. Various centres and departments at Makerere University were also top of mind when respondents commented on what organisations in Uganda had the capacity to conduct impact evaluations.

Western Africa

Compared to other regions, there was less evidence of impact evaluation capacity in Western Africa, although the capacity is growing. Evidence on countries with the greatest impact evaluation capacity in Western Africa related to five countries: Benin, Burkina Faso, Côte d’Ivoire, Nigeria and Senegal.

Central Africa

The most evidence of impact evaluation capacity in Central Africa was found in Cameroon. We found evidence that individuals from Cameroon had participated in accredited impact evaluation training presented by AERC. In addition, PASGR had presented a short course that individuals from Cameroon participated in. Despite Cameroon not featuring prominently in the search for authors, a number of local institutional affiliations were noted across five different impact evaluations: University of Yaoundé I, University of Dschang, Yaoundé Central Hospital and University Center Hospital. Other institutions in Cameroon that were mentioned as having impact evaluation capacity included CEREG (Centre de Recherches en Economie et Gestion) at University of Yaounde II and CamCoSO (Cameroon Consumer Service Organization).

Southern Africa

In Southern Africa, South Africa showed the strongest capacity to conduct impact evaluations. Looking across all of the data collected, there was strong evidence of a number of organisations in South Africa having the capacity to conduct impact evaluations; named organisations were Genesis Analytics, Benita Williams and Associates, Khulisa, JET Education Services, Southern Hemisphere, Otherwise Research and Evaluation, CLEAR-AA, HEARD, Praekelt.org, University of Johannesburg, University of Stellenbosch (AERC network), the African Microeconomic Research Unit at the University of the Witwatersrand, HSRC (Human Sciences Research Council) (Research Use and Impact Assessment Unit), University of Cape Town’s Southern Africa Labour and Development Research Unit. In terms of author affiliations on published impact evaluations, various organisations stood out as having had a substantial number of authors, including the South African Medical Research Council, University of the Witwatersrand, University of Cape Town, Humans Sciences Research Council and CAPRISA (Centre for the AIDS Programme of Research in South Africa). In addition, we have found evidence of accredited training provided by the Wits School of Governance, University of Cape Town and University of Stellenbosch, whilst short courses have been presented by the Gordon Institute of Business Science, CLEAR-AA, CESAR-Africa (Centre for Statistical Analysis and Research - Africa), IQ Business, Africa Management Services Company and PASGR.

Discussion

Summary of the results

We used a multipronged research approach including key informant interviews, an online survey and follow-up interviews, desktop research on training resources and an academic search on authors who had published impact evaluations. Using a diverse strategy enabled us to capture information on impact evaluation capacity that one single method would not have allowed. Our findings confirmed that many of the impact evaluations published on sub-Saharan Africa have been conducted by researchers from the Global North – of the 1718 impact evaluations published between 1990 and 2015, 1128 articles (71%) did not have any African authors with African affiliations. However, we also found more evidence of capacity by local researchers to publish and conduct impact evaluations than has been widely assumed to exist. In terms of impact evaluation publications, we identified 490 publications to which 1520 unique African researchers with African affiliations across 34 different countries had contributed between 1990 and 2015. For 14% of these 490 publications, all of the authors were African, indicating nascent local capacity. Looking across all of the data collected, there is evidence of strong capacity in Southern and Eastern Africa and in schools of public health and faculties of health at universities, showing disproportionate capacity in this sector and less so in non-health sectors. Despite the geographic concentration of capacity in Southern and Eastern Africa identified across our different datasets, we did find 337 authors across 14 countries in Western Africa who had published impact evaluations, indicating growing capacity in this subregion. Of the 48 African countries investigated, we found evidence that impact evaluation training has been offered in 32 of these countries, indicating more impact evaluation training opportunities than anticipated. However, respondents to the survey indicated that formal, accredited training in impact evaluations is mostly presented at universities outside Africa, particularly European universities.

Limitations of the study

Our study is subject to a number of limitations. Our focus was exclusively on those who have the capacity to produce and publish impact evaluations, as well as where training had been conducted. We were not in a position to directly investigate who provides funding for impact evaluations, which would be a further important piece in understanding the political economy of impact evaluations on the continent. In addition, although interviewees spoke of the link between the demand for and supply of impact evaluations, and the challenges of both, we did not set out to directly study the demand side, but focused on supply. As we were particularly interested in understanding local capacity, we had to make some judgements about who was considered to be local. Although we used predefined criteria that we attempted to apply systematically, we have to acknowledge some level of subjectivity in these decisions. As our focus was on sub-Saharan Africa, we are unable to comment on impact evaluation capacity in North Africa, and therefore on the continent as a whole. Finally, although we triangulated our findings through various methods and sampling strategies as outlined in our methodology section, we still might not have captured all impact evaluation capacity in sub-Saharan Africa. However, even with the methods that we used to take stock of the impact evaluation capacity in sub-Saharan Africa, there is evidence of increasing capacity across countries and regions.

Practical implications and recommendations

The impact evaluation capacity we identified across sub-Saharan Africa has come about because of decades of investment in training and capacity support (Altshuler & Staats 2019). Across the breadth of experience that exists, there appears to be disproportionate capacity in specific sectors such as health and nutrition. This should perhaps not be surprising considering the history of impact evaluation methods, with the concept of a control and experimental group first introduced in the field of health dating back to 1747, with other fields to follow later (J-PAL n.d.). This history has led to considerable investment in health care research on the continent by international agencies, including USAID, the Bill and Melinda Gates Foundation and others (Tarsilla 2014).

In addition, disparities exist in impact evaluation capacity across countries. Although the reasons for this could be multifaceted, countries such as South Africa, Uganda and Benin have had specific interventions that assisted with the development of government-wide evaluation systems (Goldman et al. 2018). Within the government context, many countries experience funding limitations that inhibit how they are able to support both the development of evaluation capacity, as well the use of evaluations. This explains to some extent the disparities in government impact evaluation capacity in Africa.

African authors, because of their awareness of local context and proximity to decision-makers, are best positioned both to shape policy-relevant research questions and to support research uptake (Altshuler & Staats 2019). Funders who rely on European or North American researchers to conduct impact evaluations would do well to consider as alternative the existing pool of more local capacity (Altshuler & Staats 2019). Similarly, those funding development projects that do include impact evaluations could, indeed should, look at the large pool of local expertise to meet this need. Ample scope exists to continue to build impact evaluation capacity across sub-Saharan Africa, particularly in sectors other than health, and to draw on the substantial capacity that already exists, particularly in the health sector.

The existing capacity has in part been built through collaboration across continents and the mutual learning from these relationships, which provides an argument for them to continue. Both African and non-African researchers reported benefits of cross-continental collaboration. For African scholars, some of the reported benefits included that co-authoring with well-known scholars from other continents helped them build credibility and created opportunities for future work. Non-African researchers in turn reported that their African co-authors provided technical capacity, context to better situate the work and relationships to help drive policy impact. However, greater reflection on, and methods for, such partnerships are needed: both African and non-African scholars said that clear communication and an equitable distribution of responsibilities are essential (Altshuler & Staats 2019). There is much to learn here from the ‘Made in Africa’ movement within M&E more broadly, in which useful frameworks (Blaser Mapitsa & Khumalo 2018; Crawley 2017) and curricula (Wotela 2017) have been developed. These shifts also need to be positioned within broader movements of decoloniality, which are increasingly prominent across the continent (Mbembe 2015).

Although not explored in great detail in our study, the interviewees expressed the need for, and the potential importance of, being able to connect with other impact evaluation practitioners. The importance of communities of practice such as NIERA and AEN in connecting researchers and users of evidence should therefore be highlighted.

Finally, the study highlighted the role that language plays, particularly English and French, as an important consideration in understanding existing impact evaluation capacity and how capacity development could be further supported. Francophone universities and think tanks might be at a disadvantage in getting their work published in international publications when writing in English, whilst a similar challenge applies in identifying funders who are willing and able to accept applications in French. This gap is something for funders to be aware of in order to create opportunities for French-speaking researchers.

Conclusion

Our research has provided a detailed account, based on various data sources, of the regions and institutions in sub-Saharan Africa where there is evidence of impact evaluation capacity. The capacity to conduct impact evaluations, produce and publish them appears to be more widespread than initially thought, although this capacity is concentrated in the health sector. The study found evidence of collaboration between local African institutions and international research organisations, but also nascent local capacity where the production of impact evaluations is by local teams. As ours was a scoping study to stake stock of the existing impact evaluation capacity in sub-Saharan Africa, it was outside of our scope to investigate why capacity appears to be concentrated in certain areas and fields, and this is an important area that needs further investigation.

Our interviews indicated that there is merit in trying to connect the different institutions more actively. Impact evaluation practitioners were sometimes not aware of others doing similar work at a different institution in their country, or even at their own institution.

Bridging the language divide, particularly the access of French-speaking impact evaluation practitioners to funding opportunities, appears to be an important challenge to address in the further strengthening of capacity in West Africa.

Accessing training opportunities remains an important means through which to strengthen capacity. There is a surprising number of M&E modules included in postgraduate accredited training in Southern, Eastern and Western Africa. The inclusion of impact evaluation methodology in these courses indicates intent from within these regions to strengthen impact evaluation capacity. However, training on its own does not mean that capacity is sustained, and trainees should receive continued support in their workplace to develop and maintain skills.

Acknowledgements

The authors would like to thank the project advisory group for their input and guidance throughout the lifespan of the project.

Competing interests

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this research article.

Authors’ contributions

Y.E. and R.S. jointly made substantial contributions to the conception and design of this work. Y.E. and S.J. implemented the project and drafted the manuscript. R.S. further drafted and revised it for important intellectual content. All three authors reviewed the final version of the manuscript.

Funding information

Funding to conduct the research on which this article is based was received in the form of a grant from the William and Flora Hewlett Foundation.

Data availability

Select data sets generated and/or analysed during the current study are available from the Africa Centre for Evidence website (www.africacentreforevidence.org).

Disclaimer

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References

Abdul Latif Jameel Poverty Action Lab, n.d., Introduction to evaluations, viewed 21 February 2020, from https://www.povertyactionlab.org/sites/default/files/research-resources/Introduction-to-Evaluations.pdf.

Abdul Latif Jameel Poverty Action Lab, 2017, Zambia to scale teaching at the right level program to 1,800 schools, viewed 21 February 2020, from https://www.povertyactionlab.org/blog/8-8-17/zambia-scale-teaching-right-level-program-1800-schools.

Altshuler, N. & Staats, S., 2019, A new look at impact evaluation capacity in Sub-Saharan Africa – Research brief, viewed 28 November 2019, from https://hewlett.org/wp-content/uploads/2019/10/A-New-Look-at-Impact-Evaluation-Capacity-in-Sub-Saharan-Africa.pdf.

Blaser Mapitsa, C. & Khumalo, L., 2018, ‘Diagnosing monitoring and evaluation capacity in Africa’, African Evaluation Journal 6(1), a255. https://doi.org/10.4102/aej.v6i1.255

Chambliss, D.F. & Schutt, R.K., 2006, Making sense of the social world: Methods of investigation, 2nd edn., Pine Forge Press, Thousand Oaks, CA.

Crawley, K., 2017, ‘The six-sphere framework: A practical tool for assessing monitoring and evaluation systems’, African Evaluation Journal 5(1), a193. https://doi.org/10.4102/aej.v5i1.193

Goldman, I., Byamugisha, A., Gounou, A., Smith, L.R., Ntakumba, S., Lubanga, T. et al., 2018, ‘The emergence of government evaluation systems in Africa: The case of Benin, Uganda and South Africa’, African Evaluation Journal 6(1), a253. https://doi.org/10.4102/aej.v6i1.253

Grosskurth, H., Todd, J., Senkoro, K., Newell, J., Klokke, A. & Changalucha, J. et al., 1995, ‘Impact of improved treatment of sexually transmitted diseases on HIV infection in rural Tanzania: randomised controlled trial’, Lancet 346(8974), 530–536. https://doi.org/10.1016/S0140-6736(95)91380-7

Kalinga, C., 2019, ‘Caught between a rock and a hard place: navigating global research partnerships in the global South as an indigenous researcher’, Journal of African Cultural Studies 31(3), 270–272. https://doi.org/10.1080/13696815.2019.1630261

Manning, R., Goldman, I. & Licona, G.H., 2020, WIDER Working Paper 2020/20. The impact of impact evaluation: Are impact evaluation and impact evaluation synthesis contributing to evidence generation and use in low- and middle income countries?, viewed 21 February 2020, from https://www.wider.unu.edu/sites/default/files/Publications/Working-paper/PDF/wp2020-20.pdf.

Mbembe, A., 2015, Decolonizing knowledge and the question of the archive, viewed 13 January 2020, from http://wiser.wits.ac.za/system/files/Achille%20Mbembe%20-%20Decolonizing%20Knowledge%20and%20the%20Question%20of%20the%20Archive.pdf.

Morkel, C. & Ramasobama, M., 2017, ‘Measuring the effect of evaluation capacity building initiatives in Africa: A review’, African Evaluation Journal 5(1), a187. https://doi.org/10.4102/aej.v5i1.187

Ritchie, J., Spencer, L. & O’Connor, W., 2003, ‘Carrying out qualitative analysis’, in J. Ritchie & J. Lewis (eds.), Qualitative research practice: A guide for social science students and researchers, pp. 219–262, Sage, London.

Siedner, M.J., Mwebesa, B.B., Moosa, Y., Paul, M., Pillay, S., McCluskey, S. et al., 2017, ‘The REVAMP trial to evaluate HIV resistance testing in sub-Saharan Africa: A case study in clinical trial design in resource limited settings to optimize effectiveness and cost effectiveness estimates’, HIV Clinical Trials 4(18), 149–155. https://doi.org/10.1080/15284336.2017.1349028

Stewart, R., Dayal, H., Langer, L. & Van Rooyen, C., 2019, ‘The evidence ecosystem in South Africa: growing resilience and institutionalisation of evidence use’, Palgrave Commun 5, 90. https://doi.org/10.1057/s41599-019-0303-0

Stewart, R., Erasmus, Y. & the UJ-BCURE team, 2017a, Working for solutions: a problem-based and relationship centred approach to building capacity to use research evidence in Africa, Africa Centre for Evidence, Johannesburg, viewed 17 February 2020, from https://africacentreforevidence.org/project-outputs-2/.

Stewart, R., Nduku, P. & Langer, L., 2017b, Capacity in Africa: The results of a survey on support for and production of evidence maps and evidence syntheses, including systematic reviews, Africa Centre for Evidence as secretariat to the Africa Evidence Network, University of Johannesburg, Johannesburg.

Tarsilla, M., 2014, ‘Evaluation capacity development in Africa: Current landscape of international partners’ initiatives, lessons learned and the way forward’, African Evaluation Journal 2(1), a89. https://doi.org/10.4102/aej.v2i1.89

Van den Berg, R.D., Naidoo, I. & Tamondong, S.D. (eds.), 2017, Evaluation for agenda 2030: Providing evidence on progress and sustainability, International Development Evaluation Association, Exeter.

World Bank, 2009, Making smart policy: Using impact evaluation for policy-making. Case studies of evaluations that influenced policy, Doing Impact Evaluation No. 14, World Bank, Washington, DC.

Wotela, K., 2017, ‘A proposed monitoring and evaluation curriculum based on a model that institutionalises monitoring and evaluation’, African Evaluation Journal 5(1), a186. https://doi.org/10.4102/aej.v5i1.186

Footnotes

1. Defined here by the Royal Geographical Society (https://www.rgs.org/CMSPages/GetFile.aspx?nodeguid=9c1ce781-9117-4741-af0a-a6a8b75f32b4&lang=en-GB).

2. Available at https://developmentevidence.3ieimpact.org/.

3. See https://data.worldbank.org/country.

4. Used to describe courses that are not followed by an assessment, but where participants receive a certificate of attendance.


 

Crossref Citations

1. Turning ‘evidence for development’ on its head: A view from Africa
Ruth Stewart
Research for All  vol: 5  issue: 1  year: 2021  
doi: 10.14324/RFA.05.1.13