Abstract
Background: South Africa and other developing countries are facing an ever-increasing demand for competent evaluators. In addition, increasing demands are being placed on those who become evaluators. What does this mean for evaluation education in its current form and state in South Africa? In addition, what possible responses can be there to the diverse drivers of change within the dynamic social context in which evaluators operate?
Objectives: This article aims to address some of the questions related to the supply and demand profile of evaluation in South Africa, which may be useful for other developing countries.
Method: A literature review and key informant interviews were carried out to answer the key research questions.
Results: The article describes the provision of formal evaluation education and the challenges currently facing university-based offerings. The study provides a framework for considering the interaction between the supply and demand elements in the field of evaluation. Strategies are proposed for strengthening the supply of evaluators and ensuring that these evaluators can respond to the growing demands being placed on them.
Conclusion: This article is valuable for all evaluation stakeholders as it provides insight into the academic landscape of evaluation in a developing context and explores practical ways to support and strengthen capacity building efforts in similar contexts.
Keywords: Evaluator education; Evaluation landscape; Evaluation curriculum; African evaluation; Monitoring and evaluation; South African evaluation.
Background
There has been a rapid increase in the development of monitoring and evaluation (M&E) courses in South Africa since 2005 (Abrahams 2015). This increase has occurred concurrently with ongoing change in the context in which evaluators work. The development context is affected by economic and financial crises, shifting inter-connected global trends, technology and communication patterns, growing inequalities and mounting concerns around climate change (Van Zeeland 2014). All these changes in the operating context have an impact on the nature of interventions and subsequently on evaluation practice (Picciotto 2011). To keep pace with a dynamic operating context, those offering evaluation education need to take the current global and local trends into consideration (Chilisa 2015; Friedrich 2015; Patton 2019 & Picciotto 2011, 2015). Evaluation education is a term covering education of consumers and various stakeholders of evaluation, evaluation practitioners and scholars (Gullickson et al. 2019:20).
Although there has been an increase in the development of evaluation education, factors around the availability of offerings, what providers should consider when developing course content and how evaluation education can contribute to producing competent evaluators has received relatively little attention. Moreover, the various contextual factors that characterise the environment and influence evaluation are seldom considered in much depth in evaluation literature (Fitzpatrick 2012:8). An examination of the state of formal evaluation education opportunities within academic institutions in South Africa, and the main factors that are currently shaping M&E discourse and practice, will provide insight into the South African evaluation education landscape and its ability to meet both the current and future needs of evaluation.
Purpose of the article
The purpose of this article is, firstly, to give evaluation stakeholders insight into the range of academic offerings available in South Africa and, secondly, to make formal and non-formal (often not institutionalised or accredited) evaluation training providers aware of the key drivers of change in the current global and local evaluation context (UNESCO 2014). These need to be addressed if evaluation course content and indeed evaluators themselves are to remain relevant. The article describes the challenges that evaluation educators are currently facing and looks at the possibly greater challenge of producing competent evaluators who can cope with the growing demands being placed on them in a context that is becoming more complex over time.
The article is guided by the following questions:
What is the state of formal evaluation education offerings in South Africa? What are the main drivers of change in the current global and local evaluation context? In what ways can South African evaluation education respond to the dynamic social context in which evaluators operate?
A literature review and key informant interviews were carried out to answer the three questions. A detailed description of these methods is provided in the following section.
Methods
Literature review
A literature review was conducted with two aims. The first aim was to provide information on the evaluation academic offerings in South Africa. For this purpose, the literature review drew primarily on a study conducted by the Evaluation Research Agency (ERA) in partnership with the Palmer Development Group (PDG) in 2018. This study is the most current description of formal evaluation education offerings in South Africa. This study aimed in part to contribute to a much-needed understanding of current academic offerings in M&E. It also focussed on scholarship in the field of evaluation in South Africa and the experiences of black evaluators. The second aim of the literature review was to gain an understanding of existing research on the current drivers of evaluation change. An inductive coding of articles was carried out using ATLAS.ti. Seven key categories of drivers of change, four demand drivers and three supply drivers, were identified. These categories were summarised, leading to the development of an analytical framework. The analytical framework is used to provide insight into the drivers of change in evaluation and the implications of these for evaluation education offerings in South Africa.
Key informant interviews
Key informant interviews were carried out to understand key informants’ views on the current drivers of change in evaluation. Eleven key informants were interviewed for the study. Purposive sampling was used as the aim of the study necessitated the selection of participants who represented key stakeholders in the field of evaluation and had a range of roles and significant experience in evaluation. The key informants were selected from the following broad categories: academia, consulting, evaluation research and members of Voluntary Organisations for Professional Evaluation (VOPEs). There were five informants who were VOPE Chairpersons, three consultants, two academics and one doctoral candidate. Although the key informants were classified according to these four categories most of them had multiple roles (even in addition to the four categories).
Five sub-Saharan countries were represented in the sample: South Africa (six participants), Zimbabwe (two participants), Zambia, Uganda and Botswana (one participant each). Interviews were carried out with people outside of South Africa to ascertain if there was a similarity in perceptions about the drivers in the region instead of limiting this to a single country. The interviews were recorded and transcribed. Excel was used for a primarily deductive coding of the transcripts based on the categories generated during the development of the analytical framework from the literature. As a result, the drivers of change are classified in the same way for both the literature and the key informant study, which allows for comparison.
Limitations
First, the description of evaluation and evaluator education offerings focuses solely on universities and so does not include the many private providers who offer training. There may still be undiscovered modules and short courses offered through private institutions and other universities as the key source for the description of university offerings is based on data from 2018. The study also does not provide information on the many non-formal providers. The description of the academic landscape is based on a single source as there are no other current descriptions available.
Second, a small number of interviews were carried out and this certainly affects the findings. The number of key informants who currently chair VOPEs was higher than other roles represented. It should also be noted that other key categories such as government departments and non-governmental organisations were not represented in the study as not all those identified as key stakeholders responded.
What is the state of the formal evaluation education offerings in South Africa?
Whilst there has been a rapid increase in the development of evaluation education courses in South Africa since 2005, there is limited understanding of which institutions are offering M&E courses. The Zenex Foundation commissioned a study (Evaluation Research Agency and Palmer Development Group 2018) to examine some of the issues related to M&E capacity in South Africa, and a key element of the study was to provide an inventory of evaluation offerings at the 26 South African universities (Universities South Africa [USAF] 2017). The study involved a web-based search and a survey of all academic institutions found to be offering evaluation education in South Africa at the time (2018). The study focused on issues such as the year the programme started, the location of the programme, admission requirements, credits, structure, duration, number of students enrolled and key challenges. In addition, staff details and qualifications were requested per module.
The researchers distinguished between three formats of evaluation education: a programme (three or more evaluation specific modules in an offering), a module (within a broader programme) and a short course (a stand-alone course). The study found 55 offerings of evaluation education in the country (ERA & PGD 2018:22), and the spread of these offerings across the three formats is shown in Table 1.
| TABLE 1: Evaluation education offerings at academic institutions in South Africa (2018). |
Table 1 shows that modules (24) and short courses (22) are the most common types of offering in evaluation education. Table 1 also shows that documentation for 37 of the 55 offerings was received by the researchers.
The study also found that evaluation offerings were distributed across 13 of the 26 South African universities, with Stellenbosch University (SU) and the University of Pretoria (UP) contributing the highest number of offerings. This is mainly because of the number of short courses on offer at the two institutions. Academic offerings at a postgraduate level are available at nine universities, whilst undergraduate educational opportunities are available at only four universities.
Evaluation programmes
The study showed that there are only three Postgraduate Diplomas (Stellenbosch University, University of Fort Hare and University of the Witwatersrand) and three Masters’ programmes (Stellenbosch University, University of Cape Town and the University of the Witwatersrand). In addition, there are only two dedicated doctoral programmes.
Table 2 shows the specific postgraduate level evaluation programmes offered by each university.
| TABLE 2: Evaluation education programmes at South African Universities. |
Evaluation modules
Modules in postgraduate programmes are the most common (24) academic offering in South Africa (ERA & PGD 2018:37). The modules are offered at 10 South African universities. Twenty-one of the modules are at postgraduate level, with over half of these (13) at Masters level and eight at Honours level. The remaining three offerings are at undergraduate level. The distribution of the modules by faculty shows that most are offered in health sciences followed closely by humanities and economic and management sciences. The study found no modules offered in education faculties (ERA & PGD 2018:48).
Evaluation short courses
The study found that 22 short courses are offered by 9 South African universities (ERA & PGD 2018:41). The 17 short courses for which data were received indicated four target groups, which are not mutually exclusive: practitioners (seven), project managers (four), public sector (four) and health sector staff (two).
Challenges
The challenges that were raised in this study focus primarily on the supply side of evaluation, that is, the number and geographical spread of institutions that offer evaluation education, capacity issues within institutions (that affects how many students can be accepted, the quality of offerings, the amount of supervision that is possible) and the research-preparedness of undergraduate students entering postgraduate courses. These are shown in Figure 1.
 |
FIGURE 1: Supply challenges for evaluation education offerings. |
|
What was not raised by respondents, perhaps as this was not the key purpose of the study, was demand challenges. To understand the nature of the demand challenges, we now turn to two additional sources of information: literature focusing on current trends in evaluation and interviews with key informants in the evaluation field.
What are the main drivers of change in the current global and local evaluation context?
The literature
When examining the relationship between supply and demand in the field of evaluation, the role of universities is critical as these institutions must supply graduates to the market who ‘possess the right skills and competencies and ought to be educated in sophisticated methodologies for conducting sound evaluations in any sector’ (Basheka & Byamugisha 2015:88). Universities must supply to the market what is demanded. But the question is what is demanded by the market? Is it competent evaluators as defined by national competency frameworks or are there other demands on evaluators that need to be considered?
When doing the literature review for this study, it became evident that six evaluation scholars (Basheka and Byamugisha 2015; Chilisa 2015; Friedrich 2015; Patton 2019; Picciotto 2011 & Picciotto 2015) had made significant contributions to the elaboration of the current drivers of evaluation change (demand and supply) and what this could mean for evaluation education offerings. The articles of the six scholars were analysed, and through an inductive coding of their articles in ATLAS.ti, seven key categories of change were identified: four demand drivers (context, agencies, methods and competencies) and three supply drivers (increased supply, professionalisation and knowledge infrastructure). The seven drivers are shown in Figure 2.
Whilst the categories are mostly self-explanatory, it is important to unpack two of them dynamic operating environment and knowledge infrastructure. Dynamic operating environment is described by Picciotto (2015:8) as constantly changing and ‘featuring complexity, non-linearity and emergence’. This is what the market for evaluation today looks like. It is dominated by interventions that are vulnerable to rapid shifts in the operating environment and to the unpredictable push and pull of a wide range of stakeholders.
The term knowledge infrastructure as opposed to technology was used as the third supply driver as this term is more comprehensive in nature than terms such as ‘technology’. Knowledge infrastructure is ‘robust networks of people, artefacts and institutions that generate, share and maintain specific knowledge about the human and natural worlds’ (Edwards et al. 2013:5). Knowledge infrastructure refers to the advances and innovations in science, technology and knowledge.
The categories for the drivers are interconnected and often interact in bidirectional ways. For example, context affects the kind of agencies involved in commissioning evaluations, which in turn affects the need for new methods and approaches to be used by evaluators. All three drivers affect the demand for particular competencies that evaluators require. The demand drivers are all focused on an ever-increasing demand for evidence, accountability and results.
On the supply side in the field of evaluation, there has been an increased supply of evaluators to meet the increased demand, a growing need for the professionalisation of the discipline (standards, ethical guidelines and competency frameworks), a critical requirement of competency in the use of knowledge infrastructure for all evaluators. We now discuss the demand and supply drivers in greater detail.
Demand drivers
Context
As global sustainability becomes a universal concern, addressing climate change and inequality is becoming part of the mission of most non-profit organisations, international agencies and governments. This is driving change in evaluation in that new priorities that are cross-cutting and global in nature have emerged, prompting a change in evaluation approaches and methodologies (Friedrich 2015; Patton 2019; Picciotto 2011, 2015).
Picciotto (2015) indicated that a wide variety of evaluative approaches will be needed to assess strategies that will be used in pursuit of:
[E]radication of poverty, empowerment of women, quality education and lifelong learning, healthy lives, food security, universal access to water and sanitation, sustainable energy, effective natural resource management, good governance, peaceful societies, reduction of inequality, and implementation of a global enabling environment for fair and inclusive development (UN 2013). (p. 4)
To remain relevant, evaluators will have to broaden the scope of their work and use a range of evaluation methods, approaches and tools that are fit for purpose and respond to dynamic operating contexts (Patton 2019).
Agencies
The literature analysed recognises the increasing role that donors and philanthropic organisations play in alleviating the threats to human security. Philanthropic organisations and international donors alike are increasingly required to demonstrate accountability and be responsive to the needs of their diverse stakeholders. The increased role of these two agencies in social interventions is bringing about change in evaluation by increasing demand for results and in turn the demand for credible evaluations.
National governments are also increasingly required to be accountable to both their citizens and intergovernmental organisations based on commitments they have made around the sustainable development goals (SDGs) (Chilisa 2015). Governments are under pressure to provide evidence of progress towards achieving goals and targets associated with the SDGs. As a result, governments are expanding their objectives to include building capacity for evaluation, developing evaluation frameworks and requiring their departments to use these (Chilisa 2015).
Significantly, the literature analysed highlighted the role of the private sector as a driver of change in M&E. The emergence of new actors in the implementation of social interventions has included the increased role of the private sector in the social sphere (Picciotto 2015). The increased involvement of the private sector has brought with it an array of innovative social interventions including market-based instruments, social impact funding and private–public partnerships. Evaluators will have to tailor existing approaches or develop new approaches to effectively measure these innovative social interventions (Picciotto 2015).
New approaches
Picciotto (2015:5) indicated that traditional development evaluation methods ‘are poorly adapted to dynamic operating contexts featuring complexity, non-linearity and emergence’. He points out that evaluation methods are often still based on ‘traditional public sector program interventions’, which do not take the rapid pace of decision-making favoured by the new actors in the operating space into account. Public demand for results is increasing, resulting in the need for effective ways to deliver adequate and timely evidence to decision makers about the likely development impact of interventions.
Competent evaluators
As the demand for accountability grows and good governance is promoted, so is the demand for credible evaluations produced by competent evaluators (Basheka & Byamugisha 2015). Developing high quality evaluation practice and strengthening an evaluation culture requires sufficient evaluation capacity. As such, the demand for accountability, evidence and result centres on having adequate (even advanced) M&E competence.
Supply drivers
Increase in formally trained evaluators
There is an increased supply of trained evaluators partly because of the increased number of institutions offering education in evaluation. In addition to academic institutions, the number of conferences, seminars and workshops dedicated to evaluation capacity building has increased significantly (Basheka & Byamugisha 2015). Trained and competent evaluators can design and implement effective monitoring frameworks and conduct rigorous evaluations, which contribute to meeting the demand for evidence, accountability and results. Considering this, the increased number of trained evaluators is a driver of change because it has the potential to contribute to strengthened M&E systems, better-quality evaluations and providing commissioners with credible evidence for making decisions.
Professionalisation of evaluation
Efforts to professionalise evaluation are also driving change in the field of evaluation and increased pressure on evaluation associations to develop standards and competency frameworks. Evaluation educators (formal and non-formal) are required to include a focus on competencies in their courses and programmes. In addition, professionalisation provides evaluators with a reference point for the quality and nature of their practice. It is therefore contributing to the increased supply of formally trained evaluators, who view formal education as a way to gain the skills, competence and status that will get them recognised as professionals (Friedrich 2015).
Knowledge infrastructure
Knowledge infrastructure was foregrounded in the literature as the key supply driver of change in M&E (Basheka & Byamugisha 2015; Friedrich 2015; Patton 2019; Picciotto 2011, 2015). In evaluation practice, knowledge infrastructure is critical for increasing the quantity, speed and accessibility of evaluation data and can reduce the costs of sharing databases and disseminating evaluation findings. Knowledge infrastructure facilitates instant connectivity and has opened avenues for real-time stakeholder involvement. New communication technologies are increasing the capacity of evaluators to deliver evaluation services and changing the way in which evaluations are conducted. Tighter timelines and responsiveness in delivering services is now required of evaluators and ‘suppliers’ of evaluators (formal and non-formal trainers) will have to ensure evaluator competency in this area (Picciotto 2015).
In summary
The six evaluation scholars paint a picture of a demanding operating space for evaluators. There is an enormous demand on those offering academic education and evaluators themselves, to be continuously informed about the changing operating space and the competencies required from both formal and non-formal education, as well as through continuous professional development. Evaluator education needs to take the demand drivers into account and ensure students of evaluation are aware of the complexity of their changing context and can respond adequately.
The question then arises, would South African stakeholders identify the same drivers of change as those identified in the literature? If this were the case, we would have an alignment between theorists and practitioners, which could be a foundation for the development of core competencies for evaluation and evaluator education and, in fact, for evaluator competency frameworks that could be common across countries and regions.
Key informant interviews
In 2019, the researchers carried out key informant interviews to understand the drivers of change from the perspective of a range of stakeholders in five sub-Saharan countries. We were interested to see if there was an alignment with the literature we had reviewed or if the regional context would impact on stakeholders’ perception of key drivers of change. Eleven people were interviewed for the study; five VOPE Chairpersons, three consultants, two academics and one doctoral candidate (although these simplistic categories do not cover their wealth of experience and many roles they have played in the evaluation field up to this point).
Figure 3 shows the analytical framework that was developed from the literature review but in this version of the framework, the overlap between the views expressed by the evaluation scholars with the key informants is shown (through a star symbol).
 |
FIGURE 3: Supply and demand drivers identified by key informants (shown by  ). |
|
What Figure 3 shows is that there is a strong alignment between the two sources – with key informants not raising the dynamic operating environment (from Context) nor foregrounding of the role of the private sector (from Agencies). We now discuss the demand and supply drivers raised in the interviews in greater detail.
Demand drivers
Context
The international development agenda, marked by the SDGs, was identified as a key driver of change in M&E in sub-Saharan Africa by the interviewees. One of the interviewees (Consultant, 3, 2019) indicated that ‘the big issue for us, from the African perspective is very much the SDGs – it’s very much about the ability to think in terms of goals and to be reporting against their targets’. The SDGs were also clearly linked to the donors who ‘are obsessed with the SDGs at this point’ (Consultant, 3, 2019), which makes this a significant driver for both those involved in interventions and in evaluations. Respondents also indicated that because the SDGs are complex, they require comprehensive M&E efforts to assess the effectiveness of strategies used to deliver on goals and their targets. These goals have created demand for more robust and innovative M&E and approaches. Once again, we see the inter-connectedness and bidirectional nature of supply and demand drivers and both types of drivers benefitting from the connection.
Agencies
Agencies was a key issue for almost all (10) of the interviewees. Interviewees raised the issue that international donors and philanthropic organisations provide substantial funding for development and as a result, they often inform the approaches, practices and standards of evaluation in the countries in which they operate, ‘you have the African Development Bank and the World Bank – they will come to you with their already structured results framework, they have their own template’ (VOPE, 3, 2019). The donors are seen to influence M&E ‘as they provide the funding. They therefore drive both the need for M&E and drive the kind of approaches used in M&E in developing countries’ (Academic, 2, 2019).
International donors are also increasingly seeking ways to support the evaluation capabilities of evaluators and evaluation stakeholders in the countries in which they operate. This is through training but is not yet seen as a strong influence, ‘the UN agencies are starting talk to the VOPE and wanting to work in country and contribute to national capacity building so that I think will be a big shift if that happens in any meaningful way’ (VOPE, 1, 2019). International donors are also seen to drive change in M&E in some countries through the development of and training on M&E systems in host countries.
Governments, particularly those that understand the role and benefits of evaluation to inform decision-making within countries, were also identified as drivers of change (three interviewees).
‘In countries like Zimbabwe and Botswana the governments are also doing a lot to develop national evaluation systems, especially in Botswana. You find that they have created what they call the National Strategic Unit that sits in the president’s office’. (PhD candidate, 1, 2019)
However, political will, sustained commitment and resources are required for governments to continue to be a driver of change in evaluation (8 of 11 interviewees), especially when competing with the power of funding agents.
One of the academics interviewed described the key relationship between the supply and the demand side as she saw those working in evaluation ‘informing decision makers about where there are weaknesses in their evidence and decision making processes’ (Academic, 1, 2019) but that at the same time, the demand element in evaluation was driving improvement in the field, ‘that demand side, that’s where the sophistication is coming from, because people have a better sense of what good M&E entails’. (Academic, 1, 2019).
Methods
Whilst the literature discussed earlier generally referred to new approaches and methodology, the key informants were asked particularly about Made in Africa Evaluation (MAE), which is a concept characterised by evaluation practice that is rooted in African cultures, development agenda philosophies, worldviews and paradigm (Chilisa 2015). Three of the interviewees identified an increasing demand for accelerated progress in the development of local evaluation approaches and methods to guide evaluation practice in Africa, also referred to as MAE. There were mixed views about the notion of MAE. The findings highlight that MAE is still nascent and requires further conceptualisation before it can be fully established. If MAE is to be included in a transformed curriculum for Southern Africa evaluation education, scholars and practitioners will have to partner to fully articulate this approach.
Competencies
Six of the interviewees raised the need for competent evaluators. One of the interviewees (Academic, 1, 2019) described the increasing demand for education as one of the key drivers in her work, ‘the big driver in the space that I work is that there is such a high demand for evaluation capacity building…we have received so many requests for training’.
This growing demand for formal and informal training is seen because of ‘the results agenda and the increasing demand for evidence informed decision-making. So what is happening is that this wave, especially with the SDGs now, even at highly strategic level of political decision making, there is an awareness at that particular level that there is a need for better evidence and that there is a need to demonstrate results to the public as well’ (Academic, 1, 2019). What is evident from the interviews is that more is being requested and required from evaluators. ‘They can no longer go into a space, do an evaluation and exit. Instead, stakeholders now want capacity building, technical assistance and skills transfer. Evaluators are now expected to be part of the broader ecosystem. This has an implication on M&E practice and is a driver of change in M&E’ (Academic, 1, 2019).
We now examine some of the supply drivers of change that were identified by the key informants interviewed for the study.
Supply drivers
Increase in formally trained evaluators
The increased supply of evaluators was the most frequently mentioned driver on the supply side, with 10 of the 11 key informants acknowledging that it was a key driver of change in M&E.
Most of the key informants indicated that there is an increase in the number of competent evaluators with the technical capacity to perform robust evaluations. They also raised the key role of emerging evaluators who bring innovative and creative ideas into the evaluation space. This is coupled with the establishment of formally recognised evaluation associations or VOPEs, which support capacity building of evaluators and raise awareness of M&E among evaluation stakeholders.
The work that associations like SAMEA and other VOPEs are doing is resulting in an increase in awareness of M&E among stakeholders. You have people who are better trained, have more knowledge, and have some theoretical monitoring and evaluation background. This in turn results in a greater demand for sophisticated M&E and stakeholders demand more accountability. (Academic, 1, 2019)
Professionalisation
Six of the 11 interviewees raised the issue of professionalisation of evaluation as a driver of change. Interviewees described how the move towards professionalisation has resulted in the development of evaluator competencies, practice and ethical guidelines and increased scholarship in the field of evaluation. This has contributed to improving the status of the profession and fostering positive views of M&E as a professional occupation.
Different people have different definitions of what it means to professionalise…but in broad sense of the word “professionalising,” I do think a driving force of evaluation globally, is that it’s being professionalised. (Academic, 2, 2019)
Knowledge infrastructure
Seven of the eleven key informants interviewed raised knowledge infrastructure as a driver. It was noted by interviewees that technology and other elements of knowledge infrastructure have not been sufficiently foregrounded in the African context. To ensure technology continues to drive change, M&E professionals should be engaging more with the different types of technology and applying them to data collection, analysis, reporting and dissemination practices.
Technology provides mechanisms and platforms that facilitate fast moving information and opportunities for people to access information. Social media enables people to demand results and accountability. This in turn is shaping how monitoring and evaluation is conducted. (VOPE, 3, 2019)
Discussion of key informant interviews
According to the key informants interviewed, change in M&E is being driven by a combination of demand and supply drivers, which are characterised by both local and global considerations. It is interesting to note that the drivers raised in the literature were closely aligned to those raised by the key informants.
Whilst a comparison between the demand and supply drivers from scholars’ and the stakeholders’ perspectives cannot be made in a detailed way, it is useful to see the areas of overlap as shown in Figure 3. The articles foreground the context and methods categories, whereas the stakeholders’ stressed the influence of agencies. This indicates that within the context of Southern Africa, evaluation is still largely driven by the source of funding and commissioners of evaluation, which is why agencies (international donors and philanthropy) are the most frequently mentioned demand driver of change. It is interesting to note that although agencies are the main demand driver for interviewees, the private sector is not mentioned by interviewees at all, which may imply that the private sector is perceived to have a limited role in evaluation in this context.
It is also important that although only three of the 11 key informants identified MAE as a demand driver (New approaches and methodologies), all key informants indicated that this is an emerging approach that needs to be conceptualised further, and perhaps then it will be more widely recognised as a demand driver.
Key informants’ responses to the perceived impact of the drivers on capacity building were grouped into four categories – issues raised related to (1) providers, (2) content, (3) modes and approaches, and (4) the objective of training. The recommendations raised by the key informants are relevant for both academic and all other evaluation education providers.
The following section builds on (1) the key informants’ perspectives on the impact of drivers of change on capacity building and combines them with (2) perspectives gained from the literature and (3) the study of South African evaluation education to inform recommendations for how academic offerings can respond to the identified drivers of change.
Not only is it important for the evaluation community to be aware of what is on offer in terms of formal opportunities in South Africa but also it is important for them to ask questions about whether the content of those courses and programmes considers the current operating environment.
In what ways can South African evaluation education respond to the dynamic social context in which evaluators operate?
The identification of supply and demand drivers foregrounds the mechanisms that are currently driving change in evaluation. This is useful for providing direction for efforts aimed at strengthening the evaluation landscape. Evaluation capacity development is recognised as one of the main solutions to strengthening the evaluation landscape in South Africa and the continent (Morkel & Mangwiro 2019). It is important to determine what academic offerings can do to address both the demand and supply drivers of change. Considering this, the effects of demand and supply drivers are discussed in turn, culminating in a list of recommendations for strengthening evaluation education.
Demand drivers
Context
Evaluators require the knowledge and skills to operate within a dynamic context, and evaluation education can contribute to developing the requisite knowledge and skills. This requires offerings to be nimble and responsive to the changes in the development context. Evaluation education curricula need to be aligned with the needs of all stakeholders within the operating environment.
Agencies
There are a variety of agencies operating in the evaluation landscape in South Africa. These include government, international funding agencies and development institutions, as well as private sector organisations. The agencies make use of different frameworks, templates and standards for evaluation-related activities (Morkel & Mangwiro 2019:211). Evaluation education offerings should be geared towards developing the skills, knowledge and experience required for evaluators to work with these diverse agencies. Through collaboration between academic institutions and a range of agencies, experiences of these agencies can be leveraged to contribute to a relevant and comprehensive evaluation education offering. Collaborations and partnerships are an ideal way to strengthen current offerings and capacity in Southern Africa.
Methods
The various agencies in the operating context require specific approaches and methods as they aim to meet the ever-increasing demands for accountability, timely results and credible evidence. For example, data visualisation has become a critical skill for evaluators as communication with a range of stakeholders in an accessible way takes precedence over lengthy reports to commissioners. A detailed examination of evaluation capacity needs and the curricula of academic offerings is therefore required to match needs with the appropriate methods.
Competencies
Developing competent evaluators requires that evaluation education align with the following: (1) the demands of the context in which they operate, (2) the agencies in the local and global context and (3) the resulting approaches and methods required to meet the evaluation needs of these agencies. For some academic institutions this will require analysing current curricula to identify gaps and considering innovative ways in which to close these gaps. Some strategies for addressing the required competencies are increased use of case studies, role plays and scenario analysis in teaching, as well as mentoring activities and internships outside of formal education.
Table 3 provides some examples of how those offering evaluation education can address demand drivers identified.
| TABLE 3: Recommendations for addressing demand drivers. |
Supply drivers
Increased supply
The evaluation landscape’s future success is dependent on sound evaluation education and training programmes that provide a continuing flow of excellently qualified and motivated evaluators (Stufflebeam 2001). Sustaining the increasing number of trained evaluators and facilitating their continued development is essential for a strong evaluation landscape. Evaluation educators should outline the options available for someone with evaluation qualifications and connect evaluators to the wider evaluation community of practice.
Knowledge infrastructure
Knowledge infrastructure will be the key method of ‘leap-frogging’ into a strengthened evaluation future. It provides opportunities to collect, analyse and visualise data in more efficient ways. However, evaluation stakeholders are behind in terms of the adoption knowledge infrastructure (Raftree 2017). To remedy this, the nature of programme content and evaluator skills will need to align with advances and innovation in science, technology and knowledge. Supporting evaluators to adopt new technologies is another role that evaluation education providers can play.
Professionalisation
Professionalisation of evaluation remains contested and unresolved. Indications of the competencies that constitute a good evaluator have been developed by VOPEs, for example, the Canadian Evaluation Society (Morkel & Mangwiro 2019). In the South African context, work towards the development of competencies requires all evaluation stakeholders including academia, VOPEs, donors and government institutions to collaborate. This will ensure that the competencies developed are inclusive and contextual. A general understanding and agreement on these competencies will result in considerable progress in the South African evaluation landscape. Table 4 provides some examples of how those offering evaluation education can address identified supply drivers.
| TABLE 4: Recommendations for addressing supply drivers. |
Conclusion
South Africa has an academic offering that needs a range of interventions to be strengthened, expanded and sustainable. There is insufficient capacity (quantity and qualifications of staff) that puts a strain on existing sites and limits access to formal education to many of those who urgently require it. Collaborations and partnerships are an ideal way to strengthen current evaluation education in Southern Africa, through both formal and non-formal training. It is also clear that alternative sources of funding (beyond universities themselves) will have to be sought to fund posts and research in the field of evaluation (and particularly new approaches like MAE). Knowledge infrastructure will be the key method of ‘leap-frogging’ into a strengthened evaluation future. A more detailed examination of evaluation education curricula is required to identify gaps and possible areas of collaboration.
Acknowledgements
The authors would like to thank the Zenex Foundation for providing funding for Study 1 and the interview respondents for participating in the interviews for Study 2.
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this research article.
Authors’ contributions
L.W. and T.R.S. contributed to the design and implementation of the research, to the analysis of the results and to the writing of the manuscript.
Ethical considerations
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.
Funding information
This research received a grant for Study 1 from the Zenex Foundation.
Data availability
Data are available by contacting the first author (L.W.) and the Zenex Foundation.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Abrahams, M.A., 2015, ‘A review of the growth of monitoring and evaluation in South Africa: Monitoring and evaluation as a profession, an industry and a governance tool’, African Evaluation Journal 3(1), 1–8. https://doi.org/10.4102/aej.v3i1.142
Basheka, B.C. & Byamugisha, A., 2015, ‘The state of monitoring and evaluation as a discipline in Africa: From infancy to adulthood?’, African Journal of Public Affairs 8(3), 75–95.
Chilisa, B., 2015. A synthesis paper on the made in Africa evaluation concept, paper commissioned by African Evaluation Association, viewed n.d., from https://afrea.org/wp-content/uploads/2018/02/MAE-Chilisa-paper-2015-docx.pdf.
Edwards, P.N., Jackson, S.J., Chalmers, M.K., Bowker, G.C., Borgman, C.L., Ribes, D. et al., 2013, Knowledge infrastructures: Intellectual frameworks and research challenges, Deep Blue, University of Michigan School of Information, viewed n.d., from https://deepblue.lib.umich.edu/bitstream/handle/2027.42/97552/Edwards_etal_2013_Knowledge_Infrastructures.pdf?sequence=3&isAllowed=y.
Evaluation Research Agency and Palmer Development Group, 2018, Strengthening monitoring and evaluation capacity: A landscape analysis, paper commissioned by the ZENEX Foundation, viewed 10 June 2020, from https://www.zenexfoundation.org.za/informationhub/publications/monitoring-evaluation/item/470-monitoring-and-evaluation-capacity-a-landscape-analysis.
Fitzpatrick, J.L., 2012, ‘An introduction to context and its role in evaluation practice’, in D.J. Roj, J.L. Fitzpatrick & R.F. Conner (eds.), Context: A framework for its influence on evaluation practice, New directions for evaluation 2012(135), 7–24, Wiley Periodicals Inc. and the American Evaluation Association, n.l. https://doi.org/10.1002/ev.20024
Friedrich, V., 2015, ‘European university-based study programs in evaluation: Characteristics and future challenges’, in R. Stockmann & W. Meyer (eds.), Evaluation as a profession: Global trends, pp. 113–128, Palgrave Macmillan, London.
Gullickson, A.M., King, J.A., LaVelle, J.M. & Clinton, J.M., 2019, ‘The current state of evaluator education: A situation analysis and call to action’, Evaluation and program planning 75, 20–30. https://doi.org/10.1016/j.evalprogplan.2019.02.012
Morkel, C. & Mangwiro, N., 2019, ‘Implications of evaluation trends for capacity development’, in C.B. Mapitsa, P. Tirivanhu & N. Pophiwa (eds.), Evaluation landscape in Africa: Context, methods and capacity, pp. 184–200, Sun Press.
Patton, M.Q., 2019, ‘Expanding futuring foresight through evaluative thinking’, World Futurist Review 11(4), 296–307. https://doi.org/10.1177/1946756719862116
Picciotto, R., 2011, Evaluating development philanthropy in a changing world, Bellagio Initiative, London.
Picciotto, R., 2015, The 5th wave: Social impact evaluation, The Rockefeller Foundation, working paper, viewed n.d., from www.rockefellerfoundation.org/report/5th-wave-social-impact-evaluation.
Raftree, L., 2017, ‘Buckets of data for MERL’, in MERL Tech: Exploring the role of technology in monitoring, evaluation, research, and learning in development, viewed 20 June 2020, from http://merltech.org/buckets-of-data-for-merl/.
Stufflebeam, D.L., 2001, ‘Interdisciplinary Ph.D. programming in evaluation’, American Journal of Evaluation 22(3), 445–455. https://doi.org/10.1016/S1098-2140(01)00155-2
United Nations Educational, Scientific and Cultural Organization, 2014, ‘Formal education and training’, in Resources and Services, viewed 26 November 2019, from https://unevoc.unesco.org/go.php?q=TVETipedia+Glossary+A-Z&id=222.
United Nations, 2013, Transforming our world: The 2030 Agenda for sustainable development, viewed 20 June 2020, from https://sustainabledevelopment.un.org/post2015/transformingourworld.
Universities South Africa, 2017, ‘Public universities in South Africa’, Universities South Africa, viewed 12 June 2020, from https://www.usaf.ac.za/public-universities-in-south-africa/.
Van Zeeland, A., 2014, ‘The changing development context: A Latin American perspective’, World Council of Churches 66(3), 313–323. https://doi.org/10.1111/erev.12108
| 
Email this article 
Hide
Show all
