This section is structured essentially according to Holvoet and Renard’s characteristics (see Table 1). In terms of their use of policy, the NEPF proposes the strategic selection of priority evaluations, recognising that there is not the capacity or resources to evaluate all programmes or policies, and the selection is expressed through national, provincial and departmental evaluation plans. There have been annual National Evaluation Plans since 2012. Western Cape and Gauteng have provincial evaluation plans from 2013 and early adopter departments from a similar period.

In total, 73 evaluations have been selected in NEPs, covering around US$ 10 billion of government expenditure with 67 taken forward (DPME 2018b). Seven provinces have had provincial evaluation plans, with 102 evaluations identified in these plans (Minister of DPME’s budget speech 2018). As of October 2017, 68 of 155 national and provincial departments had departmental evaluation plans, with over 300 evaluations in those plans (Minister of DPME’s budget speech 2018). A set of evaluations in sectors such as Human Settlements, Social Development as well as Agriculture and Rural Development have been completed and offer the opportunity to synthesise evaluation findings to provide a higher level view at policy/sector level. Therefore, great strides have been made in terms of breadth. Early adopters such as the Western Cape Province (see Box 1), the Department of Trade and Industry (DTI) and Department of Basic Education (DBE) have internalised systems and they ensure that evaluations are aligned to departmental priorities. Box 1 provides an example of one of the case studies that is used to give a view of how at organisational level the evaluation system has evolved.

So the NES has widened from national to provincial and later to departmental evaluations and it is operating at significant scale. As departments have undertaken their own evaluations, those selected in NEPs have become more strategic.

The evaluation of the NES found that the bulk of the DPME’s budget for evaluation (77% in 2016/17 and 83% in 2015/16) is spent on funding evaluations, which typically cost R2–3 million, whilst proportionally less is spent on institutionalisation activities such as capacity building (0% in 2016/17 and 8% in 2015/16) and communication (1% in 2016/17 and 0.3% in 2015/16). The British Government’s Department for International Development (DFID) supported capacity building between 2012 and 2015 (DPME 2018b). To support institutionalisation, the evaluation recommended the spread of the budget needs to be more even. Respondents on both the supply side (evaluators) and the demand side (departments and provinces) noted that the evaluation process is a lengthy process which requires a considerable investment in time (DPME 2018b:16). The length of time taken is resulting in pressure to do evaluations internally and to do more rapid evaluations. A considerable amount of time is spent on pre-design and design, developing improvement plans and the communication of results. Increasing bureaucracy in the supply chain process to procure evaluations is actually lengthening the pre-evaluation process.

Holvoet and Renard’s framework proposes a centrally located unit to manage the evaluations. In South Africa’s case, at national level this is the ERU in DPME which supports all NEP evaluations and coordinates the NES across the government. Provincial offices of the premier (OTPs) play a similar role in provinces. The OTPs provide support and technical advice to departments as they progress through the evaluation ‘journey’, although in some provinces such as Kwazulu-Natal, they have struggled to do this. There are also decentralised M&E units in departments and agencies but few of these have significant capacity to support evaluations with only a few national departments such as DBE, Social Development and Rural Development and Land Reform having specialist evaluation staff. At national level an Evaluation Technical Working Group (ETWG) of national and provincial evaluation champions was established to support the NES. Some offices of the premier such as in Western Cape have established a provincial Evaluation Technical Working Group to support the provincial system (DPME 2018b). The OTP also coordinates progress reporting to DPME, assistance with improvement plans and provincial evaluation capacity building (see Box 1).⁹

Evaluations focus on policies, programmes or systems. Historically, at national level and in some provinces, there has been a call for proposals for evaluations to departments, resulting in a mix of bottom-up proposals from departments and strategic proposals from DPME and National Treasury at national level, and OTPs at provincial level (DPME 2018b:ix). These then go through a prioritisation/selection process.

The NES has characterised evaluations as design (analysing the design of the programme), diagnostic (analysing the problem, root causes, i.e., ex ante), implementation (looking at how the activities are translating into outputs and outcomes), impact (at outcome or impact levels), economic or synthesis (DPME 2011:9). Most evaluations undertaken are implementation evaluations as they have more rapid feedback into policy (see Figure 1 drawn from the evaluation report). The evaluation found that the guidelines are utilised for standardisation of processes.¹⁰

South Africa has defined the set of competencies required for evaluators and government staff who manage evaluations (Podems, Goldman & Jacob 2013) (DPME 2014). However, the capacity to undertake and to manage evaluations is weak.

Capacity building within the NES has included learning-by-doing (e.g., support by DPME evaluation staff or OTPs), the development of guidelines and templates by DPME, promotion of learning networks and forums, short courses and developing an evaluation management standard to drive establishment of evaluation capacity in departments. Evaluation courses have been developed and they have been rolled out in collaboration with the Centre for Learning on Evaluation and Results for Anglophone Africa (CLEAR-AA) at the University of Witwatersrand and recently with the National School of Government (NSG).

The evaluation found that templates and guidelines are considered helpful to both departments and provinces. Departments and provinces with less experience in evaluations found some of the guidelines were difficult to implement without further support whilst those who had been in the NES for a longer period recommended adding more flexibility to existing templates and guidelines to better suit their varied contexts. Respondents also expressed a need for additional guidelines for more complex evaluations. (DPME 2018b:18)

Between 2012/13 and 2016/17, 1989 government staff undertook training (mostly either programme managers or M&E staff) on courses ranging from theory of change to introduction to evaluation. The investment in capacity building has been limited because DFID funding ended in 2015 and with the constraints of moving the training function to the NSG. However, a set of trainings was rolled out with the NSG at the end of 2017/early 2018 (DPME training records quoted in DPME 2018b:19).

The courses have been well received with respondents acknowledging the importance of ‘on-the-job’ training. Some of them have wished to deepen their training with more advanced courses (DPME 2018:19). An innovation has been a partnership with the University of Cape Town to train over 330 members of the top three levels of the public service in the importance of evidence (Goldman et al. 2018). This training for senior officials was seen by interviewees as helpful in advocating for evidence and evaluation but it was suggested that training of programme managers is also needed. In fact, such a course was piloted in 2018.

Some provinces and departments highlighted the need for additional staff to cover the evaluation function (DPME 2018b:19). However, fiscal constraints have led to a cap on appointing new staff and the limited funding is affecting capacity building overall and not just human capacity.

The quality assurance mechanisms of the NES are important for the credibility of evaluations coming out of the NES.

The wide range of tools used to promote quality assurance include: design clinics, where the initial theories of change for the programme and outline terms of reference (TORs) for NEP evaluations for the following year are developed; steering committees; support by DPME evaluation directors; guidelines; peer reviews; and quality assessment and scoring of completed evaluations through a contract with an independent service provider. There was general consensus from respondents that the system was useful, contributing to the production of better quality evaluations. Suggestions to improve its working in practice included using peer reviewers from the beginning of an evaluation and to strengthen communication between reviewers, programme managers and the steering committee (DPME 2018b:20).

The quality assessment scores are reported when evaluations are tabled in Cabinet¹¹ giving ministers an idea of the validity and reliability of evaluation findings. The NES evaluation noted some evidence of questionable assessment scores. It is important that the quality assessment is credible and there is a view that government officials should conduct the assessments.

With regard to evaluation standards, it is recognised that when conducting evaluations, there has to be shared norms and standards, and a common understanding of exactly what we aim to improve and how to measure it. This is crucial to ensure that there are good standards to benchmark against. The NES evaluation recommends provincial departments work closely with DPME to strengthen alignment with national policy frameworks and M&E guidelines and standards (DPME 2018c:163).

Stakeholders ranging from academics, think tanks, political parties and so on need access to the guidelines, evaluation reports and improvement plans. The evaluation found there had been a considerable investment in communication. DPME has an evaluation communication strategy and it has communicated results through the media and its electronic newsletter (Evaluation Update¹²), publications such as policy briefs and annual reports.¹³ It has also presented widely on the NESs at conferences and other forums. DPME sends completed evaluations to parliamentary portfolio committees and most years does a presentation on the NES to the chairs of portfolio committees. There have also been occasional presentations to parliamentary researchers (DPME 2018b:21–22; Goldman et al. 2015).

There is an evaluation section on the DPME website including a publically accessible national evaluation repository.¹⁴ In the Western Cape, evaluations can be accessed through the provincial project management system (Biz Brain), which also houses related documents such as the evaluation update, dictionary and guidelines. M&E officials in the Western Cape reported that having access to the evaluations conducted by other departments was very useful as it showcased the work completed, its benefits, successes and lessons learnt.

Whilst a lot of work has been performed in terms of communication, respondents noted that a number of areas should be strengthened. These include more use of the media (e.g., radio) and more formal and informal sharing of learnings within the public sector (DPME 2018b:21–22).

One of the key elements of Holvoet and Renard’s (2007) framework is the participation of other actors outside of government as part of the broader ecosystem. The architects of the NES have worked to build an ecosystem to support the NES. It has been challenging to determine the boundary of the ecosystem and identify who are the actors within the system and their relationship with one another. Boundary partners include related government structures such as the Public Service Commission, whilst non-government organisations include universities, SAMEA, CLEAR-AA, donors, civil society organisations, parliamentarians, private evaluators and so forth (Goldman et al. 2015).¹⁵

SAMEA and CLEAR-AA have played key roles in the development of the NES from the development of the original NEPF to their participation in the Steering Committee for the evaluation of the NES. Universities deliver capacity development work and they may also bid for undertaking evaluations. Academic institutions also play an important learning function in the development of evaluation culture and practices and some also offer M&E courses (Goldman et al. 2015). In provinces, universities are important in providing peer reviewers and supplementing evaluation budgets by co-funding evaluations or in some cases using student theses to undertake evaluations. The evaluation found more work is needed to bring evidence brokers such as think tanks on board (DPME 2018b:30).

A view that came out strongly during the interviews is that the roles of DPME and other actors in the evaluation space are not always clear and there is not always a shared vision for the NES across centre of government institutions. The evaluation found that the role of DPME needs to be clarified whilst the roles of the Department of Public Service and Administration (DPSA), National Treasury and the NSG need to be strengthened (DPME 2018b:15).

The NES evaluation also offers important lessons for stakeholders beyond the principal audience of government institutions. Most important amongst these lessons is that the NES has an opportunity to play a role in strengthening accountability and deepening democracy in South Africa. South Africa is characterised by deep divisions and profound exclusion. In a noisy and tumultuous market place, approaches such as Deliberative Democratic Evaluation seek to make evaluation ‘an institution that stands apart, reliable in the accuracy and integrity of its claims’ (House & Howe 2000:4) but inclusive to enable dialogue and facilitate deliberation. Such dialogue could be promoted by drawing on the evidence base that is being constructed through the NES and which can inform ongoing and sustained engagements between the state and non-state actors in both formal, institutionalised civil society and in less structured formations. The evaluation of the NES found that ‘civil society is under-utilised and under-engaged in the NES’ (DPME 2018b:15).

Some departments also have good connections to private sector partners such as farmer organisations, who participate in evaluations by forming part of steering committees and commenting on the TORs for the evaluations (DPME 2018c:177).

The reporting requirements from the Auditor General and National Treasury meant that the 2007 Government-Wide Monitoring and Evaluation System was largely compliance driven. After the establishment of DPME in 2010 the NEPF retained some of this accountability emphasis whilst seeking to promote a performance-oriented approach (Goldman et al. 2015). DPME avoided a legislative route that would make the M&E system mandatory. In terms of the NES, this meant a voluntary approach aimed at creating an enabling environment to encourage departments to engage proactively in the system. The broader intent was to shift behaviour of senior management away from a compliance orientation towards recognising the value of learning from evaluation findings to improve programme performance.

We have referred earlier to the extension of evaluations across the state, to provinces and departments, and the establishment of a range of systems to support evaluations. Lazaro (2015:110) in reviewing institutionalisation in Europe and Latin America mentions that ‘a certain degree of formal institutionalisation may help also to foster a predictable and regular evaluation practice that adheres to suitable quality standards that promote its effective use’.

Growing institutionalisation of evaluation can be seen in the complex system of plans, guidelines, standards and so on; and the fact that all evaluations in the NEP are debated in Cabinet. However, the system does not carry the weight of legislation which, for example, underlies Treasury processes. To address this, a draft Planning Bill has been presented to Cabinet which includes some elements of M&E. However, extensive revisions are required and will be delayed beyond the May 2019 elections.

As mentioned earlier, over 330 of the top three levels of the public service have been trained in evidence. The intent was to create a more conducive environment for using M&E evidence. There are examples of where this has had an effect with National Treasury, the Department of Justice and Constitutional Development and Home Affairs undertaking evaluations as a result of participation in the training.

The NES has a focus on ensuring the use of evaluations. Findings are discussed with stakeholders and senior management. There is a process of dissemination through policy briefs and thematic workshops. Results of evaluations are presented to Cabinet which gives weight to implementation. There is a formal follow-up process through management responses and improvement plans, and there are meant to be six monthly progress reports for two years, all available on the repository (link previously provided) (Goldman et al. 2015).

Key in evaluating the NES is looking at the extent to which evaluations are being used (the wider outcome in the theory of change). Enablers of use were found to be the improvement plan system and the work of DPME. Respondents noted that DPME enhances the credibility of the evaluation and pushes the improvement plan forward. By late 2017, when the evaluation was underway, there were 25 NEP evaluations with an improvement plan in place. In some cases, it has been problematic to get the six monthly progress reports; therefore, the use component is not reflected adequately in DPME’s tracking system (DPME 2018b:22).

Table 2 summarises the instances of use¹⁶ observed in the departmental case studies and Table 3 in the provincial case studies. The preliminary evidence for use of evaluations is encouraging. Instrumental use can be seen in seven out of nine cases. The second highest recorded use is process use at five out of nine. This implies that respondents felt that they gained value from being part of an evaluation process. The evaluation found that departments and provinces appear to understand the value of evaluations and attempt to use them to inform decisions. However, this use has not yet been captured accurately through the system for monitoring of improvement plans because of erratic progress reports and therefore it is not possible to reliably assert the extent of use. In order to support greater value, the improvement plan process requires greater adherence and a stronger system to track evaluation improvement plans. M&E legislation will assist.

Some respondents did signal the challenge of resourcing recommendations in evaluation improvement plans. For example, the very first evaluation on Early Childhood Development (ECD) recommended expansion of focus to include the first 1000 days from conception and a broader basket of services, both of which have significant financial implications (Davids et al. 2015). As a result, a costing exercise had to be performed to identify the costs of services and these had to be prioritised.

To respond to this issue the evaluation categorised recommendations from a selection of improvement plans in terms of whether they had financial implications or not. For example, recommendations that relate to setting up a new unit, developing a new system, hiring additional staff and increasing funding have financial implications. Of the 400 recommendations in the 24 evaluations selected, 22% had financial implications. The bulk (70%) have non-financial recommendations which largely relate to strategy, policy and operations. This data suggests that financial constraints should only impact the implementation of 22% of the evaluation recommendations that were assessed (DPME 2018b:22). Capacity constraints, on the other hand, would impact a far larger proportion of recommendations which do have capacity implications in terms of staff and time. These are the greatest in evaluations dealing with the economy.

The institutionalisation of evaluation should be a key factor to drive budgets and efficiencies and effectiveness of service delivery. At national level evaluations have influenced budgets, for example the ECD evaluation resulted in a new ECD policy which was approved by Cabinet and an ECD conditional grant (DPME 2017:19). However, a formal link with the planning and budget process is only starting. In 2016, evaluation results were used for the first time in the national budget process. Whilst a number of evaluations are being conducted at provincial level, evaluation evidence has not surfaced in budget conversations nor has it been mentioned in pre-budget conversations when the fiscal framework for the provincial sphere is being discussed,¹⁷ with the exception of the Western Cape. A key recommendation¹⁸ in the evaluation is that evaluation informs the budget process so that departments have to engage vigorously with performance outcomes and cost-benefit analyses.

The key unintended benefits of the NES came from process use. The broader unintended benefits reported by the departments and provinces who participated in this study were: (1) An improved strategic vision as a result of using theories of change, (2) The use of ‘good practice’ in internal research after having been exposed to external evaluations, (3) An enhanced use of evaluative thinking and (4) the need to harmonise learning across structures (DPME 2018b:23).

The evaluators calculated the cost-benefit ratios for three sample evaluations selected based on the completeness of the data available on the costs and benefits of the evaluations. The ratios were 1:7¹⁹, 1:10²⁰ and 1:13.²¹ In these instances, the cost of evaluation is heavily outweighed by the benefits and implies that investing in evaluation is very beneficial for government. The evaluators concluded that whilst there is certainly value in the system, tracking the costs and benefits of the system as a whole and of individual evaluations needs to be performed more systematically so that the value of the system can be accurately assessed (DPME 2018b:18).

The evaluation found that considerable progress has been made in terms of establishing the NES particularly through evaluation plans, capacity building, quality assurance mechanisms and communication. However, there is room for improvement. The recommendations from the evaluation cover five main areas – strengthening the evaluation mandate, making sure budgets are available for evaluations, capacity development, managing and tracking evaluations, and strengthening use of evaluation results through communication and improvement plans.

The evaluations offer tremendous opportunities for interest groups to hold government to account and to push for improvements in certain specific areas based on the evidence derived from these evaluations. However, stakeholders outside of government are often unaware of this opportunity and they are, therefore, unlikely to use it.

The evaluation of the NES makes it clear that the DPME needs to broaden and deepen its view of the operations of the governance system, how policy is made and how bias and interests impact on the use of evidence. Recent thinking on how evidence is ‘governed’ suggests that DPME should think more deeply about how the evidence being generated by the NES is used in the policymaking and public domains (Parkhurst 2017). In this regard, the recommendation that the role of civil society and of think tanks in particular be clarified is especially pertinent.

A number of the other recommendations made in the evaluation report raise important issues from a system-strengthening perspective. The NES evaluation highlights the need for DPME and SAMEA to clarify their respective roles in building and growing the national evaluation ecosystem. This is an important, long-term endeavour that will need sustained engagement particularly around the twin imperatives of transforming the pool of professional evaluators and strengthening the whole evidence-utilisation pipeline from policy and strategy development through to performance improvement.