Developing South Africa ’ s national evaluation policy and system : First lessons learned

This article describes the development of the national evaluation system in South Africa, which has been implemented since 2012, led by the Department of Planning, Monitoring and Evaluation (DPME, previously the Department of Performance Monitoring and Evaluation) in the Presidency. It suggests emerging results but an evaluation of the evaluation being carried out in 2015 will address this formally. Responding to dissatisfaction with government services, in 2009 the government placed a major emphasis on monitoring and evaluation (M&E). A ministry and department were created, initially focusing on monitoring but in 2011 developing a national evaluation policy framework, which has been rolled out from 2012. The system has focused on improving performance, as well as improved accountability. Evaluations are proposed by national government departments and selected for a national evaluation plan. The relevant department implements the evaluations with the DPME and findings go to Cabinet and are made public. So far 39 evaluations have been completed or are underway, covering around R50 billion (approximately $5 billion) of government expenditure over a three-year expenditure framework. There is evidence that the first evaluations to be completed are having significant influence on the programmes concerned. The big challenge facing South Africa is to increase capacity of service providers and government staff so as to be able to have more and better quality evaluations taking place outside of as well as through the DPME.

This article describes the development of the national evaluation system in South Africa, which has been implemented since 2012, led by the Department of Planning, Monitoring and Evaluation (DPME, previously the Department of Performance Monitoring and Evaluation) in the Presidency.It suggests emerging results but an evaluation of the evaluation being carried out in 2015 will address this formally.Responding to dissatisfaction with government services, in 2009 the government placed a major emphasis on monitoring and evaluation (M&E).A ministry and department were created, initially focusing on monitoring but in 2011 developing a national evaluation policy framework, which has been rolled out from 2012.The system has focused on improving performance, as well as improved accountability.Evaluations are proposed by national government departments and selected for a national evaluation plan.The relevant department implements the evaluations with the DPME and findings go to Cabinet and are made public.So far 39 evaluations have been completed or are underway, covering around R50 billion (approximately $5 billion) of government expenditure over a three-year expenditure framework.There is evidence that the first evaluations to be completed are having significant influence on the programmes concerned.The big challenge facing South Africa is to increase capacity of service providers and government staff so as to be able to have more and better quality evaluations taking place outside of as well as through the DPME.

Background
This article documents the development of South Africa's national evaluation system (NES) from late 2011 to mid 2014, a period which involved establishing many basic systems and in which 39 evaluations were started and 11 completed.The article discusses how the system developed, the approach used, some of the key systems, initial outcomes in terms of improved programmes and reflects on the lessons at this stage.It builds on a framework for institutionalising evaluation systems by Goldman and Mathe (2014).It provides an indication of emerging results, but does not attempt a systematic assessment, which will follow an evaluation of the evaluation system being carried out in 2015 and 2016.
Major investment by government in monitoring and evaluation (M&E) started in the 2000s.In 2007, the Presidency issued the policy framework on the government-wide M&E system, which linked performance information, official statistics and evaluations and coordination of various role-players at the administrative centre of government to champion M&E practices in government.At that stage there was no formal evaluation system, although there were emerging pockets of practice in some few sectors such as the Department of Social Development (DSD), Public Service Commission (PSC), as well as donor-driven evaluations.
Leading up to elections in 2009 there had been significant changes in South Africa's economy and society.The economy emerged from a long period of stagnation, achieving growth rates of over 5% by the mid 2000s.Access to potable water rose from around 62% of households in 1994 to over 90% of households.By 2008 and 2009 over 13 million people were receiving social grants of some sort.However, inequality remained a major problem and the Gini coefficient (based on expenditure) was stubbornly high, falling slightly from 0.64 in 2000 but still 0.63 (DPME 2013a).Despite the improvements there was a lot of dissatisfaction in the country, with the achievements not keeping pace with people's expectations.The discourse was of problems with 'service delivery' and there were widespread service delivery protests, which was of significant concern to the ruling party approaching the 2009 election (Van Holdt 2013).The incoming administration saw M&E as a tool for improving government performance and so addressing issues around delivery (Phillips et al. 2014).The policy decision to establish the Ministry in the Presidency for Performance Monitoring and Evaluation in 2009 and a Department for Performance Monitoring and Evaluation (DPME) in 2010 was a watershed for M&E in the country.DPME's initial efforts focused on planning and monitoring the 12 priority outcomes, a management performance assessment system and monitoring front-line service delivery (Phillips et al. 2014).A project in the Presidency (Programme to Support Pro-poor Policy Development, 1 PSPPD) started working with DPME as soon as it was established and supported the development of the evaluation mandate of the DPME.In early 2011 DPME with the support of PSPPD brought together those departments already undertaking evaluation to share their experiences and to learn from the foremost exponents of evaluation systems, notably Mexico, Colombia and the US.A study tour was organised in June and July 2011, taking the deputy minister, director general and a team of officials across government.The study tour provided some important learning and a major impetus for development of the evaluation system.Some of the key learnings from these countries were (DPME/PSPPD 2011:i): • To ensure the credibility of evaluation, one needs to show the independence and quality of evaluation.• The need for different types of evaluations, standardised systems to overcome limited capacity and an annual or rolling multi-year evaluation plan.• A budget allocation for evaluation in the range of 2% -5% of programme budgets.• A central capacity is needed to support evaluations in government, both developing policy, systems and supporting methodology and quality assurance.• Improvement plans should be developed based on the evaluations and their implementation closely monitored.
In August 2011 the group that went on the study tour met in a writeshop 2 to develop the policy framework and a final version was approved by Cabinet on 23 November 2011.Goldman and Mathe (2014) have drawn from experiences of different countries analysed by Mackay (2006;2007;2009) and Kusek and Rist (2004), as well as the framework for large-scale organisation change developed by Goldman (2001), to draw out an analytical framework for institutionalising an evaluation system that has impact (see Box 1).This is used to reflect on some of the learnings later in this article.

What the NES aimed to achieve
The national evaluation policy framework (NEPF) describes evaluation as: The systematic collection and objective analysis of evidence on public policies, programmes, projects, functions and organisations to assess issues such as relevance, performance (effectiveness and efficiency), value for money, impact and sustainability and recommend ways forward.(DPME 2011:iii) 1.A partnership between the presidency and the European Union focusing on evidence-based policymaking.

2.
A workshop during which the outline content was developed together; then different authors wrote up different sections, all within the workshop environment.
The NEPF suggests the purposes of evaluation are (DPME 2011:vi): • Improving performance (evaluation for learning).
• Evaluation for improving accountability.
• Evaluation for generating knowledge (for research) about what works and what does not.• Improving decision-making.
The M&E system in government when DPME was established was predominantly a compliance-based system, above all in terms of accountability to National Treasury and the Auditor General (see Chitepo & Umlaw in press;Phillips et al. 2014).To ensure that evaluation results were used, it was important to change this culture.In addition, capacity in government to undertake or manage evaluations was limited.As a result the approach adopted in the NEPF included certain considerations, which are discussed below (DPME 2011).
To start with enthusiasts are required who would be supportive and likely to implement the findings; hence, departments are invited to submit proposals for evaluations.Ownership of the evaluation is kept with the custodian department, although DPME is a full partner (and sometimes other departments).
With the limited capacity the focus had to be on important programmes or policies -either because they were large or strategic -and selection of policies or programmes to evaluate is on this basis.DPME also needed to work closely with departments in a learning-by-doing process and so the core support is from a DPME evaluation specialist who supports evaluation steering committees through the whole evaluation process, thus ensuring that standard systems are used.DPME needed to build agreement across government around the approach so that it is not seen as imposed.Hence, an early decision was to create a cross-government Evaluation BOX 1: Analytical framework for looking at institutionalisation of monitoring and evaluation systems.

Enabling conditions
1. Key role of a powerful and capable central 'champion' with sustained political will for the long haul and a coalition to support.2. Utilisation seen as the measure of 'success'.3. Substantive government demand.4. The importance of establishing incentives (including the ability to use hard and soft authority effectively to enforce change). 5. Performance management and M&E system that promotes interaction and variety and is dynamic.

The process
6.A clear diagnosis of the existing situation and an understanding of where delivery must improve.7. The reform strategy and plan defined before the structure, so a clear policy direction with a commitment to results.8.The process should not rely on legislation and regulations to be implemented.9.A clear and effective implementation strategy.10.A talented team to drive the system and solve problems early and rigorously.11.The courage to rethink processes completely.12. Experimentation, piloting and scaling up.13.A major investment in communication.14.Care not to over-engineer the system.15.Establishing the culture and capacity to analyse, learn and use M&E evidence.16.Role of structural arrangements to ensure M&E objectivity and quality and reliable ministry data systems.
Technical Working Group (ETWG) as a sounding board and to be the advocate of the system.To start with the system was not made compulsory except for evaluations under the National Evaluation Plan (NEP); later, the system would become government-wide.
Evaluations have to be credible, enabling confidence in findings and recommendations.Except in the case of design evaluations, all evaluations are commissioned to external service providers to ensure independence and credibility.As expressed by Minister Radebe at his first meeting with DPME managers in September 2014, 'we can't expect departments to evaluate themselves'.
Evaluations go to Cabinet and are made public, so that results have to be faced and used.
The system does not stop with the evaluation itself but an improvement plan process follows completion of the evaluation, wherein the departments involved have to act on the recommendations.

Implementation of the system
The stages of implementation At the same time as the NEPF was being developed the decision was taken to start some pilot evaluations.The initial concept for an outcomes evaluation and research unit to drive the evaluation system was approved in September 2011.The purpose was 'to coordinate the evaluation function in government, ensuring that high quality evaluation and research underpins public policy and programming, so maximising the impact of government policy and services' (Goldman 2012).By September 2014 the team had grown to 15 staff.The key roles envisaged for the unit were (Goldman 2012): • Developing and maintaining the policy framework for evaluation in government, as well as a three-year and annual evaluation plan.• Building a cross-government approach to take forward evaluation and to ensure that evaluations are used to inform plans and budgets.

Support systems
Experience has shown that a top-down approach is often characterised by resistance and failure to utilise evidence from evaluations (e.g.see Mackay 2007:96).DPME's approach aimed to create broad buy-in across government to stimulate demand for evaluation and to leverage on the scarce evaluation skills in South Africa.
As evaluations were not conducted widely in government, a number of support systems were established, including standards for the quality of evaluations, guidelines for how to conduct elements of the evaluation system, evaluation competencies, various capacity development elements, a quality assessment system and communication elements.
All of the guidelines and tools are available on the DPME website.
Building a coalition to support evaluation started from the outset.The initial study tour to Mexico and Colombia included officials from departments already undertaking evaluation (PSC, Department of Basic Education and DSD), who participated in writing the NEPF; this involvement created broad ownership of the framework.The group involved in the study tour were the embryo for a crossgovernment ETWG.The ETWG has been involved in major decisions on the reflecting on emerging lessons and selecting evaluations to be proposed to Cabinet for the NEP.
This coalition has been used to build intergovernmental commitment to the system; as potential champions emerge in other spheres they are being included, for example involving the two first provinces in the ETWG.The explicit intent of the system was to start national, then develop provincial involvement and evaluations plans for each department.This is indeed what has happened with 15 of the 46 national departments now involved, five out of nine provinces and five departments developing DEPs.This staggered approach is to enable testing of the system, but also being realistic of the capacity within DPME to support the system.For this reason too the approach has been not to focus on local government at this stage, but to systematically start focusing on the big cities (the metro municipalities) in 2016 and 2017.
As evaluations start emerging at provincial level, issues of how to iterate or align evaluations across spheres is starting to emerge.
In 2012 DPME developed a set of evaluation standards, building on international experience (DPME 2014e).DPME have applied these standards in developing a quality assessment tool, which is applied to all evaluations once completed (see Goldman et al. in press).
One of the ways to ensure minimum standards and ensure that evaluations follow the NES is the use of guidelines.DPME has developed 18 practical guidelines and templates on various components of the evaluation process.These range from a guideline on developing evaluation terms of reference (DPME 2014c) to developing an improvement plan (DPME 2014d) and are available on the DPME website.The guidelines and templates have helped to drive a common understanding on evaluation issues and terminology.
A range of capacity development tools have been used, ranging from developing competencies for evaluation (DPME 2014a), learning-by-doing support through direct experience of undertaking evaluations, just-in-time short courses to help staff working on evaluations and building capacity of senior managers and MPs to demand and use evaluation results.This involves working with a wide range of stakeholders of which a key partner is the Centre for Learning on Evaluation and Results (CLEAR).
Other aspects to improve quality are a peer review systemwhere a methodology and content peer reviewer are involved in each evaluation.The system drew from experience with the International Initiative for Impact Evaluation (3ie): the contracting process used by 3ie was adapted to DPME (DPME 2013c).In addition, a system of design clinics was developed using top national and international evaluators to support evaluation teams to develop the theory of change, evaluation purpose, evaluation questions and methodology.Three design clinics have now been run in 2013 and 2014 (DPME 2014b).
DPME has undertaken advocacy and promotion of evaluation as a discipline.This has involved publications and presentations in national and international conferences, to senior management of departments, to Cabinet and to Parliament.DPME has also signed a memorandum of understanding with the South African Monitoring and Evaluation Association (SAMEA) to promote M&E in South Africa.Areas of collaboration include capacity development and learning activities, dissemination of M&E, evaluation standards and competencies and professionalising evaluation.Beney et al. (in press) discuss this collaboration in more detail.
Effective communication of evaluation results is needed to ensure utilisation of evaluation findings.Once the evaluation has been to Cabinet, all evaluation reports, the management response to the evaluation, the improvement plan and progress reports are placed on the website.As the evaluation is primarily 'owned' by the line department, the extent to which evaluation results can be communicated is dependent on the line department's willingness to engage a wider audience on the evaluation results, which may not be the case when results reflect negative outcomes.Amisi (in press) discusses this aspect of DPME's work in more depth.
The change management approach adopted requires senior managers to see the point of doing evaluations.In order to stimulate demand for evaluations a course has been developed in evidence for directors general (DGs) and deputy DGs.Another focus has been on MPs and training and awareness raising activities have been run to see how evaluation could assist them in their oversight function, including taking the Parliamentary portfolio committee to which DPME reports to the US, Canada, Kenya and Uganda.

Results from the evaluation system
The proposed impact of the NES in the evaluation logframe is 'improved performance and accountability of government programmes and policies as a result of evaluation' and, at outcome level, 'evaluation and research evidence [that] informs changes to government interventions' (DPME 2013d).The relationship between evaluation and its use is complex (Jones, Adatta & Jones 2009), contingent on a number of factors such as political imperatives, capabilities to use evidence and so on (Young 2007).As many evaluations start to be presented to Cabinet from 2014, how Cabinet responds will set the precedent of how seriously evaluations are taken.The demand-driven approach used in South Africa has sought to maximise the likelihood of alignment between the evaluation and departmental willingness to use the findings.
As indicated earlier the evaluations undertaken are implementation (26), impact (7), diagnostic (3), design (1) and economic (1) (DPME 2014b).Whilst DPME is promoting an outcome-based approach, rather than activities and outputs, to some extent the initial evaluations have been constrained by the lack of data to enable impact evaluations.This means that the majority of the initial evaluations are looking more at efficiency and relevance rather than effectiveness in achieving outcomes and impacts, although in most cases some outcome level data is being used.
Where impact evaluations have been undertaken (e.g. on Grade R), they are proving powerful.As the evaluation system becomes established it is very important to ensure that more impact evaluations are undertaken; the revised guidance on programme planning is bringing this in at the design stage.
The first way to see progress in the use of evaluations is through implementation of improvement plans produced after each evaluation.As most evaluations only started in October 2012 it is early days to see concrete impacts of the evaluations (whether symbolic, conceptual or instrumental), but some initial results are indicated and an evaluation is planned for 2016 and 2017 to help understand their use.There is already evidence of significant followup on some of the first evaluations to be undertaken.On ECD (see Davids et al. in  Programme and land restitution) led Cabinet to decide that there needed to be an integrated implementation strategy; a policy evaluation of support for smallholder farming is starting to bring together the findings from a range of programme evaluations for an integrated response.The evaluation of nutrition interventions for children under 5 highlighted the challenge of stunting and is raising the profile of nutrition, an important symbolic use.As a result a target has been adopted in the medium-term strategic framework to drop stunting of children under 5 from 21% to 10% in five years, even before the report went to Cabinet.The Business Process Services Scheme has been relaunched incorporating the findings from the evaluation, with a number of instrumental recommendations that have been adopted.
Despite these encouraging results there is also evidence of delays by departments in producing and reporting on improvement plans (DPME 2014b) and some evaluations are proving more difficult to take forward, depending on the political dynamics in the sector.
Beyond their use, the evaluations that have been completed to date (11) have all scored well in the independent quality assessment, averaging 3.7 out of 5 which is well above the minimum quality threshold of 3, so the quality is acceptable (Goldman et al. 2015).However, the quality can improve, reflecting challenges with the quality of service providers -and in some cases DPME staff are having to put in considerable effort to get evaluation reports to a suitable quality.
Apart from the effects of specific evaluations, a significant effect of the NES has been the elevation of the status of evaluation as an essential element of the work of government.
The NES 'forces' the placement of evaluation on the agenda of government departments, responding to the request from DPME to government departments to submit proposals for evaluations in the NEP.Two departments that have been involved in the NES see the following benefits for government departments: 3 • It provides a framework from which to develop departmental evaluation policies and guidelines.• It provides a mandate and an impetus to initiate evaluations.• It provides technical and budgetary support to facilitate the commissioning and management of evaluations.• It lends credibility to evaluation findings, by facilitating built-in quality assurance mechanisms.
3.These were provided by Thabani Buthelezi and Hersheela Narsee, heads of evaluation in their respective departments of DSD and DHET respectively, and co-authors of this article.There is a lack of congruence between the main planning system (strategic plans and annual performance plans), the budget system and implementation systems, often via implementation programmes.This is one of the reasons for poor implementation (DPME 2013d).This guidance includes undertaking a diagnostic prior to developing the new programme, the development of a theory of change, development of a logframe, an evaluation cycle, risk matrix, budget and GANTT chart.In fact, the practice that has emerged with the NEP evaluations is to retrospectively do theories of changes and logframes for each of the programmes being evaluated, against which they are evaluated.This has proved very beneficial to departments in understanding their programmes, as well as in undertaking the evaluations.
The second issue is that actual spending on implementation programmes across government is often not known.For this reason National Treasury and DPME are undertaking expenditure reviews to ascertain the real levels of expenditure for these implementation units across the spheres of government (DPME 2014b).A third issue is that the plans of implementation programmes are often poor: 38% of programmes being evaluated do not have clear programme documents; further evaluations are showing that where plans are in place many need major redesign (DPME 2014b).This means that a good number of programmes are not effective or efficient as currently designed and are, therefore, not achieving what government intends.
To address this DPME has developed with National Treasury a guideline on planning implementation programmes (DPME 2013d), which was approved by Cabinet in August 2014.DPME has also developed a guideline for design evaluations (DPME 2014f.) to be conducted by M&E units of departments as an assessment of the rigour of the design and the likelihood that they will succeed.

Lessons and emerging challenges
This section summarises lessons about the system and the change management approach adopted, drawing from the analytical framework introduced earlier (Table 2).This shows that a number of the proposed elements identified do seem to be important, and there is evidence that they are functioning more or less as intended.Despite a number of evaluations being completed, and evidence of early impacts, there is a long way to go before culture will be changed so that senior management and programme managers take the initiative to encompass evaluations as part of their routine work.Furthermore, much remains to be done to strengthen M&E units in government departments to enable them to initiate and support evaluations, as well as amongst contractors who provide evaluation services.In the main, major policy and programme reviews continue to remain the purview of ministers -they can be kept confidential unlike NEP evaluations -and M&E units in departments play an insignificant role in these processes.Notwithstanding current realities, the NES does provide important enabling conditions to institutionalise evaluations in government departments, the forum of DGs is encouraging DPME to submit proposals for evaluations where departments are not putting key programmes forward, and Cabinet is responding very positively to evaluations.
There are a number of areas where problems have emerged around the planning, budget and M&E system, and where the evaluation system and its linkages could be strengthened.These are shown in Table 3 as well as areas that need to be addressed in future (DPME 2014b: 18-19).These range from poor quality of programme plans, inadequate capacity of service providers, the poor quality of programme monitoring data and departments not planning impact evaluations when programmes were designed.
As stakeholders become familiar with the system there is evidence that they wish to upscale it, for example departments such as DTI, DST and DSD developing DEPs to undertake further evaluations, or the Western Cape Office of the Premier approving three PEPs with 21 evaluations.As of March 2015, eight evaluations have been to Cabinet and Cabinet's response to the evaluations has been very encouraging, extensively discussing the findings and taking the results very seriously.As the pipeline continues in 2015 this response will become very clear.DPME is working with three new provinces to develop PEPs in 2014 and 2015, so potentially five out of the nine provinces would have plans in place by 2015 and 2016.
At present DPME's capacity to support DEPs is limited, but this is intended to be an important component from 2016 to 2017, so that evaluation is internalised across government.
To do this a major drive is needed to build capacity both in service providers and government.This will require a close collaboration between training providers such as universities, SAMEA and DPME.Original Research

Element
Importance Degree to which it has been achieved in the national evaluation system or other cases

Enabling conditions
Key role of a powerful and capable central 'champion' with sustained political will for the long haul and a coalition to support.
Critical.† DPME has played an institutional role in South Africa and similar champions in Mexico, Uganda, Benin and Colombia.† Utilisation seen as the measure of 'success'.
Critical and the focus has been appreciated.† Key focus of the system.Early indications are that utilisation is likely to happen but more evidence is required.‡ Substantive government demand.
The system has been designed as demand-led.This does seem to be critical.† Formalised in the NEP.Requests coming from new sectors but some sectors avoiding.DPME responded by proposing in areas where requests were not forthcoming.Some enthusiasts emerging.Also promoting collaborative approach so departments feel they are co-owners of the system.† The importance of establishing incentives (including the ability to use hard and soft authority effectively to enforce change).
Incentives are critical and both soft (e.g.part-funding by DPME) and hard are likely to be important.† At the moment concentrated on soft authority to create a pull not push environment.Evidence that does increase likelihood of uptake of findings.† Performance management and M&E system that promotes interaction and variety and is dynamic.The effort to make a system that is seen to be widely owned, that departments can influence and that develops seems to be contributing to acceptance.† Interaction is being promoted between programme managers and M&E staff.
Buy-in of DGs is being sought.
A variety of evaluations are possible and departments are supported to develop the most appropriate.Simplistic marking of achievement is not encouraged and rather deeper understanding of the processes is occurring.
A dynamic approach is being used that focuses not only on products, but also on the evaluation process, how interventions work and how they can be strengthened.There is a strong accent on learning, not judging.† The process A clear diagnosis of the existing situation and an understanding of where delivery must improve.
This aspect does not seem so essential if the right group of stakeholders with deep knowledge of the system comes together to design.‡ This has happened based on a group of departments who are active in evaluations.DPME has subsequently undertaken other diagnostic exercises around M&E. † The reform strategy and plan defined before the structure, so a clear policy direction with a commitment to results.It would appear to be critical to have a clear intent and approach before setting up structures.† South Africa developed the evaluation policy before starting to implement the system which has given a strong direction.This is not true of other countries.† The process should not rely on legislation and regulations to be implemented.Legislation does not appear to be necessary before starting processes.
Not being legislated also allows for systems to be developed and tested before freezing them in laws and regulations.† Appears to be valid.Not present in SA; the system has been established based on executive decisions and has survived a new administration.May benefit more going forward.Other countries such as Mexico and Colombia have more legalistic systems where legislation may be essential.† A clear and effective implementation strategy.
Critical.† This is present.† A talented team to drive the system and solve problems early and rigorously.

Critical. †
As above.Also true in Mexico, Uganda, Benin and Colombia.† The courage to rethink processes completely.
New systems may be needed.Where there are none this is easier than where systems exist.† In evaluation new systems were created as there were no national evaluation systems.This has helped in trying to change the predominant compliance culture.† Experimentation, piloting and scaling up.
Essential.† Visible in SA, also Uganda and Benin.Some departments emerging as early adopters.DPME working in learning-by-doing process with departments.Flexible support from PSPPD critical in getting system started as well as learning from other countries.† A major investment in communication.
Essential.† Significant efforts within government but not yet to the public and wider stakeholders.‡ Care not to over-engineer the system.

Conclusion
South Africa is one of a few countries that have developed an NES -Mexico, Colombia, Canada, Chile, Uganda and Benin are the notable examples.In these countries evaluation is a means of understanding in depth what is working and not working in government programmes and policies and to see how to strengthen them.In some cases (notably Chile) there is a strong budget linkage (Mackay 2007), as is likely to happen with the expenditure reviews in South Africa.
In order to develop a credible system a fairly complex system has been developed in South Africa.The system is working: overall satisfactory evaluations are emerging and first evidence is that these are having a significant impact on programme design and implementation.
The South African model has drawn considerably from Mexico and Colombia in particular.Some of the particular characteristics of the South African model emerging are: 4 • The NEPF gave clear direction to the system from the beginning (Diego Dorado, World Bank, personal communication).• The system has developed very quickly, essentially in two years.This is partly as it was able to build systematically from previous experience and has maintained strong links to other exponents.
4.Note that this list was developed from a list circulated to international colleagues asking for their views on areas where South Africa's system was making a contribution; some endorsed suggestion by the authors and suggestions by international colleagues (who are named).
• It is led from the Presidency -so is very central (rather like Uganda and unlike Mexico, whilst Colombia has strong links with the Presidency).• It is explicitly working at the supply and demand sides of evaluation, with the latter working with DGs and parliamentarians to stimulate demand for evaluations and other forms of evidence (others are to some extent too).• It has a demand-driven approach, stimulating departments to ask for evaluations and to encourage them to use evaluation results.Not getting evaluations from some sectors Raising gaps with Cabinet and proposing possible evaluations for them to select from.Targeted work with areas of low uptake (e.g.health, local government and public service).
Cabinet to consider priorities they would like to be evaluated.
Discuss what cross-cutting evaluations are key for local government.
Some departments taking a very long time to procure DPME to procure wherever possible.Evaluations where departments procure not prioritised in the NEP but rather included in departmental evaluation plans.
Inadequate supply of strong evaluators Advocacy work at universities to encourage them to participate.Develop course to assist researchers to understand evaluation.
Capacity building work with service providers.
Developing training courses and briefings in 2014-2015.Undertake rating system of service providers and publicise the results.
Diagnostic on the supply of qualified evaluators.Fundraising for this.
New call for evaluation panel in August 2014 to expand the group to draw from.
In process.
Inadequate data for some evaluations to be viable Developing model for evaluability assessment and apply in 2015-2016.
Work to improve administrative data quality and also programme data collection.
Encourage all first evaluations to be implementation evaluations, only after which do we consider an impact evaluation.
Departments to plan impact evaluations at programme inception.
Departments taking too long to take forward evaluation results, including improvement plans Standard now being applied that DPME takes the evaluation to Cabinet.DPME is having to keep reminding departments about completing the improvement plans, and progress reports.
Cabinet to note the problem.Include this in Auditor General monitoring and Management Performance Assessment standards.

Improve communication of evaluation findings
Testing out with next evaluations including policy briefs, seminars and development of a communication strategy.
See how this works and additional inputs needed.
Departments slow to produce improvement plan progress reports Repeated requests and highlighting the problem.See whether the Auditor General can audit reporting on improvement plans.Also include in Management Performance Assessment standards.
Additional capacity needed to support provincial and departmental evaluations Supported two provincial evaluation plans in Western Cape and Gauteng to test the system.Now working with five other provinces.
Strengthen imperative to take forward.In 2016 and 2017 major focus on DEPs.
DPME, Department of Planning, Monitoring and Evaluation; DEP, departmental evaluation plans; NEP, national evaluation policy.
years.This suggests a minimum rate of return of over 30:1.Financially this suggests a strong argument that evaluation is a good investment to ensure programme effectiveness, efficiency and sustainability.
A challenge will be upscaling so this can happen across government.Over the next 10 years major capacity development is needed both for service providers and also in government.Parallel work will be needed on programme planning and also on budgeting.Together with expenditure reviews these promise to provide major opportunities to improve government performance and accountability.

TABLE 1 :
Status of evaluations as of 31 July 2014.

Served at Cabinet Approved reports † Evaluation underway Terms of Reference approved Preparation stage Stuck
Source: DPME, 2014b, Annual report on national evaluation system 2013-14, DPME, Pretoria †, Approved by Evaluation Steering Committees.
• It seeks to ensure that evaluation reports do not sit on shelves, but are used to improve interventions.• It draws attention to key policy issues by ensuring that evaluation reports are tabled in Cabinet.• It facilitates inter-departmental cooperation and coordination on issues that are cross-cutting.

TABLE 2 :
Lessons against the analytical framework.
Major efforts made at supply and demand side but major capacity deficits in SA.Long way to go.Improvement plan is the key system to ensure learnings implemented -not yet clear if the improvement plan system is working.‡ Role of structural arrangements to ensure M&E objectivity and quality and reliable departmental data systems.Critical.The belief that M&E systems are objective and valid is critical to the integrity and trust in the systems.† Major efforts have been made to ensure the rigour of the findings.There is less success around administrative data which is problematic.Also issues around capacity of service providers.‡ Source: adapted from Goldman, I. & Mathe, J., 2014, 'Institutionalisation philosophy and approach underlying the GWM&ES in South Africa', in F. Cloete, B. Rabie & C. De Coning (eds.),Evaluation management in South Africa and Africa, pp.554-571, African Sun Media, Stellenbosch †, indicates likelihood of success, ‡, where it is not sure.M&E, monitoring and evaluation; DPME, Department of Planning, Monitoring and Evaluation; NEP, national evaluation policy; SA, South Africa; PSPPD, Programme to Support Pro-poor Policy Development; DG, directors general.

TABLE 3 :
Key challenges and how these are being addressed.Poor programme planningDevelopment of guideline on planning new implementation programmes approved by Cabinet, as well as design evaluations.Do audit of implementation programmes in government.Refine course in planning implementation programmes.Develop course in design evaluation and roll out.