Measuring the effect of Evaluation Capacity Building Initiatives in Africa : A review

Background: The growing demand for evidence to support policy decisions, guide resource allocation and demonstrate results has elevated the need for expertise in monitoring and evaluation (M&E). Despite the mushrooming of short courses in M&E, their impact on improving the capacity to meet the demand has not been adequately and comprehensively measured or evaluated. The purpose of this article was to highlight the need for improving the measurement of evaluation capacity building (ECB) to better understand what works in building M&E capacity in Africa. Objectives: This article provides important insights into the need for empirical and rigorous measurement of ECB interventions and their role in strengthening evaluation practice. Method: The study was primarily a desktop review of existing literature, corroborated by a survey of a few senior representatives of organisations responsible for capacity building across the African continent. Results: The review found that there remains little empirical evidence that indicates whether ECB processes, activities and outcomes are ultimately effective. There is also very little empirical evidence that helps to interpret how change happens, and how this may shape ECB efforts. Training is acknowledged as only one element of ECB, and there is a need for a multi-pronged approach to ECB. Conclusion: Much more empirical and rigorous research is needed to build a clear understanding of what conditions are needed in ECB in Africa to strengthen evaluation practice. This article is useful for guiding further research into measuring the effect of ECB, as well as implementing more effective models of ECB towards strengthening evaluation practice in Africa.


Introduction Problem statement
Evaluation capacity building (ECB) has been recognised as a critical issue for development by international agreements such as the Accra Agenda for Action, the Cairo Consensus on Capacity Development as well as the Busan High-Level Forum (Lucas 2013).The realisation that development interventions need to be informed by good evidence of what works -and why -has meant that capacity building in monitoring and evaluation (M&E) has grown in leaps and bounds, largely due to the growth in the demand for evaluation by national as well as international donors, government agencies and others.Building effective M&E systems has long been recognised as a critical support mechanism for resource allocation, accountability, decision-making and programme design in the development sector (The World Bank in Ross & Hopson 2006).The growing demand for functional M&E systems in Africa has led to a rising need for skilled evaluators who are able to produce the kind of evidence that can reliably inform decision-making in policy and development programmes (Porter & Goldman 2013).There is also an emerging demand across the African continent (e.g.South Africa, Benin and Uganda) for evaluation in particular, apart from the demand for monitoring systems (Porter & Goldman 2013).The first decade of the 21st century has therefore seen the rise (and rise) of capacity building initiatives and an increasing commitment and enthusiasm for undertaking and measuring the impact of ECB (Preskill & Boyle 2008).What is yet insufficiently known is the precise relationship between ECB interventions and strengthening evaluation systems and practice, and the best methods of measuring this.
The authors designate 10 different strategies, of which training is one, in their taxonomy of factors to consider when embarking on any ECB intervention (Preskill and Boyle 2008).ECB is therefore multidimensional and training is one component of thereof.Nonetheless, the mushrooming of M&E training courses in academic institutions, private consultancies and other institutions across the African continent has been significant, and it forms the cornerstone of any ECB intervention, such that 'capacity building' is often used synonymously with the concept 'training'.
The challenge in the evaluation community is that there is still no globally ratified consensus on the guidelines for desired evaluator practice and essential evaluator competencies that is used consistently in ECB in Africa (despite the existence of legitimate competencies and standards produced by, for example, the International Development Evaluation Association, the African Evaluation Association, the Joint Committee on Standards for Educational Evaluation and others).There is also no professional body overseeing the content of M&E training material on the African continent.The quality, content and composition of training courses vary between institutions.Of the many training courses that are undertaken every year, it is not known whether these assist individuals to become better evaluators, or if they strengthen organisational evaluation practice.There is little empirical evidence available that tests whether ECB processes, activities and outcomes in general are ultimately effective (Preskill & Boyle 2008;Tarsilla 2014).Post-training evaluation forms are often completed; however, these are more about the quality of the course and perspectives about the facilitators, rather than tools to assess changes in knowledge, attitudes, practice or behaviour.In recognition of this challenge, this article seeks to contribute to the research agenda by providing newer reflections on moving beyond the challenges in the measurement of the effectiveness of ECB interventions (in particular training) in strengthening evaluation practice, towards the enhanced achievement of development outcomes on the African continent.
The problem that this review sought to unpack was the absence of adequate attempts to measure the effectiveness of ECB in strengthening M&E capacity on the African continent.The problem is compounded by the lack of a singular and agreed upon definition of what constitutes M&E capacity.This further problematises the measurement of capacity (i.e. which indicators to use, which tools can be used to measure and the level -individual or organisational -at which capacity is measured, as well as the absence of baseline indicators, which tell us what capacity was there to start with before training began).
This review therefore explores the challenges facing ECB and measuring the effect of ECB in Africa through a document study and a survey of regional institutions (including the South African Development Community, Economic Community of West African States, the African Union Commission and Economic Community of Central African States, among others) from parts of the central, east and west African ECB community.It also makes recommendations for moving beyond the status quo in the design and measurement of ECB initiatives on the African continent.

Purpose of the review
The primary purpose of this review is to explore the extent to which the effectiveness of ECB (including its contribution to strengthening evaluation practice) is adequately measured on the African continent.

Background
Capacity building initiatives in Africa have traditionally been driven by donor agencies since the inception of development aid, and take various forms.Most commonly, they include the training of individuals as a basic tenet, supported by the provision of technical support.ECB has experienced rapid growth within public sector organisations on an international scale (Naccarella et al. 2007;Tarsilla 2014).As with capacity building in general, a large part of ECB is comprised of training.Tarsilla (2014) further argues that the short-term evaluation training programmes funded by international agencies in developing countries rarely respond to local trainees' and organisations' interests and needs.He even emphasises the complementarity -and yet distinctionbetween the terms ECB and evaluation capacity development (ECD).It appears that short course provision has become the common mode of training in ECB, regardless of which institution is the provider.Despite recent attempts to validate and develop common measures of ECB (Labin 2014;Nielsen, Lamier & Skov 2011) the impact of short courses and training within the ECB suite of interventions has not been adequately and comprehensively measured or evaluated.There is largely agreement that more research is needed to develop an empirical basis to support efforts to embark on such measurement (Nielsen et al. 2011;Wandersman 2014) The specific objectives of the review were to explore: • The various approaches to defining and measuring ECB.• The extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity.• Challenges experienced in the measurement of the effect of training on ECB in Africa, and recommendations for how these may be addressed.

Literature review
The literature review focused on the following key concepts: evaluation capacity; ECB and the measurement of these.It further explores the aspect of competencies in evaluation, and how this interfaces with the concepts of capacity and capacity building, which form the back-drop to an exploration of the efforts at, and complexities of, the measurement of ECB interventions.

Defining evaluation capacity and evaluation capacity building
Significant resources have been invested in building evaluation capacity (particularly by international donor agencies) in Africa; however, it is not clear whether these efforts are yielding results (Tarsilla 2014) • The ability to sustain the capacity to use evaluative knowledge in decision-making.• Aligning the organisation's functional elements in a way that is coherent, and supports effectiveness.• Allowing evaluative knowledge in its varied forms to be used in all phases of the policy cycle.• The ability to use evaluation knowledge at several levels of practice and decision-making to improve organisational effectiveness.
This study challenged the notion that evaluation capacity is limited to the ability to conduct evaluations and use the findings appropriately (Nielsen et al. 2011).
Brinkerhoff and Morgan (2010:3) define capacity at a systems level as the 'the evolving combination of attributes, capabilities and relationships that enables a system to exist, adapt and perform'.Horton (1999:156) similarly describes capacity as 'the ability of individuals and organizations or organizational units to perform functions effectively, efficiently and sustainably'.In all of these definitions, capacity is not confined to the individual, while the notion of the ability to perform is identified as an important characteristic.
An assumption is often made that capacity leads to an improvement in performance; however, LaFond, Brown and Macintyre (2002)  The authors further posit that capacity is a 'latent phenomenon', and only becomes explicit once it is exercised in order to attain a certain outcome.Defining evaluation capacity is therefore complex and multifaceted, including the provision of financial as well as other resources (Crisp, Swerissen & Duckett 2000).
Building and measuring capacity is equally complex, it can take many forms and may occur at an individual, organisational or systems level.LaFond et al. (2002:5), describe capacity building as a 'process or activity that improves the ability of a person or entity to carry out stated objectives'.Horton (1999) on the other hand argues that capacity building is something like 'social experimentation'.
A study conducted by Tarsilla (2014), involving a sample of 21 international development organisations who perform ECB activities across the African continent, measured whether their interventions were yielding the desired results.Among the findings were the following: 92% of the respondents (n = 33) could not provide any evidence of the ECB programme suitability and 89% of the respondents (n = 31) cited that there is a need to employ practical evaluation theories and principles as opposed to the current sporadic evaluation training initiatives.These results are concerning in light of the assumption that ECB is good and leads to positive outcomes.
The lack of consensus in defining evaluation capacity and capacity building poses challenges in the measurement of ECB interventions.

Training, evaluation capacity building and the building of individual competencies
The study conducted by Tarsilla (2014) points out that international development partners have invested significant financial resources in ECB in Africa, without satisfactory results.One of the challenges highlighted by the author includes a continued focus on short-term courses despite evidence of their deficiencies in building capacity (Tarsilla 2014).Preskill and Boyle (2008) identify training as only one of 10 strategies within the ambit of ECB, and it is largely agreed that training, on its own, is insufficient as a comprehensive ECB strategy.During the provision of training, case studies may provide some insight into this and provide an answer to the need for more evidence; however, the challenge is that the learning may be very specific to the cases and context.A useful case study by Cohen (2006) illuminates how an embedded approach, where capacity building took on multiple forms (facilitation, coaching and mentorship) throughout the ECB intervention, provided positive results in strengthening evaluation capacity.The intensity of this strategy is not feasible with the current multiplying demand for ECB across the African continent, although some of the lessons regarding the 'ingredients' of effective capacity building are useful.These include, for example, the need to assess organisational readiness, the importance of teaching in context, providing opportunities for reflection and maintaining a partnership stance (Cohen 2006).Dillman (2012:280) argues that there is a need to employ a 'deliberate and intentional attitude' in how education and training interventions are designed.A combination of classroom-based and field-based work must be employed in order to build evaluation competencies.Coursework remains important in the transfer of knowledge (theory); however, to build on this knowledge and develop competencies that may strengthen evaluation practice, an eclectic mix of professional development, fieldwork and mentorship is needed (Dillman 2012).This can be supplemented by innovative methods, for example evaluation learning circles, developed by Carolyn Cohen (2006), which is a long-term immersion experience that involves participant-led learning sessions, and which resulted in very positive learning outcomes.
Training therefore remains important, but not in isolation.
A study conducted on ECB interventions in 13 communitybased organisations revealed that ECB does increase organisational evaluation capacity (Stevenson et al. 2002).
It is significant, however, that the intervention consisted of more than just training, and included a needs assessment, the identification of 'exemplary' evaluations that would be used as models for other programmes, as well as providing on-site and telephonic technical assistance (Stevenson et al. 2002).Dillman (2012) concurs that multiple and varied training experiences are needed to develop essential evaluator competencies.Although training will always be a very significant part of ECB efforts, research trends are indicating that capacity is a multidimensional concept, and therefore needs to incorporate multiple approaches to be successful (cf.Naccarella et al. 2007;Preskill & Boyle 2008;Suarez-Balcazar & Taylor-Ritzler et al. 2013).
Podems (2014) contributes to the discourse and posits that a list of competencies is a useful guide to those vested with the responsibility to design, provide guidance and teach evaluation courses.The author (Podems 2014:132) states that 'the assumption is that a person having those core skills would be a competent evaluator who would have the abilities to implement feasible and credible evaluations'.It is further asserted that the competencies guide will provide direction on the areas of knowledge and skills to be improved, further supporting practitioners to use the list as a guideline to nurture the evaluation profession rather than acting as a prerequisite to entry into the profession.The Department of Planning, Monitoring and Evaluation (DPME) in the Presidency in South Africa currently uses such a competency guideline to develop evaluation capacity short courses (Podems 2014).There are ongoing debates among 'pro' and 'anti-competency' scholars: those who are pro use of competency guidelines are of the view that the initiative will eliminate unqualified evaluators who are currently parading as consultants with the sole objective of increasing their bottom line.Dillman (2012) offers the caution that despite literature available on evaluator competencies and training programmes, there is very little evidence of how these evaluation competencies are gained.Podems and Dillman both acknowledge the danger that such guidelines may have the unintended consequence of being perceived as a barrier for aspiring new professional evaluation entrants, because it can be used to determine who is and is not an evaluator.
Training, and by extension efforts at building individual competencies, can only go as far as imparting and facilitating skills and knowledge transfer, but cannot determine the capacity to implement these.This has implications for measurement and attribution, and how the evaluation community may determine if a single training intervention has the ability to build capacity in a comprehensive sense.(Medina et al. 2015).However, these are inadequate as they do not answer the question of impact (Medina et al. 2015).'Striking differences' have been identified between the acquisition of evaluation skills and competencies (e.g.effective communication skills, project management) and evaluation knowledge in a review conducted by Dillman (2012) on evaluator skill acquisition.One of these differences is that fieldwork and mentorship were perceived as having had the greatest effect on skill acquisition, while coursework was rated as being the most impactful in contributing to the acquisition of theoretical knowledge (Dillman 2012).This was also confirmed in a tracer review

Beyond the literature review 1
The review was conducted through a literature review on ECB and its measurement, focusing specifically on the adequacy of training as one aspect of ECB, supported by a survey of key strategic institutions operating on the African continent and linked to ECB activities.The survey was conducted with a small convenience sample of key stakeholders (including organisations responsible for capacity building or specialising specifically in ECB across the African continent) who were invited to participate in a self-administered survey in the context of their responsibility for or linkages to ECB interventions across the African continent.The organisations were participants of a workshop, hosted by the African Capacity Building Foundation (ACBF) in October 2016, to identify the potential content, design and implementation modalities for an Africa-wide M&E capacity building programme.CLEAR-AA had also been invited to the workshop.It was a closed meeting, attended by a select number of organisations who are actively involved in regional integration, capacity building or M&E activities across the African continent.Permission was requested from the ACBF to distribute the survey instrument, and all of the organisations present were invited to complete the survey.
The survey instrument was developed using the following thematic areas as the point of departure: (1) the extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity and (2) challenges and recommendations in the measurement of the effect of training on ECB.The items on the survey included biographical information on the organisation represented by each respondent, including the countries within which it operated.This provided insight into the 'footprint' of the organisations on the African continent.In addition to the biographical information, the survey consisted of eight closed-ended items and three openended questions covering the following issues: the extent to which organisations were involved in M&E training, whether, as well as how, organisations measured the effect of their training (knowledge, behaviours, attitudes, practices) and challenges and recommendations to measuring the effect of ECB efforts on the African continent.
1.The 12th organisation remained anonymous In total, 13 surveys were returned by senior representatives of 12 regional organisations, listed in Table 1 (numbered 1-11, as one respondent did not provide the name of the organisation they represented).
The snap survey provided perspectives of senior staff members of key regional organisations who are extensively involved in the coordination or implementation of capacity building programmes in Africa in general and ECB specifically.Their views or responses, however, might not be representative of their entire respective organisations.
Perspectives were sought around training and its effect on strengthening evaluation capacity.The results of the survey were analysed using simple descriptive statistics (tabulated in the results section of this article), due to the sample size being too small to draw any inferences to a larger population.Qualitative data were analysed using content analysis, as no preconceptions about the categories of findings were articulated ahead of the analysis.The findings emerging from the survey results were corroborated with the findings from the literature review, and the results categorised into themes and discussed in response to the stated objectives of the review.

Trustworthiness Representativeness
There was no expectation to generalise or transfer the results of this review beyond the organisations represented in the sample.Convenience sampling, as a non-probability sampling method, precludes the drawing of inferences to a larger population.However, the corroboration of the survey results with the findings emerging from the literature studied allowed for the drawing of some general conclusions about the measurement of the effect of training on evaluation practice, and extrapolate this (in broad terms) across the African continent.

The extent to which the effect of training is measured
The survey results indicated that training remains an important component of ECB in Africa.According to Table 2, The survey results also revealed that very few of the respondents who conduct M&E training consistently measure the effect of their efforts.Table 3 illustrates that almost all the respondents surveyed (10 out of the 13) reported that they only rarely or sometimes measure the effect of their training.

Approaches to measurement and the use of findings
Table 4 illustrates that, of those who do measure the effect of their training in some way, most (7 to 8 out of 13) reported that they: • Only sometimes or rarely measure the effect of training on behavioural change (7 out of 13).• Only sometimes or rarely measure the effect of training of knowledge (8 out of 13).• Rarely measure the effect of training on attitudes (8 out of 13).• Rarely measure whether participants use the training in their day-to-day work (8 out of 13).
Respondents listed the following as the various approaches and methods used to measure the effect of their training interventions: •  5).More than half (7 out of 13) respondents only sometimes or rarely do so.
This could be related to the challenges highlighted by respondents around the measurement of their training programmes, which included inadequate data, limited resources as well as poor attention paid to evaluating the impact of ECB interventions.These challenges are further discussed below.

Challenges and recommendations in measuring the effect of training
The challenges raised by survey respondents can be categorised into four broad areas: challenges related to planning and design, weaknesses in data collection and information management, resource challenges and challenges in the capacity to measure the effect of training programmes.In terms of planning and design, survey respondents highlighted the challenge of weak planning of ECB impact assessments in particular.It was emphasised that not enough attention is paid to measuring the effect or impact of ECB interventions.One reason that was provided for this is that donor interests are in what was termed 'programmatic initiatives', and therefore activities related to measuring the impact of the training programmes are not prioritised.At least one respondent pointed out that the selection of appropriate indicators to measure the effect of training was a challenge in measuring the effect or impact of ECB.In relation to data collection and information management, respondents highlighted that tracing beneficiaries (i.e.training participants) in order to measure the effect of the training intervention posed a challenge.For those who managed to collect information from participants, the low response rate as well as what were held to be 'poor/inaccurate responses to questions' were added to the list of challenges.Inadequate monitoring data, the loss of institutional memory as well as inadequate knowledge management were also listed as areas of concern.The third category of challenges revolves around the availability of resources to support the measurement of training effects.This included both human and financial resource constraints for some respondents, where one respondent noted 'resources in terms of skills and financial to monitor impact studies of any kind' as a challenge.
One respondent also noted that the capacity to measure the effect of training posed a challenge to their ability to do any measurement of this kind.
The survey respondents also made a number of suggestions to address these challenges.These included improving the availability of proper monitoring data as well as improved knowledge management.At least one respondent suggested that tracer studies and some form of post-training follow-up must be introduced.More rigorous methods of measuring the effect of training were also recommended, including the use of impact evaluations.

Discussion
The primary objectives of this article were to: (1) explore the extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity and (2) discuss the challenges of and make recommendations for improving the measurement of the effect of training on ECB in Africa.Each of these is discussed separately below.

The extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity
The findings of the review, supported by literature (Tarsilla 2014), revealed that the measurement of the impact of training as an element of ECB is not commonplace, and that it is difficult to find empirical research on how the acquisition of skills and competencies takes place, or how to measure it.
The survey results indicated that there are very limited attempts by the institutions sampled, who have quite a significant training 'footprint' in Africa, to measure the effect of training on behavioural change, knowledge, attitudes and practices.Some of the approaches and methods listed by a small number of respondents included: questionnaires, interviews, experimental designs (i.e.treatment vs control groups), pre-post-tests (before-after comparison), focus groups as well as individual follow-up.These methods are mostly used to determine effects at the individual level.However, there is widespread agreement that capacity is acquired at multiple levels: individual, organisational and systems (LaFond et al. 2002;Taylor-Ritzler et al. 2013).Preskill and Boyle's (2008) model of ECB is instructive in this sense, as it provides an overarching approach that incorporates all three levels of learning.This has implications for how the effects of any kinds of ECB interventions are actually measured.If the measurement is confined to the individual level (as is the case with the respondents sampled in this review), important findings around organisational and systemic change may be missed.Strengthening evaluation practice in Africa should not be confined to building the capacity of individuals to become better M&E practitioners, but should be linked to a broader programme of transformation of organisations and systems to ensure better use of evidence for better development results.Perhaps the issue is not so much improving the measurement of training, but on the design of ECB interventions, of which training should form one component.
Capacity, in all its complexity, is in fact hard to measure (Tarsilla 2014).The findings also revealed that it may be easier to measure the acquisition of technical knowledge by individuals, but much more difficult to measure the effect of evaluation training on attitudes, behaviours and practices.This may explain, therefore, why the institutions surveyed in this review were more likely to measure knowledge acquisition, as compared to changes in attitudes and behaviours.According to Dillman (2012), coursework has been rated as being the most impactful in contributing to the acquisition of theoretical knowledge.Both the measurement and acquisition of knowledge are easier than more complex measurements of changes in behaviour and practices, as illustrated by Kirkpatrick's four-level model of assessing training effectiveness (Bates 2004).In his model, the extent to which skills, knowledge and attitudes are measured is located at the second stage of the evaluation of learning, rendering it more complex and difficult to measure than the transfer of learning as well as the achievement of results (Bates 2004).
The intersecting variables (such as incentives to learn, institutional support, etc.) that play a role in the extent to which evaluation practice is strengthened in individuals and organisations contribute to this complexity.The review found that there is little empirical evidence available that tests whether ECB processes, activities and outcomes in general are ultimately effective (Preskill & Boyle 2008).There is also very little empirical evidence that helps to interpret how change happens, and how this may shape ECB efforts in the future.The critical pathways that lead from ECB to strengthened evaluation practice are often hidden (at best), or not measured at all (at worst).There is broad consensus that training is only one element of ECB, and that there is a need for a multi-pronged approach to ECB, in line with what some authors (Tarsilla 2014) define as ECD.
Even within the formal academic education sector, there are hardly any evaluations of changes in knowledge, attitudes and behaviour as a result of teaching and learning that the ECB sector can learn from (Burgess & Carpenter 2008).The measurement of short-term effects has been the focus of most donor agencies involved in ECB in Africa, while the measurement of long-term results has been less forthcoming (Tarsilla 2014).Much more needs to be done around pedagogic research and ECB, as well as the development of methodologies to evaluate strengthened evaluation capacities and capabilities, which are not well developed (LaFond et al. 2002).
The need for more empirical research in this area of ECB could perhaps be filled by the use of randomised control trials (RCTs, Suarez-Balcazar & Taylor-Ritzler 2013).The argument against the use of RCTs in ECB evaluation is that the variables are too vast to control or offer predictions, while the costs are prohibitive (Tarsilla 2014).The use of multiple case studies and their comparison may be far better suited to a deeper understanding of the specific conditions that facilitate improved evaluation capacity in individuals, organisations and systems (Tarsilla 2014).Some successful research was done on the success case method, which demonstrated that evaluating a small number of cases for impact can provide justification for the future resourcing of successful training programmes (Medina et al. 2015).Overall, there is little consensus on which approaches work -and work best -in assessing the effectiveness of capacity building efforts (Burgess & Carpenter 2008), and more research is needed in this area.
The tracer study conducted by CLEAR-AA in 2015 on the perceptions of participants about the effect of in-service evaluation training conducted on behalf of the South African DPME, revealed that the highest perceived gains were made in the area of knowledge transfer and less in the management of evaluations (CLEAR-AA 2015).The survey findings give an indication that some institutions are embarking on post-training evaluation that goes beyond the testing of perceptions about the course, its content and the facilitator; however, these studies are seemingly not used internally, and are not available widely for learning to be shared.In general, very few respondents regularly use the findings of their aforementioned pre-training or post-training measurement efforts.

Challenges and recommendations in the measurement of the effect of training on evaluation capacity building
ECB initiatives in Africa are often poorly designed and planned (Tarsilla 2014), and continue to be focused on training primarily, despite its (known) shortcomings as a standalone offering (Dillman 2012;Tarsilla 2014)

Limitations of the review
The small sample size and non-probability sampling method in this review does not allow for a generalisation of the results to the entire ECB community across the continent of Africa.Furthermore, the view of the individual respondents cannot be held to be representative of their respective organisations.What we can learn from the extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity The findings have shown that capacity building is a multidimensional, dynamic and complex phenomenon, which is influenced by many elements including individuals' existing capacity, their knowledge, behaviours and attitudes.
It is also influenced by the context within which the individual or institution finds themselves.M&E capacity is not measured in a standardised manner on the African continent, and therefore it is difficult to build an empirical case for why there is a need to continue providing short course training to an ever-growing clientele.There is a critical research gap in evaluating the impact of training on changes in attitudes, behaviours and practices in the evaluation sector (as well as to some degree knowledge), and therefore more research is therefore required in this area.It is also recommended that all training interventions should include, as a matter of course, a plan for the evaluation of the effect of the training on the acquisition of evaluation competencies, which need to be agreed upon at the design phase of the ECB intervention.
There is scant use of tracer studies to track M&E training participants' acquisition of knowledge and skills through coursework programmes in the ECB sector.One is even less likely to find studies that track whether training has improved participants' ability to use their newfound skills in a way that improves evaluation practice.Evaluative thinking is a fundamental tenet of M&E, and it is a foregone conclusion that any interventions of this nature need to be monitored and evaluated to test the soundness, efficiency and effectiveness of the intervention logic.All institutions embarking on ECB interventions should therefore ensure that tracer studies are built into all training of this nature, so that ongoing empirical evidence may be built up over time about the effect of training on strengthening evaluation practice.
The write-up and publication of the results of those who do conduct studies would be very useful and are critical for the evaluation community in Africa to learn more about the strategies and methods of ECB (in particular training) that work well, and those that do not.There is also an opportunity to develop a database of findings on the evaluation of ECB efforts, which would be valuable to the evaluation community in general, and ECB individuals and institutions in particular, to inform the trajectory of future ECB interventions.
Much more empirical and rigorous research is needed to build a clear understanding of what conditions are ideal for the transfer of evaluation skills, competencies and knowledge, and the strengthening of evaluation practice at large.
What are some of the main challenges that need to be addressed in the measurement of ECB efforts, and how can these be addressed?
The multiplicity of challenges in ECB in Africa hampers the measurement of capacity building interventions.
These include the lack of clarity and consensus on the meaning of evaluation capacity, the various dimensions that constitute capacity, the levels at which capacity may be identifiable (e.g.individual and organisation) as well as the absence of indicators and baseline information on evaluation capacity on the continent.The absence of a variety of robust measures of evaluation capacity, and in some cases lack of knowledge around those which do exist, means that (1) more empirical research is needed on measuring the attribution of various dimensions of ECB on strengthening evaluation practice and (2) more work is needed to mainstream the use of instruments that do exists (e.g.Taylor-Ritzler et al. 2013, ECAI).It is also important to isolate training as a specific capacity building intervention in such research, due to its ubiquitous nature in ECB across the continent.
ECB interventions must be better designed to achieve the results they set out to achieve, and a longer-term view of capacity building needs to be adopted by institutions who are actively involved in ECB.

Conclusion
This article presents the findings of a desktop review, combined with a snapshot survey of institutions from parts of the African continent, of the measurement of ECB in strengthening evaluation practice.The review explored the challenges facing the ECB sector in general, as well as the difficulties in measuring the effect of ECB interventions, in particular training, as only one aspect of ECB.The findings revealed that the complexity of defining ECB renders it almost impossible to test whether ECB processes, activities and outcomes in general are ultimately effective.The many intersecting variables that play a role in the extent to which evaluation practice is strengthened in individuals and organisations would best be measured through empirical studies on the continent, of which there are currently very few.The review also found that there is a general gap in the research in Africa on the impact of training on improving the competencies required for quality evaluations.More work needs to be done to determine, through research, what conditions are ideal for the transfer of evaluation skills, competencies (which a number of evaluation associations are currently working on) and knowledge, and the strengthening of evaluation practice at large.
ECB has become a 'hot topic' within the evaluation field (Preskill & Boyle 2008:443).This desktop review is important as it begins to highlight the research gaps in measuring the effect of ECB and training in particular on strengthening evaluation practice in Africa.There is a need for more empirical evidence about 'what works' to inform the ways in which capacity building is defined, and how capacity building interventions are developed, implemented and measured.Sobeck and Agius (2007:237) posit that information on the processes that lead to successful capacity building, and the strategies that lead to better outcomes in this regard, can provide support to capacity building funders on decisions around where to invest.This article will be useful in guiding future empirical research into more rigorous methods of measuring the effect of training on the broader efforts at ECB, and more importantly, shed light on the quest to strengthening evaluation practice.
Questionnaires and evaluation forms • Interviews (individual and group) • Simple chi-squared test • Treatment and control (experimental design) • Pre-post-test (before-after comparison) • Most-significant change (MSC) technique • Individual follow-up • Site visits In general, very few respondents regularly use the findings of their abovementioned pre-training or post-training measurement efforts.Only four respondents reported that they always or often use the findings of the studies they conduct to adjust their training interventions (Table

.
It is important to interrogate the effects of capacity building interventions in Africa, especially in light of the mushrooming of M&E training offerings across the continent.The questions that remain unanswered include: (Volkov & King 2007)nnan 2011 efforts, and to use these results to shape future ECB efforts(Suarez-Balcazar & Taylor- Ritzler 2013:190).The instrument, consisting of 68 items, measures participants' perceptions of their ability to mainstream and use evaluation findings(Suarez-Balcazar &  Taylor-Ritzler 2013).A number of other tools have also been developed, for example, the Capacity and Organizational Readiness for Evaluation (CORE) tool(Morariu, Reed & Brennan 2011) and the checklist Building Organisational Evaluation Capacity(Volkov & King 2007); however, these do not address the question around whether or how ECB efforts strengthen evaluation capacity.Often there is an underlying assumption that capacity building interventions have a direct link with performance (whether organisations or individuals).Thus far, the research is nascent: very little empirical evidence has been established to correlate the two, or even to confirm the effects of capacity building in general(Sobeck & Agius 2007:237; Suarez-Balcazar &  Taylor-Ritzler 2013).
(Dillman 2012;kay (2014:419)assPreskill & Boyle 2008;ning opportunities for evaluators will always be important to the evaluation profession'.While this is true, there is very little consensus as to what evaluators need to know, understand and to be defined as 'trained'.The literature is clear that training programmes are limited in what they can achieve in strengthening evaluation practice and although short-term courses may be able to upgrade individual capacities around specific areas of learning, significant and lasting change is not possible with training alone(Dillman 2012; Horton 1999:182;Preskill & Boyle 2008; Suarez-Balcazar & Taylor-Ritzler 2013).
Tarsilla, in support of the Organisation for Economic Cooperation and Development (OECD) definition of evaluation capacity 'development' sees it as 'a process whereby individuals, organisations and society as a whole embarks on a path to strengthen, create and maintain evaluation capacities over a period of time'.This definition supports the notion that training is only one aspect of a much larger process in building evaluation capacity, and is focused on a much broader goal of effective development results.
scholars in measuring the impact of ECB in organisations.The success of training interventions is usually measured by using self-reporting tools and answering questions around the satisfaction of participants, knowledge gained and utilisation of the knowledge and skills acquisition address the challenges and gaps of ECB measurement (cf.LaFond et al. 2002; Lucas 2013; Sobeck & Agius 2007; Wandersman 2014); however, the problem in practice is that what is measured is not helpful and does not contribute to advancing the discourse on ECB.Taylor-Ritzler et al. (2013) acknowledge that there is a growing interest among practitioners and

TABLE 2 :
Number of respondents who conduct training in M&E.
Source: Survey results -Responses to Question 1

TABLE 3 :
Number of organisations who measure the effect of training.
Source: Survey results -Responses to Question 2

TABLE 5 :
Whether findings are used to adjust training interventions.
Source: Survey results: Responses to Question 3

TABLE 4 :
What gets measured.
Source: Survey results -Responses to Questions 4-8 http://www.aejonline.orgOpenAccess Taylor-Ritzler et al. (2013) a standalone intervention to individuals, outside of any other supporting mechanisms of ECB, it is not effective, while it has been lamented that more practical efforts are required to enable participants to apply what had been learnt in practice(Tarsilla 2014).The findings of the study conducted byTaylor-Ritzler et al. (2013)

What can we learn about the role of training in ECB and its effect on improving knowledge, attitudes, behaviour and practices in evaluation practice, and how this can be better measured?
Some of the institutions sampled are also hosts to other institutions who specialise in training provision and other ECB interventions, and this distinction was not made in the instrument.The survey instrument also did not assess whether institutions supplemented their training in M&E with other ECB interventions, which should form part of future research of this nature.The findings of this review indicate some of the research gaps in ECB as well as the kinds of empirical evidence that is required to adequately measure the effect of training on changing knowledge, behaviours, attitudes and practices.Future studies of this nature would need to adopt a more robust probability sampling method and possibly employ more extensive methods of both quantitative and qualitative data collection (such as interviews and focus groups) in order to emerge with a deeper and more nuanced analysis of the conditions under which ECB (and training in particular) are most suited to delivering on development outcomes.
coaching and other means of support) is more effective in strengthening evaluation practice.The design of ECB interventions for institutions should, ideally, include a multiplicity of strategies and activities and should (as far as possible) not constitute of training as a standalone event.