No theoretical framework currently exists that can readily be used for studying the effectiveness of NES (as one type of CoP) in building M&E capacity. To identify factors that may influence NES’ performance in capacity building, we have found inspiration in the relatively scarce literature on COP’s effectiveness (see Li et al. 2009a; Ranmuthugala et al. 2011a, 2011b), the organisational performance literature (including resource mobilisation theory, see e.g. Edwards & Mc Carthy 2004; and political opportunity structure theory, see e.g. Tarrow 1994) and the evaluation capacity building literature (see e.g. Labin 2014; Labin et al. 2012; Nakrosis 2014; Nielsen et al. 2011). An online survey in English and French was elaborated in 2011 to collect data on national evaluation societies in low- and middle-income countries. Forty NES participated in the survey out of a total of 67 identified national evaluation societies, yielding a response rate of 60%, which is relatively high for an online survey. The survey focused on the goals and activities, the perceived performance of the evaluation societies as well as on organisational features. The ‘most knowledgeable respondent’ was targeted (see Bryman 1989) resulting in the majority of the respondents (67%) being president/board member/founding member, while the others identified themselves as mere members of the evaluation societies (19%) or M&E experts (14%). Information from the survey was triangulated with other sources of information such as interviews and informal discussions with some members of the African evaluation networks and our own participant observations.

In this article we focus specifically on ‘M&E capacity building’, which NES labelled as their second most important goal of the five goals which were identified (see Holvoet, Dewachter & Gildemyn 2011) Survey findings highlight that our respondents perceive the evaluation capacity of the NES members to have changed over the years (since 2006). Only one NES indicated deteriorating evaluation capacity, 25 NES felt evaluation capacity had (much) improved, while eight NES indicated a status quo (see Figure 1).

However, this increase in the members’ capacity is not necessarily brought about by efforts from the NES. Hence, a separate question in the survey attempts to gauge the extent to which NES have contributed to capacity building of the evaluation capacities of their members. Figure 1 shows that (at least in perception) some 35% of NES have contributed much or very much to capacity building among their members, while 65% indicated that they did so only to a limited extent or even not at all.

Secondly, survey respondents were asked to rate to what extent organisational capacity factors (human resources, financial resources, social resources, etc.) and context-related factors (political openness, government interest, donor interest, etc.) have been beneficial in increasing the M&E capacity of NES members. More specifically, the respondents were asked to indicate (on a seven-point scale, where 1 = completely insufficient and 7 = completely sufficient) the degree to which they felt the resources under review were sufficiently available to the evaluation association to strengthen the M&E capacity of the members. The factors identified by the survey respondents as the least at their disposal (= lowest score) and most available (= highest score) to strengthen M&E capacity are included in our analytical framework. For strengthening the evaluation capacity of the members, NES indicated that financial and human resources were the most important stumbling blocks. Given that 75% of the NES in our survey had no paid staff and 70% of NES had a budget of less than $10 000, the lack of financial means and staff seems to be indeed widespread among NES. Evaluation societies therefore build heavily on the work of volunteers. These findings are in line with the evaluation capacity literature in which the availability of resources (staff/time/money) is often highlighted as one of the most important organizational-level factors for capacity building (see Labin et al. 2012). Similarly, the resource mobilisation theory highlights the importance of the level of resources available to an organisation for the amount of action an organisation can undertake (see Cress & Snow 1996; Edwards & McCarthy 2004; Eisinger 2002).

In line with the literature on CoP success ingredients (see e.g. ADB n.d.; James 2010; Meessen et al. 2011), NES’ most positively rated asset was found to be their membership base (average score = 5.62). Members have very diverse professional backgrounds, ranging from government agencies and civil society organisations, to academia, the private sector, donor agencies, parliament and even the media. There is a plethora of expertise in different evaluation methodologies and skills available (quantitative, qualitative, mixed methods) among the membership. While the diversity in methodological skills enables mutual learning and cooperation among members, the width in terms of professional sector and the concomitant networks facilitates rapid diffusion and dissemination of information and, in the long run, it can build trust beyond the borders of one’s own sector. NES members with different professional backgrounds also tend to have access to different types of information and resources. Parliamentarians often have easier access to national-level policy data, and academics often have a comparative advantage in terms of analytical and evaluation tools, while civil society organisations often have easy access to information about the actual implementation on the ground and citizens’ perceptions of policy effects at local level. This allows for useful triangulation of data from various sources, and may stimulate discussion and counter-analysis, while also helping to address the policy-to-implementation and research-to-policy gap (see Holvoet, Gildemyn & Inberg 2012). Finally, the diverse membership base nurtures the creation of interfaces between M&E demand and supply-side actors, which is likely to contribute to tacit knowledge on how to use M&E.

Another critical ingredient for success that is often referred to in CoP as well as M&E capacity building literature is face-to-face contact (see e.g. Fleming & Easton 2010; Labin et al. 2012). While the upsurge of ICT has led to an increase in virtual exchange opportunities, various authors (see e.g. Duguid 2005; Meessen et al. 2011; Orlikowksi 2002) refer to the continued importance of face-to-face interaction, as ‘tacit’ knowledge in particular cannot easily be transmitted but rather needs displaying and/or exemplification (see Duguid 2005:113). In this regard, Brown and Duguid (2001) and Vaast and Walsham (2009) usefully distinguish between ‘community of practice’ and ‘network of practice’, whereby the main difference is the degree of face-to-face interaction between members, resulting in closer (community) or looser (network) relationships. In line with this, we identified different interaction modalities among NES members as a potentially important factor of influence on the effectiveness of NES’ capacity building. Our sample diversity in this regard allows one to explore whether it is enough to interconnect a large group of people with similar interests and professional challenges online and offer them the opportunities to exchange knowledge and best practices, or whether more frequent face-to-face interaction is necessary in order to build relations of trust, foster cooperation and make exchange networks sustainable.

In a final step, we focus on the context in which evaluation societies function and, more specifically, which contextual conditions are important to explain NES’ ability to nurture their members’ evaluation capacity (see also Labin et al. 2012; Nakrosis 2014). For NES, the donor setting could be an important element of a context that is more-or-less conducive to evaluation capacity building. Given the specific context of low-income countries, we have incorporated as a context variable whether or not the NES was based in a country that signed the Paris Declaration. Against the background of the 2005 Paris Declaration (and the follow-up Accra Agenda for Action, 2008, and the Busan Partnership for Effective Development Cooperation, 2011) and its five key principles of country-ownership, harmonisation, alignment, results orientation and mutual accountability, there has been a renewed interest in M&E (Holvoet, Gildemyn & Inberg 2012). More specifically, an M&E reform agenda has unfolded whereby it is expected that donors break down their own M&E and align it with recipient countries’ M&E systems which need strengthening. From this vantage point, the assumption is that there may be more incentives to invest in evaluation capacity in PRSP countries. Additionally, seeing how donors are often pushing for M&E systems to be in place and improved (at various levels), we could also hypothesise that donor-supported NES have more incentives to invest in M&E capacity building. Finally, it could also be possible that such a positive effect only works when both conditions are present, namely when an NES is donor funded in a PRSP country.

In sum, our conceptual framework specifies three actor dimensions and one context dimension, each comprising two variables (Figure 2), namely, Resources (human and financial resources), Membership (the number of members and level of diversity in terms of the sectors in which they work), Exchange modalities (the frequency of face-to-face interaction and online activity) and Context (PRSP country and donor funded).

As we are interested in what combinations of actor and context conditions can facilitate effective M&E capacity building, we have used a crisp-set Qualitative Comparative Analysis (csQCA) to get an insight into NES’ effectiveness in capacity building. Prior to an overview and discussion of the main findings, the next section explains the QCA method.

To analyse which combinations of factors are conducive for NES capacity building and which configurations rather lead to limited capacity building, we first operationalise our analytical model. Since we use csQCA, which requires dichotomous data (Schneider & Wagemann 2010), we also identify calibration points in order to dichotomise the data (see Table 1).

With respect to the actor conditions, various operationalisations of the factors were possible. ‘Financial resources’ was operationalised as the availability of paid staff and dichotomised based on having at least one or not having any paid staff as a ‘natural’ cut off point. Given the fact that NES mostly build on volunteer input, taking into account only paid workforce sells NES short. Moreover, not only the number of volunteers matters, but also the skills and competencies with which they are endowed. Unfortunately, as we had no survey data on the educational profile of all the members of a NES, we used a proxy indicator that captures the overall education level in a country (i.e., secondary and tertiary level education enrolment and completion rates at national level). Admittedly, this is a distant proxy, but we felt we had to control for the overall education level as there were substantial differences among countries (e.g. between DRC and Argentina). NES in countries scoring high on two out of the three higher education indicators were labelled as ‘high overall higher education level’ and those who did not as ‘not high overall higher education level’ (see UNESCO 2016). As regards the operationalisation of ‘NES membership’, we included both the number of individual NES members and the number of different sectors represented among their members. The cut off points for ‘high membership’ and ‘inclusive membership’ were set at more than 50 members and at least five different sectors, respectively. The reasoning behind the latter cut-off point was to distinguish between truly multisectoral membership (at least five sectors of the following list: Government; PRSP M&E Unit; Parliament; National civil society organisations; International civil society organisations; Universities/Research institutes; Consultancy firms; Donor agencies; Private sector) and those that do not have such an extensive multisector membership. The final actor dimension is the type of interaction modality the NES uses, whereby we operationalise ‘Frequency of face-to-face activity’ of NES as the frequency with which the NES organised meetings. The cut-off point was put at twice a year or more, for having frequent face-to-face contact, based on our appreciation of NES practices. Standard practice among NES is to have one ‘general assembly’ a year and, as we want to distinguish those with frequent face-to-face contact from the other NES, we thus consider all NES with meeting frequencies higher than the ‘standard once a year meeting’ as having frequent face-to-face contact. Having an operational website should provide us with an indication of online NES activity and was based on a survey question (yes/no). The context conditions in this research were made operational as follows: ‘being donor supported’ was operationalised by asking whether the NES received any donor support for its budget. Similarly, ‘PRSP context’ was also easy to operationalise and calibrate; we classified countries that have produced at least one PRSP and one progress report as ‘PRSP countries’ as this implies that the PRSP process has been institutionalised at least to some extent.

Finally, the outcome, ‘NES M&E capacity building’, was operationalised by the survey question which gauges NES’ perceived contribution to strengthening members’ M&E capacity (see Figure 1). The cut-off point was set so that only those having strengthened ‘much’ and ‘very much’ are considered as having high NES performance on M&E capacity building.

Firstly, a raw data table was constructed in which the original data from the survey for six actor conditions and two context conditions were dichotomised into 1 (present/high) and 0 (absent/low). Eight dichotomised conditions (0/1) generated 263 logically possible combinations of conditions, of which 31 combinations were observed in reality (10 with a positive outcome, whereas 21 conjunctural combinations were linked to non-occurrence of the outcome (i.e., no/modest contribution to strengthening members’ M&E capacity).

The data were analysed using Tosmana (Cronqvist 2011). Originally, eight conditions in total were introduced for 35 cases, which yielded a proportion of conditions to cases of 0.23, still significantly less than is suggested as the upper limit (ratio = 0.33) for conducting meaningful QCA analyses (Marx 2010), but above the upper limit of seven conditions put forward by Marx (2010) as a threshold for generating non-random meaningful conjunctural paths. Given that one of the conditions could be removed without drastically changing the outcome and that this condition – namely, educational level – was actually not measured at NES level but rather at national level, we therefore decided to remove the educational level as it was only a distant proxy for capturing the availability of human resources for NES. The probability of QCA generating a model based on random data with 35 cases and seven conditions is zero (Marx 2010). Our 35-case analysis also easily surpasses the minimum of 12 cases and, with seven conditions, it does not surpass the upper limit of seven conditions put forward by Marx (2010) as a threshold for generating non-random meaningful conjunctural paths.

As mentioned previously, QCA is asymmetrical by assuming that the factors explaining the lack of NES M&E capacity building are not necessarily the inverse of the factors explaining successful NES M&E capacity building. Hence, the following sections subsequently focus on the analysis of perceived high NES performance in strengthening M&E capacity of their members (outcome = 1) and perceived ‘not high NES performance’ in strengthening M&E capacity building (outcome = 0). For each of the outcomes, we first concentrate on the necessity analysis before moving to the sufficiency analysis.

As suggested by Schneider and Wagemann (2010), the necessity analysis should precede the actual analysis of the sufficient conditions in a QCA analysis. Lewegie defines necessity as ‘Condition A is necessary for outcome Y if the occurrence of Y is not possible without the presence of A, but A alone is not enough to produce Y’ (Lewegie 2013:5) and suggests using 0.90 as a consistency threshold and 0.50 as a coverage threshold, when considering if a condition is or is not necessary (Lewegie 2013). He further emphasises that necessary conditions are not often identified in empirical studies (George & Bennet 2005; Schneider & Wagemann 2007).

Applying the thresholds put forward by Lewegie (2013), the findings in Table 2 show that frequent face-to-face meetings is the only necessary (but not sufficient) condition for NES M&E capacity building. This is an interesting finding, particularly given the potential that is ascribed to online exchanges of best practices, tips and knowledge for capacity building. Finally, the necessity analysis also supports our removal of education as an explanatory condition as it is also the condition with the lowest value on necessity (both in consistency and coverage).

The next step in a QCA analysis is to compose a ‘Truth table’ (Table 3) in which the cases are grouped according to the identical causal paths displayed; that is to say, NES that have the same (dichotomised) score for capacity building and the same score on the actor and context conditions are grouped together.

QCA then applies Boolean algebra to minimise the combinations for the (non) occurrence of the outcome. Boolean algebra employs ‘+’ as OR, and ‘*’ as AND. Capital letters indicate the presence of a condition, while lower-case letters indicate the absence of a condition. Two context conditions, namely, ‘D’ (having donor support) and ‘P’ (being in a PRSP country), and five actor conditions, namely, ‘A’ (high frequency of face-to-face meetings), ‘W’ (operational website), ‘M’ (extensive NES membership), ‘C’ (diverse composition of membership) and ‘E’ (paid employee) were included in the analysis. The QCA output summarised below is based on a process of logical minimisation (Grofman & Schneider 2009; Ragin 1987).

The overall consistency value is 1.00 as there are no contradictory cases, which comfortably surpasses the minimum threshold of 0.75 suggested by Ragin (2008:118). Nine different sufficient conjunctural configurations were identified (see solution below), which indicates that all nine combinations of context and actor conditions lead to a positive outcome (i.e., perceived high NES performance on M&E capacity building). By way of example, the first causal pathway leading to high NES perceived effectiveness in terms of M&E capacity building (eAwMCdP) reads as follows: an NES having frequent face-to-face meetings (A), with many members (M) and from many different sectors (C), operating in a PRSP country (P) – even without paid employees (e), in the absence of an operational website (w) and without donor support (d), is able to – at least in the perception – contribute greatly to strengthening NES members’ M&E capacities. The full solution reads:

eAwMCdP + EAWMCD + AWMCDp + EAwmCDP + EAWmcDp + eawMCDP + eAwmcDP + eAWMcDP + eAWMcDP + eAWmCdP → CAPACITY BUILDING

Besides the full solution, Appendix 1 also provides the parsimonious solution of our QCA analysis for reasons of transparency. The parsimonious solution includes hypothetical cases (logical remainders) which are based on theoretical knowledge and is generally used in QCA analysis to minimise the outcome (Ragin & Sonnett 2004; Rihoux & Ragin 2009). Due to the lack of prior theoretical knowledge on the topic of study, we did not use the parsimonious solution in our analysis.

The above full solution highlights that each of the nine pathways is a sufficient combination of conditions for perceived high NES performance on M&E capacity building, which implies that if the combination of conditions is present, the outcome will be positive.

Different paths can thus lead to high NES performance on M&E capacity building. Table 4 highlights that the most empirically-prominent causal path (EAWMCD with a unique coverage of 0.23) is also theoretically the most likely path to lead to significant capacity building as all conditions are positive/present: the NES has a paid full-time employee, has frequent face-to face interaction, an operational website, many members from different sectors, and is receiving donor funding. The latter result seems to confirm the importance, and direction of, the hypotheses formulated in our analytical framework for explaining effective M&E capacity building. Only the condition of being in a PRSP country is not defined in either a positive or negative direction in this combination. The NES in the Philippines, Niger and Burkina Faso follow this pathway to substantial M&E capacity building. Two other different pathways have a unique coverage of 0.15 (as they each cover two cases), namely, eAwMCdP and EAwmCDP. The first pathway combines frequent face-to-face activity, with membership strengths both in size and composition, and operates in a PRSP country, with no paid employees, an operational website and no donor support (Senegal, Pakistan). The second pathway (Madagascar, Georgia) can build on a paid employee, having frequent face-to-face interaction though no operational website, a diverse but low level of membership, and being donor supported in a PRSP country. The quasi-necessity of having frequent face-to-face interaction also becomes clear again. All pathways – except eawMCDP (Ethiopia) – have A (Frequent face-to-face interaction) in the configuration. Our QCA-findings are also supported by qualitative evidence provided by several of the survey respondents. The Réseau Burkinabe de Suivi-Evaluation, for instance, emphasises the importance of their ‘Evaluation days’, during which actors from different professional backgrounds meet together to discuss ideas and current thinking, as well as practical experiences. As to trigger evaluation use, they also deliberately invite policy makers. The same holds for the case of Senegal, where the ‘Senegalese days of evaluation’ are considered an important moment for strengthening capacities through different face-to-face training sessions. As a follow-up to their evaluation days, a committee was created which met with several government authorities to inform them about the importance and benefits of evaluation. In addition, the Réseau Nigérien de Suivi-Evaluation organises face-to-face meetings between the actors of the M&E supply and demand side, which facilitates joint identification of the existing M&E capacity buildings needs while at the same time creating mutual understanding. Similarly, the representative of Pakistan Evaluation Network emphasises the importance of creating a sense of ‘togetherness’, which facilitates the exchange of information and literature among participants from local non-governmental organisations, donor and/or United Nations organisations. He furthermore points out that ‘in such settings, there is also little hesitancy among members to come up with a problem and ask peers to suggest a solution or share similar experience’.

Similar to the above analysis for high NES performance on M&E capacity building, explaining the perceived low NES performance on M&E capacity building should start with a necessity analysis, to map potentially necessary conditions before proceeding to the analysis of sufficient conditions.

The findings in Table 5 clearly show that the absence of frequent face-to-face contact is not a necessary condition for low performance on M&E capacity building (0.55 < 0.90). Therefore, there is no perfect symmetrical causality between ‘frequent face-to-face contact’ and ‘capacity building’. Whereas having frequent face-to-face contact is a necessary condition for having high NES performance on M&E capacity building, lacking face-to-face contact in the NES is not a necessary condition for failure to excel in M&E capacity building. Put differently, even some NES that do have face-to-face contact can still have limited performance on M&E capacity building. The condition that comes closest to being a necessary condition for not having substantial M&E capacity building is the absence of paid staff, with a consistency value of 0.82, or the absence of donor funding with 0.77 (both with coverage surpassing the 0.50 threshold).

Table 6 summarises the findings from the QCA analysis that was performed with regard to the zero outcome or, put differently, perceived low NES performance on M&E capacity building. The formula for the zero outcome is:

ewmdP + eAWmcp + EAWmdP +AWmcdP + eawcdp + eawMCd + eaWmCP + eaMCdp + eAWmdp + EaWMCDP + EaWMcdP + eAwMCDP + eAWMCdP → capacity building (see Appendix 1 for the parsimonious solution)

The most prominent empirical path, namely, ewmdP, is also the path that is most similar to what we theoretically expect (all factors absent), except for being in a PRSP country. Similarly, eawcdP is also a pathway where the factors that were predicted to be important for capacity building are absent (except PRSP country) and that leads to the absence of capacity building. However, we must admit that many other pathways to limited capacity building are possible, which makes it difficult to distill a clear pattern.

This study analyses the contribution of National Evaluation Societies as Communities of Practice to the monitoring and evaluation capacity building of their members.

The most compelling finding from our QCA analysis is that frequent face-to-face contact is a necessary condition for substantial M&E capacity building by NES. While there is no perfect necessity, the necessity value is above the threshold and this is quite an exceptional finding. Moreover, if we plot ‘frequent face-to-face contact’ against the outcome of ‘significant M&E capacity building’ (see Figure 3), it becomes clear that the only exception to frequent face-to-face contact being a necessary condition for capacity building, namely Ethiopia, is a borderline case that is as close as possible to also being a necessary condition.

Disregarding Ethiopia, which is a borderline case, the upper left quadrant (NES able to build M&E capacity without having frequent face-to-face contact) is empty. However, the reverse is not true. While most of the NES that are unable to significantly contribute to M&E capacity building have limited or no face-to-face contact, a considerable portion of them do (bottom right quadrant).

Reverting back to the original discussion in this article on modalities of capacity building and how online communities are often praised as the way to facilitate capacity building, in light of the findings of our analysis more modesty is probably in order when it comes to presenting the potential of online networks relating to practices for M&E capacity building. To compare the effect of offline and online M&E capacity-building modalities (admittedly using crude proxy variables) and reverting to classical statistics, we find that the average score on M&E capacity building is significantly higher for NES with frequent face-to-face interaction (average capacity building [frequent face-to-face interaction] = 3.59; average capacity building [infrequent face-to-face interaction] = 2.62; p = 0.04), while we do not find a significant difference in the average scores on capacity building for NES with operational websites, compared to those without.

The important policy-relevant message to retain from this analysis is to keep in mind the importance of face-to-face contact for M&E capacity building, which is most likely related to the substantial involvement of unstandardised and localised hands-on tacit knowledge. Furthermore, capacity building in conducting, and particularly using, evaluations entails building networks between the M&E supply and demand side which can most easily be done through regular face-to-face interaction. This finding is particularly important against the growing realisation that building nationally-owned M&E systems is not a stroke-of-the-pen exercise. While national M&E systems are key ingredients for the realisation of the Paris/Accra/Bhusan principles of results orientation, alignment and mutual accountability, progress on the ground turns out to be among the most challenging components of the Paris Declaration reform agenda (see OECD 2011; OECD/DAC 2008). Disaggregating scores on indicator 11 (‘building results-oriented frameworks’) into its three sub-components, namely, ‘quality of information’, ‘access to information’ and ‘coordinated M&E system’, highlights that it is particularly the latter, more institutional, component that is most difficult to implement (World Bank 2007). This does not entirely come as a surprise as forging collaboration among different staff and entities dealing with M&E is a long-term process of institutional and personal trust building (see also Bedi et al. 2006). From this vantage point, the value added by NES which offer a unique forum for repeated and informal face-to-face interaction among much of the nationally available M&E expertise scattered over different sectors and institutions, is obvious. Supporting NES in the organisation of face-to-face activities may thus be one efficient way forward for those donors who have national M&E capacity building in their mandates and face challenges in making this objective operational against the backdrop of their own persistent capacity-related weaknesses (OECD/DAC 2010). As the lack of financial and/or human resources was identified as the condition closest to being a necessary condition for poor capacity-building performance, allocating resources to enable NES to have at least a minimal level of permanent human resources at their disposal could make a significant difference in NES performance.

Finally, the findings from our QCA analysis lend support to the conceptual framework we have elaborated, which is evident from the fact that the most prominent path to high NES performance on M&E capacity building is the one which includes the actor and context conditions included in our conceptual framework.