Sector-wide approaches (SWAps) were introduced as a response to donor fragmentation and non-adjusted and parallel programming (Peters, Paina & Schleimann 2013). In a SWAp funds by development partners contribute directly to developing and implementing a defined sector policy under a government authority (Cassels 1997). Key principles of a SWAp are that the partner government leads and takes ownership of the sector programme and that there is a shared effort by development partners to support the programme (Cassels 1997; Peters et al. 2013; Vaillancourt 2009). Ideally, different aid modalities are combined, including joint financing and technical assistance.

Since the early 1990s SWAps have evolved in the health sector of developing countries and received considerable support by the international donor community due to their potential to reduce inefficiencies through alignment to common procedures and hence to increase development effectiveness (Cassels 1997; Vaillancourt 2009). Whilst this led some scholars to declare SWAps as a panacea for development and also influenced the policy process initiated by the 2005 Paris Declaration on Aid Effectiveness, recent experiences from Africa provide a less positive assessment with regard to development effectiveness of interventions embedded in the context of a health SWAps (Walford 2007). Moreover, new global health initiatives introduced to target specific diseases in the course of the health-related Millennium Development Goals (MDGs), such as the Global Fund for AIDS, TB and Malaria (GFATM) or the President's Emergency Plan for HIV and/or AIDS (PEPFAR), are often found to bypass policies and management processes of partner countries (Peters et al. 2013).

In spite of a wide strand of literature about the SWAp model itself, there is still limited evidence from empirical case studies investigating the capacities and performance of interventions embedded in a SWAp environment and the factors that determine their success (or failure). The authors argue that this also stems from a lack of methodological approaches suitable to disentangle the cumulative efforts of multiple development partners, influencing factors and alternative explanations (cf. Leeuw & Vaessen 2009; Pawson & Tilley 1997) for evaluating highly aligned development interventions in complex sector environments for the purpose of learning and (mutual) accountability.

In this article, the authors propose contribution analysis, a theory-based evaluation approach, to manage the methodological challenges using the Rwandan-German cooperation on performance-based financing as an example. To date, the applied methodological approach has not, to the knowledge of the authors, been used for development evaluation in the context of a (health) SWAp. The rest of the article is organised as follows: the next section outlines the approach and specific features of the Rwandan-German cooperation in health. The subsequent section introduces the methodological strategy, a roadmap for its implementation and the results. Finally, the authors discuss some methodological limitations and present conclusions.

Rwanda has made remarkable development progress since the dramatic genocide in the mid-1990s, which severely set back the country's social and economic development (Binagwaho et al. 2014). Although strong economic growth and pro-poor development together with large investments into the public health system brought Rwanda on track to achieve most of the health-related MDG targets before 2015 (International Health Partnership + related initiatives 2011), Rwanda remains one of the poorest and most aid-dependent countries in the world as it struggles with persistent poverty and economic inequality (Abbott & Rwirahira 2012). As a result, the Rwandan health sector is still highly dependent on foreign support which, in turn, manifests in a multitude of donors whose support has to be managed.

Looking back on more than 30 years of development cooperation in health, the Rwandan-German cooperation evolved along the Paris Declaration paradigm and was intensified by aligning it to the umbrella of a SWAp and incorporating joint financing modalities since 2007. In compliance with a national policy on the division of labour of development partners, introduced in 2011 (cf. GoR & MINECOFIN 2013), Germany phased out its support at the end of 2012.

In the course of the health SWAp, major development partners, including Belgium, the United States of America, the United Kingdom, Switzerland, Luxembourg and Germany, aligned their interventions to the health sector strategic plans and the overall national Economic Development and Poverty Reduction Strategy of Rwanda. The process was accompanied by a national health policy, a strategy which followed the policy process initiated by the Paris Declaration (GoR 2005). At the national level, donor coordination was managed in a development partner group, a health sector working group which coordinated the SWAp and about 30 subordinate technical working groups (GoR & MoH 2012). At the decentralised level, development partners were involved in joint action development forums. Besides various forms of technical assistance, joint financing modalities such as sector budget support and basket funding were established.

Within the SWAp structure, the Rwandan-German strategy focused on health system strengthening by promoting primary health care. The cooperation followed a multi-level approach by operating on all health system levels: from support to the technical working groups at the national level, over cooperation with health district administrations to the support of community-based initiatives. Support to the SWAp was provided to improve coordination and cooperation between the Rwandan government (GoR) and development partners. Joint financing modalities were used to increase ownership of the Rwandan Ministry of Health (MoH) and to improve the capacities of public financial management. These outcomes were envisaged to enhance the quality of care and to contribute to a health system that better responds to the needs of the population, especially the poor. Besides support to the SWAp itself and in coordination with the MoH as well as other development partners, Germany concentrated its support on three components including (1) human resource development, (2) sexual and reproductive health and (3) health financing.

In this article, the authors exemplarily investigate the support by German Development Cooperation (GDC) to one selected intervention of the health financing component: performance-based financing (PBF). PBF is a flagship amongst the Rwandan health sector reforms designed to overcome low-quality services and has sparked special international interest (Logie, Rowson & Ndagije 2008). PBF can be defined as the transfer of conditional financial support on the basis of performance targets (Eichler & Levine 2009). Rwanda piloted PBF approaches in 2002 and rolled out a nation-wide program in 2005 (cf. Basinga et al. 2011; Soeters, Musango & Meessen 2005). Whilst PBF encompasses all care levels, in the following the authors only refer to the administrative procedure at the level of hospitals (cf. Rusa et al. 2009): PBF budgets for hospitals are set prospectively, based on an annual value of about $600 per bed. Each quarter, a team from a peer hospital assesses the hospital's performance according to a checklist of over 50 indicators. The percentage of the prospective budget disbursed as incentive is contingent on the degree of indicator achievement. The district hospital decides on how to use these performance fees. Following a multi-level approach, GDC support to PBF included technical advice at the national level, and direct financial assistance to one specific hospital, Ruhengeri in the Musanze district, between 2007 and 2011.

In this section, the authors present their methodological approach for evaluating the effectiveness of development interventions in the context of a health SWAp in Rwanda. Firstly, the nature and suitability of contribution analysis as a methodological strategy for this evaluation are introduced. Secondly, Mayne's (2011; 2012) steps to contribution analysis are outlined and applied using the example of the Rwandan-German PBF intervention. Moreover, the authors also account for suggestions made by Delahais and Toulemonde (2012).

In this section, the effectiveness of GDC support to the Rwandan strategy to overcome low-quality services in the health sector by investigating GDC interventions focusing on PBF is assessed. For this the authors examine in detail the extent to which a single causal link examined in the previous section contributes to the overall contribution story for the two strands of the PBF programme theory of change. The causal link A.3.→A.4., nevertheless, still serves as an example for an evidence analysis table, as outlined in Table 3. Special emphasis is placed on the programme's effectiveness towards increased pro-poor responsiveness of the Rwandan health system.

The self-reported German ambition regarding PBF was to put stronger emphasis on assessing service quality with the PBF indicator system. For this, GDC actively participated through SWAp mechanisms in technical working groups at the national level. Other contributors were the GoR followed by the Belgian cooperation, the World Bank, and non-governmental organisations (World Health Organization 2009).

Regarding decentralised support, GDC contributed to the training of PBF evaluators and data managers, participated in peer reviews of hospitals and disbursed about €0.5 million as performance incentives to the supported district hospital Ruhengeri. Over the time period of interest, Ruhengeri hospital has made far-reaching administrative rearrangements. Whilst the PBF evaluation system was not well understood and accepted by hospital staff at its introduction in 2008 (Deutsche Gesellschaft Für Technische Zusammenarbeit 2008), the comparative case study sheds light on the most recent perceptions: focus group discussants at hospitals and resource persons in the districts (including Ruhengeri) acknowledged the alignment of PBF to the district performance contract system. The respondents often perceived the PBF indicator system as too rigid and at times unreachable and questioned whether the indicators validly measure the defined targets. Nevertheless, health staff self-report higher motivation for their work due to the PBF system. These self-assessments do not, however, allow determining whether this effect is attributable to setting incentives rather than to a general increase in money.

The analysis of the items of evidence also sheds light on the formulated assumptions and risks as presented above (see Table 1). Evidence on the assumption of low motivation of health professionals as the main impediment to high-quality services is mixed: several interviews indicate that intrinsic motivation might be low amongst health workers, whilst secondary interview data by Paul (2009) suggest that Rwandan health workers abide by a high work ethos. Whilst this inconclusiveness impedes a final effectiveness assessment for this aspect of the causal link, it underscores the importance of making explicit and testing one's assumptions about how a programme works. Regarding the programme, it becomes obvious that the relationship between motivation and performance is conditional on more than (the postulated) monetary rewards. This, in turn, lends further plausibility to the assumption that negative side effects occurred. Regarding the identified risks, negative effects of PBF in Ruhengeri such as crowding out and gaming have been consistently reported and researched Kalk (2011; Kalk, Paul & Grabosch 2010; Paul 2009) and pose a threat to improving the structural quality of health services, usually under-researched (Ireland et al. 2011). Whilst these experiences in Ruhengeri also led GDC to more strongly focus its support on other interventions, it is unclear if these effects prevail today. The nationwide PBF roll-out has been conducted in a standardised format, thus lending plausibility to the assumption that other district hospitals have faced similar problems.

Indubitably, it can be concluded from interviews, reports and studies that PBF has induced positive changes: quality and quantity of health services have increased (Sherry, Bauhoff & Mohanan 2013), health staff self-report higher personal performance, less absenteeism and improved internal communication and supervision (Kalk et al. 2010; Paul 2009). According to the PBF indicator score, the overall performance of Ruhengeri hospital rose from 78% in 2008 to 84% in 2010 (data provided by GDC).

Next to the improvement of the structural quality of health service, it is assumed that PBF increases capacity to manage for results and autonomy at the provider level. End users report in focus group discussions to have been treated with more respect at health facilities since the introduction of PBF. Improved client satisfaction can be seen as an indicator for service responsiveness (Kruk & Freedman 2008). Moreover, health professionals report that due to improved supervision, the management can indeed tailor the facilities’ services more closely to the clients’ needs. The same key informants voiced additional major risks: firstly, PBF might exacerbate inequities between health providers because the inflexible indicator system will favour well-capacitated over resource-poor facilities in the long run; secondly, most respondents considered that their work environment suffered from problems not tackled by PBF, such as high turnover rates and lack of equipment. In particular, the raised affordability of health services due to the introduction of a nation-wide health insurance scheme has markedly increased their workload and the strain on hospital capacities, so that other resource constraints may become the bottleneck to high-quality health services.

The finding that hospital management and staff report an insufficient number of health professionals, high turnover rates and insufficient equipment and materials again illustrates that insufficient motivation is not the sole impediment to high-quality health services, thus questioning one of the main assumptions underlying PBF. On the other hand, the recurrent theme of a poorly capacitated work environment somewhat invalidates the rival explanation that an upward trend in Rwanda's total health expenditure has trickled down to improve health services on facility level to such an extent that it would override the positive changes induced by PBF.

To conclude, the evidence confirms that PBF does increase service orientation and outputs of health professionals, but also suggests that negative motivational side effects and resource constraints are real and constrain the intervention's overall effectiveness.

On the basis of the presented results, in this section the authors discuss their experience with contribution analysis as a methodological strategy and its strengths and weaknesses in assessing the effectiveness of development interventions in a complex programme environment.

The authors applied contribution analysis in order to formally manage the high complexity surrounding the interventions under evaluation when attribution is not possible. To this end, they firstly reconstructed the programme's theory of change, before the main analysis concentrated on single causal links by using evidence analysis tables (cf. Delahais & Toulemonde 2012). By doing so the authors ended up with a multitude of causal links to be tested. With regard to the evaluation of similar interventions, two fundamental problems arise: firstly, every additional link under examination increases the resource demands for the overall study. This necessarily leads to the above-mentioned trade-off between scope and causal strength of evidence (Stern et al. 2012). Only deliberate decisions regarding the scope of an evaluation by the evaluation stakeholders and team, taking into account the principle of evaluation efficiency, can thus ensure a utilisation-focus of similarly complex evaluations like that of a highly integrated programme-based approach in this article. Secondly, and related to the first problem, programmes operating in a SWAp environment reshape the attribution problem. Rather than an attribution gap, the attribution problem is arguably better framed as gradient or continuum: in the case that support is strongly aligned to national priorities and well harmonised amongst development partners, even tracing more proximal effects back to the partners’ deliverables becomes increasingly difficult, let alone attributing the more highly aggregated outcome levels to the interventions. A resulting lesson learnt was that sufficient resources should be devoted to explicitly formulating and collecting other influencing factors and rival explanations questioning the programme's contribution. The authors allocated considerable time and resources to this step when reconstructing the theory of change and are confident they have gathered sufficient evidence on the mechanisms of the PBF intervention to solidly back the conclusions drawn in this article. For future complex evaluations, however, the authors see an untapped potential for more strictly testing theories of change by placing more emphasis on rival explanations and invalidating their postulated influence and putting more focus on validating the theory of change with key stakeholders.

In this article the authors analysed the effectiveness of a development intervention in the context of a health SWAp in Rwanda applying contribution analysis as the overarching evaluation strategy. They conclude that contribution analysis is suitable to provide orientation for the analysis of single interventions in a complex sector environment when the programme is strongly aligned to and integrated into the policy framework of the partner country. As such, contribution analysis was capable to also consider alternative explanations which are prevalent in a sector environment with multiple donor activities in place. Finally, the evaluation strategy was flexible enough to incorporate different data collection tools and analytical methods to test different causal links in a systematic manner.

Challenges, however, were found with regard to the efficiency of the investigation as each additional link under examination increases the resource demands for the overall study. This necessarily leads to a trade-off between scope and causal strength of evidence. Moreover, for similar complex evaluations, the authors see an untapped potential for more strictly testing theories of change by placing more emphasis on rival explanations and invalidating their postulated influence and putting more focus on validating the theory of change with key stakeholders.

The evidence gathered confirms that the Rwandan system of PBF does increase service orientation and outputs of health professionals, but also suggests that negative motivational side effects are real and resource constraints limit the overall effectiveness. GDC's contribution to PBF is thus rated as moderate. For further probing the potential of PBF in Rwanda, the aspects of effectiveness and efficiency would merit further methodologically rigorous research, especially with regard to ruling out alternative explanations such as a motivational increase by raising health professionals’ salaries without performance orientation and appraising the cost-effectiveness of the laborious verification and reporting process. Future PBF interventions should be coupled more strongly with capacity development for the supervising and managerial staff and, at best, with interventions improving the physical work environment. Otherwise the goals and indicators set are perceived as out of reach and might lead to demotivation or faked reporting on goal achievement.