Abstract
Background: Programme efficiency is a vital factor in achieving Sustainable Development Goals (SDGs) in an environment of scarce resources. The branch performance tool (BPT) developed by the International Planned Parenthood Federation (IPPF) is an effective tool for continuous monitoring of programme efficiency in the health sector.
Objectives: The aim of this study was to assess the effectiveness of BPT in improving programme efficiencies and provide recommendations for further improvements.
Methods: The service delivery data, collected through an Internet-based Monitoring and Evaluation Information Management System (MEIMS), as well as costing data over a period of three consecutive years (2013–2015), were fed into the BPT. Data on performance against key efficiency indicators (KPI), measured through data envelopment analysis at each service delivery point (SDP), were used to inform the development of an action plan at the end of each year. Lessons learnt on the best practices of well-performing SDPs and on the improvement opportunities of low-efficient SDPs identified through the BPT were presented during programmatic review workshops and formed the basis of management action.
Results: The organisation met all its efficiency targets in 2013–2015, including increase in number of clients per staff day from 1.9 to 3.0, spike in the cost recovery ratio from 20% to 29%, reduction in cost per SRH service from $3.6 to $2.7 and decrease in overhead cost as a percentage of total organisation level cost from 20.5% to 12.8%.
Conclusions: Branch performance tool is effective for evidence-based decision-making on programme efficiency of service delivery interventions in the health sector. It has potential for further improvement and replication in the health sector, which will contribute to the pursuit of SDG3.
Keywords: monitoring programme efficiency; cost effectiveness; medical service delivery; data envelop technique; relative efficiency.
Introduction
Ensuring healthy lives and promoting the well-being for all and at all ages are essential for sustainable development. This explains why the third United Nations Sustainable Development Goal (SDG3) focuses exactly on ensuring healthy lives and promoting well-being for all and at all ages. Despite being associated with multiple health-related targets, SDG3 aims to ensure universal access to sexual and reproductive health (SRH): from care services, including for family planning, information and education, to the integration of reproductive health into national strategies and programmes (United Nations 2015). However, one of the most important challenges to the fulfilment of SDG3 in many countries is that healthcare cost is constantly rising, which in turn affects healthcare service providers, patients and the overall health systems in hospitals. Therefore, the rising healthcare cost can ultimately lead to reduction in service uptake, which adversely affects the achievement of all development goals, including SDG3. Therefore, as a remedy, policymakers, private players and system leaders are looking for ways to reduce the underutilisation of resources, increase the efficiency of healthcare delivery and allocate more resources to improve value for money in healthcare. The overall objective is to produce more and more services with limited available resources while maintaining the quality of care at a high standard. On the other hand, donors who are providing funding for healthcare service delivery demand maximising the value for the money they invest (Health Research & Educational Trust 2008). So, the continuous improvement of programme efficiency is a vital factor in achieving these challenging SDG targets in an environment of scarce resources.
The term ‘programme efficiency’ refers to a measure of how well a programme deploys its development resources (loans and/or grants) to support development outcomes and impacts. More specifically, a programme’s efficiency is defined by how well the various activities have transformed available resources into the intended results in terms of quantity, quality and time. Much of the current discussion on programme efficiency focuses on how to measure it. However, discussions regarding programme efficiency must go beyond measurement to improvement, that is, reducing unnecessary cost and waste related to the use of resources while maintaining or improving quality.
In light of this global context, the International Planned Parenthood Federation (IPPF)1 is building a ‘performance culture’ at every level of the federation – a culture that revolves around maximising the federation’s contribution to advancing SRH and rights and achievement of SDG targets. However, economic instability and austerity measures have made donors’ funding scarce. They are increasingly demanding measures of progress and tangible signs of achievements. In this environment, maintaining pace is not sufficient: IPPF wanted to demonstrate improvement and prove to a sceptical public that its unique combination of rights-focused services and advocacy is the surest path to universal access to sexual and reproductive healthcare. If the federation cannot achieve and demonstrate efficient use of resources, it risks losing funding and shrinking operations (International Planned Parenthood Federation 2013). In light of this larger context, IPPF is increasing the emphasis on performance in its funding allocation decisions, as are many of the donors that fund individual member associations (MAs).2 As a result, in 2012, the IPPF secretariat introduced a special monitoring tool called branch performance tool (BPT) to its MAs so that they would be able to serve more clients efficiently and effectively, improve their resource mobilisation effort, and maximise their funding allocations under the performance-based funding (PBF) allocation system.
Branch performance tool allows IPPF MAs to visualise the operating performance of individual branches, compare across branches and use the resulting information to guide performance improvements. To do this, the tool estimates relative efficiency, as well as key operational ratios, across a set of branches or clinics. Additionally, where data permit, a user can visualise detailed statistics (e.g. cost and service mixes) to understand differences in branch performance. The goal is to use these data to facilitate a small number of performance improvements in specific branches that will significantly benefit the MA’s ability to deliver quality SRH services (International Planned Parenthood Federation 2013).
As an MA of IPPF, the Family Planning Association (FPA) Sri Lanka3 implemented the BPT across all its service delivery points (SDPs) 2013 onwards. This article presents FPA Sri Lanka’s experience in implementing the BPT. As FPA Sri Lanka does not have a separate administration unit called ‘branch’, the BPT was administered among its SDPs as comparable units. The BPT was administered by the Monitoring and Evaluation (M&E) Unit using the data collected from Programme, Finance and Human Resource (HR) units. Some SDPs attached to FPA Sri Lanka were successful in the achievement of programme efficiencies through continuous monitoring of cost-effectiveness indicators using BPT (Suranga & Rajakaruna 2016). The aim of this study is to assess the effectiveness of BPT and share the lesson learnt and best practices.
Methodology
The service delivery data collected through a centralised web-based monitoring and evaluation information management system (MEIMS) and costing data captured from the organisational finance system (TALLY ERP) for three consecutive years (2013–2015) were fed into the BPT. MEIMS is a web-based, centralised, real-time data collection, aggregation, analysis and reporting system developed by FPA Sri Lanka and is managed by FPA Sri Lanka’s Monitoring and Evaluation unit (The Family Planning Association of Sri Lanka, 2016). While the programmatic data required for BPT were collected using MEIMS, financial and costing data were captured using TALLY ERP.
The branch performance tool
The BPT is an Excel-based tool developed by IPPF to enable users to enter data and view auto-generated reports on a package of predefined indicators on programme efficiency and cost-effectiveness. The BPT allows users to calculate relative efficiency of each branch, identify the best practice branch for each indicator, identify improvement opportunities of each branch compared to the best practice branch and compare two branches. One of the important features of BPT is that it allows a user to apply the tool at two levels depending on the data availability – firstly, the key ratio calculation in situations with less availability of data and, secondly, detailed comparison when the user can provide data for all input indicators.
Data requirement
The BPT analysis requires three types of SDP-wise data – service statistics data, costing data and HR data (permanent, daily pay and contract basis) – to perform efficiency analysis. The M&E Unit, Finance Unit and Human Resource Unit of FPA Sri Lanka are responsible for the development of relevant data capturing mechanisms and provision of high-quality data for these three areas, respectively (The Family Planning Association of Sri Lanka 2016). Table 1 shows the data requirement for the smooth operation of BPT. The BPT data entry sheet is organised into three main sections (as shown, the last section contains several subsections). This BPT data entry sheet is the place where the user can enter (system input) the progress of BPT indicators (described in Table 1) related to programmatic, financial and human resources data. The quality of the programmatic data is ensured using the routine data quality assessments conducted by the M&E Unit of FPA Sri Lanka, and the quality of finance and HR reports are carried out by the internal auditor. These sections and subsections correspond roughly to distinct pieces of analysis, organised by level of detail. Data entered under the ‘Core efficiency data’ and ‘Key ratio analysis’ sections enable the basic functionality of the performance tool, while data under ‘Branch detail data’ are necessary for detailed breakout charts (International Planned Parenthood Federation 2013).
| TABLE 1: Data requirements for a sound administration of the branch performance tool (the layout of the branch performance tool data entry sheet). |
Key efficiency indicators
A statistical technique called data envelopment analysis (DEA) is used to estimate the efficiency of a branch based on how well it uses a given set of inputs to produce a given set of outputs (Anderson 1996). DEA is a mathematical programming technique that has found a number of practical applications for measuring the performance of similar units, such as a set of hospitals, a set of schools, a set of banks, and others. This technique aims to measure how efficiently a decision-making unit (DMU) uses the resources available to generate a set of desired outputs. The performance of DMUs is assessed in DEA using the concept of efficiency or productivity, which is the ratio of total outputs to total inputs. Efficiencies estimated using DEA are relative, that is, relative to the best-performing DMU (or DMUs if there is more than one best-performing DMUs). The best-performing DMU is assigned an efficiency score of unity or 100%, and the performance of other DMUs vary, between 0% and 100% relative to this best performance (Ramanathan 2003). DEA is often used in healthcare settings because it does not require users to define the relative value of each input and output. For instance, if counselling services and medical procedures are included as two separate outputs in a DEA model, these two outcomes are not compared against one another, and branches will appear relatively efficient by efficiently providing one, the other or both (International Planned Parenthood Federation 2013).
By default, the BPT uses one input and two outputs to evaluate branch performance. The input is total cost, while the outputs are couple years of protection (CYP) and non-contraceptive SRH services. However, the facilities are available to customise this model as per the requirement. Efficiency is provided as an efficiency score from 0 to 1, with 1 being most efficient (Figure 1). While the inputs and outputs to DEA cannot fully capture all branch operations, the model provides a first-cut estimate of the relative efficiency of branches. Additional inputs and outputs could result in more branch activities being captured using other indicators listed in Table 2. More details on these indicators are available in the Core Indicator Reference Guide published by FPA Sri Lanka (Monitoring and Evaluation Unit, FPA Sri Lanka, 2014a).
 |
FIGURE 1: Calculation of relative efficiency using data envelop techniques. A branch performance tool output using hypothetical data. |
|
| TABLE 2: Key efficiency indicators and indicator description. |
Additional decision-making tools available in branch performance tool
Apart from the relative efficiency and key performance indicators described in Table 2, BPT can perform several other analyses to support users to make a concrete decision. These features4 include two-metric comparison, additional ratio comparison, detailed comparison, identification of improvement opportunities, analysis of performance over time and identification of best practice branch for each indicator.
The two-metric comparison of the BPT dashboard provides a scatter plot of all branches based on their performance on two metrics defined by the user. The user can test hypotheses about factors that may influence efficiency scores, cost per CYP, or non-contraceptive SRH service, or other relationships by selecting two factors (Figure 2). By interpreting the resulting scatter plot, the user can identify most influential factors for efficiency score in the given context. The additional ratio comparison enables the user to select a ratio (i.e. cost per SRH service and cost per client) to compare branches using a bar chart. The detailed comparison allows the user to visualise detailed breakouts of factors contributing to the performance (e.g. cost structure and CYP composition).
 |
FIGURE 2: Two-metric comparison analysis. A branch performance tool output using hypothetical data. |
|
The purpose of the detailed comparison is to enable users to investigate the causes of differences in the efficiency scores and key ratios. For example, a branch with relatively high cost and a breakout of costs may reveal that overhead costs are an especially large share of costs (Figure 3).
 |
FIGURE 3: Detailed comparison analysis. A branch performance tool output using hypothetical data. |
|
One of the important features of BPT is that it allows users to identify improvement opportunities that they have missed so far. This feature directly provides inputs for the SDP managers and other stakeholders to formulate management actions. On the Improvement Opportunities sheet of the BPT, the efficiency score and key ratios can be used to identify potential opportunities for the improvement of a focused or selected branch. This sheet relies on ‘best practices’ branches to provide benchmarks for branch performance. For example, the potential cost-saving estimates the total amount of savings that may be possible by bringing a less-efficient branch up to an efficiency score of 1. A branch with an efficiency score of 0.75 and a total cost of $100 would have a potential cost saving of $25. Performance gaps on key ratios calculate the gap between the selected branch and the best practices branch for each ratio (Figure 4). By default, the best practices branch is the branch with an efficiency score of 1 that performs best on a given ratio. The goal of identifying best practices branches is to provide MAs a standard against which to compare their branches. Hopefully, low-performing branches can adopt lessons from the best practices branches, which should come across during the final group discussion.
 |
FIGURE 4: Improvement opportunities of a selected branch (i.e. Centre for Family Health) compared to best practices branch for a given indicator. A branch performance tool output using hypothetical data. |
|
The performance of any two branches can be compared using the Branch Comparison sheet of the BPT. This sheet is similar to the Improvement Opportunities sheet but compares any two branches rather than comparing one branch to a best practices branch. By offering customised, one-to-one comparisons between branches, this feature can help explain differences in performance among branches more precisely. The best comparisons will be informed by existing knowledge about which branches operate in similar contexts, serve similar client populations, have similar resources, etc. If data are available for multiple periods over time, the trend graphs can be auto-generated for data elements and ratio (Figure 5).
 |
FIGURE 5: Trend over time (i.e. cost per non-contraceptive sexual and reproductive health (NC SRH) service from 2013 to 2017). A branch performance tool output using hypothetical data. |
|
Implementation of the branch performance tool at service delivery point level
Performance of key efficiency indicators and relative efficiency derived though DEA for each service delivery point were used to develop an action plan at the end of each year. Lessons learnt from the best practices SDPs and improvement opportunities of low-efficient SDPs identified by the BPT were presented at the review workshops as the basis for management action. The workshop was facilitated by the M&E Unit of FPA Sri Lanka and representatives from the SDP management, senior management and service providers participated in the workshop. The management actions were participatory, identified using the results of the BPT and other ground realities which were revealed during the discussion. The SDP managers and their supervisors agreed on specific management actions to be implemented during the following year. The year 2013 was considered as the pilot testing period, and data of 2014 and 2015 were used for decision-making.
Results and discussion
The year 2013 was considered as the pilot testing period for the implementation of BPT. Therefore, the data of 2013 were not considered for this study. The time period of this study was limited to 2014 and 2015. BPT provides the achievement of key performance indicators for each branch separately. Nevertheless, the result on relative efficiency derived through DEA compares the performance of all other branches against the most efficient branch. The relative position of each SDP varied from 2013 to 2015 based on their relative efficiency. All the SDP managers (seven SDPs) tried to increase the relative position of their SDP during the following year by implementing management actions. However, it is important to note that there were no significant changes in programme efficiencies for two SDPs.
Although the BPT provides the achievement of key performance indicators for each SDP separately, this study tried to analyse the organisation-level changes that occurred between 2014 and 2015. Table 3 presents the organisation-level achievement of key performance indicators during 2014–2015.
| TABLE 3: Organisation-level achievement of key performance indicators, 2014–2015. |
There was a significant achievement in all efficiency indicators from 2013 to 2015. More specifically, the following changes were observed: an increase in the number of clients per staff day (from 1.9 to 3.0), an increase in the cost recovery ratio (from 20% to 29%), a reduction in the cost per SRH service (from $3.6 to $2.7) and a decrease in the overhead cost as a percentage of the total cost (from 20.5% to 12.8%) at the organisational level. At the debriefing meeting with SDP managers and Finance Unit representatives, it was identified that the increase in cost per CYP could be attributed to the promotion of Jadelle (an implant which is expensive compared to other contraceptives) rather than a change in cost-effectiveness.
The branch performance tool can be considered as an important tool to monitor the efficiency of non-profit organisations, where the measurement of performance efficiency is difficult. Note that the efficiency of commercial organisations can be assessed easily by their yearly profits, or their stock market indices. However, such measurable factors are not applicable to non-profit organisations. The problem is complicated by the fact that the SDPs consume a variety of identical inputs and produce a variety of identical outputs. Therefore, the methodological approach described in this article would be important and replicable in measuring programme efficiencies of service delivery interventions implemented by non-profit organisations.
However, while the BPT can raise important questions regarding performance and can help guide discussions around operational improvements, the unique environment in which each branch operates should be taken into account when making any management decision. For example, the operational cost of a SDP located in an underserved resettlement area is naturally higher than the operational cost of an urban SDP which serves the general public. Results from the BPT are therefore intended to serve as one component of a broader evaluation of branch performance, including factors such as location, client demographics, community involvement and others. In addition, users should ensure that improving service statistics does not come at the expense of providing high quality of care, which is a core value of IPPF. For example, focusing only on cost effectiveness may motivate SDP managers to provide less-expensive family planning methods, which can adversely affect the client’s right for the choice of method mix.
Challenges, limitations and lessons learnt
During the administration of the BPT tools, a number of challenges and limitations, as well as lessons learnt, were identified.
- Capturing costing data was a real challenge, which demands close budget monitoring, periodic stock-taking and financial system strengthening. However, as FPA Sri Lanka moved from Tally financial system to SAGE ERP, the accuracy of the costing data was improved. Furthermore, selected finance staff who are working with BPT were trained on costing and requirement of the tool. The financial data were entered into the system after closing annual accounts and completion of the annual financial audit to ensure the accuracy of the financial data.
- Costing shared resources and shared staff time are challenging as most of the SDPs are sharing their resources (especially human resources) which have not been properly documented.
- Cost per service highly depends on the type of service and medical procedure undertaken which has not been taken into consideration in the current version of BPT. As an example, medical procedures are more time-consuming and costly than general consultation services. Assigning a weight for each service type based on the cost of the procedure/service may improve the usefulness of the BPT.
- Clarity on definitions of the services and other terms is essential for accuracy of the BPT output. All the services and terms were clearly defined and translated into local languages (Sinhala and Tamil) in 2014. All the service providers and data entry operators were trained on service statistics definitions and glossary of terms to ensure the consistency of reporting (Monitoring and Evaluation Unit, FPA Sri Lanka, 2014b).
- Monitoring the cost-effectiveness without considering the quality of care will compromise the quality of the programme. Therefore, cost-effectiveness and quality of care must go hand-in-hand.
- Close monitoring of cost per service will direct the SDP management to promote low-cost contraceptive methods (Ex: IUD), which may affect the client’s right for choice.
- There must be a way to capture volunteer contribution which is not captured in the current version of the BPT. For example, newly developed tools such as the tools developed by Laurie Mook, University of Torontom5 may be incorporated with BPT.
- There is a room for improvement of the current version of the BPT in consultation with MAs and technical experts. Specially, the user-friendliness of the tool needs to be improved as there are some bugs in macros. This tool needs to be integrated with existing data collection systems (e.g. MEIMS) to minimise the duplication of effort. The current user manual needs to be improved and should interpret the reports with its business value.
- Results from the BPT are intended to serve as one component of a broader evaluation of branch performance, including factors such as location, client demographics, community involvement, and others.
- The latest version of the BPT is customised to measure the efficiency of service delivery interventions in the health sector, with special focus on family planning and SRH. Therefore, BPT supports the monitoring of SDG3. However, the tool can be customised using the same approach to support other SDGs as well. Improvements and pilot testing to discover the potential need to be implemented in the future.
Intellectual property rights
The BPT was developed using IPPF core funding by an external consultancy agency. Therefore, BPT is an IPPF intellectual property. However, the latest version of the tool, along with the user manual, can be obtained free of charge from the principal author of this article for the use of development practitioners and/or organisations.
Conclusion
The findings of this article suggest that the use of the BPT tool during the described pilot exercise seems to provide useful information for decision-makers to deliver more efficient services. Therefore, it can be concluded that BPT is effective for evidence-based decision-making on programme efficiency of health service delivery interventions. Furthermore, BPT has the potential for further improvement and replication in the health sector, which will contribute towards the achievement of SDG3.
Acknowledgements
The authors would like to acknowledge the contributions made by RedstoneStrategy, Mr Jhon Good, Mr Varun Anand, Mr Ataur Rahman, Ms Thushara Agus, Ms Natasha De Rosayro, Dr Sumithra Tissera, Ms Sureka Perera, Mr Lasantha Indika Gunaratne, Mr Amal Bandara, Mr Janaranga Dewasurendra, Mr Sanjeewa Chandrasekara and Mr Thivanka De Silva.
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Authors’ contributions
S.S.S. involved in the designing and conceptualisation of the study, including designing of data collection tools and systems, analysing data and writing the manuscript. R.M.D.K.R. and K.S. contributed in training the service providers, coordinating with service delivery points and other units (finance and HR), collecting data and assessing data quality.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Anderson, T., 1996, A data envelopment analysis (DEA), viewed 15 February 2016, from http://www.emp.pdx.edu/dea/homedea.html#DEA_Title
Health Research & Educational Trust, 2008, ‘Improving efficiency and value in health care’, Health Service Research 43(5).
International Planned Parenthood Federation, 2013, Branch performance tool user manual, International Planned Parenthood Federation, London.
Monitoring and Evaluation Unit, FPA Sri Lanka, 2014a, Core indicator reference guide, an operational guideline on defining and management of reporting against indicators, viewed 29 November 2016, from https://www.researchgate.net/publication/268580394_CORE_ESSENTIAL_INDICATORS_TO_MEASURE_SEXUAL_AND_REPRODUCTIVE_HEALTH_INITIATIVES_A_Case_Study_from_Family_Planing_Association_of_Sri_Lanka
Monitoring and Evaluation Unit, FPA Sri Lanka, 2014b, Service statistic manual; service statistic definitions and glossary of terms, viewed 29 November 2016, from https://www.researchgate.net/publication/299454450_-_-_Sexual_and_Reproductive_Health_Service_Statistic_Definitions_and_Glossary_of_Terms_-_Sinhala_Translation
Ramanathan, R., 2003, An introduction to data envelopment analysis: A tool for performance measurement, 1st edn., Sage Publications India Pvt Ltd, New Delhi, India.
Suranga, M.S. & Rajakaruna, D.K., 2016, A programmatic review of the Centre for Family Health (2012–2015), Monitoring and Evaluation Unit, FPA Sri Lanka, Colombo, Sri Lanka.
The Family Planning Association of Sri Lanka, 2015, FPA Sri Lanka strategic plan (2016–2022), viewed 23 November 2016, from http://www.fpasrilanka.org/index.php?option=com_content&view=article&id=312&Itemid=884&lang=en
The Family Planning Association of Sri Lanka, 2016, Monitoring and evaluation in FPA Sri Lanka – An operational manual for good practice, viewed 23 November 2016, from http://www.fpasrilanka.org/index.php?option=com_content&view=article&id=243&Itemid=880&lang=en
United Nations, 2015, Sustainable development goals, 17 goals to transform our goal, viewed 25 February 2017, from http://www.un.org/sustainabledevelopment/health
Footnotes
1. IPPF is a locally owned, globally connected civil society movement which is working for sexual and reproductive health and rights.
2. IPPF works in more than 170 countries to provide help, advice, services and supplies relating to any aspect of sexual and reproductive health. These services are delivered through an IPPF member association (MA). MAs are IPPF-accredited organisations within a country.
3. The Family Planning Association (FPA Sri Lanka) is an accredited MA of IPPF. Established in 1953, FPA Sri Lanka serves as a non-governmental organisation (NGO) that explores innovative and challenging processes of family planning in Sri Lanka. FPA Sri Lanka is one of the most expansive and well-known NGOs in Sri Lanka that focuses on family planning, and sexual and reproductive health and rights (The Family Planning Association of Sri Lanka 2015).
4. This article does not describe these functionalities in detail as it is beyond its scope.
5. http://www.academia.edu/1258722/Calculating_the_value_of_volunteer_contributions_for_financial_statements
|
Hide
Show all
Email this article (Login required)
