Programme efficiency is a vital factor in achieving Sustainable Development Goals (SDGs) in an environment of scarce resources. The branch performance tool (BPT) developed by the International Planned Parenthood Federation (IPPF) is an effective tool for continuous monitoring of programme efficiency in the health sector.
The aim of this study was to assess the effectiveness of BPT in improving programme efficiencies and provide recommendations for further improvements.
The service delivery data, collected through an Internet-based Monitoring and Evaluation Information Management System (MEIMS), as well as costing data over a period of three consecutive years (2013–2015), were fed into the BPT. Data on performance against key efficiency indicators (KPI), measured through data envelopment analysis at each service delivery point (SDP), were used to inform the development of an action plan at the end of each year. Lessons learnt on the
The organisation met all its efficiency targets in 2013–2015, including increase in number of clients per staff day from 1.9 to 3.0, spike in the cost recovery ratio from 20% to 29%, reduction in cost per SRH service from $3.6 to $2.7 and decrease in overhead cost as a percentage of total organisation level cost from 20.5% to 12.8%.
Branch performance tool is effective for evidence-based decision-making on programme efficiency of service delivery interventions in the health sector. It has potential for further improvement and replication in the health sector, which will contribute to the pursuit of SDG3.
monitoring programme efficiency; cost effectiveness; medical service delivery; data envelop technique; relative efficiency.
Ensuring healthy lives and promoting the well-being for all and at all ages are essential for sustainable development. This explains why the third United Nations Sustainable Development Goal (SDG3) focuses exactly on ensuring healthy lives and promoting well-being for all and at all ages. Despite being associated with multiple health-related targets, SDG3 aims to ensure universal access to sexual and reproductive health (SRH): from care services, including for family planning, information and education, to the integration of reproductive health into national strategies and programmes (United Nations
The term ‘programme efficiency’ refers to a measure of how well a programme deploys its development resources (loans and/or grants) to support development outcomes and impacts. More specifically, a programme’s efficiency is defined by how well the various activities have transformed available resources into the intended results in terms of quantity, quality and time. Much of the current discussion on programme efficiency focuses on how to measure it. However, discussions regarding programme efficiency must go beyond measurement to improvement, that is, reducing unnecessary cost and waste related to the use of resources while maintaining or improving quality.
In light of this global context, the International Planned Parenthood Federation (IPPF)
Branch performance tool allows IPPF MAs to visualise the operating performance of individual branches, compare across branches and use the resulting information to guide performance improvements. To do this, the tool estimates relative efficiency, as well as key operational ratios, across a set of branches or clinics. Additionally, where data permit, a user can visualise detailed statistics (e.g. cost and service mixes) to understand differences in branch performance. The goal is to use these data to facilitate a small number of performance improvements in specific branches that will significantly benefit the MA’s ability to deliver quality SRH services (International Planned Parenthood Federation
As an MA of IPPF, the Family Planning Association (FPA) Sri Lanka
The service delivery data collected through a centralised web-based monitoring and evaluation information management system (MEIMS) and costing data captured from the organisational finance system (TALLY ERP) for three consecutive years (2013–2015) were fed into the BPT. MEIMS is a web-based, centralised, real-time data collection, aggregation, analysis and reporting system developed by FPA Sri Lanka and is managed by FPA Sri Lanka’s Monitoring and Evaluation unit (The Family Planning Association of Sri Lanka,
The BPT is an Excel-based tool developed by IPPF to enable users to enter data and view auto-generated reports on a package of predefined indicators on programme efficiency and cost-effectiveness. The BPT allows users to calculate relative efficiency of each branch, identify the best practice branch for each indicator, identify improvement opportunities of each branch compared to the best practice branch and compare two branches. One of the important features of BPT is that it allows a user to apply the tool at two levels depending on the data availability – firstly, the key ratio calculation in situations with less availability of data and, secondly, detailed comparison when the user can provide data for all input indicators.
The BPT analysis requires three types of SDP-wise data – service statistics data, costing data and HR data (permanent, daily pay and contract basis) – to perform efficiency analysis. The M&E Unit, Finance Unit and Human Resource Unit of FPA Sri Lanka are responsible for the development of relevant data capturing mechanisms and provision of high-quality data for these three areas, respectively (The Family Planning Association of Sri Lanka
Data requirements for a sound administration of the branch performance tool (the layout of the branch performance tool data entry sheet).
Category | Item | Description |
---|---|---|
Core efficiency data | Total CYP provided | Total couple years of protection provided through all contraceptive methods. Use standard IPPF conversions (see electronic management information system [EMIS]). |
Total non-contraceptive SRH services provided | All non-contraceptive services that directly relate to a client’s sexual and reproductive health (including counselling and medical procedures). MAs should refer to the Global Indicators and the EMIS for the definitions of SRH services. | |
Total overhead cost ($) | Non-staff costs not directly related to service provision, for example facility rent, transportation, utilities, office supplies, etc. Enter in US dollars. | |
Total cost for salaries ($) | Salary of all branch personnel plus an allocated portion of salary for any shared employees that benefit the branch. Enter in US dollars. | |
Total variable cost ($) | Cost of any drugs, supplies and contraceptives that contribute directly to service provision (e.g. medicine, gauze, and contraceptives). Enter in US dollars. | |
Total cost ($) | Total branch cost. This cost should equal the sum of all salaries, overhead costs, and variable costs (entered under ‘Core efficiency data’). Enter in US dollars. | |
Key ratio analysis data | Total staff (FTEs) | Number of FTEs working at the branch. For example, two staff members each working half-time would count as one FTE. |
Days worked by full time staff in a year | Number of days a full-time staff member works at the branch. Staff members working less than this number of days should be recorded as a fraction of an FTE (e.g. a staff member working half the number of days shown should be recorded as 0.5 FTEs). | |
Total clients | Total unique clients seen at the clinic (e.g. one person seen for two separate services on separate day’s counts as one client). | |
PMSEU clients | Number of unique clients that are poor, marginalised, socially excluded and/or underserved. MAs should refer to the global indicators and the EMIS for the definition of PMSEU clients. | |
Youth clients | Number of unique clients that are youth. MAs should refer to the global indicators and the EMIS for the definition of youth clients. | |
Non-grant income | Total branch income excluding grants and support from the MA’s central office. | |
Branch detail data | Number of services provided (disaggregated by service type) | Number of services provided disaggregated by service type contraceptive services, STI services, HIV services, counselling services, subfertility services, etc. |
CYP provided disaggregated by the method | CYP provided disaggregated by the family planning method condoms, oral contraceptive pills, etc. | |
Disaggregated variable cost ($) | Variable cost disaggregated by contraceptive variable cost and non-contraceptive variable cost, facility cost, transport cost, office supply cost, etc. | |
Disaggregated cost for staff salaries ($) | Staff salary cost disaggregated by staff category, medical staff, counselling staff, management staff, support staff, other. | |
FTEs disaggregated by staff category | FTEs (full time equivalents) disaggregated by staff category, medical staff, counselling staff, management staff, support staff, other. | |
Clients served through clinic | Number of unique clients served through the clinic. | |
Clients served through outreach | Number of unique clients served through outreach efforts. |
CYP, couple years of protection; IPPF, International Planned Parenthood Federation; SRH, sexual and reproductive health; MA, member associations; FTE, full-time equivalent; PMSEU, poor, marginalised, socially excluded and/or underserved.
A statistical technique called data envelopment analysis (DEA) is used to estimate the efficiency of a branch based on how well it uses a given set of inputs to produce a given set of outputs (Anderson
By default, the BPT uses one input and two outputs to evaluate branch performance. The input is total cost, while the outputs are couple years of protection (CYP) and non-contraceptive SRH services. However, the facilities are available to customise this model as per the requirement. Efficiency is provided as an efficiency score from 0 to 1, with 1 being most efficient (
Calculation of relative efficiency using data envelop techniques. A branch performance tool output using hypothetical data.
Key efficiency indicators and indicator description.
Number | Performance indicator | Indicator description |
---|---|---|
1 | Cost per non-contraceptive SRH service ($) | Total cost divided by non-contraceptive SRH services. A lower score indicates more cost-effective provision of non-contraceptive SRH services. |
2 | Cost per CYP ($) | The total cost divided by couple years of protection. A lower score indicates more cost-effective provision of CYPs. |
3 | Clients per staff per day | Average number of daily clients seen by the clinic personnel, divided by the number of staff. This ratio roughly captures the capacity utilisation of the clinic staff. As a point of reference, if all staff members were full-time service providers and saw 2 unique clients an hour for 7 h a day in all working days; this would result in 14 clients per staff per day. |
4 | Overhead as a percentage of cost | Overhead cost divided by total cost. This metric measures how efficiently the clinic employs facilities, equipment, etc. A lower score indicates that the clinic has managed to keep spending on these areas relatively low compared to the levels of services it provides. Larger clinics are generally expected to have lower scores, as there are likely economies of scale (i.e. fixed costs like overheads become a smaller share of total costs as service volume increases). |
5 | Cost recovery ratio | The percentage of the clinic costs recovered through non-grant income. A higher percentage indicates that a clinic is more self-sustaining and relies less on grant support. While a higher score can be desirable, it is critical that the clinic continues to offer affordable SRH services to any client who demands them. |
SRH, sexual and reproductive health; CYP, couple years of protection.
Apart from the relative efficiency and key performance indicators described in
The two-metric comparison of the BPT dashboard provides a scatter plot of all branches based on their performance on two metrics defined by the user. The user can test hypotheses about factors that may influence efficiency scores, cost per CYP, or non-contraceptive SRH service, or other relationships by selecting two factors (
Two-metric comparison analysis. A branch performance tool output using hypothetical data.
The purpose of the detailed comparison is to enable users to investigate the causes of differences in the efficiency scores and key ratios. For example, a branch with relatively high cost and a breakout of costs may reveal that overhead costs are an especially large share of costs (
Detailed comparison analysis. A branch performance tool output using hypothetical data.
One of the important features of BPT is that it allows users to identify improvement opportunities that they have missed so far. This feature directly provides inputs for the SDP managers and other stakeholders to formulate management actions. On the Improvement Opportunities sheet of the BPT, the efficiency score and key ratios can be used to identify potential opportunities for the improvement of a focused or selected branch. This sheet relies on ‘best practices’ branches to provide benchmarks for branch performance. For example, the potential cost-saving estimates the total amount of savings that may be possible by bringing a less-efficient branch up to an efficiency score of 1. A branch with an efficiency score of 0.75 and a total cost of $100 would have a potential cost saving of $25. Performance gaps on key ratios calculate the gap between the selected branch and the best practices branch for each ratio (
Improvement opportunities of a selected branch (i.e. Centre for Family Health) compared to best practices branch for a given indicator. A branch performance tool output using hypothetical data.
The performance of any two branches can be compared using the Branch Comparison sheet of the BPT. This sheet is similar to the Improvement Opportunities sheet but compares any two branches rather than comparing one branch to a best practices branch. By offering customised, one-to-one comparisons between branches, this feature can help explain differences in performance among branches more precisely. The best comparisons will be informed by existing knowledge about which branches operate in similar contexts, serve similar client populations, have similar resources, etc. If data are available for multiple periods over time, the trend graphs can be auto-generated for data elements and ratio (
Trend over time (i.e. cost per non-contraceptive sexual and reproductive health (NC SRH) service from 2013 to 2017). A branch performance tool output using hypothetical data.
Performance of key efficiency indicators and relative efficiency derived though DEA for each service delivery point were used to develop an action plan at the end of each year. Lessons learnt from the
The year 2013 was considered as the pilot testing period for the implementation of BPT. Therefore, the data of 2013 were not considered for this study. The time period of this study was limited to 2014 and 2015. BPT provides the achievement of key performance indicators for each branch separately. Nevertheless, the result on relative efficiency derived through DEA compares the performance of all other branches against the most efficient branch. The relative position of each SDP varied from 2013 to 2015 based on their relative efficiency. All the SDP managers (seven SDPs) tried to increase the relative position of their SDP during the following year by implementing management actions. However, it is important to note that there were no significant changes in programme efficiencies for two SDPs.
Although the BPT provides the achievement of key performance indicators for each SDP separately, this study tried to analyse the organisation-level changes that occurred between 2014 and 2015.
Organisation-level achievement of key performance indicators, 2014–2015.
Number | Performance indicator | 2014 | 2015 | Percentage Change |
---|---|---|---|---|
1 | Cost per non-contraceptive SRH service ($) | 3.60 | 2.70 | −25.00 |
2 | Cost per CYP ($) | 29.10 | 34.22 | 18.00 |
3 | Clients per staff per day | 1.90 | 3.00 | 57.00 |
4 | Overhead as a percentage of cost (%) | 20.50 | 12.80 | −08.00 |
5 | Cost recovery ratio (%) | 20.50 | 29.50 | 09.00 |
SRH, sexual and reproductive health; CYP, couple years of protection.
There was a significant achievement in all efficiency indicators from 2013 to 2015. More specifically, the following changes were observed: an increase in the number of clients per staff day (from 1.9 to 3.0), an increase in the cost recovery ratio (from 20% to 29%), a reduction in the cost per SRH service (from $3.6 to $2.7) and a decrease in the overhead cost as a percentage of the total cost (from 20.5% to 12.8%) at the organisational level. At the debriefing meeting with SDP managers and Finance Unit representatives, it was identified that the increase in cost per CYP could be attributed to the promotion of Jadelle (an implant which is expensive compared to other contraceptives) rather than a change in cost-effectiveness.
The branch performance tool can be considered as an important tool to monitor the efficiency of non-profit organisations, where the measurement of performance efficiency is difficult. Note that the efficiency of commercial organisations can be assessed easily by their yearly profits, or their stock market indices. However, such measurable factors are not applicable to non-profit organisations. The problem is complicated by the fact that the SDPs consume a variety of identical inputs and produce a variety of identical outputs. Therefore, the methodological approach described in this article would be important and replicable in measuring programme efficiencies of service delivery interventions implemented by non-profit organisations.
However, while the BPT can raise important questions regarding performance and can help guide discussions around operational improvements, the unique environment in which each branch operates should be taken into account when making any management decision. For example, the operational cost of a SDP located in an underserved resettlement area is naturally higher than the operational cost of an urban SDP which serves the general public. Results from the BPT are therefore intended to serve as one component of a broader evaluation of branch performance, including factors such as location, client demographics, community involvement and others. In addition, users should ensure that improving service statistics does not come at the expense of providing high quality of care, which is a core value of IPPF. For example, focusing only on cost effectiveness may motivate SDP managers to provide less-expensive family planning methods, which can adversely affect the client’s right for the choice of method mix.
During the administration of the BPT tools, a number of challenges and limitations, as well as lessons learnt, were identified.
Capturing costing data was a real challenge, which demands close budget monitoring, periodic stock-taking and financial system strengthening. However, as FPA Sri Lanka moved from Tally financial system to SAGE ERP, the accuracy of the costing data was improved. Furthermore, selected finance staff who are working with BPT were trained on costing and requirement of the tool. The financial data were entered into the system after closing annual accounts and completion of the annual financial audit to ensure the accuracy of the financial data.
Costing shared resources and shared staff time are challenging as most of the SDPs are sharing their resources (especially human resources) which have not been properly documented.
Cost per service highly depends on the type of service and medical procedure undertaken which has not been taken into consideration in the current version of BPT. As an example, medical procedures are more time-consuming and costly than general consultation services. Assigning a weight for each service type based on the cost of the procedure/service may improve the usefulness of the BPT.
Clarity on definitions of the services and other terms is essential for accuracy of the BPT output. All the services and terms were clearly defined and translated into local languages (Sinhala and Tamil) in 2014. All the service providers and data entry operators were trained on service statistics definitions and glossary of terms to ensure the consistency of reporting (Monitoring and Evaluation Unit, FPA Sri Lanka,
Monitoring the cost-effectiveness without considering the quality of care will compromise the quality of the programme. Therefore, cost-effectiveness and quality of care must go hand-in-hand.
Close monitoring of cost per service will direct the SDP management to promote low-cost contraceptive methods (Ex: IUD), which may affect the client’s right for choice.
There must be a way to capture volunteer contribution which is not captured in the current version of the BPT. For example, newly developed tools such as the tools developed by Laurie Mook, University of Torontom
There is a room for improvement of the current version of the BPT in consultation with MAs and technical experts. Specially, the user-friendliness of the tool needs to be improved as there are some bugs in macros. This tool needs to be integrated with existing data collection systems (e.g. MEIMS) to minimise the duplication of effort. The current user manual needs to be improved and should interpret the reports with its business value.
Results from the BPT are intended to serve as one component of a broader evaluation of branch performance, including factors such as location, client demographics, community involvement, and others.
The latest version of the BPT is customised to measure the efficiency of service delivery interventions in the health sector, with special focus on family planning and SRH. Therefore, BPT supports the monitoring of SDG3. However, the tool can be customised using the same approach to support other SDGs as well. Improvements and pilot testing to discover the potential need to be implemented in the future.
The BPT was developed using IPPF core funding by an external consultancy agency. Therefore, BPT is an IPPF intellectual property. However, the latest version of the tool, along with the user manual, can be obtained free of charge from the principal author of this article for the use of development practitioners and/or organisations.
The findings of this article suggest that the use of the BPT tool during the described pilot exercise seems to provide useful information for decision-makers to deliver more efficient services. Therefore, it can be concluded that BPT is effective for evidence-based decision-making on programme efficiency of health service delivery interventions. Furthermore, BPT has the potential for further improvement and replication in the health sector, which will contribute towards the achievement of SDG3.
The authors would like to acknowledge the contributions made by RedstoneStrategy, Mr Jhon Good, Mr Varun Anand, Mr Ataur Rahman, Ms Thushara Agus, Ms Natasha De Rosayro, Dr Sumithra Tissera, Ms Sureka Perera, Mr Lasantha Indika Gunaratne, Mr Amal Bandara, Mr Janaranga Dewasurendra, Mr Sanjeewa Chandrasekara and Mr Thivanka De Silva.
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
S.S.S. involved in the designing and conceptualisation of the study, including designing of data collection tools and systems, analysing data and writing the manuscript. R.M.D.K.R. and K.S. contributed in training the service providers, coordinating with service delivery points and other units (finance and HR), collecting data and assessing data quality.
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
IPPF is a locally owned, globally connected civil society movement which is working for sexual and reproductive health and rights.
IPPF works in more than 170 countries to provide help, advice, services and supplies relating to any aspect of sexual and reproductive health. These services are delivered through an IPPF member association (MA). MAs are IPPF-accredited organisations within a country.
The Family Planning Association (FPA Sri Lanka) is an accredited MA of IPPF. Established in 1953, FPA Sri Lanka serves as a non-governmental organisation (NGO) that explores innovative and challenging processes of family planning in Sri Lanka. FPA Sri Lanka is one of the most expansive and well-known NGOs in Sri Lanka that focuses on family planning, and sexual and reproductive health and rights (The Family Planning Association of Sri Lanka
This article does not describe these functionalities in detail as it is beyond its scope.