Original Research

The use of continuous household surveys to generate timely data for annual programme outcome monitoring: Experience from the Integrated Family Health Program in Ethiopia

Girma Kassie, Mengistu A. Kibret, Bekele B. Tefera, Kidest L. Hagos, Habtamu Zerihun, Ismael Ali

Received: 19 July 2017; Accepted: 30 Oct. 2017; Published: 28 Mar. 2018

Copyright: © 2018. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: Lack of timely household (HH) level data to inform evidence-based decision-making for improved, timely and targeted health interventions is challenging. Integrated Family Health Program (IFHP) developed and implemented a system for collecting data on an on-going basis at household levels.

Objective: To discuss the implementation of the continuous household surveys of the IFHP.

Methods: Continuous surveys were used for generating timely data for monitoring outcome indicators. Integrated Family Health Program conducted continuous cross-sectional household surveys in its four major target regions over the past 6 years. This approach facilitated the aggregation of data annually, thus enabling progress to be tracked over time and timely decision-making. A total of 2560 households were interviewed each year and trend report synthesised for use.

Result: Key performance indicators in the areas of family planning, and maternal and child health were considered for the study. Time series analysis data revealed overall progress in each outcome indicator. The report was shared among key programme staff to provide data for informed programmatic decision-making.

Conclusion: The surveys as part of other programme activities proved to be a feasible and efficient way to generate timely data for programme outcome monitoring. Documenting the design and implementing data collection and feedback mechanism for prospective description and outcome monitoring potentially represents a new paradigm for course correction.

Recommendation: Programme implementers should plan integrated continuous programme outcome monitoring mechanism as part of programme design rather than waiting for mid-term or end-term project evaluation.

Introduction

Lack of timely outcome-level data for evidence-based decision-making presents important challenges to most health and development programmes in low-income countries (Hirschhorn et al. 2013; Maluka et al. 2010; Mutemwa 2006). The main reasons for this include the lack of time, resources and expertise to generate outcome-level data on a regular basis. Outcome-level data that reflect local variations are generally unavailable and prevent health and development professionals from monitoring interventions and adapting programmatic activities.

Variations of the continuous survey concept have already been applied in different settings. For example, those conducted by the US Bureau of Labour Statistics (2008) report on monthly economic indicators and are a key source of real-time information for decision makers. In Malawi, sub-district continuous surveys were implemented during a period of rapid scale-up of malaria control efforts in 2010 (Roca-Feltrer et al. 2012). The implementer has reported that these surveys were feasible and affordable, had provided valuable information about where to target malaria control efforts and were recommended as a promising district-level monitoring and evaluation tool for short-term processes. In Peru, a continuous survey was implemented nationally for five consecutive years (2003–2008) and concluded to be successful in its design and implementation, and recommended for other countries that currently rely on large-scale surveys (USAID 2016). More recently, continuous household and health facility surveys have been implemented in Sengal (DHS 2013).

A number of existing large-scale community-based surveys have been conducted to provide accurate data on the health status of low-income countries at one point in time, such as Demographic and Health Survey (DHS) (Central Statistical Agency 2016), Multiple Indicator Cluster Survey (MICS) (UNICEF MICS 2016) and Health Information System (HIS) (Mutale et al. 2013) data. These surveys unfortunately have several limitations: Firstly, the periodicity of these surveys is usually once every 2–5 years, which means that programme managers have to wait several years to assess performance of interventions on key health outcomes of interest in order to consider actions for course correction (Central Statistical Agency 2016; UNICEF MICS 2016). HIS data which capture health service statistics at individual health facilities and aggregated up to the district, regional and national levels have the potential for supporting the programmatic decision-making. Unfortunately, HIS data often have limited practical utility because of problems with completeness, timeliness, representativeness, accuracy of data capture and aggregation at multiple levels of the health system. Moreover, HIS data often do not have household-level statistics, which limits the utility of this data source (Mutale et al. 2013). There is limited recorded continuous household survey implemented in Ethiopia.

Since 2011, the Integrated Family Health Program (IFHP) continuous household survey was proposed as a possible means of routinely collecting data on key health outcome indicators of interest. The survey was conceptualised to be undertaken in the regions, with a programmatic level estimate for a broad range of indicators, using sample sizes that have reasonable precision for regional and outcome-level analysis. The IFHP is a project funded by USAID and implemented by Pathfinder International and John Snow, Inc. since July 2008. The project focuses on providing integrated family planning (FP); maternal, newborn and child health (CH) services; as well as improving the quality of reproductive health services specifically in public-sector facilities of the four major target intervention regions (Tigray, Amhara, Oromia and SNNP) and to a lesser extent in Benshangul Gumuz and Somali regions in the country. The programme supports and strengthens the primary health care services through health extension program (HEP) at community and health post (HP) level. The IFHP provides different types of technical and logistics support at all levels of the public system in 300 woredas (districts) of which 286 are found in the survey regions. For programme outcome monitoring, IFHP conducted a continuous household survey annually for 6 years (2011–2016) for the purpose of meaningful time series analysis of key health outcomes (IFHP 2016).

This manuscript describes the IFHP experience in implementing the continuous household surveys in project intervention areas as part of programme outcome monitoring. In our understanding, this is an example of implementing continuous household surveys designed to generate timely data and monitor programmatic health outcomes in Ethiopia. The IFHP continuous survey represents a new paradigm that places equal weight on data systems for course correction as well as programme outcome monitoring. The aim of this report is to discuss the implementation of the continuous household surveys in the IFHP.

Methods

Survey implementation

The IFHP continuous survey is based on a multistage cross-sectional survey design. IFHP had been supporting 286 woredas (districts) in its four major target regions (77 in Amhara, 120 in Oromia, 54 in SNNP and 35 in Tigray). Sixteen cluster-level offices (CLOs) were established in these regions (four in Amhara, six in Oromia, three in SNNP and three in Tigray) in order to support programme implementation in the intervention areas. All 16 CLOs were considered for the survey. A sampling frame was prepared by listing the Woreda Health Offices (WorHOs) under each CLO, health centres (HCs) under each WorHOs and HPs under each HC. A total of 128 woreda health offices, 256 health centres and 512 HPs were selected randomly using proportion to the size of the region. If the HP was randomly selected for the IFHP continuous household survey, then the Keble (village) associated with the selected HP was chosen for the next stage of sampling. From each selected Keble, five households (HH) were selected using a random-walk technique (Figure 1). As a result, a total of 2560 households were selected. If there is more than one eligible respondent found in a household, simple random sampling was used to select one eligible respondent.

FIGURE 1: Sampling process flow chart.

A household questionnaire was developed in consultation with programme technical advisors. Key questions on CH, maternal and newborn health (MNH), FP and nutrition were considered. Regional and CLO staff members received training to use the survey questionnaire and were responsible for data collection in their respective catchment areas. Data collection for the IFHP continuous survey was conducted annually between January and March.

The data management process was managed by the IFHP Monitoring and Evaluation (M&E) team at regional and country office levels. Data checks were conducted by the region and the country office (CO) M&E team. Access database was used to enter the data and finally transferred to SPSS Version 20 for analysis. Ten per cent of the questionnaire was selected and re-entered and cross-checked for data consistency and completeness. It was observed that about 99% of the questionnaires were entered accurately.

Results

Six rounds of data collection for the IFHP continuous survey took place throughout the life of the project. Trend analyses were conducted with 95% confidence intervals around all point estimates. These results were shared with the regional programme manager and programme advisors during project review meetings. As a result, the programme made significant efforts to ensure the use of FP as a norm in the community – the FP-focused interventions have generated more than 23 million couple years of protection. For example, data suggest that 27.4% of women at baseline in 2008–2009 and 40.1% of women at first survey in 2011 aged between 15 and 49 years used a modern method of contraception.

Learning about these gaps mid-implementation meant that the project had introduced different initiatives to train health extension workers (HEWs), a cadre of front-line health workers and adapt HEWs refresher courses as necessary. Drawing on lessons, IFHP introduced different programme support strategies to increase FP service access and uptake, such as (1) extending clinical practicum session for HEWs from 2 days to 3 days to ensure sufficient time to exercise their skills; (2) increasing the number of practicum sites and clinical trainers from 3–4 to 6 to ensure a sufficient level of client demand and HEW competency; (3) increasing the amount of post-training commodities and consumable supplies for routine services from 30 to 60 client supplies per HEW, thus meeting client demand and allowing immediate initiation of services after the training; (4) providing gap-filling of commodities and consumable supplies support during follow-up visits to ensure uninterrupted continuation of services; and (5) strengthening the linkage between HCs and their catchment area HPs through back-up support to maintain competency of HEWs’ skills and ensure the provisions of methods not available at the HP level. As a result, the trend of survey data reports that, women between the ages of 15 and 49 used a modern method of contraception increased and reached 55.6% in 2016 (Figure 2), while long-acting contraceptive prevalence rate (LCPR) also increased from 4.8% to 14.7%.

FIGURE 2: Women aged 15–49 years who are currently using family planning (2011–2016).

Results in skilled maternity service uptake – including antenatal, delivery, and postnatal care – are encouraging. This is a major government priority area. In 2008, only 7% of births were attended by a skilled provider, and antenatal care and postpartum care were sparse and inconsistent. As shown in Figure 3, the survey data show women with children 0–11 months who had at least four or more antenatal care visits at a health facility increased from 30.8% in 2012 to 64.6% in 2016. Since that, more than 7.2 million women delivered with the assistance of skilled birth attendants. Close to 4 million women received at least one postnatal care visit. A similar trend is observed in the percentage of skilled attendance at deliveries that occurred in a health facility, showing an increase from 13.3% in 2012 to 66.4% in the 2016. With respect to child feeding practice indicators, there is great improvement during the project implementation period (Figure 4). Finally, several promising practices were identified and recognised nationally as well as our shared experience with the national and international audiences. The survey learning evidence was used to inform programme decisions and actions. For example; data from the survey revealed that certain health facilities and HPs lacked the necessary supplies for functional oral re-hydration therapy, while others lacked a designated location and board for information-sharing between facility staff and clients. Learning about these gaps, the project had the opportunity to provide and distribute oral re-hydration therapy supplies and information materials to health facilities and HPs to improve their responses to dehydration and diarrhoea. In addition, the project provided pin boards to health facilities and HPs so that they were able to make information about health and child feeding practices information or post available to clients.

FIGURE 3: Trends in maternal health care (2012–2016).

FIGURE 4: Infants age 0–11 months that started first milk (colostrum) and exclusive breastfeeding (2011–2016).

Discussion

The continuous survey approach involves the application of the continuous household survey methods with programme-specific context. The intent of this discussion is to contribute to a dialogue on strengthening health systems of developing countries with the state-of-the-art methods for monitoring, evaluation and timely data use. Every year, the data were synthesised into a report to support the evidence-based decision for monitoring and evaluating team and programme staff working at different levels. The survey is also important for tracking key outcome indicator monitoring and is a source of real-time information for making informed decision across the programme intervention areas. Continuous survey systems contribute to the improvement of quality, availability, reliability and comparability of health data; strengthen M&E capacity; increase the use of health data for programme management; assess the performance of health systems; and measure and improve the quality and safety of health services (Alliance for Health Policy and Systems Research 2008; Bryce et al. 2013; Global Health Workforce Alliance 2008; Hanson et al. 2014; Institute for Health Metrics and Evaluation 2008; International Household Survey Network 2008; Partnership in Statistics for Development in the 21st Century 2008; Tanya et al. 2014).

Lessons learnt

Outcome-level data that reflect local variations are generally unavailable and they prevent health and development professionals from monitoring interventions and adapting programmatic activities. The following lessons were learnt through survey activities:

• Continuous household survey and regular analysis of data is critical to identify challenges and bottlenecks so that timely adjustment of programme direction can be ensured and additional programme needs can be identified.
• Integrating household survey is important for programme outcome monitoring without waiting for costly mid-term or end-term project evaluation.
• It is essential to strengthen programme monitoring systems to cater to the organisation’s data need annually.

Conclusion

In conclusion, continuous survey process, as part of programme activity, helped to produce outcome-level data for strategic decision-making on a timely basis, and it helps to increase the value for money. The survey process has fostered improved information utilisation and ensured learning and mid-implementation corrections in programme implementation. Integrated Family Health Program reported that these surveys were feasible and affordable, and had provided valuable information about where to target programme interventions. Integrated Family Health Program recommends that other programme implementers should be plan integrated continuous programme outcome monitoring mechanism as part of programme design rather than waiting for mid-term or end-term project evaluation.

Acknowledgements

The Integrated Family Health Program (IFHP) is a USAID-funded health programme implemented by Pathfinder International and John Snow, Inc. (JSI) in partnership with the health sector at various levels (FMOH, RHBs of Amhara, Oromia, SNNP and Tigray, woreda health offices, health centres, health posts and Kebeles) and due acknowledgement is forwarded to the above institutions. Our deep gratitude also goes to data collectors who showed their commitment to collect data during the survey. We would like to express our heartfelt gratitude to USAID for funding this survey.

The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.

G.K. was responsible for survey design, prepared the sample, planning and conception of the survey questions. M.A. and K.L. were responsible to direct the project. B.B. was responsible for analysing and interpreting the survey data, prepared, critically reviewed and revised the manuscript. H.Z. and I.A. participated in data collection supervision and manuscript review as well as data entry and analysis.

References