Role of an international non-governmental organisation in strengthening health systems in fragile-state context : Evaluation results from South Sudan

How to cite this article: Rosales, A.C., Walumbe, E., Anderson, F.W.J., Hedrick, J.A., Cherian, D.T. & Holloway, R., 2015, ‘Role of an international nongovernmental organisation in strengthening health systems in fragile-state context: Evaluation results from South Sudan’, African Evaluation Journal 3(2), Art. #162, 7 pages. http://dx.doi. org/10.4102/aej.v3i2.162 Role of an international non-governmental organisation in strengthening health systems in fragile-state context: Evaluation results from South Sudan


Introduction Problem statement
Half of global child deaths take place in fragile conflict-affected countries, 1 mainly due to weak health systems with consequent major disruptions of health service delivery.Despite the work of international non-governmental organisations in this type of setting, their role and value in supporting national health systems has not been adequately documented.

Key focus
The evaluation of the Maternal and Child Health Transformation (MaCHT) project 2 examined the contribution of an international non-governmental organisation (INGO) to the efforts of the South Sudan Ministry of Health (MoH) in strengthening the health system's capacity to deliver essential community-based services in a fragile post-conflict context.
In 2005, the Government of Southern Sudan (GOSS) (later South Sudan) operationalized three levels 3 of health care -the Primary Health Care Unit (PHCU), the Primary Health Care Centre (PHCC), and the Referral Hospital (GOSS 2005).However, the health system remains severely constrained by chronic and severe shortages in human resources for health with only 126 trained 3.Now five: referral hospitals are now tiered as county, state, and teaching hospitals.
Copyright: © 2015.The Authors.Licensee: AOSIS OpenJournals.This work is licensed under the Creative Commons Attribution License.
physicians and 211 trained midwives for the country's current population of 11.3 million and covering an area of 644 329 square km, roughly the size of France (World Health Organization 2015).Nearly 85% of health workers are provided by international and faith-based non-governmental organisations (NGOs) (Wakabi 2011).Furthermore, health indicators in South Sudan are dismal, with the world's highest maternal mortality ratio of 2054 per 100 000 live births, and an infant mortality rate of 68 per 1000 live births (HHS 2006).

Literature review
INGOs are increasingly transitioning from a role of direct service provision to one of building capacity and systems strengthening.INGOs work alongside government ministries, provide input to policy and planning, engage at the local level, and advocate at the macro level in national and international settings (Ulleberg 2009).In some conflict/ post-conflict settings, such as in South Sudan, INGOs provide health, education, and other services where the host government is unable to do so.Indeed, the United States Agency for International Development (USAID), in its threeyear strategy for South Sudan's post-comprehensive peace agreement period, noted that: A key consideration is striking the right balance between capacity building and timely, effective implementation," with recognition that "development resources may need to be redirected from capacity and systems building to service delivery.(USAID Transition Strategy 2011:15) In many low-income countries, bottlenecks to effective health service delivery include a shortage of human resources for health, poor supply chain, and limited physical and policy infrastructure to support a robust health system (Chopra et al. 2012).Shifting service delivery from clinical settings to the community can be an effective way to overcome constraints of health service delivery, whilst other strategies concurrently address the bottleneck within the clinical service delivery channel (Chopra et al. 2012)

Method
The evaluation of the MaCHT project evaluation was conducted to measure and document the INGO contribution to rebuilding the health system's capacity for delivering essential health care to the target population of women and children.The selection of participants used a two-stage random selection (two-stage 30 cluster sampling).Within each selected cluster, households and respondents (mothers of children aged 0-23 months) were randomly selected and only one respondent was selected from a household to avoid overrepresentation.
A structured questionnaire was applied to selected participants during baseline and endline.Eight enumerators and four supervisors trained and deployed to collect data in each cluster.
Data collection management and follow-up supervision was in place during data collection to minimise errors.Questionnaires were checked for completeness and consistency when they arrived at the place of data entry.Any inconsistencies observed during data entry were recorded and the feedback was given to the concerned enumerator and supervisor on a daily basis.Lessons applicable to all enumerators were also shared with them before they left for the field.
All collected data was coded and entered into Excel.Data was checked for inaccuracies and inconsistencies, and then entered into SPSS Statistical Analysis software.Data analysis was conducted in two steps.The first step consisted of the production of descriptive statistics for each variable included in the survey.The second included the calculation of p-values (using Mantel-Haenszel two-tailed tests).

Qualitative methods:
In-depth qualitative interviews and focus group discussions were conducted with stakeholders including project staff, MoH, local NGOs and communitybased organisations, district health teams, community and facility-based health workers, community members, community leaders and mothers.Two focus groups were conducted, one with mothers of children under two years of age and one with HHPs.Both of these were conducted in the program intervention area where operations research was done.Focus group questions explored the effects of the project, and identified barriers to project success.Fourteen interviews were conducted with stakeholders selected on the basis of their involvement with the project.

Secondary data:
The evaluator reviewed project reports (e.g.detailed implementation plan; annual reports; mid-term evaluation; baseline and endline KPC surveys; monitoring reports) to assess the quality of quantitative and qualitative data and to assess the project's results in relation to its design and set targets.

Limitations:
The endline survey was limited by several factors, including the distance to villages, difficult terrain, vehicle availability, and early nightfall.Due to the questionnaire's length, it was difficult for some respondents to complete and difficult to ensure accuracy of entered data.Additionally, as the data for household practices was based on mothers' responses and not observation, overreporting of recommended practices cannot be ruled out.The qualitative survey was limited by the number of beneficiaries and volunteers that could be interviewed for the project.Impediments to the evaluation included weather, long travel times, and limited number of days for the on-site qualitative evaluation process.One key informant from the state MoH was unable to keep the scheduled appointment and could not reschedule during the period set aside for qualitative interviews.

Setting
Warrap State is located in the northern part of the country and is home to an estimated 1.7 million people, mostly ethnic agro-pastoral communities of the Twic Dinka tribe.The state is located in flood plains with a long rainy season, and the region is endemic for malaria, and experiences frequent outbreaks of acute watery diarrhoea and measles.Within Warrap State, the project covered four of nine payams, across Gogrial East and Gogrial West counties.The total end-ofproject population was estimated to be 148 899.
In is staffed by midwives and community midwives.PHCUs are outposts of PHCCs and provide basic preventive and curative services to about 15 000 people (Table 1).
HHPs provide promotion care and mobilise communities, supervised by maternal and child health workers (MCHW) and CHWs.HHPs are selected by communities and are often trained by NGO-led programs.In the project area, CHWs and MCHWs are based in the PHCUs and are appointed following nine months of training.National and international NGOs run most of the PHCCs and PHCUs around the country and provide over 80% of all health services in the country.
The project's target area has seven PHCUs offering basic prevention and promotion services and two PHCCs -both staffed with community midwives and one designated to provide basic emergency obstetric services.HHPs trained to provide case management for non-severe pneumonia, malaria, and diarrhoea, as well as referrals for more severe conditions, were not in place prior to the start of the MaCHT project.

Strategy and package of services
MaCHT utilised five technical interventions and corresponding levels of effort were: Maternal and newborn care (MNC) (30%), malaria (20%), pneumonia (20%), control of diarrheal disease (CDD) (20%), and immunisation (10%).The project sought to utilise high-impact, low-cost, and feasible interventions 6 to improve maternal, neonatal, infant, and child health outcomes at the household/community level through the training of HHPs, a newly approved cadre of CHWs in South Sudan, supported by the HFMCs and mother leader groups (MLGs).
Additionally, the project sought to strengthen the capacity of the health system through investment in increased human resource capacity and improvements at health facility level.
The partners for the MaCHT project included national and state Ministries of Health (MOH and SMOH), county health departments, village and boma health management committees, and the Gogrial East Women's Association.

Results
Maternal care: All seven maternal care-related variables showed significant and positive change over the life of project implementation (see Figure 1).Nearly one-third of mothers surveyed reported having the recommended four or more antenatal checks whilst pregnant with their youngest child, showing a statistically significant increment of 17 percentage points compared with data found at baseline.Three out of five of mothers surveyed received at least two tetanus toxoid vaccinations before the birth of their youngest child, as well as iron supplementation and intermittent preventive therapy (IPTp) representing 38, 31 and 40 percentage point differences respectively from baseline data.Skilled birth personnel (doctor, nurse, midwife, or other medical staff with midwifery training) attendance increased by 5% from baseline to endline as well as active management of third stage of labour (AMTSL). 7Delivering at a health facility was a common theme.One woman noted the importance of delivery at health facility, especially for the first time, 'because the baby may not come out and you may have complications.You may be helped when something goes wrong.'Another woman said that she felt more secure at a facility.In general, women had been well educated by the HHPs and had a good relationship with them.The percentage of mothers surveyed at endline who were able to name at least two post-partum danger signs that would require them to seek immediate medical attention was 73%, compared to 32% at baseline.

Child care:
Similarly, all variables related to care-seeking behaviour and treatment coverage for diseases measured showed statistically significant increases (see Figure 2); thus care-seeking and treatment for fever increased from baseline to endline with 22 and 25 percentage points respectively.Care-seeking for acute respiratory infection (ARI) increased by 27%.Fifty-two percent were taken for care within two days of onset of symptoms, compared to only 19% at baseline; a 33% increase.The amount of treated drinking water increased by 38 percentage points from baseline.Three out of every four children with diarrhoea seek care and received oral rehydration (ORS) treatment, with a 49 percentage point difference from baseline.
The MaCHT project aimed to strengthen the capacity of the health system to deliver essential MCH services through building the supply and performance of the health workforce.
Increasing the health workforce supply was primarily achieved by a combination of recruitment and task-shifting to mitigate the acute shortage, particularly at community level (e.g. through supporting the establishment and training of HHPs).Improving the performance of the health workforce was achieved through in-service training (at health facility and community level), and performance appraisals (focused on community level service delivery).
A review of the project's monitoring data showed that over 100 women were trained as HHPs to deliver evidencebased interventions and provide education around MCH.
Qualitative data revealed that women with children under 7.Active management of third stage of labour involves three components: an injection to prevent excess bleeding, the birth attendant placing pressure on the mother's abdomen while pulling on the umbilical cord, and massage of the abdomen following delivery of the placenta.Due to contextual factors affecting the government's health system (violent conflict and government instability) project activities related to drug procurement and availability of transportation did not find a viable operational framework on which to integrate, leading to the creation of parallel tracks to address drug supply needs and mobilisation of health personnel.As a result, there was a lack of sustainable achievements in these areas.
The evaluation found that the project led to increased community awareness about the need for maternal care and supported the government's efforts to reduce maternal and infant mortality in Gogrial East and West counties (14 key in-depth interviews and two FGDs with project stakeholders, beneficiaries and community volunteers).
In addition, the MaCHT project has paved the way for a hospital improvement project, specifically to ensure Emergency Obstetric Care (EMOC) is available for residents of this geographical area.
In tandem with government priorities for rural areas (where HR shortage is highest), most projects' efforts focusing on community structures were at the level of HHPs.The overall effort was devoted to increasing HHP capacity to deliver high impact essential interventions and to strengthen operational linkages with PHCU.Thus, an evidence-based medical action plan (iCCM plus algorithms) plus its educational curriculum and teaching methodology, supported by clinical and teaching equipment, was collaboratively developed and/or adapted by the relevant branches of the INGO for use in the context of rural Warrap State.Community-based transport committees were organised and linked to the eight health facilities (seven PHCU and one PHCC 8 ).Likewise, within communities included in the project geographical area, mother-groups 9 were organised around MCH.
8.During the life of the project, only one PHCC was linked to community-based transport committees due to budget limitations.9.Mother groups are organised groups made up of mothers who support HHPs and PHCU outreach activities in fostering positive maternal, newborn and young child health and nutrition practices within their communities.

Discussion
The rebuilding of routine public health services should be the main goal of post-conflict reconstruction.Nonetheless, most often during the transition phase following a post-conflict situation, access to health services suffers a contraction due to the slow replacement of sustainable and quality health services by public health services.The MaCHT project described in this paper, and implemented by an INGO in close collaboration with the South Sudan Government, was found to be largely successful in achieving levels of essential health service coverage surpassing the overall basic health service coverage estimated at 40% in South Sudan (Canavan, Vergeer & Bormenisza 2008), and with funding at 25% of the level recommended by the 2001 Commission for Macroeconomics and Health. 10Furthermore, according to a Lives Saved Tool (LiST) analysis 11 of six prevention and treatment interventions (see Figure 2 for a list of the interventions), there was an overall 7.7% decline on under-5 child mortality over a three-year period (2.6% of annual rate of decline). 12  Over 100 HHPs were trained by the project and were found to be very effective in improving knowledge amongst community members, assessing mothers and children, and initiating treatment for malaria and diarrhoea.
Qualitative results strongly suggest that the MaCHT project increased the awareness and motivation for the MoH at the state and local level to refocus its efforts on MCH projects in the area.The Health Pooled Fund project, 13 supported by multiple donors, is actively pursuing the improvement of the Kuajok hospital (district) as a direct effect of program activities.The successful training of HHPs and the dissemination of research results pertaining to their effectiveness has garnered interest from across the health system and suggests that the project has repositioned the maternal, neonatal and child health agenda.
The quality of MCH education was improved in the project area.Over the life of the project, the components of the health system affected positively were human resources (supply and performance), service delivery, and information system.The procurement and distribution of essential drugs and other health supplies was not integrated into the national health system due to limitations associated with contextual factors outside the control of the project; resulting in the creation, as a secondary effect, a parallel system mostly supported by the 12.According to a Lives Saved Tool (LIST), analysis of results of the project (not included in this publication), the project achieved a 7.7% reduction in under-five mortality rate over a period of three years of intervention coverage.
13.Health Pooled Fund South Sudan is financing with $1.9 million over 18 mothperiods the Kuajok hospital (Warrap State) to provide emergency obstetric and newborn care.
NGO. Governance and financial components were not the focus of the project, and were thus not measured.

Limitations
The implementation of this project was affected by the effects of at least three major contextual issues -infrastructure, political/social instability and human resource capacity.South Sudan has existed as a country for less than five years.
In addition to the aftermath of a long period of war with Sudan, this newly independent country has faced numerous challenges, including gaps in health infrastructure to meet the needs of the influx of people from Sudan to South Sudan, and relief efforts taking priority over development efforts.In addition to challenges inherent to integrating with a nascent national health system in the process of building capacity at all levels, the project encountered numerous setbacks.Amongst these were chronic supply chain disruptions, including shortages of drugs, and evacuation of project staff from the project area and the national office as a result of civil unrest.The large relief efforts implemented to care for the large number of people displaced by the civil conflict shifted some of the shared resources to these more urgent issues.
Uncontrolled cross-sectional before-and-after studies are intrinsically weak evaluative designs because attribution of observed change to specific intervention is non-feasible.Likewise, over-estimation of results has been documented with this type of research design; thus inference of overall results should be done with caution.Data collection for the baseline and endline survey was affected by several factors, including the distance to villages, difficult terrain, short supply of vehicles, and early nightfall.Additionally, as the data for household practices was based on mothers' responses and not observation, overreporting of recommended practices cannot be ruled out.The qualitative survey was limited by the number of beneficiaries and volunteers that could be interviewed for the project.Impediments to this process included weather, long travel times, and number of days for the on-site qualitative evaluation process.

Recommendations
1.Fact sheet: Progress on the Millennium Development Goals.[http://.go.worldbank.org/4TZL5TGCW0.2.The Maternal and Child Health Transformation Project was funded through the United States Agency for International Development's Child Survival and Health Grants Program from 2011-2014.

TABLE 1 :
Level of health services and catchment population.
midwives.Training of health facility personnel in HBLSS and iCCM was done in ten health facilities to improve technical capacity.Likewise, project activities tangentially supported another component of the health system such as equipment and essential drugs.Obstetric delivery kits were supplied to PHCUs as part of basic infection control measures.
INGOs' efforts and resources can and should be aligned with governments' initiatives for strengthening local and district health systems.In general, these private efforts and resources should focus on four health systems strengthening components: health workforce, service delivery, health information and medical supplies and equipment.Specific areas of support include: improving partnerships and communication with the national and state MoH to build capacity and implement strategies; integrating community-based resources and activities into new projects to rapidly increase health coverage of essential interventions; addressing supply chain and infrastructure shortcomings; and filling gaps in staff shortages, turnover, and capacity.INGOs can have an important role in complementing national health systems' strengthening efforts.Weak health systems are a common challenge faced by many developing countries, particularly those with fragile-state conditions.Our findings suggest that community-based models supported by INGOs in fragile-state context effectively and efficiently complement government efforts in the areas of service delivery, human resources, information and, with due caveats, supply of medical products.In South Sudan, community-based health services provided via HHPs based on iCCM are a viable model for increasing MNCH services and practices, and improving access in an overstretched system.Further comparison design research plus additional investments in packaging the model can help prepare it for scale-up.