Health evaluations in Africa – A review of the health strand held at the 7th Biennial Conference of the African Evaluation Association

Although Africa has made significant progress in public health over the past several decades, it still faces a very high burden of disease compared to the rest of the world. This overwhelming disease burden is further aggravated by a lack of adequate financial and human resources for health, inequitable distribution of health services, and other social, economic and political factors. Given these constraints, it has become critical for African countries to ensure that health interventions are selected based on evidence and implemented efficiently and effectively to ensure desired outcomes and impact. This has led to an increasing appreciation for monitoring and evaluation as an integral element of programme planning, implementation and scale-up. The importance of M&E within the health sector was recently reflected in the fact that the health evaluation strand was the largest at AfrEA’s 7th Biennial International Conference, held over 3 days in March 2014 in Yaoundé, Cameroon. The health strand, which had nine sub-themes, was sponsored, managed, and supported by the USAID-funded African Strategies for Health (ASH) project. This review summarises the health strand presentations, and panel and roundtable discussions. The evaluations featured in the strand were diverse in terms of health area focus, evaluation methodology, language and authors’ affiliation. More than 21 African countries from all regions of sub-Saharan Africa were represented. Among the key recurrent messages highlighted during the conference were the importance of: data use for planning and improving health programmes, data quality, well-functioning M&E systems and identifying and sharing best/good practices.


Introduction
Over the past decade, increasing calls for accountability and evidence-based programming in the social sectors has led governments, donors, and international and national organisations to intensify their focus on monitoring and evaluation.In Africa, growing appreciation for the role of monitoring and evaluation in improving social programmes has led governments to establish M&E departments or, at the very least, have dedicated M&E personnel.
This increasing M&E focus in Africa is also evident in the mushrooming of evaluation associations across the continent.Currently, 38 countries in Africa have national evaluation associations or voluntary organisations for professional M&E (VOPEs) compared to only a handful in 2000 (AfrEA n.d.).
The African Evaluation Association (AfrEA), founded in 1999, serves as an umbrella association bringing national associations, VOPES, and individual members together in an effort 'to promote and strengthen evaluation in Africa'.One way AfrEA supports this is through its biennial international conference.The most recent AfrEA 7th Biennial International Conference, held over 3 days in March 2014 in Yaoundé, Cameroon, brought together more than 500 participants from over 70 countries representing a wide range of representatives from governments, universities, non-governmental organisations, national evaluation associations, individuals, and donors such as the World Bank, the African Development Bank, United Nations Children's Fund (UNICEF), the Bill and Melinda Gates Foundation, and USAID.The conference enabled participants to share evaluation findings and methodologies, and to discuss evaluative approaches and priorities.
The largest conference strand, Health Evaluation, was sponsored, managed, and supported by the USAID-funded African Strategies for Health (ASH) project. 1 ASH provided technical and logistical support including: (1) reviewing and selecting health-related abstracts, papers and presentations submitted for the conference; (2) securing and organising panel discussions, presenters, and posters within the strand; (3) disseminating health evaluation papers and exploring areas for further analytic work; and (4) organising a skills building workshop with African evaluators.The health strand's nine themes, delivered through various methods (Table 1), are summarised in this review.

Importance of health evaluation
Although Africa has made significant progress in public health over the past several decades, it still faces a very high burden of disease compared to the rest of the world.According to statistics from United Nations (UN) agencies working to improve global health for example the World Health Organization (WHO) and the UNICEF: • A child in sub-Saharan Africa has the highest risk of dying before the age of five -almost 14 times higher than a child in developed regions (UNICEF 2015).

AfrEA Conference health strand
Of 325 abstracts submitted to the AfrEA 7th Biennial International Conference, 56 were specifically submitted to the health strand from independent evaluators as well as evaluation experts from African universities and research institutions, governmental and non-governmental organisations, donor agencies, and private evaluation firms.
The ASH project reviewed these and selected 48 (representing evaluations conducted in 21 African countries) for inclusion in the health strand sessions (papers, panels, round tables and posters).Although most of the submissions were in English, several were in French.Authors of abstracts selected for presentation received technical support and guidance from the ASH project in producing their final papers.

Monitoring health standards (Panel discussion -English)
The focus of this session was on JHPIEGO's experience in implementing M&E standards across its health programmes in Africa and elsewhere.JHPIEGO's M&E standards were based on UNAID's M&E Standards (UNAIDS 2008) as well as AfrEA's evaluation guiding principles (AfrEA c. 2006).
Standards and norms were developed to ensure the presence of a well-planned, integrated, high quality M&E system that facilitates the use of data for decision making.JHPEIGO developed an Excel-based tool using a traffic-light scoring system to assess the adequacy/maturity of each country programme's M&E system against the standards and norms.
The panel discussion centred on case studies from four JHPIEGO country programmes in Côte d'Ivoire, Malawi, Mozambique and Zambia.Across all four country programmes, the implementation of standards-based M&E increased the attention paid to M&E at various stages of the project/programme cycle and helped to facilitate dialogue between M&E and programme and finance personnel.In addition to strengthening a commitment to M&E across the organisation, the M&E standards contributed to an increase in the quality and use of M&E data which led to improved quality of care and services.
Evaluation in key disease programmes (Papers -2 English)  DQA findings highlighted the need to strengthen M&E staffing and documentation of procedures.These were addressed through tailored training and on-the-job mentoring.Subsequent mini DQAs were used to monitor changes in the M&E capacity.
CTV was found to be an effective and efficient approach for identifying the strengths and weaknesses in M&E for community-based HIV programmes.In addition, household visits made it possible to monitor other indicators of child well-being and programme coverage.

Evaluation for maternal and child health (MCH) (Papers -3 English)
Three evaluators presented papers around MCH evaluation.
The first, a World Bank economist and impact evaluations expert, presented a meta-evaluation entitled Delivering the millennium development goals on maternal and child mortality -A systematic review.The meta-evaluation aimed to answer the following questions

Evaluation for development (Papers -1 English, 2 French)
The first paper, 'Evaluating development cooperation in a sector-wide approach (SWAp) -The case of the Rwandan-German cooperation in health', described the use of contribution analysis to assess the performance effectiveness of a highly complex and integrated Rwandan health programme with multiple interventions operating at different levels in a multi-donor sector-wide approach (SWAp) environment.The evaluation's analytical approach linked the expected contributions of organisations involved in the SWAp to the SWAp's theory of change, assumptions, risks, and rival explanations.The approach involved six steps: (1) Establishing the cause-effect issue to be addressed; (2) Developing the postulated theory of change and risks to it, including rival explanations; (3) Gathering existing evidence on the theory of change; (4) Assembling and assessing the contribution claim, and challenges to it; (5) Seeking out additional evidence; and (6) Revising and strengthening the contribution story.This evaluation found that performancebased financing (PBF) contributed to intended improvements in health service delivery by increasing motivation and performance of health professionals despite the prevalence of negative motivational side-effects and resource constraints.
The Rwandan-German contribution to PBF through the SWAp was rated as moderate.
The The study measured the practice of policy evaluation in the Senegalese government and some of its institutions by collecting key actors' understanding and perceptions of evaluation, as well as the role that these institutions play or are likely to play in the development of evaluation or evaluation practices in the country.Amongst the factors hampering the effective institutionalisation of evaluationa clearly stated priority for the Government -the study identified a lack of a reliable information system, a lack of monitoring and evaluation units in many ministries, and attempts by some actors to evade carrying out evaluations.The evaluation found that the regulatory requirement for institutionalising evaluation in Senegal was 'more symbolic than effective'.The evaluator concluded by affirming the importance of regulatory requirements to ensure evaluation of public investments, but stressed that these needed to be supported by improved evaluation capacity, oversight, and enforcement.The second presentation entitled Linking scale up theory to scale up M&E: Findings from a multi-country prospective study of scaling up a reproductive health innovation examined the use of programme theory for scaling up health innovations, the implications for M&E, and the application of systems-oriented M&E for scale up.The presentation described the variables and performance benchmarks used to monitor availability and institutionalisation of the new health innovation into health systems as well as how the monitoring of processes for coverage, quality, institutionalisation, sustainability, adherence to innovation values were measured.The presenter described what measuring success in a complex systems context could look like, and elaborated on some of the lessons learned in the scale-up process, such as the importance of wide use of M&E to guide decision making.

Evaluation methods and methodologies (Papers -3 English)
The third presentation 'Preliminary baseline survey results from the mSOS pilot project in Kenya' from JICA highlighted the mobile SMS-based disease outbreak alert system (mSOS) being introduced in Kenya, the partnerships involved, and the aim of the baseline study of health facilities.The baseline study assigned facilities to either an intervention or control group and showed that some facilities are already using technology in some form to report or manage data.

Health and poverty evaluation (Round table -English)
Two papers were presented at this round table discussion.
The first, 'Postpartum care and family planning in Tanzania: M&E challenges and solutions', presented Tanzania's postpartum care programme which aims to provide a continuum of comprehensive and integrated postpartum care (PPC) services (at both facility and community levels), including postpartum family planning and PMTCT for all women and women living with HIV and their HIVexposed infants.The presentation detailed the programme's interventions, coverage, method of routine monitoring using existing national tools, achievements, and highlighted the need for indicator definitions and reconciliation of indicators with 2014 WHO updates.The evaluator also noted that increasing human capacity, integrated reporting systems, and job aids would all positively affect data collection outcomes.
The second presentation from the International Rescue Committee (IRC) in Ethiopia, 'A performance evaluation of HIV/AIDS projects community conversations in Shimelba and My'ayni camps and host communities, Tigray Region, Ethiopia' focused on HIV and/or AIDS projects implemented in two refugee camps and host communities.
To decrease HIV and/or AIDS incidence and to increase care and support for people living with HIV and/or AIDS, IRC introduced a community conversations (CC) approach as its main implementation strategy.A performance evaluation of the first phase of the project found perceptible changes in behaviours amongst the CC group participants and communities as a result of the CC sessions and activities.The perceived behaviour changes included improved condom use, faithfulness to sexual partners, use of VCT and ART services, reduction in harmful traditional practices, as well as reduction in stigma and discrimination of PLWHA.

Documenting good practices (Round table -2 French, 1 English)
Three evaluators held a round table discussion on the topic of good practices in health evaluation.
The first presentation entitled Nécessité d'une meilleure gestion des savoirs et de partage d'expériences en Afrique sub-Saharienne: expérience innovante de la communauté de pratique prestation de services de santé [Need for better knowledge management and experience sharing in sub-Saharan Africa: Innovative experience of the 'health service delivery' community of practice] shared the experiences of a Community of Practice (COP) around strengthening health services delivery.The COP was created and supported institutionally by a multilateral mechanism known as Harmonisation for Health in Africa (HHA) and brings together a diverse network of more than 1500 members from different geographical, institutional and functional backgrounds.For circulating and sharing knowledge, three strategies were employed: weekly newsletters, online discussion forums, and conferences.Newsletters document new international policies and global health issues, as well as experiences and best practices for implementing health interventions whilst the Forum publishes blogs and articles written by experts that are enriched by member contributions.A conference commemorating the 25th anniversary of the health district in Africa was held in October 2013 at which over 170 participants representing local, national, regional and international health systems actors shared their experiences.
The second presentation entitled Evaluation des actions du pouvoir publique sur le système de sante en Côte d'Ivoire depuis la fin de la crise post-electorale de 2010-2011 [Evaluation of the Government's actions on the health system in Côte d'Ivoire since the end of the post-election crisis of 2010-2011] was presented by the Ivorian Ministry of Finance and a member of the Ivorian Monitoring and Evaluation Network (RISE).The evaluation highlighted the government's increased spending on essential health interventions and its commitment to providing free health care services and free HIV and/or AIDS treatment; its fight against the Malaria-TB, HIV and/ or AIDS; its expanded vaccination programme; human resources for health (personnel recruitment); and investment in health infrastructure.Although the free healthcare policy was regarded as the greatest benefit to women and children (0-5 years), it has been affected by implementation problems.In addition, financial constraints have limited the government's efforts to significantly improve the country's health system and infrastructure.
The third presentation, 'Identifying maternal health good practices from the United Nations Population Fund's (UNFPA's) Fifth Country Programme (CP5) in Ghana' was presented by an evaluator from the University of Toronto.UNFPA CP5 2006-2010 was focused on reproductive health, population and development, gender equity and women empowerment, with HIV and AIDS as a crosscutting issue.Based upon UNFPA's summative evaluation of CP5, independent evaluators identified good practices utilising mixed methods such as insider knowledge (UNFPA staff poll), key informant interviews, surveys, field visits and observation.The evaluation also compared selected practices against scalability, relevance to SRH, innovation, data availability and correspondence to CP5 timeframe.Unfortunately, the evaluators were unable to attribute observed good practices solely to the CP5 interventions as there were gaps in routine monitoring as well as a lack of consistent baseline, mid-project, and end-of-project data measured against clear output, outcome and impact indicators.

Indigenous knowledge (Papers -3 English, 1 French)
This session included four evaluation presentations from Benin, Guatemala, Madagascar and Uganda.
World Vision Uganda presented a paper on the local utilisation of evaluation findings.The evaluation used the Citizen's Voice Action (CVA) methodology (World Vision n.d.) to improve child health service delivery and to generate new learning and innovations.The CVA model combined several elements of social accountability approaches promoting accountability between citizens and government around health issues.
An evaluator from Madagascar presented a paper on Adapting health services utilisation behaviour models to developing countries: the case of a community health project in Madagascar.The evaluation, which was ongoing at the time of the presentation, intended to answer two main questions: (1) to what extent did project interventions contribute to the utilisation of community health volunteer services by the rural population in the project's intervention zones, and; (2) to what extent did the communal health development committees fulfil their roles and responsibilities in managing community health systems including support to community health volunteers (CHVs).Evaluation methods included (i) a descriptive analysis of target population and households characteristics data related to the utilisation of CHV services; (ii) a multidimensional analysis of the relationship between the utilisation of CHV services and a few independent variables through a Probit regression to estimate the effect of these variables, and (iii) an empirical test of the adapted model by measuring the respective weight of each variable/ factor to objectively answer the evaluation questions.
The third presentation was from Plan Benin and was entitled Résultats de l'évaluation finale du projet de prévention à base communautaire du paludisme par l'approche collaborative au Bénin [Results of the final evaluation of a community-based malaria prevention project in Benin using a collaborative approach].
The presentation showcased the results of a final evaluation of a community-based malaria prevention project in Benin which had used a collaborative approach to improve the behaviours associated with the prevention and treatment of malaria amongst the community.Using lot quality assurance sampling (LQAS), the evaluation ensured participation and autonomy of the community for quality assurance during data collection at the community level.
The fourth presentation from the Instituto de Investigación e Incidencia Ciudadana in Guatemala 'Questioning indicators in Guatemalan health policy: Building a bridge between users and policy makers', focused on examining reproductive health indicators in the national Guatemalan health policy.
The evaluation critiqued indicators used in monitoring reproductive health and aimed to build a bridge between health service users and policy makers.The presentation highlighted areas requiring strengthening within the Guatemalan health system with regards to M&E knowledge, stakeholder knowledge and participation, information resources, financial resources and human resources.

Applications of M&E systems (English)
Of the four planned presentations, only one presenter was available (inability to travel to the conference was a potential reason).The presenter, an MPH candidate at the University of Kentucky and originally from Swaziland, presented a paper on the Application of the M&E systems strengthening tool in evaluation of TB and malaria programmes in Swaziland.
The assessment was necessitated by: the renewal phase for Swaziland's Global Fund grants, the need for routine M&E systems assessments, and the need to test the application of the global fund's M&E strengthening (MESS) tool.The objectives of the evaluation were to (1) assess the M&E plan and capacities of the programme's/project's implementing entities; (2) determine how the M&E activities of the programs/projects are linked and integrated within the national M&E system; and to (3) lead to the development of a costed action plan to strengthen M&E systems.A three-day workshop with stakeholders using the MESS tool provided a rating of several aspects of the national health information system, particularly for the Malaria and TB programmes and found that: • Skills and capacities for M&E are in place but resources are not assigned for M&E functions.

Skills building workshop
In addition to the papers, panels, and round tables discussed above, the ASH project joined forces with USAID/Africa Bureau to facilitate a skills building workshop for African evaluators focused on USAID's evaluation policy.The workshop, entitled 'The USAID evaluation policy: Quality standards, lessons learned, and experiences' was attended by 35 participants and employed role playing, small group discussions, and a panel discussion featuring USAID M&E officers and representatives from African and international evaluation firms.
The workshop enabled participants to: (1) become familiar with key aspects of the USAID evaluation policy; (2) define and apply USAID evaluation quality standards and checklists to design and implement useful and high quality evaluations for USAID; (3) understand the key elements of USAID evaluation scopes of work and reports and assess their quality, and; (4) gain better insight into the issues USAID and its partners face in meeting evaluation quality standards.Key issues highlighted during the workshop included the need for: • Realistic alignment of USAID evaluation scopes of work with budgets and timelines to ensure delivery of quality products.• Closer collaboration between USAID and evaluators with clearly defined roles and responsibilities.• Strengthening the capacity of local evaluators while ensuring the achievement of quality standards.• Host government participation in USAID evaluations to facilitate buy-in and use of evaluation results for future programming.

Discussion
The various evaluations featured in the AfrEA Health evaluation strand were diverse in terms of countries represented, health area focus, evaluation methodology, language and authors' affiliation.More than 21 African Invited Paper countries from all regions of sub-Saharan Africa were represented.Most authors were affiliated with international and regional development research (including universities) or evaluation institutions, whilst a few were independent consultants or affiliated with governmental departments.
Of the 48 abstracts accepted under the health strand, 11 were posters and some of those accepted for papers, panels and round tables were no shows -often due to financial constraints.
The strand's presentations covered a diverse set of health topics and programmes, including health systems development and strengthening, reflecting the broad range of health challenges affecting Africa today.The strand's programme focus was largely around targeted both communicable and non-communicable diseases, including TB, HIV and/or AIDS, malaria, diarrheal diseases, and malnutrition, amongst others.The evaluations covered a range of prevention and treatment interventions for maternal, new-born and child health, sexual and reproductive health, and water and sanitation, using different modalities such as community-based, mobile and facility based programmes.
A wide array of methodologies were utilised in the various evaluations including: • Case studies.

Conclusion
The importance of data use for planning and improving health programmes was a recurrent theme in most presentations delivered under the AfrEA health strand.To some degree, all of the evaluations highlighted the importance of learning, improving, and identifying what works and what does not work, including best/good practices -and then sharing that knowledge/information.Another recurring theme was the important role that data quality and a well-functioning M&E system play in improving implementation effectiveness and efficiency of health interventions.
Indeed, the past decade has seen a growing appreciation of M&E as a way to improve implementation of health programmes and for informing policy making in developing countries.2 work in a different context.Despite general consensus around the scientific evidence underpinning many public health interventions, there is continued need for better understanding of effective implementation approaches in different social, cultural, economic and political contexts; and once found, this understanding and learning must link back to health policy and practices.

TABLE 1 :
The nine themes of the Health Strand, AfrEA 7th Biennial International Conference.
that health and social services in Tanzania suffered from inadequate and poorly coordinated M&E systems as well as insufficient human and organisational capacity, which in turn led to poor-quality data for programme planning, management, and reporting.The presenter described MEASURE's approach to strengthening individual and organisational M&E capacity using routine data quality assessments at facility level and Community Trace and Verify (CTV) -a supervisory tool developed by MEASURE to be used at the household level for verifying services received during visits to households served by the program and identifying M&E challenges in community-based HIV programmes.
improvements in skilled birth attendance can be achieved through conditional cash transfers and vouchers to households as well as interventions that bundle quality improvements with increased accessibility; (b) Only training the health workforce or increasing awareness of safe motherhood did not yield significant results; and (c) Effects are larger for more disadvantaged households.
: (1) What interventions demonstrate reductions in maternal and child mortality and increase skilled birth attendance?(2) What do we know about the effects of increasing skilled birth attendance?(3) What important knowledge gaps remain on interventions to reduce maternal and child mortality?The evaluation found that: (a) The third presentation from World Vision entitled Essential new-born care in a rural setting: The case of Warrap State in South Sudan focused on the results of a baseline survey which looked at the status of implementation of essential new-born care in four payams (counties).The survey was part of a four-year USAID-funded child survival project (2010-2014) focused on improving new-born care service delivery by integrating essential new-born care into community-based activities.The survey results confirmed low coverage of systematic application of essential new-born care measures, with the exception of early wrapping of the new-born and early initiation of breastfeeding.
second presentation from GRAAD (Groupe d'Analyse et de Recherché Appliquées pour le Développement) in Burkina Faso was entitled Evaluation des centres de santé par les communautés [Assessment of health centres by communities].The paper sought to determine whether the satisfaction of communities is dependent on the quality of services provided by the health centres.The study used baseline data collected in 2011 for the World Bank project entitled Support to the participatory monitoring and evaluation of projects of the World Bank in the education and health sectors in Burkina Faso and data from respondents in both the test zone and a control zone of the project.The key health service delivery issues identified through these community assessments were the stock outs of drugs and long waiting times.The evaluators concluded that communities have the capacity to evaluate the services they receive from health facilities.
The third and final paper was from Senegal and entitled, L'évaluation dans les ISC au Sénégal: nouveau métier ou refus de rénovation institutionnelle?[Evaluation within ISCs in Senegal: A new profession, or rejection of institutional reform?].