Editorial

Editorial: Mixed methods and mixed messages in evaluation. Engaging stakeholders, including children in evaluation

Mark Abrahams

Copyright: © 2024. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

What matters most about evaluation is that there is absolute clarity about the purpose of the evaluation. This is easily said about the many evaluations commissioned in Africa where the purpose is variously articulated as ‘formative’, ‘for improvement’, ‘for enlightenment’, ‘for clarity’, ‘to assess progress’, ‘to assess outcomes,’ or ‘to assess impact’. The upfront declaration of the purpose of the evaluation provides the justification for the methods to be deployed to gather data that will provide the information for the subsequent report on the purpose statement that framed the evaluation. This ‘logical’ framework quickly gets turned on its head during implementation because it failed to consider stakeholder interests, needs, requirements and their orientation to the intended purpose of the evaluation. Even if the stakeholders are very positive about the evaluation and the articulated purpose, the timing of the evaluation can conflict with organisational or community patterns and rhythms that will adversely affect the quality and depth of the collected data. Similarly, a mixed methods approach that is ostensibly framed to ‘catch all angles’ through gathering quantitative as well as qualitative data cannot be put forward as the magic methodological bullet that will ensure credible evidence to support the findings and recommendations. The point is that the political circumstances of interventions can undermine the methodology of evaluation and affect the kind of quality of available evidence. Therefore, specific measures must be taken to foster the trust and openness required to do evaluation.

This edition grapples directly with the challenges of purpose and methodologies in the evaluation process. Mapulanga’s (2024) article deals with the evidence used by policymakers in Malawi. The article is based on a study aimed to determine the post-research endeavours of Malawian health researchers to incorporate research findings into policy and decision-making. He found that traditional channels, such as journals and conferences, dominate the dissemination of health research. However, limited evidence suggests the need for broader communication strategies. The current landscape lacks effective products and methods, urging researchers to produce accessible formats with clear, key messages to address policy questions. The article by Abebe et al. (2024) shares their experiences conducting a needs assessment in a context where people had limited or no understanding of the need being assessed. This resonates with the point made earlier, that having the purpose clearly in sight, does not mean that it should be implemented. Their findings supported the necessity of expanding rehabilitation services in Ethiopia, but with a shift from a purely medical focus to addressing issues associated with daily functioning and community engagement. The article extends methodological suggestions to inform future evaluations and contributes valuable insights to the broader discourse on needs assessment and programme development in a context where people have limited awareness of services, such as occupational therapy.

A desktop review by Nshimyimana and Rabie (2024) of available literature on M&E systems is complemented by a review of strategic reports and programme documentation of the M&E System in Namibia. Data gathered through interviews with senior and middle level programme managers responsible for the implementation of a national programme, assessed the suitability and gaps of these arrangements as well as the proposed recommendations to strengthen the existing M&E arrangements. According to them, the improved results-based M&E system for the programme they studied offers a comparative base for the implementation of results-based M&E systems for child-focused cash transfer programmes in similar contexts. The involvement of stakeholders in the monitoring of governance is discussed by Matlala (2024) whose article presents a tailored model for the South African government. Utilising social media, the model seeks to transform Citizen Based Management (CBM) practices, enhancing governance and ensuring project success. The limitations of current CBM tools, according to him, necessitate such a model in South Africa. The model is tailored to leverage social media’s transformative potential.

In the article, ‘Child-centred evaluations from child-friendly design to dissemination’: draws on lessons from 5 African countries’, Chloe Maillard (2024) argues strongly that child participation in evaluation and research is key to ensure horizontal accountability and respect the rights of the children to voice their opinions on actions that have a direct impact on their lives. The author uses illustrative examples to describe how child-focused evaluation and research can be implemented in practice and discusses the main challenges and lessons learnt. She concludes that ensuring the meaningful participation of children in evaluation and research processes is not only a right for children but also a great opportunity to enhance the design of evaluation and research pieces to make them more engaging and human-centred. She further states that the use of a child-focused approach brings its own set of challenges and questions that evaluators can navigate with the support from relevant stakeholders. It requires authors to think outside the box, be creative and put themselves in the shoes of their main audience to develop the most suitable engagement strategies.

Engaging children to make evaluation products more child-focused is the title of the article by Jansen van Rensburg (2024) that reflects on the process of developing child-friendly materials by engaging with children throughout the process. According to the author, the children enjoyed the development of the documents. The documents were more user-friendly for a child audience, which ensured that the information shared was more likely to be understood and used and contribute to child rights work. Flexibility in timing the activities and roles played by children and a celebratory event that acknowledged the contributions of every child contributed to the success. The author is convinced that the engagement of children in this manner will enhance the likelihood of children claiming their rights and holding programmes accountable. The theme of children engagement in the evaluation is continued by Andrea Mari (2024) who states that research with young students is not as frequently undertaken as one might expect. The author shows that the employment of questionnaires with young students is likely to yield valid and reliable data when three conditions are met: (1) questions are tailored to the respondents’ cognitive skills and cultural background; (2) questions cover content that is meaningful to the respondents; and (3) questionnaires are administered in settings in which respondents can freely interact with each other. The article attempts to set a foundation for enhancing the effectiveness of teacher evaluation methods in educational settings.

The final article in this issue shares the exploration of the strategies that the UNICEF West and Central Africa Regional Office (WCARO) has implemented in order to fulfil children’s rights through evaluations. The strategies were intended to support evaluation managers and practitioners by discussing challenges and proposed corrective actions encountered when adapting adult-centric evaluation processes into child-friendly ones, both in development and humanitarian settings. According to the authors, Ahamed and Tarsilla (2024), the UNICEF WCARO’s evaluation practice has shifted from a rather marginal integration of child-focused methods in its evaluations to a more substantial mainstreaming of such methods. The article highlights how important it is to engage children during all key phases, while keeping ethical considerations into account. The authors are convinced that implementing child-focused evaluations enhances the inclusivity of evaluation processes under the prism of 2030 Sustainable Development Goals. Moreover, the article highlights the significance of integrating children’s perspectives to strengthen equity-based approaches in contemporary evaluation and enhance the validity of evaluation results.

References