Undertaking child-focused evaluations using a participatory approach has received recognition in recent years. Such an approach is critical not only to build capacity amongst children and youth but also to increase the rigour, validity and usefulness of evaluation findings.
Objectives
The current paper builds on the methods used in a longitudinal evaluation of a parenting programme on reintegration outcomes of children, ages 1–13 years, living in residential care facilities in Uganda. The procedures used to select and modify measures to enable 8–13-year-old children to self-report on their own outcomes are described.
Method
Using a grounded theory of child development, the authors describe the data collection protocols and child-friendly measures used as well as the piloting work that was done by engaging children in the feedback process.
Results
The study underscored the importance of adapting hybrid methods to the local context of a child-focused evaluation, especially in collecting data from young children on sensitive topics across a variety of situations.
Conclusion
The process described in this article can be replicated for designing and conducting evaluations that are child centric and have children as informants of their own well-being.
Contribution
The article contributes to a growing body of knowledge on child-focused evaluations by building on a study conducted in Uganda that focused on developing child-centric measures and data collection procedures. This study shows how to involve children as respondents and assists evaluators to design studies that are ethical, safe, and sensitive to the needs of the children.
Children’s participation in child-focused evaluations has increased significantly in recent years, with the growing recognition and appreciation within the evaluation community of the important role children can play as both evaluators and respondents. A participatory approach has shown to be beneficial to children and youth in terms of building their capacity and sense of agency (Shamrova & Cummings 2017). In addition, it is their right to be involved in matters and decisions concerning themselves and their everyday lives (United Nations 1990). The rising interest in children’s perspectives has made child-focused research and evaluation a critical aspect of the design and methodology of more recent studies, furthering the attention on finding ways to empower and listen to children’s voices and for adult–child participatory processes to thrive.
Child-focused evaluations require thorough consideration, flexibility and openness and a change in mindset about the capacity of children to be involved as co-researchers in the study. The extent to which evaluations are by children, as opposed to on or with children, varies considerably (Tarsilla 2021; UNICEF WCARO Child Focused Evaluation Webinar Series 2022) and accordingly requires resources, time and effort. The goal of these child-focused evaluations has been to learn from children themselves about their views and experiences, so that eventually, the efforts to address their needs can be appropriately targeted based on the reality of their lives. Undertaking evaluation among vulnerable populations, especially children and youth, has its own unique ethical and methodological challenges, which are further compounded when researchers1 seek the perspective of children and youth as informants of their own well-being (Johnson, Hart & Colwell 2014).
Objectives
The authors describe below the child- and youth-centric methods that they used in the multimethod longitudinal study conducted in Uganda (2015–2019). The study examined the effects of a household-based parenting programme, Esanyu Mu Maka, or Happiness in the Home, on the outcomes of reintegrating children living in residential care back into family-based care. The parenting programme was an adaptation of the evidence-based Sinovuyo Kids curriculum, to include components specifically designed to address parenting challenges under reunification conditions (Cluver et al. 2016; Lachman et al. 2017). The study population comprises children (ages 1–13 years) living in residential care facilities (RCFs) and their families. Funded by the United States Agency for International Development (USAID),2 the study included observations of standardised assessments of children and youth, interviews and questionnaires administered at baseline and follow-up to children, their caregivers and to the residential care staff at baseline. The research component of the Keeping Children Healthy in Protective Families (KCHPF) programme was managed by 4Children consortium partner, Westat, in close collaboration with the Department of Social Work and Social Administration at Makerere University, which served as the local research partner (hereafter referred to in the collective as the ‘team’ or ‘we’).
Child-level data on the entire cohort of children aged 1–13 years were obtained in three ways: (1) from primary caregivers reporting on behalf of the child (at baseline in the institution and two follow-up occasions in the home), (2) observer ratings of the child by trained data collectors and (3) child self-reported data collected from children 8–13 years of age. For this article, we focus on the piloting work that we did prior to data collection to make the study more child-focused and make the data collection more child-friendly (see Walakira et al. 2022 for a qualitative analysis of interviews we conducted on a smaller group of children).
Research methods and designDesigning the study to be child-focused
Drawing upon the team’s expertise in child development and the knowledge of developmental changes in children’s competencies and cognition as they grow older, we carefully considered the outcomes we needed to assess in the study. We used a framework called The Whole School Whole Community Whole Child (Centers for Disease Control and Prevention & ASCD 2014) that looked at the child in a holistic way and not focus solely on specific aspects of child functioning, such as school grades or health, as shown in Figure 1:
Types of child outcomes evaluated.
Firstly, we examined the literature on developmental constructs3 that were responsive to changes in parenting and environment, given that the children would experience a change in both when they were reintegrated from RCFs. That is, we selected different constructs that would be important to the child given the change in their living situation and not restrict ourselves to only those that were documented in the literature to be affected by parenting programmes.
Secondly, we used a strengths-based approach to select constructs that assessed positive aspects of child development as protective factors that would help the child cope with the changes in their environment. A strengths-based approach focuses on children’s strengths and abilities instead of focusing solely on risks and deficits (Rashid & Ostermann 2009). That is, instead of assessing just the traditional constructs such as problem behaviours (e.g. withdrawal, getting into fights, acting out), we also included factors such as community belonging, positive interactions with caregivers. Protective factors have shown to foster the development of resilience in children who have faced adverse childhood experiences (ACEs) (Manian 2021).
Thirdly, instead of relying exclusively on caregiver reports of children’s well-being, both in the RCFs and home, we designed the study to include self-reports of the children. Towards this end, we selected constructs that were within the ‘zone of proximal development’ of the children in our study.4 The zone of proximal development refers to the difference between what a child (at his or her age) can do without help and what he or she can do with guidance and encouragement from a skilled person, usually an adult (Fernández et al. 2001). That is, constructs such as self-esteem or social support can be derived from child self-reports if there is guidance from the researcher in terms of how the construct is operationalised (e.g. ‘how do you feel about yourself’). Using this framework, we were able to include constructs that could be based on children’s own self-reports (in addition to adult-reports of the child) of their own well-being, their sense of social and community belonging, attitudes towards harsh punishment, self-esteem, as well as their perspectives on their relationship with their caregiver.
Following the above considerations, we identified six key domains to assess the effects of the parenting programme and the experience of reintegration, as shown in Figure 1. These domains were assessed both from the child’s perspective (for children 8–13 years) and from other multiple sources (observations, RCF caregiver’s perspective and home caregiver’s perspective). They include:
Child health and development – children’s physical health in terms of malnutrition, food security and access to healthcare, as well as cognitive changes that characterise normative development.
Psychosocial health and wellbeing – children’s and their caregivers’ psychological health and wellbeing. The constructs within this domain include stress, mental health, self-worth and belonging and self-confidence.
Child protection and safety – older children’s and caregiver attitudes towards physical punishment; experiences of physical, emotional, sexual abuse and neglect.
Caregiver–child relationship – children’s relationship with their caregiver, including spending dedicated time with each other, building meaningful connections and working together to solve problems. It involves communication, attachment, play and the child feeling a sense of love, acceptance and protection from the caregiver.
Social and community belonging – children’s feelings of acceptance, welcome, inclusion and support within their wider community.
Education access, quality and achievement – children’s access to schooling including school enrolment, attendance and progression, and satisfaction and happiness with school.
Selecting developmentally appropriate measures
The longitudinal nature of the study required the measures to be comparable across the different time points, as well as across the informants, in addition to being developmentally appropriate for the child respondent. This meant that given the age of the child, he or she had to understand what was asked of them, formulate the response by drawing upon their experience and memory and generate a response that aligned with the specific measure. Given the longitudinal nature of the study, we faced specific issues in selecting the measures to assess the constructs listed above:
Firstly, the measure had to be appropriate and meaningful to children’s ages. That is, for constructs such as child externalising, internalising or parental monitoring, the measure needed to tap into different aspects of child-caregiver relationship and child behaviour that were salient for a 2-year-old and a 12-year-old, for instance. ‘Externalizing’ refers to children’s acting-out behaviours such as aggressive and disruptive behaviours (Achenbach & Ruffle 2000, 2010). Internalising refers to self-directed behaviour such as being withdrawn, sad or anxious (Manian & Bornstein 2009). Parental monitoring refers to the tracking and supervision of children’s whereabouts and activities (Stattin & Kerr 2000).
Secondly, the measure had to be appropriate as self-reports for the older children and as caregiver reports for the younger children. That is, a measure had to be appropriate for older children to understand and think about their own feelings such as the child feeling anxious or wanting to be alone. In addition, the measure had to also have observable aspects that caregivers could report on, such as a child not participating in play or being quiet and sullen (Van Roy et al. 2010).
To select measures that satisfied all these criteria and to maintain the validity and reliability of the measures, the team reviewed questionnaires that were previously tested in Africa and globally, especially in the context of parenting intervention programmes, and selected separate measures for the same construct to accommodate the age range. For example, we selected two scales on parenting to be able to compare across ages. The Alabama Parenting Questionnaire (APQ) had a ‘setting limits’ subscale that was appropriate for younger ages (1–5 years) and a ‘monitoring and supervision’ subscale for older ages (age 6–12 years) and was available as a parent report as well as a child or youth report (Frick 1991, Sasagawa & Frick 2006).
In addition to including child self-reports, we included observable aspects of child cognitive functioning. For example, to assess child cognitive functioning, we selected the Mullen Scales Early Learning (MSEL; Mullen 1995) for children aged 1–4 years (at baseline) and the Kaufman Assessment Battery for Children (KABC-II; Kaufman & Kaufman 2004) for children aged 5 years–13 years (at baseline). We selected these instruments for several reasons. Cognitive functioning refers to the learning skills – how children think, learn, figure things out and how they understand the world around them. MSEL and KABC test cognitive functioning through assessing children’s attention, memory, problem-solving abilities and comprehension of text and pictures. The scales had excellent psychometric properties (Koura et al. 2013) and were appropriate for the ages of children and youth in our study, both in terms of age ranges in our study as well as the comparability across ages as the children grew older because of the longitudinal nature of the study. More importantly, the instruments had been used in Uganda for similar-aged children (Bangirana et al. 2011) and could be adapted to local context by substituting the stimuli. Lastly, these tests were fun for the children because they essentially mimicked playing with toys (e.g. placing cards with drawings of people along a sequence and making up a story or remembering names of colourful fishes or jumping a certain distance or kicking a ball around).
Given the developmental nature of the study, we administered separate cognitive functioning measures to accommodate the age ranges of the children in the study. We administered MSEL subscales (Gross Motor, Fine Motor, Visual Reception, Receptive Language and Expressive Language) as well as the Early Learning Composite (ELC) for all children ages 1–4 years and the KABC-II subscales (Sequential, Simultaneous, Learning and planning) administered to children 5 years–14 years. With few exceptions, we were able to administer the same measures to the child at baseline and the follow-up time points, given the age ranges that the tests were appropriate for. Figure 2 and Figure 3 3 indicate the sample material used.
Sample materials from Mullen Scales of Early Learning.
Sample questions to be considered for planning the participatory approach.
Designing the data collection to be child focused
We used the participatory action research approach (Reason & Bradbury 2008; Shamrova & Cummings 2017) in selecting the constructs and the measures to ensure a child-focused evaluation. Participatory action research with children and youth showed evidence of positive outcomes for children, organisations and communities (Shamrova & Cummings 2017; Jansen van Rensburg & Jansen van Rensburg 2020). This framework addresses the challenges of involving very young participants, providing meaningful participation opportunities and addressing power differences between children and adults in diverse cultural contexts (Klein et al. 2016). Participatory methods strongly value local knowledge and participation of the study staff. We considered several key questions in order to decide the participatory nature of children and other stakeholders in our evaluation (see insert, modified from Johnson et al. 2014). First and foremost, to be able to use a participatory approach in the present study, we budgeted enough time in the design of the evaluation to involve data collectors, the RCF staff and children. We allowed enough time to understand the local culture and talked to several stakeholders pertaining to the outcomes of the study (e.g. ‘what is considered to be positive reinforcement for younger children’, ‘what does community acceptance look like’). We then used this knowledge to design the questionnaires as well as the data collection process to be more child focused and reflective of the cultural norms and practices (see the next section on Piloting).
Piloting measures and data collection procedures
We conducted several iterative rounds of testing all the measures as well as the protocols for data collection, to ensure the validity and accuracy of the data and increase efficiency of data collection. We used versions of the tests that were available in the local language and modified to the local context. For example, researchers at Makerere University had already modified KABC-II by substituting locally available toys and objects as stimuli into the assessments, as well as substituting locally used terms in the instructions (Bangirana et al. 2009).5 However, given the different settings in which the tests were applied – such as the rooms at the RCFs or in their courtyard or at the child’s home, we ensured that the stimulus materials were amenable and appropriate to be used across all settings.
In addition to this, we further pilot tested our measures with similar-aged children in local schools and in the RCFs and asked them for input into the stimuli materials we used as well as ideas to make the instructions more clear and concise. We pilot tested in English and Luganda and could accommodate individual preferences of the child. We modified the procedures slightly so we could use the tests in different physical settings (on the floor, on the table, under a tree), used tablets with built-in skip patterns for ease of administering the tests and made all the measures available in different languages (English, Luganda) so the data collectors could offer children a choice for their preferred language. All questionnaires were translated and back-translated to Luganda; questions were programmed into the table in both English and Luganda so children could choose the language that they wanted to respond in. In addition, when we piloted in schools, we asked the children for feedback as to which questions were difficult to understand and which of the tasks were harder or more fun, so we could sequence the subtests accordingly to make the data collection less burdensome for the children.
Training research staff
Participatory approaches also call for reducing the power imbalances and biases of involving and collaborating with children in an evaluation, which can be expressed in subtle ways during interactions with children. Training data collectors and other research staff become even more critical in child-focused evaluations that involve children in vulnerable situations such as the present context that included children living in RCFs away from families of origin. In such studies, data collectors are required to have a high level of self-awareness and who can reflect on their own actions on an ongoing basis (Klein et al. 2016). That is, they are required to not only focus on collecting data but also be observant of children’s reactions and body language during the entire time. Hence, a rigorous training process is critical to conducting child-focused studies. We conducted several training sessions that involved peer-led discussions, demonstrations, role-playing, provided videotaped feedback for data collectors, as noted in Figure 4. Integrated into the training process was assessing the effectiveness of the training in terms of changes in knowledge about the developmental capabilities of children, their attitudes towards researchers, the mindset of children while doing the assessments and responding to questions and potential for unintended harm while collecting data.
Overview of the training process.
Trauma-informed training
In recognition of the potential trauma and ACEs that children in our study could have experienced, we used a trauma-informed approach in our training and data collection activities (Manian, Rog, Lieberman & Kerr 2022; SAMHSA 2015). We provided a basic informational session on the potential neglect and abuse experienced by the children at RCF and the potential negative effects on children’s attachment, distrust of adults, increased need to appease adults, fear of close contact and so on (San Cristobal, Santelices & Miranda Fuenzalida 2017). We then had informational and peer-led sessions on how data collectors needed to integrate this knowledge into their data collection and other interactions with children and adults. Given that the questions in the data collection tools ask children and adults to reveal highly personal information, answering these kinds of questions could be difficult – and sometimes retraumatising – for children and adults with trauma histories (Manian 2021). Hence, we trained interviewers to move through questions and assessments during data collection in a sensitive and timely manner, while collecting the most complete and accurate data possible.
Child safeguarding protocols
Children need to have in-built protections within a study, both as subjects in the study and as stakeholders. Towards this end, we developed a detailed ‘child safeguarding protocol’ and a ‘distress protocol’ in collaboration with the RCF staff and the personnel doing the reintegration and used both protocols to train the data collectors. We developed the child safeguarding protocol in alignment with the applicable Ugandan child welfare and protection laws, to ensure that participation in study does not pose any risk to the child. Procedures to report concerns or suspicions of abuse or exploitation were outlined during the training. All staff were also trained on how to manage sensitive information respectfully, professionally and in compliance with best practice standards regarding confidentiality. We developed the distress protocol for handling emergent issues during staff interactions with children and to pause data collection, if the child or adult appeared distressed.
Data storage and confidentiality
Data collection included audio recordings and notes from interviews. Audio files were copied to access-protected laptops at the end of each data collection day and then transcribed into English and transferred on access-protected laptops. Handwritten notes were kept in locked file cabinets. All data were de-identified; after completion of data collection, qualitative data was stored, managed or destroyed according to Institutional Review Board (IRB) approval requirements (UNCST SS 4551 and MUREC REC 0503-2017). Transcripts and notes from interviews were imported into NVivo 10, a qualitative data analysis software package. This was used to manage coding, following a systematic and inductive procedure (see Walakira et al. 2022).
Modifying data collection procedures to be child-friendly
There were several key design features we introduced during data collection to ensure we were using child- and youth-centric methods.
The programme staff made several visits to the RCF as part of the implementation of the reintegration programme and visited the homes as part of the implementation of the parenting programme. This ensured, to the extent possible, that the RCF staff understood the requirements of the study, including the confidentiality procedures, and minimised any negative repercussions on the child.
The study team built in extra time for the data collector to build rapport and establish trust with the child. The interviewers were required to spend the initial part of the interview conversing with the child. The interviewer also brought snacks and started with a story of a recent newsworthy event or a popular sportsperson or media star to converse with the child and commence the interview process. In addition to assuring the child that no harm would befall them by participating in the study, we felt it was also important to assure them that their answers will not benefit them in any way, so they were not focused on giving socially desirable responses.
We adapted the testing format to ensure that the children felt safe. We devised several strategies around positioning of the data collector with respect to the child, minimising initial physical contact with the child, letting children know that their RCF caregivers were on board with the study and explaining to them that their responses would not affect them or their relationship with their RCF caregiver in any way. Rather than just explaining the process to the children, we assessed their understanding about the entire situation by asking questions. Data collectors initially showed children how they recorded their responses on the tablets or the paper-pencil measures to let the child get a basic understanding of the procedure.
An ethical issue in building rapport mentioned above is that it may lead to the possibility of developing close relationships with children that have to end when the study draws to a close. Given the longitudinal nature of the study, it was possible for both children and data collectors to become familiar with each other. The data collector ensured that the interviews ended on a positive note, and the children were able to engage in an activity at the conclusion of the interview, for which the support of the participants’ caregivers at the RCFs and homes was invaluable. In addition, the research team made arrangements for further ongoing support to individual children who needed it, by making referrals to local providers.
Despite the different settings where the interviews occurred, such as the RCFs, homes and playgrounds, data collection was conducted in a private area where the children could speak freely and comfortably without the conversation being overheard and yet could seek the comfort of a caregiver, if needed. We worked around the daily schedules of the younger children, around their sleeping and eating schedules, to ensure we were not putting undue strain on them or their caregivers.
The different data collection techniques were geared towards ensuring that children remain interested during the process of data collection. For example, even though the Gross Motor Scale from MSEL was not part of the study assessment, we administered a few items from the scale because it was engaging and entertaining to the child. The tasks within the scale required a child to move about and to do fun movements such as hopping, jumping and kicking a ball, which served to provide a break to both the child and the data collector from filling out questionnaires. We also interspersed the questionnaires with questions that were positive and ‘fun’ to break the monotony of testing.
Following the work of Ebrahim (2008), who argues for a concept of consent as a process, such that while primary caregivers give their consent for participation (as is standard practice), we allowed children to give continued assent to participate in various aspects of the study. The data collectors were trained to explain the study to the children in ways they would understand. In addition, data collectors were sensitised to practices such as monitoring body language and facial expressions to determine unwanted activities or interactions. In these instances, children were at first diverted to another activity (such as choosing a sticker) to see whether that helped to regain their interest, and if not, interviews were terminated and the appropriate procedures to mitigate distress were followed.
Ethical considerations
Ethical clearance to conduct this study was obtained from the Mildmay Uganda Research Ethics Committee (No. REF 0503-2017).
Results
While a full report of the results is not within the scope of this paper, we present the following excerpts from data analysis in progress and from previous papers published by the team, to underscore the ability of child-centric methods to speak about child outcomes in the same way as parent- or caregiver-focused evaluations.
In order to assess the experience of the reintegration, we felt it was important to understand the circumstances that led children to enter RCFs. We collected data through in-depth interviews conducted in Luganda with children aged 8–13 years and their caregivers. These children had completed 12 months post-reunification from six of the nine districts, namely, Lwengo, Mukono, Mpigi, Masaka, Rakai and Ssembabule. Out of the 23 children followed up, we conducted interviews with 11 who were old enough (8 years–13 years; three females and eight males) with data on the remaining 12 collected from their caregivers. Only four children and four caregivers were from the control group. In the presentation of findings below, we have aggregated the respondents across the two groups of those who received the parenting programme and those who did not, and we added in findings from questionnaires that both caregivers and children and filled out (see Walakira et al. 2022 for details).
Based on the relative comparison of the reasons offered by caregivers and children, one can see the similarities and differences between caregiver and child reports (see Figure 5). Children mentioned schooling and ‘don’t know’ as the dominant reason for their placement while parents mentioned lack of money and schooling. Understandably, children may not be privy to reasons such as lack of money or the lack of means for the caregiver to provide care. The results underscore the importance of including children as another set of respondents to assess their perspective, which may or may not be similar to the caregivers’ and lend credibility to child-centric methods.
Responses from (a) home caregivers (n = 77) and (b) children (n = 44) to reasons for entering residential care facilities.
Another set of preliminary findings relates to the child cognitive assessments, as described in the Methods section. We examined the composite score from the KABC-II measure. Based on a generalised linear modeling (GLM), the composite score showed a significant treatment effect (effect size = 4.11, 95% CI 1.32–6.91, p = 0.0047). Children in the intervention group showed significantly higher overall cognitive functioning (M = 71.7, SE = 0.96) than children in the comparison group (M = 67.6, SE = 1.03) at 6 months, after adjusting for their baseline scores. This shows that children in the intervention group, whose caregivers participated in the parenting intervention had a higher learning ability compared to children in the comparison group at 6 months, even after adjusting for their learning score at baseline.
Although obtaining statistically significant effects of the intervention is not a pre-condition to the validity of child-centric methods, it further bolsters the ability of the modified methods and the quality of the derived data to lend themselves to rigorous statistical analysis.
In summary, the study underscored the importance of hybrid methods, especially in collecting data from young children on sensitive topics across a variety of situations. We had to intentionally balance the rigour of data collection with the time and cost incurred in a prolonged piloting process and including children’s feedback in modifications of questionnaires and data collection protocol.
Any data collection from children and youth requires study personnel to being mindful of the power relations between adults and children and take time to build rapport, establish trust and assure the child that answering questions will not harm (or benefit) the child. These two elements are particularly important to ensure that the responses given by children during data collection are as valid and reliable as possible, and that children who are likely to have been through traumatic experiences are able to trust the researcher. Our approach to training by incorporating trauma-informed practices was helpful in avoiding negative situations during data collection, putting the children (and data collectors) at ease, which, presumably, increased the accuracy of data collected. We realised the importance of selecting a good and reliable cadre of data collectors and the lengthy process of training them. They were selected and trained to do whatever was required in the field at any given time, to ensure the trust and welfare of the child over and above the study requirements.
We also learnt that child-centred practices, such as the ones mentioned above, inevitably lead to missing data wherein the data obtained may not lend itself to straightforward analysis, or comparison across measures. Although missing data are a concern even with the traditional adult-focused methods, especially in a multi-methods longitudinal study such as the present one, the time and cost of the study to ensure complete responses from children for the entire length of the questionnaire or interview, need to be taken into account. In addition, the permissions required to obtain child reports can be more time consuming, especially if the granting authorities change over the course of the study – such as a different school or different set of caregivers. As such, statistical techniques that take into account the repeated assessments and changes within and across participants are critical. This can include the imputation of missing data using different techniques available (Huque et al. 2018) or using statistical techniques such as Generalised Estimating Equations (GEE) (Paik 1997).
Programme monitoring is essential, especially for programmes that are large in scope and involves many teams conducting multiple steps in implementing the programme across a period. The programme personnel who visited the RCFs and vetted families for child placement entered programme monitoring data from the field, such as the number of visits to the families, the number of parenting sessions attended that is critical to the implementation and evaluation of the programme.
Conclusion
Most evaluation methods that investigate children’s experiences are grounded in ‘research on’ rather than ‘research with’ or ‘research for’ children (Darbyshire, MacDougall & Schiller 2005; Oakley 1994).
Although we tried to have participatory approaches to modify the data collection procedures and measures, it would have been better to include these approaches while designing the study. Our suggestion is to conduct a ‘pre-evaluation’ while designing the study, to understand the critical constructs that are meaningful for the study context from the child and youth perspective – such as, what constitutes ‘successful reintegration’, what aspects of the RCFs are important to assess in the questionnaires, what aspects of a caregiver–child relationship is most meaningful to them. Although the responses to questions such as these will differ depending on child characteristics such as gender and age, it would be more meaningful to select a few representative respondents to assess their perspectives and to inform the study design and measures.
Mixed methods are important, especially in collecting data from young children on sensitive topics (e.g. modes of punishment). That entails the use of qualitative, open-ended responses as well as structured responses. The open-ended questions allow children to express responses that the adults may not have thought of and provide for a richer understanding of the realities of their lives. However, the length of the questionnaires and the number of variables that were assessed made it difficult to render all of them to be child focused. It also increased the time and resources required for the study to track down and follow-up the children, in addition to the caregivers.
Training adults to listen and being flexible is critical in child-focused research and evaluations. Conducting these studies essentially involves power differentials between adults and children. Training data collectors and the research team, at large, requires a frank discussion of these topics and a deep understanding of the consequences of research and doing unintended harm. Training adults to prioritise children’s feedback and to acknowledge children’s capacities is essential. Most importantly, the research team has to build in the training time and for the study to be child-focused right from the beginning.
Acknowledgements
We would like to acknowledge the support provided by Kato Francis, Makerere University, in training the field staff, collecting the materials for testing and providing other resources needed for data collection.
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Authors’ contributions
N.M. conceptualised and led the writing of the manuscript. E.J.W. contributed to the design of the study, headed the local data collection and training and reviewed the manuscript. K.M. was the project director; she contributed to the design of the study and managed the data collection and analysis and reviewed the manuscript. D.O. oversaw the study and reviewed the manuscript. P.B. worked with the data collection team in modifying the measures, headed parts of data collection and reviewed the manuscript. K.F. helped train the data collection team and collect data.
Funding information
The study was funded under the Coordinating Comprehensive Care for Children (four Children) programme supported by the United States Agency for International Development (USAID Co-operative Agreement #AIDOAA-A-14-00061) (see ULS link: https://www.usaid.gov/) and was code named ‘Keeping Children in Healthy and Protective families). It was implemented by partners including Catholic Relief Services (CRS), Intra-Health and Maestral USA. Westat Inc. (USA) and Makerere University (Uganda) were responsible for the implementation of the study and worked in collaboration with CRS, Child’s I Foundation and TPO among other local partners. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Data availability
The data that support the findings of this study are available upon reasonable request from the corresponding author, (N.M.).
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
ReferencesAchenbach, T. & Ruffle, T., 2000, ‘The child behavior checklist and related forms for assessing behavioral/emotional problems and competencies’, Pediatrics in Review21(1), 265–271.Achenbach, T.M., 2010, ‘Multicultural evidence-based assessment of child and adolescent psychopathology’, Transcultural Psychiatry47(5), 707–26https://doi.org/10.1177/1363461510382590Bangirana, P., Allebeck, P., Boivin, M.J., John, C.C., Page, C., Ehnvall, A. et al., 2011, ‘Cognition, behaviour and academic skills after cognitive rehabilitation in Ugandan children surviving severe malaria: A randomised trial’, BMC Neurology11, 96. https://doi.org/10.1186/1471-2377-11-96Bangirana, P., Seggane-Musisi, Allebeck, P., Giordani, B., John, C.C., Opoka, O.R. et al., 2009, ‘A preliminary examination of the construct validity of the KABC-II in Ugandan children with a history of cerebral malaria’, African Health Sciences9(3), 186–192.Bornstein, M.H., Putnick, D.L., Cote, L.R., Haynes, O.M. & Suwalsky, J.T.D., 2015, ‘Mother-infant contingent vocalizations in 11 countries’, Association for Psychological Science26(8), 1272–1284. https://doi.org/10.1177/0956797615586796Centers for Disease Control and Prevention & ASCD, 2014, Whole school whole community whole child: A collaborative approach to learning and health, viewed 03 March 2023, from https://www.cdc.gov/healthyschools/wscc/WSCCmodel_update_508tagged.pdf.Cluver, L.D., Lachman, J.M., Ward, C.L., Gardner, F., Peterson, T., Hutchings, J.M. et al., 2016, ‘Development of a parenting support program to prevent abuse of adolescents in South Africa’, Research on Social Work Practice27(7), 758–766. https://doi.org/10.1177/1049731516628647Darbyshire, P., MacDougall, C. & Schiller, W., 2005, ‘Multiple methods in qualitative research with children: More insight or just more?’, Qualitative Research5, 417–436. https://doi.org/10.1177/1468794105056921Ebrahim, H.B., 2008, ‘Situated ethics: Possibilities for young children as research participants in the South African context’, Early Child Development and Care180(3), 289–298. https://doi.org/10.1080/03004430701822958Essau, C.A., Sasagawa, S. & Frick, P.J., 2006, ‘Callous-unemotional traits in a community sample of adolescents’, Assessment13(4), 454–469. https://doi.org/10.1177/1073191106287354Fernández, M., Wegerif, R., Mercer, N. & Rojas-Drummond, S.M., 2001, ‘Reconceptualising “scaffolding” and the zone of proximal development in the context of symmetrical collaborative learning’, Journal of Classroom Interaction36, 40–54.Frick, P.J., 1991, ‘The Alabama Parenting Questionnaire. Unpublished rating scale, University of Alabama’, Journal of Child and Family Studies15(5), 597–616. https://doi.org/10.1037/t58031-000Harland, T., 2003, ‘Vygotsky’s zone of proximal development and problem-based learning: Linking a theoretical concept with practice through action research’, Teaching in Higher Education8(2), 263–272. https://doi.org/10.1080/1356251032000052483Huque, M.H., Carlin, J.B., Simpson, J.A. & Lee, K.J., 2018, ‘A comparison of multiple imputation methods for missing data in longitudinal studies’, BMC Medical Research Methodology18, 168. https://doi.org/10.1186/s12874-018-0615-6Jansen van Rensburg, M.S. & Jansen van Rensburg, D., 2020, ‘Child participation: Child contributors’ reflections on their experiences’, African Evaluation Journal8(1), a475. https://doi.org/10.4102/aej.v8i1.475Johnson, V., Hart, R. & Colwell, J., 2014, Steps to engaging young children in research: The guide, The Bernard van Leer Foundation, viewed 03 March 2023, from http://www.bernardvanleer.org/steps-to-engaging-young-children-in-research.Kaufman, A.S. & Kaufman, N.L., 2004a, Kaufman assessment battery for children: Second edition (KABC-II), American Guidance Service, Circle Pines, MN.Klein, E.M., Brähler, E., Dreier, M., Reinecke, L., Müller, K.W., Schmutzer, G. et al., 2016, ‘The German version of the Perceived Stress Scale – Psychometric characteristics in a representative German community sample’, BMC Psychiatry16, 159. https://doi.org/10.1186/s12888-016-0875-9Koura, K.G., Boivin, M.J., Davidson, L.L., Ouédraogo, S., Zoumenou, R., Alao, M.J. et al., 2013, ‘Usefulness of child development assessments for low-resource settings in francophone Africa’, Journal of Developmental and Behavioral Pediatrics34(7), 486–493. https://doi.org/10.1097/DBP.0b013e31829d211cLachman, J.M., Cluver, L., Ward, C.L., Hutchings, J., Mlotshwa, S., Wessels, I. et al., 2017, ‘Randomized controlled trial of a parenting program to reduce the risk of child maltreatment in South Africa’, Child Abuse & Neglect72, 338–351. https://doi.org/10.1016/j.chiabu.2017.08.014ManianN., 2021, Becoming trauma informed adverse childhood experiences and trauma: Implications for schools, TI Brief #1, The National Comprehensive Center, Rockville, MD, viewed n.d., from https://www.compcenternetwork.org/sites/default/files/National-Center-2021_Brief-1-ACEs.pdf.Manian, N. & Bornstein, M.H., 2009, ‘Infant regulation during maternal still-face: Dynamics and group differences’, Journal of Child Psychology and Psychiatry50(11), 1410–1418. https://doi.org/10.1111/j.1469-7610.2009.02166.xManian, N., Rog, D.J., Lieberman, L. & Kerr, E.M., 2022, ‘The organizational trauma-informed practices tool (O-TIPs): Development and preliminary validation’, Journal of Community Psychology50(1), 515–540. https://doi.org/10.1002/jcop.22628Mullen, E.M., 1995, Mullen scales of early learning, AGS edn., American Guidance Service Inc., Circle Pines, MN.Mullen, M.E., 2023, Mullen scales of early learning, viewed n.d., from https://www.wpspublish.com/mullen-scales-of-early-learning.Oakley, A., 1994, ‘Women and children first and last: Parallels and differences between children’s and women’s studies’, in B. Mayall (ed.) Children’s childhoods observed and experienced, pp. 13–32, The Falmer Press, London.Paik, M.C., 1997, ‘The generalized estimating equation approach when data are not missing completely at random’, Journal of the American Statistical Association92(440), 1320–1329. https://doi.org/10.1080/01621459.1997.10473653Rashid, T. & Ostermann, R.F., 2009, ‘Strength-based assessment in clinical practice’, Journal of Clinical Psychology65(5), 488–498. https://doi.org/10.1002/jclp.20595Reason, P. & Bradbury, H., 2008, The SAGE handbook of action research, SAGE, Newbury Park, CA.San Cristobal, P., Santelices, M.P. & Miranda Fuenzalida, D.A., 2017, ‘Manifestation of trauma: The effect of early traumatic experiences and adult attachment on parental reflective functioning’, Frontiers in Psychology8, 449. https://doi.org/10.3389/fpsyg.2017.00449Shabani, K., Khatib, M. & Ebadi, S., 2010, ‘Vygotsky’s zone of proximal development: Instructional implications and teachers’ professional development’, English Language Teaching3, 237–248. https://doi.org/10.5539/elt.v3n4p237Shamrova, D.P. & Cummings, C.E., 2017, ‘Participatory action research (PAR) with children and youth: An integrative review of methodology and PAR outcomes for participants, organizations, and communities’, Children and Youth Services Review81, 400–412. https://doi.org/10.1016/j.childyouth.2017.08.022Stattin, H. & Kerr, M., 2000, ‘Parental monitoring: A reinterpretation’, Child Development71(4), 1072–1085. https://doi.org/10.1111/1467-8624.00210Substance Abuse and Mental Health Services Administration (SAMHSA), 2015, A guide to GPRA data collection using trauma-informed interviewing skills, SAMHSA, Rockville, MD, viewed 03 March 2023, from https://www.pacarepartnership.org/uploads/Trauma-informed_Interviewing_Skills_Guide.pdf.Tarsilla, M., 2021, ‘How engaging with children and young people in evaluation will give a renewed sense of purpose to our profession’, Online presentation presented at the European Evaluation Society Conference, 08–10 September 2021, EES.UNICEF WCARO Child Focused Evaluation Webinar Series, 2022, Engaging with children in evaluation: A critical review of transformative methodological approaches, UNICEF, from Webinar 1 – Engaging with Children in Evaluation: A critical Review of Transformative Methodological Approaches.pdf (unicef.org). viewed n.d., from https://www.unicef.org/evaluation/media/2796/file/Webinar%201%20-%20Engaging%20with%20Children%20in%20Evaluation:%20A%20critical%20Review%20of%20Transformative%20Methodological%20Approaches.pdf.United Nations, 1990, United Nations Convention on the Rights of the Child, United Nations Commission on Human Rights, Geneva.Van Roy, B., Groholt, B., Heyerdahl, S. & Clench-Aas, J., 2010, ‘Understanding discrepancies in parent-child reporting of emotional and behavioural problems: Effects of relational and socio-demographic factors’, BMC Psychiatry10, 56. https://doi.org/10.1186/1471-244X-10-56Walakira, E., Nnyombi, A., Ssenfuuma, J., Kyamulabi, A., Kato, F., Natukunda, H. et al., 2022, ‘Qualitative insight into children’s and care-givers’ psychosocial experience following re-integration from Uganda’s residential care facilities into family-based care’, Global Studies of Childhood,1–15.
Research Project Registration:
Project Number: NCT03498469
How to cite this article: Manian, N., Walakira, E.J., Megazzini, K., Oliver, D., Bangirana, P. & Francis, K., 2023, ‘Child-focused evaluation: Involving children as their own respondents’, African Evaluation Journal 11(1), a688. https://doi.org/10.4102/aej.v11i1.688
We use the term ‘researcher’ throughout to refer to anyone involved in designing and conducting research or evaluation studies including programme staff, data collectors, training staff and so on.
The Coordinating Comprehensive Care for Children (4Children) programme was a six-year United States Agency for International Development (USAID)-funded consortium of organizations led by Catholic Relief Services (CRS) with partners IntraHealth, Maestral, Pact, Plan International, and Westat. 4Children was designed to improve health and wellbeing outcomes for orphans and vulnerable children (OVC).
We use the term ‘construct’ to note the abstract psychological or behavioural component (e.g. self-esteem, attitudes towards punishment) and the term ‘measure’ to refer to the instrument to assess attributes associated with a construct (e.g. Mullen scales to measure cognitive functioning or Rosenberg’s scale to assess self-esteem).
The Zone of Proximal Development (ZPD) was a key construct in Lev Vygotsky’s theory of learning and development according to Vygotsky, 1978, as cited in Shabani et al., (2010) The main idea of the ZPD is that a person with more knowledge can enhance a child’s learning by guiding them through a task slightly above their capacity/aptitude (see Bornstein et al. 2015; Harland, 2003).