Original Research

Implementation of a project-based comprehensive monitoring and evaluation strategy in Ghana and Côte d’Ivoire: Key lessons

Daniel O. Chachu

Received: 01 Mar. 2019; Accepted: 07 May 2019; Published: 19 Aug. 2019

Copyright: © 2019. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: The quest for an appropriate monitoring and evaluation (M&E) design that delivers accountability, supports management and facilitates learning is one that many organisations grapple with. Over the years, experiences in project and/or programme development and delivery led the International Programme for the Elimination of Child Labour of the International Labour Organization (ILO-IPEC) to consolidate M&E efforts towards the development of a comprehensive monitoring and evaluation strategy.

Objectives: The aim of this article is to present lessons from the design and implementation of a theory of change-driven comprehensive monitoring and evaluation strategy in a child labour project rolled out in Ghana and Côte d’Ivoire. The 5-year project was implemented during 2011–2015 by ILO-IPEC with support from the United States Department of Labour (USDOL).

Methods: This article critically analyses project documents (including evaluations) and captures the reflections and experiences of key project staff involved in the project.

Results: Timeless lessons are distilled, along with key phases of the project cycle. Critical markers include the importance of stakeholders’ involvement in the design and development of a M&E strategy as a pre-requisite for buy-in and uptake. We find capacity building not just as a box to be ticked but an iterative process to improve knowledge, transfer skills and support learning. In addition to paying attention to technical elements, the soft issues of patience, flexibility and simplicity are discussed as invaluable ingredients for realising M&E goals.

Conclusion: While not exhaustive, it is hoped that these lessons would contribute to a minimum set of guidelines for improving M&E practice within projects and programmes.

Keywords: theory of change; monitoring; evaluation; results; child labour.

Background

The article presents an ex-post reflection of one of the latest generations of child labour projects being implemented by the International Labour Organization’s (ILO) International Programme on the Elimination of Child Labour (ILO-IPEC) across the world. The flagship project, ‘Towards child labour free growing communities in Côte d’Ivoire and Ghana through an integrated area-based approach’, was also known as the Cocoa Communities Project (CCP). The article presents learning experiences from the implementation of a comprehensive monitoring and evaluation strategy (CMES) within ILO-IPEC projects in Ghana and Côte d’Ivoire. In addition to serving as a reflective essay based on experience, it is also a product of interactions between the project teams in the two countries and technical staff of the then evaluation and impact assessment (EIA) section of ILO-IPEC.

The article has four sections. Firstly, a brief background information on the project and its context, including a working definition of the concept of child labour. Secondly, we discuss the CMES, followed by key lessons from its implementation and then a conclusion.

Key terminology – Child labour¹

Children carry out a wide range of tasks and activities when they work. Considerable differences exist between the many kinds of work they do. Some are difficult and demanding when juxtaposed against the health and well-being of the children. Others are hazardous, morally reprehensible and legally proscribed. According to ILO-IPEC, not all work done by children should be classified as child labour, which is to be targeted for elimination. Some kinds of work are indeed necessary for the socialisation of children. The term ‘child labour’ is often defined as work that deprives children of their childhood, their potential and their dignity and that is harmful to their physical and mental development. It refers to work that is mentally, physically, socially or morally dangerous, harmful to children and interferes with their schooling.

Whether or not particular forms of ‘work’ can be called ‘child labour’ depends on the child’s age, the type and hours of work performed, the conditions under which it is performed and the objectives pursued by individual countries. The answer varies from country to country, as well as among sectors within the countries. Attempts at these answers are usually elaborated in legal text, not least constitutions, child rights and/or protection laws, labour laws and other legislative instruments. These modes of expression apply to Ghana and Côte d’Ivoire.

International Programme for the Elimination of Child Labour’s contribution to child labour elimination

Child labour is a complex decent work deficit that must be challenged through a rights-based decent work approach to economic and social development. The approach must emphasise and enable the realisation of human right qualities as enshrined in the ILO Conventions (e.g. Convention numbers 138 and 182) and the various ILO Declarations (e.g. 1998 and 2008 Declarations).

The International Programme on the Elimination of Child Labour (IPEC), a programme of the International Labour Organization, is operational in over 80 countries. Its objective is the effective abolition of child labour. The programme assists member states to design and implement policies and programmes to prevent and eliminate child labour. International Programme for the Elimination of Child Labour works to enhance the awareness of member states, workers’ and employers’ organisations, the international community and a wide range of actors about the problem of child labour. It assists countries in implementing international labour standards on child labour and in taking the necessary comprehensive and integrated measures for sustained and effective action. These international standards are usually articulated in National Action Plans, which act as the policy framework for the elimination of child labour at the country level.

IPEC delivers technical and operational support to countries through its projects under integrated complementary strategies. Elements of this comprehensive strategy include:

• development of legal and policy framework, as well as enforcement capacity
• support to national capacity development and mainstreaming efforts
• targeted direct action (i.e. support to community livelihood schemes and social services)
• support to developing an integrated monitoring and evaluation (M&E) capacity
• research
• data collection and analysis
• external partnerships
• advocacy, awareness raising and social mobilisation.

The Cocoa Communities project

The CCP project was initially scheduled for implementation between January 2011 and August 2014, a total of 44 months. The project was, however, extended into the first quarter of 2015 to, among others, consolidate on the project gains and facilitate a smooth exit. The goal of the USD 5 million project was to prevent and eliminate child labour, particularly its worst forms, in the cocoa sector in 80 communities in Ghana and Côte d’Ivoire. This was to be achieved through strengthening local mechanisms for community action, social surveillance of child labour, and enhancing and linking national and local stakeholders’ efforts against child labour. Consistent with IPEC’s targeted direct action, the project was also aimed at improving the livelihood of households and children’s access to quality education.

Country contexts at the time of project design: Ghana and Côte d’Ivoire

During the project design phase, available national survey results conducted by the national statistics office of both countries indicated that about one in five children was involved in child labour. A 2001 National Child Labour Survey in Ghana indicated that about 64% of the children were in school. A cocoa sector survey conducted in 2009 by the government indicated that child labour in the sector stood at about 23%. An earlier cocoa sector survey in 2007/2008 found that although approximately 9 out of 10 children were in school, about half could neither read nor write. More than three-quarters of cocoa production in both countries were from individual small farm holdings.

In the Republic of Côte d’Ivoire (RCI’s) cocoa sector, studies by ANADER, a national rural development agency under the Ministry of Agriculture, revealed that nearly 9 in 10 children were involved in cocoa production. Reasons for children’s involvement in economic activities included not only contribution to family income but also participation in work as part of the socialisation process. The cocoa survey in RCI further revealed that 63% of the children were in school while about 21% had never attended school. Simultaneously, in 2007, a survey in the education sector revealed that approximately two in five children were not in primary school. The RCI government’s role in addressing child labour in the country in general and the cocoa sector, in particular, was dealt a big blow when the political crisis erupted in 2002. In addition to the north–south geographical split, many of the country’s institutions for addressing child labour were weakened. In 2007, a multisectoral National Child Labour Steering Committee was established in RCI to coordinate efforts in tackling child labour in cocoa and other sectors of the economy.

The comprehensive monitoring and evaluation strategy

Since 1993, ILO-IPEC projects have been designed and implemented with a M&E component. The product, as well as the process, has evolved over the years in keeping with the complexity of projects and emerging trends within the M&E space. The emphasis has been on relevance, utility and rigour.

The elements of M&E in IPEC projects have included strategic planning through a Strategic Programme Impact Framework (SPIF) methodology, the precursor to a project’s Theory of Change (TOC). In addition, there is scope for developing Project Monitoring Plans (PMPs), conducting baseline studies, monitoring delivery of goods and services and other high-level project results, in particular the number of children prevented and withdrawn from child labour (CL). Other elements are an independent mid-term and final evaluations, and in some cases ex-post tracking of or tracer studies on target children. These elements have been central to most IPEC projects. While these elements were included in most projects, they were not clearly and systematically articulated as part of an integrated plan. At the end of 2010, with funding from the Department of Labour of United States (USDOL), ILO-IPEC and the then EIA (IPEC’s evaluation unit) led a process to design an M&E strategy that articulates all the critical M&E elements of a project. This is known as the Comprehensive M&E Strategy (CMES).² The term ‘Comprehensive’ is used to reflect the intention to ‘provide evidence of progress and linkages at different levels’ (ILO-IPEC, July 2010). Comprehensiveness also attests to monitoring not just ‘what happens’ (results) but also capturing ‘how change happened’ (implementation features and time frame) and ‘why it happened’, that is, understanding different contributions and/or attribution (ILO-IPEC, July 2010; ILO-IPEC, August 2013). Since late 2011, IPEC projects have integrated a CMES in all new projects, adapting it to suit each context.

The CMES implementation in Ghana and Côte d’Ivoire had five key components (as shown in Figure 1):

• The TOC: The TOC served as the foundation and central guide for the articulation of the CMES. It provided the overall framework for understanding the changes envisaged in the project, and how these changes/results were to be realised. As a precursor to the development of a TOC, a SPIF analysis was conducted with stakeholders. It allowed stakeholders in each country to consider and identify contextual issues underlying the causes of child labour, the desired outcomes and the chain of results. Examples of direct outputs from this process included the development of a Problem Tree and a Project Outcome Tree for each country. This approach was to reflect nuances and differences in the legal and policy framework for tackling child labour between the two countries. For example, the minimum age for entry into work in Côte d’Ivoire is 14, a year lower than Ghana’s. School progression in RCI was premised on the ability to provide birth certificates, which in some cases constituted a considerable barrier to education access. The volatile security situation in some communities following RCI’s emergence from a civil war meant that key assumptions underlying the TOC would differ between the two countries.
• An Outcome Measurement Framework (OMF): It comprised indicators that covered four main levels of results from inputs to impact. The OMF goes beyond articulating internal (project) contributions by recognising ‘external’ influence. It, therefore, captures context indicators (pre-existing or third-party factors that had implications for project results). The OMF also envisaged the way data will be collected, by whom, when and how. This was backed by a data collection plan. The development and application of the OMF allowed for the contextual differences between the two countries.
• Baseline study: The baseline surveys served as the basis for populating baseline indicators, selecting target direct beneficiaries (DB), assessing progress and measuring impact. With regard to the latter, a repeated baseline study was envisaged.
• Monitoring component: It provided a basis for monitoring key project results including services to beneficiaries, capacity building and context. This was backed with a verification process to ensure quality control while meeting ILO-IPEC standards for reporting children as having been withdrawn or prevented from child labour. The CMES also made allowance for mandatory monitoring requirements negotiated with the donor for purposes of accountability.
• Evaluation component: Three types were envisaged – an independent mid-term, final evaluation and an impact evaluation. The mid-term evaluation was replaced by a project implementation review (Upton 2013). An expanded final evaluation was carried out instead of an originally proposed impact evaluation exercise.

FIGURE 1: Components of the comprehensive monitoring and evaluation strategy.

The planned impact evaluation (IE) was experimental in nature, which meant randomly assigning treatment and control communities to assess the impact of the overall set of interventions on child welfare outcomes (Understanding Child Work [UCW], 2012b).³ The approach was decided based on the government and other key stakeholders’ demands. At the design phase, adequate resources were planned for all the elements of the IE, including benefitting from technical expertise from the Understanding Child Work Program of the ILO, World Bank and the United Nations Children’s Fund (UNICEF). During the implementation stage, contextual issues such as sky-rocketing costs for the data collection exercises, ‘costly’ internal budget review processes vis-à-vis time available and new demands from key partners meant that the project management needed to make relevant adjustments to the evaluation design. Towards the end of the project, it was decided that a ‘comprehensive’ final evaluation would be fit for purpose. This was an expanded final evaluation characterised by special sub-studies that focused on key sustainability dimensions of the project (e.g. the role of enhanced livelihood strategies in eliminating child labour).

Key lessons for different phases of the project cycle

This section discusses key lessons from the project, along with three main points in the project cycle: the design phase, implementation phase and the completion phase (i.e. evaluation and post-evaluation phase).

Lessons learnt at the design phase

Building ownership from the beginning: IPEC’s long-held tradition of involving stakeholders from the beginning through the SPIF, which then evolved to the use of the TOC, meant that communities had a say in shaping the nature and outcome of the CCP project. Project stakeholders’ involvement in the development of the TOC was central to reflecting their views in the CMES. In addition to community members (i.e. main target beneficiaries), project stakeholders (for example national and local government offices, implementing nongovernmental organisations [NGOs], private service providers and development partners) could develop a shared understanding of the goals, intervention strategy and envisaged results. This was important for promoting the use of a ‘common language’ among implementing partners and other stakeholders. The CMES became an important vehicle for building ownership and generating a commitment to various elements along the project cycle. Working out a deliberate strategy at the design phase to maintain this ownership was an important learning point. To this end, the use of project reviews and other knowledge-sharing platforms was very instrumental.

The ownership and understanding of the CMES was a necessary ‘learning-by-doing’ process. The development of the OMF was a painstaking process that required consultations on the appropriateness, accuracy and usefulness of indicators in describing project results. This was important because the OMF was going to determine how success was defined and measured. The process of validating indicators identified during the planning phase took more than a year in an incremental process of quality improvement. This was necessary as it constituted a shared process of assessing progress and measuring outcomes. The project team, direct beneficiaries, implementing agencies or partners and other stakeholders were more likely to appreciate and value the information generated if they were involved in the identification of their own information needs. Consequently, this process required its own pace. After understanding the usefulness of specific indicators, it would follow logically to discuss their value since the collection of data on any single variable and/or indicator comes with an associated cost, which can be estimated. Thus, these requirements should logically precede the data collection process.

Lessons at the implementation phase

Capacity building as an enabler of the ‘learning-by-doing’ process: Among other things, the purpose of training is to share knowledge, transfer skills, and develop capacity for action. Training provided was designed to be flexible, taking place at different levels from community to national but also based on specific needs. Training activities on the CMES was designed to follow this approach, including visiting offices and work spaces of implementing partners and direct beneficiaries. For example, following the launch of the CMES, project stakeholders (including implementing partners) attended an initial training workshop at the national capitals. The initial reaction from participants was that the CMES was too ambitious and complex. The capacities and capabilities were heterogeneous. This meant a more customised and hands-on approach for those who needed support most. The initial project-wide training workshops were complemented by various follow-up technical support services at the local level. Ongoing technical support was also based on a number of informal channels such as meetings on the side-lines of other project meetings or workshops and conference calls (including use of Skype). The project M&E officer, with support of the project team,⁴ also revised critically data and/or information on a quarterly basis, with necessary adjustments to make the products of the CMES more useful and analytical for the agencies concerned. In this way, the value-addition of the CMES became more obvious while contributing to changing attitudes and building a culture.

The effective take-up of an M&E strategy does not just happen. It requires creating incentives: An effective M&E strategy without the necessary organisational structure, resources and motivation to execute is dead on arrival. The project strategy to have an M&E Officer overseeing the CMES with support from other project staff strategically located in working centres close to project sites was an important element for lending ongoing support. As expected, the requirement to recruit staff with some knowledge and/or training in M&E was useful but not sufficient. For many on the labour market, but also for organisations (state and non-state), capacity for M&E was embryonic, and therefore a strict requirement for set skills was going to be counterproductive. Instead, the opportunity for training on the job was going to be less intimidating for ‘newcomers’ while at the same time an important means of backstopping.

Adequate planning and budgeting for M&E are critical for a meaningful rollout of an M&E strategy. This includes enough contingency measures for the unforeseen. The rule-of-thumb allocation of 3% – 5% of the total project to M&E is a starting point for many projects. Enough resources for the various elements of the CMES – surveys, field visits, training and data-use workshops, and so on – were considered for implementing the CMES; however, actual cost for the surveys diverged significantly from what was anticipated. The implication was a general delay in the rollout of the full package of interventions due to a mandatory organisational requirement for a budget review to free more resources from other budget lines towards funding the baseline survey. With the latter out of the way, adequate resources meant many planned activities under the CMES could be carried out.

Incentives for implementing the CMES stem from a sense of understanding of the tools, a deep appreciation of the value of the data to be collected and confidence in the ability to apply the tools. In addition to building capacity in these areas, reporting procedures add to the incentive structure. Periodic reports with CMES indicators, for example semi-annual reports to the donor and updates to the National Steering Committee on Child Labour, were tasks that carried intrinsic motivation for compliance. Periodic updates to local authorities were another important element of ensuring accountability. The regular use of CMES tools provided enough exposure to appreciating their value beyond the scope of the project. For instance, in the case of Ghana, the project strategy to engage with the National Development Planning Commission (responsible for the country’s M&E system) and its sub-national support structures meant that there was an opportunity to contribute to refining national and sub-national indicators for monitoring and reporting on child labour. The project team was also well positioned to make input into the development of a much-awaited National Monitoring and Evaluation Manual. Furthermore, there was an opportunity to contribute to ILO’s technical and financial support for the development of the sixth round of the Ghana Living Standards Survey, which had a special module on child labour.

Contextual factors matter enough to have data requirements: The M&E strategy looks beyond data and/or information gathering on project intervention, by including data on external factors (for example, the role of other actors in the treatment and control communities), and the monitoring of other contextual factors. The latter would include capturing the local and national policy context over the period of implementation and how these relate to the project intervention. Issues of conflict and/or potential conflict could also be captured in order to advise project management on an appropriate response. This was very important in some project communities in Côte d’Ivoire. The application of this systematic monitoring of context is a new dimension of M&E for stakeholders, who have come to appreciate its value-addition for project efficiency and effectiveness. Context monitoring made project staff and implementing agencies aware of how to avoid duplication with other organisations in services support to communities. Additionally, it helped implementing agencies to identify partners they could work with towards influencing project efficiency and effectiveness. At the evaluation stage, context monitoring was helpful for attribution analysis.

Keeping it simple but not more so: While maintaining the essential building blocks of the CMES, the project in Ghana and Côte d’Ivoire used an iterative approach that supported the adoption of simplified CMES elements not only within the project but also by interested stakeholders such as national and local government staff. In both Ghana and Côte d’Ivoire, the process of identification and review of project indicators at outcome level meant that the project was strategically positioned to support ongoing national and local government processes to develop a national surveillance system on child labour. Furthermore, a local government authority in Ghana requested technical support to analyse project baseline data on child labour with a view to supporting their preparation of a district medium-term development plan. Keeping the tools for the CMES simple made it easier for stakeholders to identify potentially useful elements for their own projects and programmes. Another advantage was the opportunity for the project team to influence other ongoing national processes such as the national survey and a national action plan on child labour.

Going beyond donor reporting to positioning CMES as a means for learning: At the beginning, the CMES’s systematic use was more focused on reporting to the project, donor and government. The implementation agencies (government organisations, NGOs, etc.) were also more focused on the delivery of outputs. There were also some trade-offs with the need to speed up implementation given the late start of project field activities. Consequently, what mattered more to implementers was outputs delivery and basic narrative and financial reporting. Thus, analysis of results was lacking with limited exploration of lessons learned. Methodologies and tools to facilitate learning such as guides or questions to aid periodic analysis and/or learning meetings, ways to approach the results-based analysis of information and use of self-learning modules on how to question data were examples of elements identified as lacking and thus constituting an important weakness in the CMES implementation process.

Tools and organisational incentives, such as more learning-oriented reporting as well as facilitation of analysis based on the TOC, were required to encourage learning. Applying CMES as a learning and knowledge management tool also required a closer interaction between the M&E officer and other project staff on one hand and the implementing agencies and other stakeholders on the other. Though there were information-sharing platforms, they lacked the appropriate framework to facilitate continual learning but also sustain their usefulness. The internal project review exercises clearly indicated the need to build more capacity to orient M&E towards learning, particularly for complex thematic areas like child labour interventions (Upton 2013). Development of training modules, on-site training and mentoring activities that facilitate learning could have been better explored.

Lessons at the completion phase: Evaluation and post-evaluation

A desire to know matched with an ability to show: Demand for in-country training on tackling child labour at national and sub-national levels (e.g. through a National Action Plan with special features such as a national surveillance system) backed by a strong M&E strategy became clear. The consultation process leading to the development of impact evaluation design revealed that there was a demand from key national stakeholders regarding evidence about achievements. Particularly, there was a desire to understand what worked (including how and why) as far as addressing child labour. The gap in knowledge also called for an approach that was acceptable and useful to national stakeholders. The use of experimental methods in impact evaluation has been found to be helpful in policy-making. While some have described the method as a ‘gold standard’ in conducting evaluations, it may not be applicable in every case. Capacity, in terms of resources, technical ability or feasibility and project context, is important in deciding on evaluation methods. Alternative approaches can prove equally useful. In other words, the best need not be the enemy of the good.

Flexibility in making adjustments and the essence of time: As argued above, flexibility would be required in any successful project implementation. The switch from an impact evaluation to an expanded final evaluation is one example of how projects must continually evaluate options available in the face of unforeseen developments.

In addition to being flexible, the importance of patience, continual support and time in the rollout of the CMES could not be overemphasised. Every effort to reduce implementation delays is key and would have an implication for the cost of executing any action. Time is also of essence in understanding, adopting and using new tools and methodologies. Project managers must keep their eye on the bigger picture as part of strategies to deal with nascent challenges and frustrations that can trigger conflict and reduce morale. Over the passage of time (and certainly with continuous support), challenges potentially turn to opportunities for making adjustments, learning and, in some cases, changing course altogether.

Successful project closure starts at the beginning and not at the end: An upfront investment into strategic analysis for sustaining project outcomes is a key element of any good M&E strategy. This includes identifying critical stakeholders, in particular state officials whose assigned mandates border on the project intervention. Such analysis should be included as a key component of plans and strategies of implementation agencies, whatever their levels of involvement and whether they are government, employer or worker organisations, or NGOs. The CMES documentation had such an analysis, which also featured in the project document of the various implementation agencies.

Beyond the analysis stage, a periodic assessment to review the state of readiness for handing over to state-mandated agencies must form an essential part of the monitoring process. This provides enough time and room to build capacity for smooth project closure and exit.

Conclusion

The CMES constitutes an important element of ILO-IPEC’s global strategy for tackling child labour in a comprehensive and effective manner. The design and implementation of the CMES have presented useful lessons that span the project cycle.

At the design stage, developing a shared understanding and building ownership of project goals are found to be essential for satisfactory achievement. Building capacity and providing incentives for realising project targets are implicit in this process. Other relevant elements of the implementation phase include keeping an eye on contextual (third-party or external) factors that can influence project outcomes. Then, there is a need to go beyond project reporting to donors to promoting learning among stakeholders. Tools and methodologies for M&E must also be designed to be fit for purpose without being unwieldy or overly complex.

Finally, experience from the rollout of the CMES indicates that time matters, patience is key and ongoing support for uptake is invaluable.

Acknowledgements

I am indebted to Ricardo Furman (ILO), whose initiative and guidance in our earlier attempt to share preliminary lessons from Ghana and El Salvador at the 2013 American Evaluation Association Conference inspired this article.

The author declares that he has no financial or personal relationships that may have inappropriately influenced him in writing this article.

D.O.C. is the sole author of this article.

This article followed all ethical standards for research without direct contact with human or animal subjects.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Data sharing is not applicable to this article as no new data were created or analysed in this study.

The views expressed here are solely those of the author. They do not necessarily reflect the views of the ILO, its affiliate bodies, the US government and its agencies (including USDOL) or any other organisation referred to in this publication. The author takes full responsibility for all errors.

References