Theory of Change for Development is a free online course developed at an African institution to strengthen evaluation capacity in the region. Massive Open Online Courses (MOOCs) provide a platform for building skills at scale in the region. Scholars of evaluation have long pointed to a gap between supply and demand that frustrates both evaluation practitioners and commissioners. This article explores the possibilities and limitations of MOOCs to bridge this gap.
This article seeks to explore the role open online training can play in evaluation capacity development. It considers the implications of one course’s experience of evaluation-sector-blended learning and instructional design in Africa for strengthening regional communities of practice, and for evaluation capacity development.
This article draws on current theories on online learning, specific course data from the Theory of Change course as well as experiential learning from two processes of revision and delivery of the MOOC to explore open online learning as an approach to evaluation capacity development.
This study has found that MOOCs can play important roles in evaluation capacity development. They can raise awareness of methods and tools that are not part of traditional monitoring and evaluation curriculum, their reach for evaluation practitioners is wide and they build much needed networks in the sector. However, their role in skills development has limits, particularly if they are delivered in isolation of other accessible contact learning options.
Massive Open Online Courses should be one part of an evaluation capacity development strategy. They support blended learning, expand access to certain core practical skills and link evaluation practitioners to other real outlets of learning and exchange.
This manuscript explores the role of Massive Open Online Courses (MOOCs) as a tool for evaluation capacity development (ECD) in Africa. Massive Open Online Courses, as their name suggests, are online courses that were initially offered for free, without admissions criteria, although this landscape is rapidly changing. These courses are usually offered by accredited higher education institutions through external online learning platforms. They were first introduced in 2006, and over the last decade, have demonstrated possibilities and limitations in improving access to education, expanding the development impact of education, building user-generated instructional material and revolutionising the flexibility and collaboration available through traditional learning outlets. Having developed and delivered a series of evaluation-focused MOOCs from an African academic institution, lessons are emerging about the promise and limitations MOOCs have in contributing to skills development in this area.
This article illustrates the role MOOCs can play in ECD in Africa. National evaluation systems are a growing priority in the region, as an approach to measuring progress towards achieving the sustainable development goals (SDGs), and strengthening public sector reform and democratic decision-making. As these systems are still new, the evaluation capacity landscape is varied, with gaps on both the demand side of evaluation commissioning and the supply side in evaluation practice and capacity development providers which are mandated to fill these gaps on both sides. The effectiveness of these systems depends on the wider ECD ecosystem, which must include communities of practice, contact training and other activities. The purpose of this article is to understand how to locate MOOCs within this broader landscape of ECD tools and approaches, to better understand the usefulness and relevance of open online courses in building evaluation capacity in the region. To respond to this question, this article will draw on the case study of a course, Theory of Change for Development, which was run as an instructor led MOOC on the edX platform twice, and is currently open as a self-paced course. It will draw on course data, as well as the experiences of instructional design and student feedback. This article will then explore the delivery of the course in the broader context of the regions ECD landscape.
Various studies have pointed to the gap between evaluation demand and supply, noting that demand has grown for functional monitoring and evaluation (M&E) systems in Africa that can result in effective evidence-informed policy-making and related programme interventions (Basheka & Byamugisha,
As a multidimensional concept, the value of including multiple approaches (such as coursework, mentorship, peer exchange, etc.) in broader ECD processes has been highlighted in developing evaluation competencies (Dillman
These discussions on effective ECD have evolved at the same time that the opportunities and constraints to online learning have become apparent. Following 10 years of MOOCs revolutionising online learning, disaggregated data are emerging to better understand the reality of the promise of MOOCs. The initial promise of MOOCs, because they are fee-free, was to expand access to communities that could otherwise not afford these educational opportunities. Most research on MOOCs has focused on America, and to a lesser extent European audiences, because this is both where the majority of institutions have supported MOOC creation, as well as where the majority of the study body has been based (Czerniewicz & Naidoo,
It has become apparent that for a range of reasons, MOOCs are not a panacea for access to education. Exclusion from higher education corresponds strongly to lack of access to technology, as well as the skills needed to navigate self-paced online learning (Liyanagunawardena et al.
Theory of change is both a process and product that underpins most theory-based evaluations. While it is one of the fundamental tools available to programme evaluators, many evaluators, particularly in Africa, come to the field through cognate disciplines (Tarsilla
In 2016, Wits University developed an online MOOC on theory of change for development. The course was identified because of a popular accredited short course provided by the Centre for Learning on Evaluation and Results (CLEAR-AA); the course is in demand and was identified in a training needs assessment as a priority area for skills development. Theory of change is considered a core skill for programme evaluators, but there are few accredited courses on it available in Africa. Additionally, the scope of the course was seen as commensurate with the time students may be willing to commit to a free online course.
Furthermore, because of staffing capacity constraints, it is not always possible to deliver contact versions of the course on demand. As staff capacity was a consideration, this was a course that already had training materials developed, and several experienced teaching staff who could easily deliver the content, or facilitate online discussions to support the delivery of the course. Finally, MOOCs can be more engaging if all suggested materials are open source, and there is a wide range of comprehensive, high-quality, open source material that covers both practical and theoretical approaches to theory of change (Gaebel
The Theory of Change for Development course was designed around six modules: introduction to theory of change, conducting a problem analysis, building a results chain, mapping stakeholders, building a theory of action and interpreting and evaluating a theory of change. Each module firstly identifies the objective of the exercise as part of a theory of change process, then provides an example of the exercise as part of a case study and finally provides practical tools an evaluation practitioner could use to apply the exercise in their own context. The six modules are delivered over 4 weeks of content, and both self-assessments and graded assessments take place at the end of each week of material. As a contact course, it is often delivered in one of two ways. One is to a cohort of programmatic staff working on a single programme to develop the theory of change for that programme. The second is to a wide variety of staff who include internal evaluators, programme managers and evaluation commissioners working on a range of programmes. In this situation, a multidisciplinary case study approach is used. Given the diversity of a global online student cohort, a fictional case study approach was chosen to allow for something that would support engagement across thematic areas of expertise and national origin.
This article uses data from the two theory of change courses to understand the profile of students participating in the course, understand the learning outcomes of the course and reflect on student feedback about the beneficial elements and limitations of the course. Data include pre- and post-course surveys administered to all students, as well as course data available to instructors. Because of the structure of course data, disaggregating by geographical origin of participants was not possible, limiting the possibility for authors to compare the participation and performance of students in Africa to other regions. However, given multiple data sources, the authors believe that it is still possible to draw valuable conclusions from the data available.
The 2017 course run attracted 2005 participants, while the 2018 run attracted 2242 participants in total. With MOOCs, completion rates are typically low, around 5%, and participation rates vary significantly through the duration of the course. Additionally, because of the varied online analytical data available, it is not possible to track the precise representation of the students who completed different components of the course. Therefore, the survey results and participation information reflect only the subset of the students who participated in that component of the course, and whose student information was therefore available to analyse.
The majority of the participants enrolled in the theory of change course in both 2017 and 2018 had postgraduate qualifications; 72% held a master’s or doctoral degree, demonstrating that the course largely composed of a skilled cohort of participants. Further, participants came from varied disciplines with economics, development studies, business management, public health and international relations backgrounds standing out. The majority of the course participants (74%) had job responsibilities related to M&E (including managing and reporting M&E information, working with or supervising individuals or teams with M&E responsibilities). Moreover, 88% were employed (mostly by International Organizations and Non-Governmental Organisations [NGOs]). This highlights that MOOCs are not opening access to ECD, but rather attracting highly educated, current professionals in the African evaluation sector.
Interestingly, a limited number of participants who took the course had no direct M&E job responsibilities; only 17% had no responsibilities related to M&E – these could represent people from other disciplines who had interest in moving into M&E-related roles as indicated by the progression of new practitioners into the discipline mostly from social sciences and economics backgrounds over the past years.
The majority of course participants stated their primary motivation of taking the course as being the convenience that is offered by a free online course that face-to-face does not necessarily offer. These advantages include the flexibility of scheduling and not incurring financial costs while receiving learning. Notably primary motivation for taking the course did not significantly indicate interest in the subject area but centred on the convenience offered by online learning.
With regard to learners’ skills, most participants indicated that their understanding of theories of change content and application prior to taking the course ranged between good and average (58%), while the rest of the participants had limited knowledge. Consequently, the MOOC provided an adequate space for both those who wanted to sharpen their skills as well as those who wanted to familiarise themselves with building new skills towards theories of change.
Remarkably, only 23% of the participants were familiar with other theory of change courses available in their countries; this shows the crucial gap that MOOCs such as these can fill to provide training on content areas that enhance evaluation capacity at a low cost as well as enhance access to such learning opportunities. It is also important to note that the majority of enrolments, 88%, were taking a theory of change (TOC) course for the first time, although over 50% had taken prior courses that made mention of TOC.
At around 2000 participants per course run, the theory of change course did not attract high student numbers by edX MOOC averages (Dennis
Engagement across the course modules shifted between 2017 and 2018, in part because of some improvements that were made in the course’s instructional design, as a result of student feedback. Certificated completion rates for each course were 7% and 8.4%, respectively, in 2017 and 2018, with 100% of those students purchasing a certificate on completing the course. While this seems low as an overall completion rate, a 5% completion rate is respectable for an MOOC, which is one of the key criticisms of this medium of teaching.
Massive open online course participants represented 139 countries in 2017 and 147 in 2018. While the largest cohort of students came from the USA both years, South Africa was the next largest cohort (about 11% of all students), and Kenya, Nigeria, Tanzania and Ghana were among the top 15 countries, which is far stronger representation from Africa than could be expected from most online courses. Thirteen African countries had more than 10 students enrolled. Additionally, there was also strong participation from other countries in the global South, with representatives from India, Thailand, Brazil and Peru.
Map of Theory of Change for Development course participants.
There are only limited analytics to suggest why this demographic make-up may be true, but the available data highlight that many students found the course through searching for ‘Wits University’, suggesting that the institution already has significant name recognition, and the nature of the course appealed to people across geographies. Furthermore, course marketing targeted African audiences.
The student body attracted to the course was composed primarily of professionals who are working in the monitoring and evaluation sector, and who have higher education degrees. While this suggests that the course did not necessarily attract sectoral professionals into the monitoring and evaluation space, it does suggest firstly that the course was well tailored to the needs of the evaluation sector in the region. Many course participants indicated that they chose to participate in the course because of a lack of available training opportunities on theory of change in their geographic location. Forty-two per cent of students indicated that they did not have alternative training opportunities available to them on theory of change. Seventy-six per cent of the participants in the course have direct responsibilities either conducting evaluations, managing evaluations or managing teams with evaluation functions. This indicates that the course was well matched to the needs of evaluation practitioners.
Sixty-eight per cent of participants from the 2017 and 2018 cohorts rated their levels of understanding and application TOC (after completion of the course) as being good or excellent, while only 22% of participants rated their understanding of theory of change as good or excellent prior to the course. This supports the success of the course in achieving its learning outcomes in broadly improving knowledge in TOC. This suggests that the learning opportunity (building and improving skills) that is offered by this course is valued by participants. A limitation of this study is that pre- and post-tests on content knowledge were not administered, so this study can only speak to student perceptions of the extent to which skills were built. In addition to the MOOC making progress towards its learning outcome, the majority (84%) of participants from both years agreed that the TOC course provided them with practical skills that are useful to apply in their current and future work in M&E.
This article followed all ethical standards for research without direct contact with human or animal subjects.
The case study approach was central to designing the course, and while it plays an important role in both the contact and online versions of the course, developing a case appropriate for an MOOC was critical to the course success. While the contact course is 3 days, with roughly two modules covered a day, the online version was adapted to be a 4-week course in the first run, and a 5-week course in the second run. The additional week was added despite the scope of content remaining the same, because of a need expressed by participants in the first course run to have time allocated with no additional course content to complete assignments and examinations.
The fictional case study was largely well received by most participants as a useful approach providing means to learning and application of learning. Eighty-two per cent of participants from both years considered the case study approach as being useful to their learning experience on the MOOC.
Student agreement with the statement ‘The online case study was helpful to my learning’.
However, integrating the case study design into the online instruction was a key lesson for the instructors. Consistency in introducing components of the case study, ensuring that each piece of the course material spoke to the case study, was a key challenge in course development. The investment was worthwhile, however, given that the case study material was then included in future contact courses. A key lesson is that in contact teaching, case studies are created to allow students to follow through an engaged logic process. Online, the purpose of a case study shifts, to facilitating collaboration across diverse contexts which can be difficult to achieve without content that can bring students into a common conversation. This again supports the idea that MOOCs can play an important role in strengthening collaboration and communities of practice in the evaluation space, though they may play a more limited role in building knowledge.
A number of comments pointed towards needing extra time to complete the course. Notably as most participants were employed, they took the course in between other commitments. Massive open online courses are thus beneficial in providing flexibility to a diverse range of participants although the completion time may still be limiting for other participants as this quote would indicate:
Allow a little more time to complete the course. Usually doing online courses is done in-between other work and commitments. (Anonymous course evaluation form, theory of change 2017)
Here, the resourcing is central, particularly given that 41% of course participants mentioned cost being a key consideration to furthering their education in the evaluation field. Most other activities that support collaboration and engagement for communities of practice are extremely costly, such as regional conferences. By nature, they exclude a subset of people, often those working in sectors or institutions that are not as well resourced. Perhaps, rather than comparing MOOC results to contact learning outcomes, it is also important to consider how they compare to other networking and knowledge sharing events, in terms of cost, engagement and results.
Finally, the content developed for an MOOC is available for future-blended learning and contact sessions. For this to work, instructors need to consider the potential and promise of an MOOC in material development, and also plan around its application in other learning contexts. As Wits University offers more blended learning options, there is potential to exchange and borrow appropriate material, which could free up capacity for contact teaching, and that, eventually, could address some of the institutional issues of capacity for contact teaching in the region.
Feedback around instructional design was a particularly important feature of the MOOC. While contact courses are often delivered with evaluation forms, the larger student numbers and multiple entry points for feedback (surveys, discussion forums, instructor contact and user-generated content) meant that there was rich student feedback to allow for improvements in the course, which can contribute to both online and contact learning. This points to a potential for MOOCs to play a role in diagnostic needs assessments in sectors where relatively little is known about the landscape of skills and training needs.
Massive Open Online Courses are not a
As much as MOOCs may not be increasing access to education in the evaluation sector, they are making important inroads around networking, strengthening communities of practice and contextualising teaching material. These have been identified as critical issues in the ECD space, making it important to further explore the contributions MOOCs can make when delivered, in a planned way, with other ECD interventions. Through increasing the integration of multidisciplinary ECD approaches, capacity development providers will better respond to the needs of the sector (Preskill & Boyle
The experience of delivering the Theory of Change for Development course in 2017 and 2018 highlighted the need for high-quality, contextually appropriate, practitioner focused curricula in Africa. Furthermore, it highlighted the geographic variability of certificated courses, as well as the variability in ability of evaluation practitioners to pay for continuing professional development. Further course delivery and engagement with the emergent analytics will fill an important information gap in the ECD market in the region.
The authors have declared that no competing interests exist.
C.B.M. was the academic lead on the Theory of Change for Development course, and both accredited the contact version of the course, and led the development of the online course materials. In addition, she took the leading role in conceptualising and drafting the article. L.K. supported the delivery of the Theory of Change for Development course and played a leading role in data analysis and discussion around the course content. H.E. was the internal beta tester of the course and led the thinking in the article around regionally effective capacity building approaches. D.W. led the technical development of the theory of change courses and brought her extensive online learning expertise to the article, contributing theory around instructional design, and the role of MOOCs in capacity development. She also developed a presentation on the course from which some of the data and discussion were drawn.
The views expressed in the submitted article are those of the authors alone, and do not reflect the views of Wits University or any other institution. While the authors are grateful to the university and edX for their support in delivering the course, any mistakes in the manuscript remain those of the authors.
Data sharing is not applicable to this article as no new data were created or analysed in this study.
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.