Abstract
Background: Theory of Change for Development is a free online course developed at an African institution to strengthen evaluation capacity in the region. Massive Open Online Courses (MOOCs) provide a platform for building skills at scale in the region. Scholars of evaluation have long pointed to a gap between supply and demand that frustrates both evaluation practitioners and commissioners. This article explores the possibilities and limitations of MOOCs to bridge this gap.
Objectives: This article seeks to explore the role open online training can play in evaluation capacity development. It considers the implications of one course’s experience of evaluation-sector-blended learning and instructional design in Africa for strengthening regional communities of practice, and for evaluation capacity development.
Method: This article draws on current theories on online learning, specific course data from the Theory of Change course as well as experiential learning from two processes of revision and delivery of the MOOC to explore open online learning as an approach to evaluation capacity development.
Results: This study has found that MOOCs can play important roles in evaluation capacity development. They can raise awareness of methods and tools that are not part of traditional monitoring and evaluation curriculum, their reach for evaluation practitioners is wide and they build much needed networks in the sector. However, their role in skills development has limits, particularly if they are delivered in isolation of other accessible contact learning options.
Conclusion: Massive Open Online Courses should be one part of an evaluation capacity development strategy. They support blended learning, expand access to certain core practical skills and link evaluation practitioners to other real outlets of learning and exchange.
Keywords: capacity development; evaluation; online learning; Africa; Massive Open Online Courses.
Introduction
This manuscript explores the role of Massive Open Online Courses (MOOCs) as a tool for evaluation capacity development (ECD) in Africa. Massive Open Online Courses, as their name suggests, are online courses that were initially offered for free, without admissions criteria, although this landscape is rapidly changing. These courses are usually offered by accredited higher education institutions through external online learning platforms. They were first introduced in 2006, and over the last decade, have demonstrated possibilities and limitations in improving access to education, expanding the development impact of education, building user-generated instructional material and revolutionising the flexibility and collaboration available through traditional learning outlets. Having developed and delivered a series of evaluation-focused MOOCs from an African academic institution, lessons are emerging about the promise and limitations MOOCs have in contributing to skills development in this area.
This article illustrates the role MOOCs can play in ECD in Africa. National evaluation systems are a growing priority in the region, as an approach to measuring progress towards achieving the sustainable development goals (SDGs), and strengthening public sector reform and democratic decision-making. As these systems are still new, the evaluation capacity landscape is varied, with gaps on both the demand side of evaluation commissioning and the supply side in evaluation practice and capacity development providers which are mandated to fill these gaps on both sides. The effectiveness of these systems depends on the wider ECD ecosystem, which must include communities of practice, contact training and other activities. The purpose of this article is to understand how to locate MOOCs within this broader landscape of ECD tools and approaches, to better understand the usefulness and relevance of open online courses in building evaluation capacity in the region. To respond to this question, this article will draw on the case study of a course, Theory of Change for Development, which was run as an instructor led MOOC on the edX platform twice, and is currently open as a self-paced course. It will draw on course data, as well as the experiences of instructional design and student feedback. This article will then explore the delivery of the course in the broader context of the regions ECD landscape.
Background
Massive open online courses in evaluation capacity development in Africa
Various studies have pointed to the gap between evaluation demand and supply, noting that demand has grown for functional monitoring and evaluation (M&E) systems in Africa that can result in effective evidence-informed policy-making and related programme interventions (Basheka & Byamugisha, 2015; Porter & Goldman 2013; Preskill & Boyle 2008). In turn, demand for ECD has seen rapid growth in recent years by various stakeholders including national governments, local and international donors and multilateral organisations among others (Cohen 2006; Morkel & Ramasobana 2017). The landscape of demand is varied, with monitoring and evaluation systems in the region still focusing disproportionately on compliance, with evaluative practice lagging behind.
As a multidimensional concept, the value of including multiple approaches (such as coursework, mentorship, peer exchange, etc.) in broader ECD processes has been highlighted in developing evaluation competencies (Dillman 2012; Naccarella et al. 2007). The study identified how the different educational approaches respond to different ECD needs, and noted that fieldwork and mentoring contribute more to skill acquisition, because of the inclusion of contextual considerations, project management and effective communication skills, while coursework tends to contribute in a more limited way to knowledge (Dillman 2012:279). While there tends to be an emphasis on ‘training’ delivered as short courses in the evaluation space, it has widely been acknowledged that training in isolation will not achieve significant and sustained change. Its value is often limited to building specific individual capacities (Dillman 2012; Morkel & Ramasobama 2017; Preskill & Boyle 2008).
These discussions on effective ECD have evolved at the same time that the opportunities and constraints to online learning have become apparent. Following 10 years of MOOCs revolutionising online learning, disaggregated data are emerging to better understand the reality of the promise of MOOCs. The initial promise of MOOCs, because they are fee-free, was to expand access to communities that could otherwise not afford these educational opportunities. Most research on MOOCs has focused on America, and to a lesser extent European audiences, because this is both where the majority of institutions have supported MOOC creation, as well as where the majority of the study body has been based (Czerniewicz & Naidoo, 2013; Liyanagunawardena, Williams & Adams 2013; Oyo & Kalema, 2014). However, the global South, in particular Africa, faces the biggest limitations to access to education. Access to higher education in Africa stands at about 6% with goals of 12%, while Europe and North American have access of around 45%. There are many reasons to believe MOOC participation and purpose may look different in Africa, and more research is needed to understand this (Boga & McGreal, 2014; Nesterko et al. 2013; Richards & Diallo 2015). Many scholars of open online education are highlighting the importance of locally relevant needs to be reflected in online teaching material (Gaebel 2013).
It has become apparent that for a range of reasons, MOOCs are not a panacea for access to education. Exclusion from higher education corresponds strongly to lack of access to technology, as well as the skills needed to navigate self-paced online learning (Liyanagunawardena et al. 2013). However, with this disappointment came a better understanding of what MOOCs can do, and this article tries to frame the analysis on the possibilities MOOCs hold around the evaluation capacity needs in Africa. They hold greater promise for collaborative learning, and because of the opportunities for student generated content, they are also useful for contextually relevant learning (Castillo et al. 2015; Dillman 2012). Importantly, these are key gaps in Africa’s ECD space, suggesting that MOOCs could be uniquely well suited to addressing certain capacity gaps in the field of evaluation in the region. This hypothesis seemed to prove correct in the Theory of Change for Development course (Garrido et al. 2016). While the course did not attract a huge African audience to the evaluation sector that was otherwise unaware of the field of study, it did play a valuable role in networking this sector and creating other opportunities for collaboration. Previous evaluations and reviews of evaluation capacity in the region have pointed to certain scarce skills in the region, but more prominently, to a need to better align available skills with evaluations demanded, and also to strengthen institutional mechanisms in the evaluation sector, such as communities of practice and mechanisms for peer learning.
Theory of change for development
Theory of change is both a process and product that underpins most theory-based evaluations. While it is one of the fundamental tools available to programme evaluators, many evaluators, particularly in Africa, come to the field through cognate disciplines (Tarsilla 2014). Even those who have explicitly studied monitoring and evaluation as a discipline often do so either through programmes which offer a theoretical lens to the field, or which embed M&E in another discipline, such as public policy, or health (Morkel & Ramasobana 2017). While many private training providers offer practice-based instruction on theory of change, these short courses are usually unaccredited, and vary tremendously in their quality (Tarsilla 2014). As a result, many evaluators have learned theory of change approaches while on the job and use theory of change approaches in their work, without necessarily having adequate resources for learning good practice, exchanging experiences, and other professional development opportunities that might be available to more professionalised sectors. This was upheld by the experience of the Theory of Change for Development course, which largely attracted educated, practising evaluation professionals in the region.
In 2016, Wits University developed an online MOOC on theory of change for development. The course was identified because of a popular accredited short course provided by the Centre for Learning on Evaluation and Results (CLEAR-AA); the course is in demand and was identified in a training needs assessment as a priority area for skills development. Theory of change is considered a core skill for programme evaluators, but there are few accredited courses on it available in Africa. Additionally, the scope of the course was seen as commensurate with the time students may be willing to commit to a free online course.
Furthermore, because of staffing capacity constraints, it is not always possible to deliver contact versions of the course on demand. As staff capacity was a consideration, this was a course that already had training materials developed, and several experienced teaching staff who could easily deliver the content, or facilitate online discussions to support the delivery of the course. Finally, MOOCs can be more engaging if all suggested materials are open source, and there is a wide range of comprehensive, high-quality, open source material that covers both practical and theoretical approaches to theory of change (Gaebel 2013).
The Theory of Change for Development course was designed around six modules: introduction to theory of change, conducting a problem analysis, building a results chain, mapping stakeholders, building a theory of action and interpreting and evaluating a theory of change. Each module firstly identifies the objective of the exercise as part of a theory of change process, then provides an example of the exercise as part of a case study and finally provides practical tools an evaluation practitioner could use to apply the exercise in their own context. The six modules are delivered over 4 weeks of content, and both self-assessments and graded assessments take place at the end of each week of material. As a contact course, it is often delivered in one of two ways. One is to a cohort of programmatic staff working on a single programme to develop the theory of change for that programme. The second is to a wide variety of staff who include internal evaluators, programme managers and evaluation commissioners working on a range of programmes. In this situation, a multidisciplinary case study approach is used. Given the diversity of a global online student cohort, a fictional case study approach was chosen to allow for something that would support engagement across thematic areas of expertise and national origin.
This article uses data from the two theory of change courses to understand the profile of students participating in the course, understand the learning outcomes of the course and reflect on student feedback about the beneficial elements and limitations of the course. Data include pre- and post-course surveys administered to all students, as well as course data available to instructors. Because of the structure of course data, disaggregating by geographical origin of participants was not possible, limiting the possibility for authors to compare the participation and performance of students in Africa to other regions. However, given multiple data sources, the authors believe that it is still possible to draw valuable conclusions from the data available.
Overview of the participating students
The 2017 course run attracted 2005 participants, while the 2018 run attracted 2242 participants in total. With MOOCs, completion rates are typically low, around 5%, and participation rates vary significantly through the duration of the course. Additionally, because of the varied online analytical data available, it is not possible to track the precise representation of the students who completed different components of the course. Therefore, the survey results and participation information reflect only the subset of the students who participated in that component of the course, and whose student information was therefore available to analyse.
The majority of the participants enrolled in the theory of change course in both 2017 and 2018 had postgraduate qualifications; 72% held a master’s or doctoral degree, demonstrating that the course largely composed of a skilled cohort of participants. Further, participants came from varied disciplines with economics, development studies, business management, public health and international relations backgrounds standing out. The majority of the course participants (74%) had job responsibilities related to M&E (including managing and reporting M&E information, working with or supervising individuals or teams with M&E responsibilities). Moreover, 88% were employed (mostly by International Organizations and Non-Governmental Organisations [NGOs]). This highlights that MOOCs are not opening access to ECD, but rather attracting highly educated, current professionals in the African evaluation sector.
Interestingly, a limited number of participants who took the course had no direct M&E job responsibilities; only 17% had no responsibilities related to M&E – these could represent people from other disciplines who had interest in moving into M&E-related roles as indicated by the progression of new practitioners into the discipline mostly from social sciences and economics backgrounds over the past years.
The majority of course participants stated their primary motivation of taking the course as being the convenience that is offered by a free online course that face-to-face does not necessarily offer. These advantages include the flexibility of scheduling and not incurring financial costs while receiving learning. Notably primary motivation for taking the course did not significantly indicate interest in the subject area but centred on the convenience offered by online learning.
With regard to learners’ skills, most participants indicated that their understanding of theories of change content and application prior to taking the course ranged between good and average (58%), while the rest of the participants had limited knowledge. Consequently, the MOOC provided an adequate space for both those who wanted to sharpen their skills as well as those who wanted to familiarise themselves with building new skills towards theories of change.
Remarkably, only 23% of the participants were familiar with other theory of change courses available in their countries; this shows the crucial gap that MOOCs such as these can fill to provide training on content areas that enhance evaluation capacity at a low cost as well as enhance access to such learning opportunities. It is also important to note that the majority of enrolments, 88%, were taking a theory of change (TOC) course for the first time, although over 50% had taken prior courses that made mention of TOC.
Participation, course engagement and learning
At around 2000 participants per course run, the theory of change course did not attract high student numbers by edX MOOC averages (Dennis 2012), but are high for African-institution-affiliated MOOCs. This may speak to a difference in purpose MOOCs play for institutions in different regions. An early criticism of MOOCs is that they largely served as advertising for the sponsor institutions (Rohs & Ganz 2015), while objectives for Wits University are more directly around skills development and the developmental mandate of educational institutions.
Engagement across the course modules shifted between 2017 and 2018, in part because of some improvements that were made in the course’s instructional design, as a result of student feedback. Certificated completion rates for each course were 7% and 8.4%, respectively, in 2017 and 2018, with 100% of those students purchasing a certificate on completing the course. While this seems low as an overall completion rate, a 5% completion rate is respectable for an MOOC, which is one of the key criticisms of this medium of teaching.
Massive open online course participants represented 139 countries in 2017 and 147 in 2018. While the largest cohort of students came from the USA both years, South Africa was the next largest cohort (about 11% of all students), and Kenya, Nigeria, Tanzania and Ghana were among the top 15 countries, which is far stronger representation from Africa than could be expected from most online courses. Thirteen African countries had more than 10 students enrolled. Additionally, there was also strong participation from other countries in the global South, with representatives from India, Thailand, Brazil and Peru. Figure 1 demonstrates a scaled map of the countries from which participants came in the 2017 theory of change course.
 |
FIGURE 1: Map of Theory of Change for Development course participants. |
|
There are only limited analytics to suggest why this demographic make-up may be true, but the available data highlight that many students found the course through searching for ‘Wits University’, suggesting that the institution already has significant name recognition, and the nature of the course appealed to people across geographies. Furthermore, course marketing targeted African audiences.
The student body attracted to the course was composed primarily of professionals who are working in the monitoring and evaluation sector, and who have higher education degrees. While this suggests that the course did not necessarily attract sectoral professionals into the monitoring and evaluation space, it does suggest firstly that the course was well tailored to the needs of the evaluation sector in the region. Many course participants indicated that they chose to participate in the course because of a lack of available training opportunities on theory of change in their geographic location. Forty-two per cent of students indicated that they did not have alternative training opportunities available to them on theory of change. Seventy-six per cent of the participants in the course have direct responsibilities either conducting evaluations, managing evaluations or managing teams with evaluation functions. This indicates that the course was well matched to the needs of evaluation practitioners.
Sixty-eight per cent of participants from the 2017 and 2018 cohorts rated their levels of understanding and application TOC (after completion of the course) as being good or excellent, while only 22% of participants rated their understanding of theory of change as good or excellent prior to the course. This supports the success of the course in achieving its learning outcomes in broadly improving knowledge in TOC. This suggests that the learning opportunity (building and improving skills) that is offered by this course is valued by participants. A limitation of this study is that pre- and post-tests on content knowledge were not administered, so this study can only speak to student perceptions of the extent to which skills were built. In addition to the MOOC making progress towards its learning outcome, the majority (84%) of participants from both years agreed that the TOC course provided them with practical skills that are useful to apply in their current and future work in M&E.
Ethical considerations
This article followed all ethical standards for research without direct contact with human or animal subjects.
Lessons for online instructional design in Africa
The case study approach was central to designing the course, and while it plays an important role in both the contact and online versions of the course, developing a case appropriate for an MOOC was critical to the course success. While the contact course is 3 days, with roughly two modules covered a day, the online version was adapted to be a 4-week course in the first run, and a 5-week course in the second run. The additional week was added despite the scope of content remaining the same, because of a need expressed by participants in the first course run to have time allocated with no additional course content to complete assignments and examinations.
The fictional case study was largely well received by most participants as a useful approach providing means to learning and application of learning. Eighty-two per cent of participants from both years considered the case study approach as being useful to their learning experience on the MOOC. Figure 2 illustrates the student feedback on the case study as a teaching tool from the combined post course survey for the 2017 and 2018 courses.
 |
FIGURE 2: Student agreement with the statement ‘The online case study was helpful to my learning’. |
|
However, integrating the case study design into the online instruction was a key lesson for the instructors. Consistency in introducing components of the case study, ensuring that each piece of the course material spoke to the case study, was a key challenge in course development. The investment was worthwhile, however, given that the case study material was then included in future contact courses. A key lesson is that in contact teaching, case studies are created to allow students to follow through an engaged logic process. Online, the purpose of a case study shifts, to facilitating collaboration across diverse contexts which can be difficult to achieve without content that can bring students into a common conversation. This again supports the idea that MOOCs can play an important role in strengthening collaboration and communities of practice in the evaluation space, though they may play a more limited role in building knowledge.
A number of comments pointed towards needing extra time to complete the course. Notably as most participants were employed, they took the course in between other commitments. Massive open online courses are thus beneficial in providing flexibility to a diverse range of participants although the completion time may still be limiting for other participants as this quote would indicate:
Allow a little more time to complete the course. Usually doing online courses is done in-between other work and commitments. (Anonymous course evaluation form, theory of change 2017)
Here, the resourcing is central, particularly given that 41% of course participants mentioned cost being a key consideration to furthering their education in the evaluation field. Most other activities that support collaboration and engagement for communities of practice are extremely costly, such as regional conferences. By nature, they exclude a subset of people, often those working in sectors or institutions that are not as well resourced. Perhaps, rather than comparing MOOC results to contact learning outcomes, it is also important to consider how they compare to other networking and knowledge sharing events, in terms of cost, engagement and results.
Finally, the content developed for an MOOC is available for future-blended learning and contact sessions. For this to work, instructors need to consider the potential and promise of an MOOC in material development, and also plan around its application in other learning contexts. As Wits University offers more blended learning options, there is potential to exchange and borrow appropriate material, which could free up capacity for contact teaching, and that, eventually, could address some of the institutional issues of capacity for contact teaching in the region.
Feedback around instructional design was a particularly important feature of the MOOC. While contact courses are often delivered with evaluation forms, the larger student numbers and multiple entry points for feedback (surveys, discussion forums, instructor contact and user-generated content) meant that there was rich student feedback to allow for improvements in the course, which can contribute to both online and contact learning. This points to a potential for MOOCs to play a role in diagnostic needs assessments in sectors where relatively little is known about the landscape of skills and training needs.
Conclusion: Lessons for regional capacity development
Massive Open Online Courses are not a panacea [universal remedy] for ECD in Africa. However, they do have an important contribution to make. Learning from the delivery of the Theory of Change for Development course, it is important to appropriately shape expectations around MOOC outcomes, and make sure that hidden benefits are made explicit. While MOOCs are typically expected to play a communications or advertising role, casting a wider net to attract a new audience to an area of study, this was not the case for the African evaluation audience. Rather, the student body was a well-targeted set of professional evaluators. While the course did not necessarily grow a cohort of evaluation professionals, it did mean that MOOCs may be able to serve skills building and networking roles in the African evaluation sector in a way that they cannot in other geographic or thematic areas.
As much as MOOCs may not be increasing access to education in the evaluation sector, they are making important inroads around networking, strengthening communities of practice and contextualising teaching material. These have been identified as critical issues in the ECD space, making it important to further explore the contributions MOOCs can make when delivered, in a planned way, with other ECD interventions. Through increasing the integration of multidisciplinary ECD approaches, capacity development providers will better respond to the needs of the sector (Preskill & Boyle 2008).
The experience of delivering the Theory of Change for Development course in 2017 and 2018 highlighted the need for high-quality, contextually appropriate, practitioner focused curricula in Africa. Furthermore, it highlighted the geographic variability of certificated courses, as well as the variability in ability of evaluation practitioners to pay for continuing professional development. Further course delivery and engagement with the emergent analytics will fill an important information gap in the ECD market in the region.
Acknowledgements
Competing interests
The authors have declared that no competing interests exist.
Authors’ contributions
C.B.M. was the academic lead on the Theory of Change for Development course, and both accredited the contact version of the course, and led the development of the online course materials. In addition, she took the leading role in conceptualising and drafting the article. L.K. supported the delivery of the Theory of Change for Development course and played a leading role in data analysis and discussion around the course content. H.E. was the internal beta tester of the course and led the thinking in the article around regionally effective capacity building approaches. D.W. led the technical development of the theory of change courses and brought her extensive online learning expertise to the article, contributing theory around instructional design, and the role of MOOCs in capacity development. She also developed a presentation on the course from which some of the data and discussion were drawn.
Funding
The views expressed in the submitted article are those of the authors alone, and do not reflect the views of Wits University or any other institution. While the authors are grateful to the university and edX for their support in delivering the course, any mistakes in the manuscript remain those of the authors.
Data availability statement
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Basheka, B.C. & Byamugisha, A., 2015, ‘The state of monitoring and evaluation (M&E) as a discipline in Africa’, African Journal of Public Affairs 8(3), 75–95. https://doi.org/10.4102/apsdpr.v3i2.80
Boga, S. & McGreal, R., 2014, Introducing MOOCs to Africa: New economy skills for Africa program.
Castillo, N.M., Lee, J., Zahra, F.T. & Wagner, D.A., 2015, ‘MOOCS for development: Trends, challenges, and opportunities’, International Technologies & International Development 11(2), 35.
Cohen, C., 2006, ‘Evaluation learning circles: A sole proprietor’s evaluation capacity-building strategy’, New Directions in Evaluation Fall, 85–93. https://doi.org/10.1002/ev.200
Czerniewicz, L. & Naidoo, U., 2013, MOOC-less in Africa.
Dennis, M., 2012, ‘The impact of MOOCs on higher education’, College and University 88(2), 24.
Dillman, L.M., 2012, ‘Evaluator skill acquisition: Linking educational experiences to competencies’, American Journal of Evaluation 34(2), 270–285. https://doi.org/10.1177/1098214012464512
Gaebel, M., 2013, MOOCs Massive open online courses, Occasional papers, European University Association, Brussels.
Garrido, M., Koepke, L., Anderson, S., Felipe Mena, A., Macapagal, M., & Dalvit, L., 2016, The advancing MOOCs for development initiative: An examination of MOOC usage for professional workforce development outcomes in Colombia, the Philippines, & South Africa, Technology & Social Change Group.
Liyanagunawardena, T.R., Williams, S. & Adams, A., 2013, May, ‘The impact and reach of MOOCs: A developing countries’ perspective’, E-Learning Papers, viewed 18 February 2019, from http://www.openeducationeuropa.eu/en/article/The-Impact-and-Reach-of-MOOCs%3A-A-Developing-Countries%E2%80%99-Perspective.
Morkel, C. & Ramasobana, M., 2017, ‘Measuring the effect of evaluation capacity building initiatives in Africa: A review’, African Evaluation Journal 5(1), 1–11.
Naccarella, L., Pirkis, J., Kohn, F., Morley, B., Burgess, P. & Blashki, G., 2007, ‘Building evaluation capacity: Definitional and practical implications from an Australian case study’, Evaluation and Program Planning 30(3), 231–236. https://doi.org/10.1016/j.evalprogplan.2007.05.001
Nesterko, S.O., Dotsenko, S., Han, Q., Seaton, D., Reich, J., Chuang, I. et al., 2013, December, ‘Evaluating the geographic data in MOOCs’, in Neural information processing systems.
Oyo, B. & Kalema, B.M., 2014, ‘Massive open online courses for Africa by Africa’, The International Review of Research in Open and Distributed Learning 15(6). https://doi.org/10.19173/irrodl.v15i6.1889
Porter, S. & Goldman, I., 2013, ‘A growing demand for monitoring and evaluation in Africa’, African Evaluation Journal 1(1), Art. #25, 1–9. https://doi.org/10.4102/aej.v1i1.25
Preskill, H. & Boyle, S., 2008, ‘A multidisciplinary model of evaluation capacity building’, American Journal of Evaluation 29(4), 443–459. https://doi.org/10.1177/1098214008324182
Richards, G. & Diallo, B., 2015, ‘OER and MOOCs in Africa’, in MOOCs and open education around the world, p. 255.
Rohs, M. & Ganz, M., 2015, ‘MOOCs and the claim of education for all: A disillusion by empirical data’, The International Review of Research in Open and Distributed Learning 16(6), 1–18. https://doi.org/10.19173/irrodl.v16i6.2033
Tarsilla, M., 2014, ‘Evaluation capacity development in Africa: Current landscape of internatonal partners’ initiatves, lessons learned and the way forward’, African Evaluation Journal 2(1), 13. https://doi.org/10.4102/aej.v2i1.89
|
Hide
Show all
Email this article (Login required)
