About the Author(s)


Nombeko P. Mbava Email symbol
Institute of Monitoring and Evaluation, Faculty of Commerce, University of Cape Town, Cape Town, South Africa

Peter Dahler-Larsen symbol
Department of Political Science, University of Copenhagen, Copenhagen, Denmark

Citation


Mbava, N.P. & Dahler-Larsen, P., 2019, ‘Evaluation in African contexts: The promises of participatory approaches in theory-based evaluations’, African Evaluation Journal 7(1), a383. https://doi.org/10.4102/aej.v7i1.383

Original Research

Evaluation in African contexts: The promises of participatory approaches in theory-based evaluations

Nombeko P. Mbava, Peter Dahler-Larsen

Received: 28 Feb. 2019; Accepted: 21 May 2019; Published: 11 Sept. 2019

Copyright: © 2019. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: A recent study of African evaluations identified deficiencies in present evaluation practices. Due to limited public sector expertise for the design of policy impact evaluations, expertise for such complex designs is largely external to the public sector. Consequently, recommendations made sometimes pay insufficient attention to variations in local contexts.

Objectives: The bold idea presented in this article is that theory-based evaluation (TBE) in its most recent participatory versions offers promising opportunities towards more flexible epistemology. When properly tweaked, tuned and adapted to local needs and demands in African contexts, better theory-based evaluations are possible.

Method: Three TBE-inspired criteria for better evaluations are suggested. The usefulness of including broad perspectives in theory-making was illustrated with a recent policy example, that is, the provision of tablets to school children in South Africa.

Results: A model of collaborative theory-making is presented. The pros and cons of the proposed hybrid model are discussed.

Conclusion: Recent trends in TBE point towards more participation of stakeholders in the theory-making process and towards more flexible epistemologies. The proposed innovation of TBE may have broader implications and serve as a promising inspiration for better evaluation practices in African contexts, given that existing research has demonstrated a need for such visions.

Keywords: theory-based evaluation; TBE; participatory evaluation; impact evaluation; realist evaluation; flexible epistemology; Africa public sector.

Introduction

A recent study of evaluations in South Africa has identified deficiencies in present evaluation practices (Mbava 2017; Mbava & Rabie 2018). The expertise for the design of impact evaluations specifically for complex interventions is lacking in the public sector (Basheka & Byamugisha 2015; Porter & Goldman 2013:8). Impact evaluations have therefore largely been led by multinational expert teams who had the skills and know-how to design highly complex evaluations (Mbava 2017:126). In addition, recommendations are often made in South African studies that do not pay sufficient attention to variations in local contexts where programmes are to be implemented (Mbava 2017:141). A simple adoption of evaluation practices from highly industrialised countries poses limitations and is ‘unsuitable in non-Western cultural contexts where totally different principles and practices prevail. A one-size-fits-all recipe for evaluation is therefore impractical’ (Cloete 2016:55). Others such as Ofir (2013:585) argue that methods adopted for evaluation and development have not fully appreciated the complexities of fragile contexts and developing societies and have tended to focus on simple interventions rather than on the reality of complex adaptive systems.

This raises the issue of evaluation capacity and the role of in-country evaluators. National evaluators too often play a limited role in the evaluation processes. For example, they act as liaisons with local stakeholders or as helping hands in data collection, but too often they do not have prominent roles in defining thought and intellectual leadership in the evaluation process (Porter & Goldman 2013:8; see also the debate in Ramasobana & Ngwabi 2018, including points made by Mouton and Wildschut). Chouinard and Milley (2018:77) further argue that inclusion of local evaluators could shift from their conceptualisation as ‘data sources’ towards recognising such participants as being an intrinsic part of the evaluation.

Finally, evaluation approaches are often not attuned to key traits in African cultures and philosophies (Chouinard & Hopson 2016; Cloete 2016; Ofir & Kumar 2013). In particular, the prominent role of collective deliberation and communal decision-making in African contexts has not been fully appreciated (Chilisa, Major & Khudu-Petersen 2017).

In recent years, there has been an increase in literature describing the role of contextual relevance in evaluations (Chouinard & Hopson 2016; Ofir & Kumar 2013; Pawson & Tilley 1997; SenGupta, Hopson & Thompson-Robinson 2004), the inclusion of stakeholders in participatory evaluation (Chouinard & Milley 2017; Cousins & Chouinard 2012; King, Cousins & Whitmore 2007), as well as the significant engagement of various voices in knowledge generation towards ontological and epistemic justice (Carden & Alkin 2012; Mamdani 2016; Ndlovu-Gatsheni 2018).

We articulate three critical aspects whose application in evaluation designs could potentially result in participatory and transformative evaluation practices. These can be described as follows:

  • Recommendations of interventions need to pay more attention to local variations in context.
  • A broader role of local evaluators and thinkers, not only in the role of practical fixers and data collectors, but also in the central epistemological processes which are guiding and defining the evaluation and its focus.
  • Evaluations that allocate more space to collective deliberation and communal decisions.

In our response to these challenges, we focus on theory-based evaluation. Our interest in this evaluation approach resonates with official policy in South Africa (RSA 2009: 21–22). The bold idea presented in this article is that theory-based evaluation (TBE) in its most recent participatory versions (Balle Hansen & Vedung 2010; Dahler-Larsen 2018; Funnell & Rogers 2011) offers promising opportunities towards more flexible epistemology. When properly tweaked, tuned and adapted to local needs and demands in African contexts, TBE could meet all three criteria stated above.

The purpose of this article is to unfold this hypothesis, considering both pros and cons. We recognise the contributions of responsive evaluation (Stake 2004), and culturally responsive evaluation (Hopson 2009), as well as contributions about indigenous philosophies in Africa (Ikuenobe 2017), but our claim is modest. In terms of evaluation approaches, we focus on TBE because it helps answer questions about impact, which continue to be of interest to donors, policymakers, and beneficiaries. TBE also aspires to base evaluative inferences on both critical thinking and empirical testing (Chen 2005; Rogers et al. 2000; Weiss 2000). At the same time, as these principles should be preserved, innovation is needed if TBE is to live up to the three criteria mentioned above. For example, in contrast to situations where local evaluation merely tests a theory that has been developed in the Global North (Carden & Alkin 2012:108–109), local participation should comprise genuinely epistemological questions. In the case of TBE that would mean involving local perspectives in crafting the very theories that are central in TBE. The local perspectives we refer to here are multidimensional. They include, but are not limited to citizenship, residence, language, culture, ethnicity, socio-economic positions and personal and professional experiences gained in situ.

Firstly, we present the findings of a recent review of evaluations in South Africa. Secondly, we explicate and justify three criteria for evaluations aimed at overcoming existing deficiencies. Thirdly, we describe how recent trends in TBE resonate with these criteria. Fourthly, we describe how local partners can be involved in theory-making, which constitutes a key epistemological aspect of TBE. We provide a short illustration of how such theory-making might unfold, using South African President Ramaphosa’s recently introduced policy intervention, tablets for school children, as an example. We end with a discussion of the wider applicability of the described approach and a conclusion.

A study of deficiencies of present evaluation practices regarding what works in which contexts

A recent study was conducted in South Africa which aimed to provide better understanding of the methodologies and approaches used in past programme impact evaluations in the South African public sector and to reflect on the usefulness of evaluation findings to policy decision-makers (Mbava 2017; Mbava & Rabie 2018). The study design entailed extensive literature review, assessment of four impact evaluation case studies through the lens of a prominent version of TBE and key informant interviews with seven policy decision-makers, so as to determine the usefulness of the evaluations. The aim was to ascertain the most important limitations with existing policy impact evaluations and the suitability of adopted evaluation approaches. It was established that there are important gaps and limitations with existing policy impact evaluations.

Evaluation methods and designs are not always appropriate to inform the needs of policymakers. There are limited insights on programme pathways to change as a base of establishing how the programme works, in what context and under what conditions. There is also perceived limited utilisation of evaluation evidence in policymaking, as evaluation evidence is not effectively infused in the policymaking cycle (Mbava & Rabie 2018:89).

A key finding showed concern over limited understanding of the broader programme context because contextual conditions under which programmes are implemented are critical. Pawson (2006:31–32) emphasised that the broader programme context includes key actors and agents in the programme implementation chain, who can enable or impede the implementation of the social programme based on their enthusiasm and will. Such stakeholders will invariably include intended programme beneficiaries, the programme staff, policymakers and other agents in the implementation chain. This broader context, which affects the efficacy and efficiency of a programme, was not found to be adequately interrogated to gain rich insights into programme context.

This is supported by the views of some of the seven key informants who highlighted important contextual aspects that are expected in order to elicit the most meaningful evaluations. Virtually all interviewed key informants (86%) preferred aspects of evaluation that specified ‘Who primarily benefited from the policy?’ and ‘If the intervention was successful, when and where can it be replicated?’ Insights on these aspects were seen as critical in order to ensure equity in programme design and implementation because ‘who exactly benefited and how equity is dispersed towards impact is critical in order to know whether the targeted beneficiaries were indeed the beneficiaries’ (Sector Expert Human Settlements Evaluations 2016 in Mbava 2017). Therefore, policy decision-makers want to know ‘For whom did it benefit and how did it work. To know whether policy should be targeted or done at full scale’ (Sector Expert Education Evaluations 2016 in Mbava 2017).

To answer these questions more precisely, this study explores whether participatory approaches to TBE are a promising possibility for evaluation in African contexts.

Three criteria for better evaluation practices

This section explicates and justifies three TBE-inspired criteria for better evaluation.

Firstly, recommendations of interventions must pay more attention to local variations in context (Pawson & Tilley 1997). At best, these variations are empirically described in previous evaluations and existing research. In the absence of such evidence, it is still better to base recommendations on well-reasoned theoretical conjectures about contextual factors which may be critical to the impact of interventions rather than merely assuming that the effects are the same everywhere.

Secondly, there must be more involvement of local evaluators, not only in the role of practical fixers and data collectors, but also in the central epistemological processes which are guiding and defining the evaluation and its focus. In TBE, the key epistemological process is the making of theory. Our justification for this criterion deserves to be made explicitly. As Schwandt (2002) and others (Julnes 2012) suggest, evaluation is a value-laden activity. Evaluators engage with values not only in relation to evaluation criteria, but also ‘by providing stakeholders with the opportunity to actively engage in evaluation’ (Schwandt 2015: 65).

It therefore constitutes a genuine problem if there is unequal access of voices to the key epistemic processes in evaluation. To make that argument is not the same as recommending one epistemology be replaced by another one, or for that matter that a modern set of ideas should be replaced by a traditional set of ideas or anything of that sort.1 The point is merely that unequal access to influence theory-making in TBE can be regarded as a problem of fairness and justice. In addition, we will seek to show that participation of local perspectives in theory-making may also help make these theories more context-sensitive and better adapted to local realities. Finally, participation in theory-making might well be carried out in a way that reflects African traditions for collective deliberation and communal decisions. Again, our claim is modest. We are not advocating a romantic picture of participatory evaluation. We are talking specifically about local involvement in TBE, where theory-making is subsequently exposed to critical empirical testing. Experiences with this process indicate that this process is not always harmonious or uncontroversial (Dahler-Larsen 2018).

In this modest spirit, a hybrid version of TBE developed especially for African contexts is suggested. In the next section, it will be shown that there are recent turns and developments in TBE that create more open doors for participation and epistemological flexibility in a way that makes such hybridisation credible and possible.

Theory-based evaluation: Trends towards a more flexible epistemology

A key ingredient in TBE is that programme theories are used as key tools in evaluation. A programme theory is a set of ideas or hypotheses that explain how and why an intervention will work, perhaps with a specification of why it works for a particular group of people in a particular context. The evaluation focuses on checking whether these hypotheses can be confirmed in the actual situation at hand (Coryn et al. 2011). This is usually based on empirical data, but critical thinking also includes whether a programme theory is logically consistent, credible and congruous with what is otherwise known about the intervention and the contexts in which it is supposed to operate. The use of programme theory is now commonplace and it has been mainstreamed and phenomenally applied in programme management processes in various programme areas. Programme theory is a core requirement, as evaluation commissioners require project proposals to initially specify the theory of change as a guide for assisting in programme design and evaluation (Rogers 2007:63–64). The theory of change is validated and tested by verifying the extent to which the theory assumptions are true against what is actually observed. TBE therefore provides for rigorous evaluation through systematic interrogation of programme theory as a basis for guiding the evaluation.

Wildschut (2014), Heradien (2013:79), Mbava (2017) and Abrahams (2003:268) have found TBE to be valuable and promising in African contexts. Policy decision-makers found that a variant of the TBE approach held much promise in a number of capacity development projects across Africa (Punton, Vogel & Lloyd 2016). It was suggested that TBE approaches can support the implementation of capacity development programmes on a broader scale.

South African government have recommended the strengthening of TBE approaches and articulated that the:

[A]nalysis of causal effects is currently weak, and the international good practice of theory-based evaluation needs to be strengthened. This would require, in the policy development and planning stages, a clear conceptual understanding of how, why and when the policy, programme or project will effect change, and how these changes may be measured. (RSA 2009:21–22)

We now turn to recent developments in TBE, which allow it to live up to the earlier-mentioned three criteria for better evaluation. Within the large tent of TBE, Realist Evaluation is a contribution to the evaluation body of knowledge built on the foundations of philosophical realism (Pawson & Tilley 1997). Pawson (2013:ix) points out that Realist Evaluation’s standpoint is that of pursuing the high scientific objectives of objectivity and generative causal explanation to inform real world policy and practice.

Carden and Alkin’s (2012:105) conceptual framework of the ‘evaluation theory tree’ places this TBE approach on the ‘methods branch’. The core of the Realist Evaluation method is the articulation of programme theory that explains how and why programmes work and for whom they effectively work (Pawson 2013; Pawson & Manzano-Santaella 2012; Pawson & Tilley 1997). This is explained by the interrogation of the programme’s broader context, understanding what causes the observed change and confirming that the observed outcomes validate the underlying programme theory. The success of an intervention is context-dependent and ‘things work and effects occur only if the circumstances and conditions are right, and they may have to be very particular’ (Pawson & Tilley 1995:23). This idea is captured through the conceptual framework of context-mechanism-outcome configuration or CMO (Pawson & Tilley 1997).

In contrast, constructivist programme theory breaks with the assumption so typical of the realist philosophy that causal mechanisms are ontologically given and inherent to the real world. On the contrary, constructivists argue that no human being has ever seen a mechanism with his or her bare eyes. Constructivists understand causal theories as ways in which human beings make sense of a complex and messy world. For a constructivist, there is no direct access to the real world except through interpretation. The fact that causal theories are human constructs helps explain why there are so many diverse and incomplete causal theories connected to a given policy intervention. In an interpretive perspective, there is no ultimate principle inherent in reality which guarantees that human beings can find one and only one correct set of causal theories. As a consequence, this variation of TBE departs from a view that associates truth with social and political authority and instead creates a space for broader participation in theory-making (Vattimo 2004).

Furthermore, constructivists insist that the contexts that play such a critical role in realist philosophy are in fact better understood not as physical realities but rather as social constructions amenable to change through social and political processes (Dahler-Larsen 2001, 2018). For example, some theories may not work in a totalitarian regime but work well in a democratic one. In a similar vein, some public policies may work poorly when implemented in a corrupt regime, whereas they may work much better if corruption is removed. In other words, although realists are correct in saying that some interventions work in some contexts but not in others, constructivists argue that policies can be designed to change these contextual circumstances so that intended policies may work better. Social contexts also change over time as a result of broader societal changes that are not a result of deliberate policy (Dahler-Larsen 2001). This view also expands the space for flexible epistemology in TBE because both the contexts in which policies operate and the results of the same policies are results of a collective action rather than those of a given physical reality.

‘Flexible epistemology’ does not imply relativism. Just because knowledge production is social, it does not mean it is without rules and without critical testing (Latour 2004; Longino 2002). A constructivist perspective on TBE still maintains that critical thinking and confrontation with empirical findings are key ingredients in evaluation. In practice, evaluators often do not explicate one particular underlying philosophy (Shadish, Cook & Leviton 1991: 43). Furthermore, their practical strategies are not determined by philosophical positions, as evaluators usually take situational factors into account. However, our argument is that changes in the philosophical ideas in TBE make it possible to bring TBE in position to be used pragmatically towards meeting the three criteria suggested earlier.

Flexible and pragmatic approaches in TBE have been seen in recent years, among other things in user-friendly terminology and in graphical forms of representation of programme theories that are intuitively appealing to people who are not usually comfortable with formal representation of theory (Funnell & Rogers 2011). This testifies to how TBE can be used flexibly without sacrificing the constructed theory, but to rather enhance the key ideas.

Theory-based evaluation: Trends towards collaborative theory-construction

According to the most classical and conventional model, the responsibility for articulating the programme theory rests with the evaluator. In an expansion of this model, the evaluator consults with a group of stakeholders before articulating the programme theory. The purpose of this process is twofold, partly to inform the evaluator about relevant revisions of the programme theory, and partly to ensure that the stakeholders find the programme theory relevant and hence are more likely to accept the evaluation findings based on that theory. A key principle that holds this process together is thus the assumption of consensus. Although several contributions, additions or revisions may be discussed along the way, these are not reported in the evaluation report and not seen as relevant for the final evaluation results. The programme theory is quite naturally referred to as ‘the programme theory’ in the singular (Balle Hansen & Vedung 2010).

In recent years, the assumption of one consensual programme theory has been problematised (Dahler-Larsen 2018). Balle Hansen and Vedung (2010) suggest that when stakeholders are involved in theory-making, it may be more fruitful to articulate the theories of the different stakeholders clearly and keep them separate. By making the differences among theories visible, each group of stakeholders may learn to better respect each other’s views. Furthermore, the different programme theories may be conducive to a democratic process of policymaking.

Dahler-Larsen (2018) goes a step further. He argues that very often, the same phenomenon may play two different roles, for example a positive one in one programme theory and a negative one in another. If that is the case, Dahler-Larsen recommends a combination of two conflicting programme theories in the same graphic representation so that the unfortunate interaction effects following from this double role become visible for all participants. Making the double role of some phenomena visible can facilitate a call to action among the stakeholders.

In an illustrative example, technical staff puts up a large ashtray outside a building. In one potential programme theory, the function of the ashtray is positive because it helps reduce the occurrence of cigarette stubs on the ground. In another potential theory, however, the function is negative because it undermines the prohibition of smoking, which should formally be in place. Because the ashtray is there, people think it is a good place to smoke, in spite of existing rules.

The dilemma in this double function is most clearly seen by juxtaposing the two programme theories. It is bringing the theories together, not keeping them separate, which makes the problem visible and collective action pressing. In the case at hand, it was easy to choose one solution consistent with legislation, which was simply to move the ashtray into a place where smoking was allowed (and clearly marked as such). In other situations, the double function of a phenomenon in several programme theories may be much more difficult to sort out. But at least the dilemma can be made visible. Examples of such double functions in real life might be dependency on drugs which may be negative for consumers, but profitable for the medical industry. Minimum wages may be good for people with jobs, but they may also, according to some, keep people with few qualifications out of the labour market (Dahler-Larsen 2018).

Such problems may be impossible to solve. Only after a careful process of participatory theory-making can it be determined whether reasonable solutions to such dilemmas are feasible. It is hoped that the very participation in theory-making is a source of insight in itself, consistent with the idea that quite a bit of the use of evaluations already begins with ‘process use’, that is, learning evaluative thinking from participation in the process (Forss, Rebien & Carlsson 2002). Process use might include the collaborative reactions to the differences between different programme theories held by various stakeholders.

An illustrative example of theory-making: Planned roll-out of tablets across a public schooling system

In this section, we illustrate how the potential development of theory in TBE, with the involvement of multiple stakeholders, could take place in relation to a real-life case: Tablets for school children.

The White Paper on e-Education (RSA 2004) outlines the overarching policy framework for South Africa to transform learning and teaching as part of an inclusive and innovative digital and knowledge-based society. It focuses on ensuring that every school has access to a wide choice of diverse, high-quality information communication technology (ICT) services and infrastructure, which will benefit all learners and local communities and further calls for public-private partnerships and collaboration in the provision of ICTs in education. In line with this policy position, during his 2019 State of the Nation Address (SONA), South African President Ramaphosa introduced a plan about giving tablets to school children (RSA 2019). The alleged purpose is to enhance educational goals, including the acceleration of digital literacy. On that basis, a simple official programme theory is not difficult to delineate. The key proposition is simple: Tablets enhance learning.

The increasing attractiveness in using tablets as a teaching aid in primary and secondary education is well documented (Habler, Major & Hennessey 2015; Kalolo 2019; McFarlane 2015). It has been argued that the digital era promises to transform teaching and learning in ways previously unimagined and tablets can contribute to educational improvements (Erstad, Eickelmann & Eichhorn 2015; Kalolo 2019; McFarlane 2015; Van Deursen, Ben Allouch & Ruijter 2016). Various research reviews (Habler et al. 2015; Herodotou 2018; Kalolo 2019) indicate that integrating tablets in learning tasks can improve and support learning.

Let us now imagine a participatory process where various local stakeholders engage in collaborative theory-making about this initiative. Broadening the process will take theory-making beyond the thin and decontextualised story (Fischer 2003) expressed in a programme theory such as ‘tablets enhance learning’. At the same time, the sources of programme theory will not rest solely with research, but include experiences and perspectives from a variety of stakeholders such as state planners and funders, educators, curriculum and subject advisors, book content providers and publishers, various ICT service providers who will ensure infrastructure and connectivity, device procurement specialists, school governing bodies, learners and parents. These are presumably some of the key actors and agents in the school tablet roll-out implementation chain. Unconventional types of stakeholders such as hackers may also be able to contribute with knowledge about security risks and how to avoid them.

The task for all is to contribute to a plausible school tablet roll-out programme theory that pinpoints key implementation variables and effects of tablets, including side effects.

Educators who embrace technology as an enabler in teaching and learning think tablets are useful teaching aids. Schools’ acquisition of physical stock of books is no longer a necessity. They no longer go missing. E-texts are in the latest edition, cost-effective, occupy less space and are easier to carry around. Digital tools are conducive to a dynamic learning environment (Kalolo 2019:353). Professional development and provision of educator learning materials in this regard are well received.

Although this theory is plausible, others have argued that the assumption that teachers are ready and eager to integrate tablets from the outset is not realistic in the absence of adequate initial professional development and change management, technical support and clear policy guidelines (Cantrell & Visser 2011; Habler et al. 2015). Tablets and other devices have been viewed with hostility by schools and regarded as disruptive in bringing the outside world into the school (McFarlane 2015). Cantrell and Visser (2011:282) found that lack of experience with computers, computer anxiety, suspicion towards technology and change were some of the factors inhibiting a positive attitude towards technology adoption. Traditionally minded teachers will be of the firm view that learners should read and learn from hardcopy textbooks, write on paper using pen and receive instruction from a chalk-holding teacher lecturing in front of a blackboard. This is the tried and tested method of learning and teaching, as they themselves were schooled similarly. The traditional classroom model has been in use in most developing countries, where knowledge gained by the teacher is imparted top-down to learners, and where distribution of hardcopy learning material, note-taking and oral instruction are prioritised (Kalolo 2019; Miah & Omar 2012).

There is tension between old and new models of learning with teachers as agents of change, confronted with demands to adapt and integrate technology or risk being discouraged and ambivalent (Erstad et al. 2015). In other words, teacher attitudes and preconceptions about ways of learning may be an important contextual factor that helps explain various outcomes of tablets. The effectiveness of tablets as an educational device hinges on the ability of teachers to integrate the use of tablets into meaningful pedagogical processes. Teacher education must be part of the intervention. This exemplifies how a contextual factor can be seen as either a given fact or a variable that can be influenced by policy.

Teachers and others with insights into the daily lives of school children might suggest that the motivation of students to use the tablet for educational purposes rather than, for example, for entertainment purposes is critical to the success of the programme. In fact, on a bad day, when tablet-based teaching may not be a focus in the teacher’s pedagogical plan for the day, tablets may distract learners rather than support their learning process. A double role for tablets may be suggested here, unless school rules and pedagogical interventions take steps to prevent it. An important factor influencing the intended use of tablets is thus the quality of the digital educational content to be used on the tablet. This educational content must, at best, be easy and fun to use, and at the same time it must enhance the achievement of educational goals. Otherwise, education might compete with entertainment in the use of the tablet.

Finally, how can it be ascertained that tablets are operational and charged, also for children who live in areas or households where electricity cannot be taken for granted? Will the tablet be handed out for free? If yes, who is accountable if it is lost, stolen or cannot be found? These are critical issues that policy decision-makers are confronted with. According to Kalolo (2019:347), developing countries face a particular set of problems in relation to digital technologies. The tablet plays a double role, positive in some theories and negative in others. Although the tablet may genuinely be of benefit for the child, a tablet is also an object with a market value. Carrying a tablet to school and back may therefore put children at risk, exposing them to potential theft, robbery, or threats, especially in troubled areas. Unless this problem is addressed, the tablet may in some situations be a liability rather than an asset for the learner. Would a solution be to lock away the tablets at school at the end of the school day? If so, what would be the impact on learning and when, where and how much the learners use tablets? Without someone aboard the theory-making process who has attention to socio-economic issues and variations in risk across neighbourhoods, these aspects related to the use of tablets in real life may remain blind spots in the wider implementation process.

Admittedly, what we have sketched here is a conceptual, albeit realistic example. Nevertheless, it illustrates that a composite theory that includes both contextual variables and interaction between different theory logics is bound to be complex. It is quite likely that even a good process of theory-making will not be able to account for all factors that are critical for the effectiveness of the tablets programme from the outset. However, a broad inclusion of perspectives is likely to lead to a richer and more context-dependent theory-making compared with the optimist and simple version where tablets simply enhance learning. Thin political stories become richer stories only through contextualisation (Fischer 2003).

The dynamic nature of theory-making would suggest that a pilot programme should be tested out in a few local contexts before tablets are rolled out in the entire nation. In other words, we advocate something more dynamic than a classic process of the following form: Policy decision – implementation – evaluation as indicated in Figure 1. Instead, we propose a circular process demonstrated in Figure 2, more like the following: Policy decision – collaborative theory-making –pilot projects – evaluation with improved collaborative theory-making – revision of policy design – implementation in broader scale – evaluation with improved collaborative theory-making, and then again policy decision.

FIGURE 1: The classic policy implementation process.

FIGURE 2: Theory-based infused policymaking.

Quite a few benefits of the TBE process may be harvested in early stages of the policy process by thinking critically about the intervention and adapting it to foreseeable contextual circumstances as well as by summarising learning points from early pilot projects. Further on, collaborative theory-making is used iteratively through the whole policy process representing a stock of knowledge gained so far as well as a resource for next immediate steps.

The example shows a way in which TBE can be sensitive to variations in context. It allows for extensive participation of local stakeholders, not only in superficial roles, but in the very epistemological core of TBE, that is, theory-making itself. The success criterion for theory-making is not that it provides ultimate evidence of the effect of the intervention, but that it helps stakeholders determine the extent to which it works for particular recipients in particular contexts and what the next reasonable steps are regarding improvements of the intervention.

Discussion

Developing a hybrid version of TBE that fits well into African contexts is not easy. Much work must be carried out by intermediaries, brokers, bridge-builders, evaluators and thinkers. There is a need to develop practical languages and presentation styles, which may help make programme theories and theory-making better understood and articulated from stakeholders’ perspectives.

It is commonly acknowledged that TBE is time-consuming. ‘If TBE is carried out in full detail, it is apt to be an expensive and time-consuming enterprise’ (Birckmayer & Weiss 2000:429). Up to one-third of the time designated to an evaluation process can be dedicated to theory-making. Not all stakeholders in all situations will find it possible and meaningful to set aside the necessary time for such a process. On the positive side, however, learning effects may occur already as a function of the theory-making process. Stakeholders may also learn to better appreciate each other’s perspectives (Balle Hansen & Vedung 2010). Not all use of TBE has to await evaluation findings.

Findings from TBE evaluations might suffer from lack of credibility unless they satisfy high methodological expectations that might exist among some stakeholders. Theory-driven evaluation could potentially be resource intensive as it requires rigorous testing of programme theory so as to ascertain plausibility (Marchal et al. 2012).

For those whose expectation is that without control groups, no causal inferences should be made, TBE may produce results that are perceived to be weak, even if carried with the utmost methodological sophistication. Designs with a control group are not always feasible. TBE sometimes faces a difficult trade-off between methodological rigour on the one hand and concerns for practical feasibility and immediate usefulness on the other. If TBE appears in a process that resembles Figure 1, demands on methodological rigour may have high priority. On the other hand, if it plays a more dynamic, formative role as in Figure 2, immediate usefulness may be given a relatively higher priority.

Another complication is the diversity of interests and political pressures from various stakeholders, which may be difficult to reconcile in a theory-making process. It might be difficult for TBE to deliver a constellation of programme theories, which satisfy both external and internal stakeholders, international sponsors as well as local actors. It is only by trying, under specific circumstances, that the true value of TBE can be known.

Conclusion

Recent trends in TBE point towards more participation of stakeholders in the theory-making process and towards more flexible epistemologies. This development allows for more attention to contextual variations in the effectiveness of interventions and more participation of local evaluators and thinkers in the epistemological domains of the evaluation process. This hybrid approach includes elements of participation that resonate with cultural and philosophical traits undergirding the role of collective knowledge construction and decision-making in African contexts, at the same time as TBE remains committed to continuous confrontation of the constructed theories with empirical findings. The proposed innovation of TBE may have broader implications and serve as a promising inspiration for better evaluation practices in African contexts, given that existing research has demonstrated a need for such visions.

Acknowledgements

Competing interests

We declare that we have no financial or personal relationships that may have inappropriately influenced us in writing this article.

Authors’ contributions

N.P.M. and P.D-L. jointly made a substantial contribution to the conception and design of this work. N.P.M. drafted the manuscript, P.D-L. further drafted and critically revised it for important intellectual content. Both N.P.M. and P.D-L. approved the final version to be published.

Ethical considerations

This article followed all ethical standards for research without direct contact with human or animal subjects.

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Data availability statement

Data sharing is not applicable to this article as no new data were created or analysed in this study.

Disclaimer

The views expressed in this article are those of the authors and not an official position of the respective affiliated institutions

References

Abrahams, M.A., 2003, ‘Theory-based evaluation of community development: A South African case study’, Unpublished PhD dissertation, Stellenbosch University, Bellville.

Balle Hansen, M. & Vedung, E., 2010, ‘Theory-based stakeholder evaluation’, American Journal of Evaluation 31(3), 295–313. https://doi.org/10.1177/1098214010366174

Basheka, B.C. & Byamugisha, A., 2015, ‘The state of Monitoring and Evaluation (M & E) as a discipline in Africa’, African Journal of Public Affairs 8(3), 75–95. https://doi.org/10.4102/apsdpr.v3i2.80

Birckmayer, J.D. & Weiss, C.H., 2000, ‘Theory-based evaluation in practice: What do we learn?’, Evaluation Review 24(4), 407–431. https://doi.org/10.1177/0193841X0002400404

Cantrell, S. & Visser, L., 2011, ‘Factors influencing the integration of technology to facilitate transfer of learning processes in South African, Western Cape Province Schools’, The Quarterly Review of Distance Education 12(4), 275–285.

Carden, F. & Alkin, M.C., 2012, ‘Evaluation roots: An international perspective’, Journal of Multi-Disciplinary Evaluation 8(17), 102–118.

Chen, H.T., 2005, ‘Theory-driven evaluation’, in S. Mathison (ed.), Encyclopaedia of evaluation, pp. 416–421, Sage Publications, Thousand Oaks, CA.

Chilisa, B., Major, T.E. & Khudu-Petersen, K., 2017, ‘Community engagement with a postcolonial, African-based relational paradigm’, Qualitative Research 17(3), 326–339. https://doi.org/10.1177/1468794117696176

Chouinard, J.A. & Hopson, R., 2016, ‘Critical exploration of culture in international development evaluation’, Canadian Journal of Program Evaluation 30(3), 248–276. https://doi.org/10.3138/cjpe.30.3.02

Chouinard, J.A. & Milley, P., 2018, ‘Uncovering the mysteries of inclusion: Empirical and methodological possibilities in participatory evaluation in an international context’, Evaluation and Program Planning 67, 70–78.

Cloete, F., 2016, ‘Developing an Africa-rooted programme evaluation approach’, African Journal of Public Affairs 9(4), 55–70.

Cousins, J.B. & Chouinard, J.A., 2012, Participatory evaluation up close: An integration of research-based knowledge, Information Age Publishing, Charlotte, NC.

Coryn, C.L.S., Noakes, L.A., Westine, C.D. & Schroter, D.C., 2011, ‘A systematic review of theory-driven evaluation practice from1990to2009’, American Journal of Evaluation 32(2), 199–226. https://doi.org/10.1177/1098214010389321

Dahler-Larsen, P., 2001, ‘From programme theory to constructivism: On tragic, magic and competing programmes’, Evaluation 7(3), 331–349. https://doi.org/10.1177/13563890122209711

Dahler-Larsen, P., 2018, ‘Theory-based evaluation meets ambiguity: The role of Janus variables’, American Journal of Evaluation 39(1), 6–23. https://doi.org/10.1177/1098214017716325

Erstad, O., Eickelmanm, B. & Eichhorn, K., 2015, ‘Preparing teachers for schooling in the digital age: A meta-perspective on existing strategies and future challenges’, Education and Information Technologies 20(4), 641–654. https://doi.org/10.1007/s10639-015-9431-3

Fischer, F., 2003, Reframing public policy, discursive politics and deliberative practices, Oxford University Press, Oxford.

Forss, K., Rebien, C. & Carlsson, J., 2002, ‘Process use of evaluations: Types of use that precede lessons learned and feedback’, Evaluation 8(1), 29–45. https://doi.org/10.1177/1358902002008001515

Funnell, S.C. & Rogers, P.J., 2011, Purposeful program theory: Effective use of theories of change and logic models, Wiley, San Francisco, CA.

Habler, B., Major, L. & Hennessy, S., 2016, ‘Tablet use in schools: A critical review of the evidence for learning outcomes’, Journal of Computer Assisted Learning 32(2), 139–156. https://doi.org/10.1111/jcal.12123

Hopson, R.K., 2009, ‘Reclaiming knowledge at the margins: Culturally responsive evaluation in the current evaluation moment’, in K.E. Ryan & J.B. Cousins (eds.), The SAGE International Handbook of Educational Evaluation, pp. 429–446, Sage Publications, London.

Heradien, E.J., 2013, ‘An evaluation of the theory behind the South African Expanded Public Works Programme’, Unpublished MA dissertation, Stellenbosch University, Bellville.

Herodotou, C., 2018, ‘Young children and tablets: A systematic review of effects on learning and development’, Journal of Computer Assisted Learning 34(2), 1–9. https://doi.org/10.1111/jcal.12220

Ikuenobe, P., 2017, ‘Tradition and a foundation for African renaissance’, in V. Msila (ed.), Decolonizing knowledge for Africa´s renewal. Examining African perspectives and philosophies, pp. 2–20, KR Publishing, Randburg.

Julnes, G. (ed.), 2012, Promoting value in the public Interest: Informing policies for judging value in evaluation, New Directions for Evaluation, 133, John Wiley & Sons, New Jersey.

Longino, H.E., 2002, The fate of knowledge, Princeton University Press, Princeton, NJ.

Latour, B., 2004, ‘Why has critique run Out of steam? From matters of fact to matters of concern’, Critical Inquiry 30(2), 225–248. https://doi.org/10.1086/421123

Kalolo, J.F., 2019, ‘Digital revolution and its impact on education systems in developing countries’, Education and Information Technologies 24(1), 345–358. https://doi.org/10.1007/s10639-018-9778-3

King, J.A., Cousins, J.B. & Whitmore, E., 2007, ‘Making sense of participatory evaluation: Framing participatory evaluation’, New Directions for Evaluation 114(Summer), 83–105. https://doi.org/10.1002/ev.226

Mamdani, M., 2016, ‘Between the public intellectual and the scholar: Decolonization and some post-independence initiatives in African higher education’, Inter-Asia Cultural Studies 17(1), 68–83. https://doi.org/10.1080/14649373.2016.1140260

Marchal, B., Van Belle, S., Van Olmen, J., Hoerée, T. & Kegels, G., 2012, ‘Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research’, Evaluation 18(2), 192–212. https://doi.org/10.1177/1356389012442444

Mbava, P.N., 2017, ‘The potential value of the realist evaluation method in programme impact evaluations in South Africa’, Unpublished PhD dissertation, Stellenbosch University, Bellville.

Mbava, P.N. & Rabie, B., 2018, ‘Strengthening impact evaluations in the South African National evaluation system’, Administratio Publica 92(26), 75–98.

McFarlane, A., 2015, Authentic learning for the digital generation. Realising the potential of technology in the classroom, Routledge, London, UK.

Miah, M. & Omar, A., 2012, ‘Technology advancement in developing countries during digital age’, International Journal of Science and Applied Information Technology 1(1), 30–38.

Ndlovu-Gatsheni, S.J., 2018, ‘The dynamics of epistemological decolonization in the 21st century: Towards epistemic freedom’, Strategic Review for Southern Africa 40(1), 16–45. https://doi.org/10.4324/9780429492204

Ofir, Z., 2013, ‘Strengthening evaluation for development’, American Journal of Evaluation 34(4), 582–586. https://doi.org/10.1177/1098214013497531

Ofir, Z. & Kumar, S., 2013, ‘Evaluation in developing countries: What makes it different?’, in S.I. Donaldson, T. Azzam & R.F. Conner (eds.), Emerging practices in international development evaluation, pp. 11–24, Information Age, Charlotte, NC.

Pawson, R., 2006, Evidence-based policy: A realist perspective, Sage Publications, London.

Pawson, R., 2013, The science of evaluation: A realist manifesto, Sage Publications, London.

Pawson, R. & Manzano-Santaella, A., 2012, ‘A realist diagnostic workshop’, Evaluation 18(2), 176–191. https://doi.org/10.1177/1356389012440912

Pawson, R. & Tilley, N., 1995, ‘Whither (European) evaluation methodology?’, Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 8(3), 20–33. https://doi.org/10.1007/BF02904928

Pawson, R. & Tilley, N., 1997, Realistic evaluation, Sage Publications, London.

Porter, S. & Goldman, I., 2013, ‘A growing demand for monitoring and evaluation in Africa’, African Evaluation Journal 1(1), 10–18. https://doi.org/10.4102/aej.v1i1.25

Punton, M., Vogel, I. & Lloyd, R., 2016, Reflections from a realist evaluation in progress: Scaling ladders and stitching theory, IDS.CDI Practise Paper 18, pp. 1–11, IDS, Brighton.

Ramasobana, M. & Ngwabi, N., 2018, From infancy to maturity: Constraints to the “Made in Africa Evaluation” (MAE) concept (Part 2), viewed 19 February 2019, from http://www.africaevidencenetwork.org/from-infancy-to-maturity-constraints-to-the-made-in-africa-evaluation-mae-concept-part-2/.

Republic of South Africa (RSA), 2004, Draft White Paper on e-Education. Transforming learning and teaching through information and communication technologies, The Presidency, Department of Education, Pretoria.

Republic of South Africa (RSA), 2009, Improving government performance: Our approach, The Presidency, Department of Performance Monitoring and Evaluation (DPME), Pretoria.

Republic of South Africa (RSA), 2019, President Cyril Ramaphosa: 2019 State of the Nation Address, The Presidency, Pretoria, viewed 8 February 2019, from https://www.gov.za/speeches/president-cyril-ramaphosa-2019-state-nation-address-7-feb-2019-0000.

Rogers, P., 2007, ‘Theory-based evaluation: Reflections ten years on’, New Directions for Evaluation 114(Summer), 63–67. https://doi.org/10.1002/ev.225

Rogers, P., Petrosino, A., Huebner, T.A. & Hacsi, T.A., 2000, ‘Program theory evaluation: Practice, promise, and problems’, New Directions for Evaluations 87(Fall), 5–13. https://doi.org/10.1002/ev.1177

SenGupta, S., Hopson, R. & Thompson-Robinson, M., 2004, ‘Cultural competence in evaluation; an overview’, New Directions for |Evaluation 102(Summer), 5–19. https://doi.org/10.1002/ev.112

Schwandt, T.A., 2002, Evaluation practice reconsidered, Peter Lang, New York.

Schwandt, T.A., 2015, Evaluation foundations revisited: Cultivating a life of the mind for practice, Stanford University Press, Stanford, CA.

Shadish, W.R., Cook, T.D. & Leviton, L.C., 1991, Foundations of program evaluation: Theories of practice, SAGE, Newbury Park, CA.

Stake, R.E., 2004, ‘Responsive evaluation’, in R.E. Stake (ed.), Standard-based and responsive evaluation, pp. 86–104, Sage Publications, Thousand Oaks, CA.

Van Deursen, A.J.A.M., Ben Allouch, S. & Ruijter, L.P., 2016, ‘Tablet use in primary education: Adoption hurdles and attitude determinants’, Education and Information Technologies 21(5), 971–990. https://doi.org/10.1007/s10639-014-9363-3

Vattimo, G., 2004, Nihilism and emancipation: Ethics, politics, & law, Columbia University Press, New York.

Weiss, C.H., 2000, ‘Which links in which theories shall we evaluate?’, New Directions for Evaluation 87(Fall), 35–45. https://doi.org/10.1002/ev.1180

Wildschut, L.P., 2014, ‘Theory-based evaluation, logic modelling and the experience of SA non-governmental organisations’, Unpublished PhD dissertation, Stellenbosch University, Bellville.

Footnotes

1. No such wholesale type of argument is helpful. Instead, consistent with Ikuenobe (2017), we see both modernity and cultural tradition as multidimensional. This calls for hybridisation of ideas woven together in different ways to fit particular local needs.


 

Crossref Citations

1. Implementation of medicines pricing policies in sub-Saharan Africa: systematic review
Augustina Koduah, Leonard Baatiema, Anna Cronin de Chavez, Anthony Danso-Appiah, Irene A. Kretchy, Irene Akua Agyepong, Natalie King, Timothy Ensor, Tolib Mirzoev
Systematic Reviews  vol: 11  issue: 1  year: 2022  
doi: 10.1186/s13643-022-02114-z