Abstract
Background: Globally, the theory and practice of evaluation largely emerged from Euro-Western worldviews and continue to evolve in a manner that addresses the needs of Euro-Western interests. The extent to which the closely held values and general outlook of diverse stakeholders underpins how evaluation is processed and designed is interrogated, given the global and multicultural nature of evaluation.
Objectives: We summarised contemporary debates on Made in Africa evaluation and reflected on what these debates might mean for evaluation practice. We also reflected on what it might mean to adapt a quintessentially Euro-Western evaluation method as an integral part of indigenising evaluation theory and practice in African contexts.
Method: Research that adopted a case study approach and a realist evaluation meta-analysis framework was conducted in South Africa, which aimed to provide better understanding of the methodologies and approaches used in past programme impact evaluations. The aim was to reflect on the usefulness of evaluation findings to policy decision makers and to determine the suitability of the realist evaluation method in the implementation of evaluations.
Results: While potential constraints were involved in its application, the realist evaluation held promise in addressing some of the requirements of Made in Africa evaluation. A model of an adapted realist evaluation cycle was presented. It was suggested that the model might enable better implementation of the realist evaluation framework in some African contexts.
Conclusion: Adaptation of evaluation models, theories and instruments, towards contextual relevance and cultural appropriateness was a progressive continuum towards originating indigenous African evaluation approaches.
Keywords: Africa-centric evaluation; adaptation; realist evaluation; evaluation cycle; evaluation model; Made in Africa evaluation.
Introduction
Evaluation has advanced and broadened to a highly globalised world since its earlier prevalent implementation in American government social reform programmes such as the Great Society and War on Poverty (Shadish & Luellen 2011:184–186).
Evaluation may have been phenomenally embedded through international development (Cloete 2016; Ofir 2013), however in recent years governments have increasingly started to build state capacity to evaluate (Porter & Goldman 2013; Mbava 2017).
Similarly, private sector and non-profit organisations have increasingly used evaluation as a tool for accountability (Bisgard 2017; Abrahams 2015:2–5; Wildschut 2014). Globally, all manners and aspects of our lives are increasingly subjected to evaluation and assessed against quality criteria, resulting in a fervent evaluation wave within society at large (Dahler-Larsen 2011, 2019).
It has been asked: whose values and world views inform the evaluation process and design? (Chouinard & Hopson 2016:248). This inquiry is underpinned by an evolving discourse that interrogates asymmetries of power structures between the regions of the Global South and the Global North, epistemic justice and issues of identity and representation (Ndlovu-Gatsheni 2015:13–40), bringing the extent to which evaluation practices and theoretical lenses are contemplative and inclusive of those involved in the evaluation process under interrogation. While the interest in interrogating evaluation theory and practice through the lens of cultural competence and responsiveness, contextual relevance and cultural validity is not new (Chouinard & Hopson 2016; SenGupta, Hopson & Thompson-Robinson 2004), most rigorous expositions have focused on localities of Native Americans, New Zealand Maoris and Australian Aborigines. Meaningful engagement with these issues as they resonate in an African context has not received due consideration. In an African context, the implication has been the perpetuation of one–size-fits-all evaluation methods that have not fully served both beneficiaries and policy decision makers (Mbava and Dahler-Larsen 2019:1).
This article seeks to address this gap through a consideration of how existing evaluation models, theories and instruments might be adapted towards contextual relevance, cultural responsiveness and inclusivity of local stakeholders and African evaluators. Specifically, we reflect critically on the ‘Made in Africa’ evaluation discourse and summarise what scholars have proposed as the way forward for moving this discourse in evaluation practice in the African continent. Drawing on empirical research that seeks to better understand the methodologies and approaches used in past programme impact evaluations in the South African public sector and drawing on the perspectives of both policy decision makers and evaluation service commissioners, we reflect on the practical steps and challenges involved in taking an adaptive approach to operationalising ‘Made in Africa’ evaluation, which involves taking progressive step towards originating indigenous and authentic African evaluation approaches into existing Euro-Western traditions. To focus this discussion, we use the example of realist evaluation, which we propose by its acknowledgement of the need to accommodate unique evaluation contexts, which may serve as a promising beacon for better evaluation practices in African contexts.
Realist evaluation is an addition to evaluation body of knowledge from European theorists (Pawson & Tilley 1997) and built on the assumptions of realism’s philosophical foundation. Its central assumption lies in articulating a programme’s theory of change as a basis of explicating how and why interventions work and for whom they effectually work for (Pawson 2006:25). This is largely described through the interrogation of the context of the intervention, the explanation of what it is that triggers the observed change and further ascertaining that the observed outcomes corroborate the original programme theory. This idea is expressed through the conceptual framework of context–mechanism–outcome configuration or CMO (Pawson & Tilley 1997; Mbava 2017).
Using this realist evaluation approach as an example that might be adapted to serve the purposes of a Made in Africa evaluation agenda, we discuss the steps that might be involved in putting this adaptation into practice. The suggestion is that acting in conducive contextual environments (the context), the programmes will have successful outcomes through a unique combination of the innate thought processes and actions of programme participants (the mechanism). Therefore, successful interventions are a result of interpreting suitable context and the underlying mechanism that result in planned outcomes (Mbava 2017:59). This study ends with a self-critical discussion of the limitations and potential applications of this kind of adaptive approach as well as directions for future theory and practice.
A brief overview of evaluation
From its earlier roots in evaluating United States government social programmes in the eras of the ‘New Deal’ and ‘Great Society’ policies (Shadish & Luellen 2011:184–186; Mbava 2017) evaluation has through development advanced and broadened to a highly globalised world and is now practiced in a multicultural world and in complex contexts, impacting the lives of various and diverse communities globally. Demanded by governments (Porter & Goldman 2013; Mbava 2017), embedded in development (Cloete 2016; Ofir 2013) and increasingly utilised in private and not-for-profit sectors (Bisgard 2017; Abrahams 2015; Wildschut 2014), an evaluation wave fuelled by performance and quality standards is creating an evaluating society (Dahler-Larsen 2011, 2019).
Given this context, a brief overview of some of the scholarly views and descriptions is useful.
It has been claimed that ‘programme evaluation is primarily concerned with judging the merit, worth, quality, or value of programmes’ (Scriven 1999:521). Furthermore:
‘… the evaluation process identifies relevant values or standards that apply to what is being evaluated, performs empirical investigation using techniques from the social sciences, and then integrates conclusions with the standards into an overall evaluation or set of evaluations’. (Scriven 2003:7)
Others (House 1993; Mark, Henry & Julnes 1999; Scriven 1997, 1999, 2003; Stufflebeam 2001) have interpreted evaluation’s main purpose as that of giving precedence to the appraisal of merit and value of a programme or policy. The centrality of this characterisation encompasses the value judgement nature of evaluation. Fournier (2005:140–141) has suggested that the inquiry and inferences of evaluation have a value judgement that is both experimental and normative. Mark et al. (1999:188) support the view of an intrinsic judgement underpinned by values with regard to the merit and worth of an intervention and its influence on further knowledge regarding programme efficiency. Patton (2008:39) described programme evaluation as:
‘… the systematic collection of information about the activities, characteristics, and results of programmes to make judgements about the programme, improve or further develop programme effectiveness, inform decisions about future programming and/or increase understanding’.
In the context of Africa-centric evaluation, the African Evaluation Association (AfrEA; Chilisa 2015: 9) has thus resolved that the role of values and culture in African contexts should be evident in current knowledge systems and infused in modelling evaluation in Africa.
It has also been argued that evaluation is part of applied research with a focus on empirical enquiry (Rossi, Freeman & Lipsey 2004; Weiss 2004). Rabie and Cloete (2009:81) have suggested that ‘evaluation is nothing more than an applied social research activity’. Weiss (in Alkin 1990:83) maintains that evaluation largely rest upon policy research evidence that aims to support decision-making and the formulation of effective programmes. Similarly, Lincoln and Guba (1986:73–84) claimed that evaluation is a ‘disciplined inquiry’ underpinned by scientific rigour as a basis for programme planning. Others equally support this observation (Rossi & Freeman 1993) noting that evaluation applies scientific methods on social research to enable the assessment of how interventions were conceptualised, planned and implemented.
The field of evaluation encompasses various definitions, this article does not claim to provide a definitive conclusion on completeness of the various perspectives, rather these few explanations provide some perspectives reflected in the literature.
Research method
A recent study was conducted in South Africa, which aimed to provide better understanding of the methodologies and approaches used in past programme impact evaluations in the South African public sector and to reflect on the usefulness of evaluation findings to policy decision makers (Mbava & Dahler-Larsen 2019; Mbava & Rabie 2018; Mbava 2017).
In addition to a comprehensive literature review, the study adopted a case study design. Four publicly available impact evaluations were identified for assessment.
A meta-analysis framework that was informed by existing knowledge from the literature review as well as the realist evaluation theoretical framework was applied to the analysis of selected case studies to explore the extent to which the evaluation questions, methods and findings of the impact case studies offered insight into what works, for whom, why it works and under which conditions it works. In addition, key informant interviews were conducted with seven policy decision makers, commissioners, implementers of evaluations in the South African public sector and people familiar with the case studies to determine the usefulness of the evaluations as well as determine the suitability of the realist evaluation method in conducting policy evaluations in the South African public sector. For a detailed discussion on the study and its findings, see Mbava (2017).
Towards ‘Made in Africa’ evaluation
The AfrEA has largely been championing a call for African ownership of evaluations of the highest standards from within Africa as well as evaluation theory and practice pertinent to the lived realities of the continent (Chilisa 2015). The interest in ‘making evaluation our own’ emerged as a strong statement in the 4th AfrEA, Niamey, Niger Conference in 2007 (AfrEA 2007). It was resolved to drive an evaluation agenda that is led and owned by Africans including exploring the possibilities of adopting African-based methods and practices in evaluation and supporting the incorporation of such within the body of evaluation (AfrEA, 2007). This had been a consistent call for the adoption of Africa-centric evaluation in both theory and practice as a response to the imperatives of African cultural contexts (AfrEA 2007; Bellagio Centre 2012; Chilisa & Malunga 2012; Chilisa 2015; Chilisa, Major, Gaotlhobogwe & Mokgolodi 2016; Cloete 2016; Carden & Alkin 2012; Ofir & Kumar 2013).
These new voices from the Global South have increasingly called for full participation in the construction of evaluation theory and practice that is relevant to the lived realities of all evaluation stakeholders, including beneficiaries, located in the developing world (Mbava 2019). Such realities reflect the incorporation of their knowledge, value systems and perspectives in the evaluation process and designs (Chournaid & Hopson 2016:248). It has been argued that the credible involvement of local evaluators in more ways than in the typecast ‘role of practical fixers and data collectors’ results in better evaluation (Mbava & Dahler-Larsen 2019:3). For example, local voices can go beyond simply testing a programme theory, which has been developed elsewhere. Local evaluators should, for example, participate in formulating programme theory as informed by their lived realities as well as in the evaluation process through collective deliberation and communal decision-making (Mbava 2019:19).
As said by Chilisa (2015:17), we learn from each other and, therefore, adapting Euro-Western evaluation approaches to suit local contexts is ‘a good practice that is supported by African value systems’. Drawing from the knowledge systems embedded in African cultural context, this has the potential to strengthen ways of thinking about evaluation and its impact.
Given this agenda, it is apparent that the pillars of an Africa-centric paradigm include appreciation of pluralistic methods in evaluation; active participation of key stakeholders in the construction of what is evaluated, when, by whom and how; the interrogation of the broader programme context and promotion of evaluative thinking; and finally due cognisance and infusion of African epistemology, ontology and axiology in evaluation.
As one perspective of evaluation enquiry necessitates empirical research to estimate how interventions are theorised, planned and implemented, we need to ask ‘Whose research methods and whose scientific enquiry process are applied in evaluation?’ Furthermore, cognisant that evaluation inquiry and inferences as gleaned through the literature have an intrinsic value judgement that is underpinned by both normative and scientific bases, we need to further ask ‘Whose value systems informs what is defined as evaluation evidence’. These are some of the questions that a Made in Africa theoretical lens seek to critically analyse and interrogate. For example, it has been argued that in many African contextual environments, embedded societal values are central in regulating social and communal structures (Mbava 2019:17; Idang 201:101). Given this state of affairs, African values and worldviews should be infused and guide the development of credible evaluations.
Whose value system?
Reflection and critical analysis on both the epistemological and axiological underpinning of evaluation enquiry is important, this article addresses the questions of whose value system informs evaluation enquiry within an Africa-centric viewpoint. Choices and decision-making are not only driven by the available evidence but also influenced by values, beliefs and assumptions that have shaped Euro-Western worldviews. The extent to which African values and worldviews guide and shape evaluation in Africa is important for consideration. Our own research suggests a correlation between this and the degree to which evaluation findings are ultimately utilised by African policymakers and citizens engaged with the development of the continent (Mbava 2017; Mbava & Rabie 2018).
Accepting that the evaluator brings an inherent value judgement in the evaluation processes has implication for objectivity in evaluation. The role of evaluations in appraising the worth and significance of a programme or policy and decision-making has implications for what is deemed legitimate and credible. We suggest this point is particularly relevant to our discussion about Africa-centric evaluation approaches, because as Mark et al. (1999:179) argued, ‘the evaluator’s background may be the most important determinant of the type of evaluation that is done, rather than the context and the information needs of the affected groups and the public’. Thus, whether an evaluator stems from a western background or an African one, the values are critical to informing the very process of value judgment in evaluation. As Chilisa further points out (2015:12), these axiological, ontological and epistemological assumptions ‘inform the realities that we see, how we see them, how we interpret them and how we communicate them’. We argue that this perspective has not been dealt with frankly and sufficiently by the evaluation community, especially because a significant majority of evaluations on the African context are undertaken by non-African evaluators – a trend that appears to be increasing rather than decreasing over time (Cameron, Mishra & Brown 2016).
Evaluation influences and theoretical models
The trajectory of evaluation into Africa has been extensively documented elsewhere (Cloete 2016; Mouton 2010; Mouton, Rabie, Cloete & de Coning 2014; Wildschut 2014). The node of this trajectory indicates deep-rooted emergence of this phenomenon in development, where practices in this sphere have, to a large extent, defined and influenced evaluations in much of the developing world today. The colonised past of the African continent and Western influences on how evaluation is conceptualised, theorised, practiced and taught have had lasting impact on the current adopted evaluation standards and approaches as well as the teaching of evaluation that dominantly draws from Western texts and theory within the continent. This has often largely resulted in one-size-fits-all programme evaluation approaches as evaluation frameworks and adopted approaches were institutionalised and formulated from the viewpoint of development institutions and donors. Cloete (2016:57) goes so far as to argue that these prescriptive ‘mental models’ and approaches were largely brought to Africa through structural adjustment policies – where as a condition of receiving financial loans from multilateral institutions, African economies had to adjust their economic policies as prescribed by the demands of development interventions. The consequence is that these Euro-Western influences have been thoroughly engendered and embedded through international development efforts, which explains why they have endured and continued to influence the trajectory of evaluation practice in Africa. It has been argued that this uncritical transfer of Euro-Western models has failed to transform millions of lives particularly in sub-Saharan Africa (Briggs 2013).
In contrast, the Africa-centric discourse emphasises that alternative theoretical frameworks should be learned from the lived realities of African stakeholders, informed by indigenous ideas and factors in such contexts (Chilisa 2015; Cloete 2016; Ofir & Kumar 2013). It is a counterbalance to the enduring tendency for theory and practice of evaluation to be largely centred on and derived from Euro-Western worldviews and perspectives. These dominant Euro-Western frameworks continue to evolve in a manner that primarily addresses the needs of donors and international agencies without sufficiently considering the realities of African beneficiaries. According to Ofir (2013:585), adopted methods in evaluation and development have not fully strengthened evaluation in developing contexts and rather have focused on simple interventions other than the reality of complex adaptive systems. Research (Mbava 2017) has identified limitations with evaluation methodologies that are inconclusive in providing coherent explanations of why interventions have demonstrated results in a particular manner. Consequently, the validity and usefulness of evaluation results from these analyses have been interrogated for their efficacy to in-country users and resulted in their under-utilisation by African policymakers (Mbava 2017; Mbava & Rabie 2018). Bamberger (2008:134) has claimed that poorly understood evaluation methods potentially result in poor use of evaluation findings. Therefore, the credibility of applied methodologies could have implications for the effective use of evaluation findings. Given this context, the role of evaluation enquiry in the transformation of society and its evolution cannot be marginalised.
Types of approaches to indigenous evaluation
Africa-centric evaluation is appreciative of African knowledge systems, philosophies and value systems. Reaching this point in evaluation research, according to Chilisa (2015:14), will entail the restructuring of power relations in how evaluation knowledge is constructed, ‘such that African people can actively participate in the construction of what is evaluated, when it is evaluated, by whom and with what methodologies’. The imbalance of power between the global north and south countries of the world has epitomised the north as the epicentre of knowledge and the southern countries, such as those in Africa, as the recipients of permanent tutelage from the enlightened Euro-Western metropole. Mamdani (2016:70) argues that ‘rather than acknowledge the plurality of experience and perspective, the universalism born of the European enlightenment sought to craft a world civilization as an expression of sameness’. As a legacy of coloniality, indigenous knowledge systems have generally remained on the periphery of what is deemed as knowledge. The people of the Global South are left with the challenge of undoing the impact of the epistemic violence of imperialism (Spivak 1985:254). Therefore, the indigenous paradigm contests the Euro-western epistemic hegemony (Smith 1999).
‘Epistemological decolonization’ (Mamdani 2016:79) is critical in positioning active African participation and African voices in the construction of evaluation theory and practice. This conversation on decolonising evaluation continues to garner interest. Most recently the 9th AfrEA Conference held a decolonisation debate within its presidential strand that focused on the imperatives of Made in Africa evaluation. Key issues that were debated in this regard were on ‘indigenous knowledge, metaphors, philosophies, relational ontology, reciprocity, epistemology, axiology’ amongst others (AfrEA 2019:14). A theoretical and conceptual understanding of a Made in Africa framework remain useful, the next step in this trajectory is the practical application of the concept. Given that Africa is a vast and diverse continent it is expected that such demonstration of application will vary as dictated by the contextual environment. In moving forward with the implementation of Made in Africa evaluation, firstly it is expected that practitioners will practically apply this theoretical thrust in evaluations that are planned or unfolding in African contexts, then proceed to document lessons learnt to further inform practice. Secondly as the proof of concept is strengthened, it is further anticipated that commissioners of evaluations will lucidly specify the need for a Made in Africa lens in their evaluation terms of reference. Finally, with the view towards advancing Africa and her people and the achievement of the sustainable development goals, there is an expectation that progressive development partners and funders adopt a longterm view on Made in Africa evaluation and progressively seek to promote specific donor-funded evaluations that adopt this African lens.
Examples of indigenisation of evaluation in the Global South and the pre-eminence of culturally relevant evaluation approaches from a global perspective have been documented (Chilisa 2015; Chilisa, Major, Gaotlhobogwe & Mokgolodi 2016). According to Chilisa (2015), the indigenisation of evaluation is informed by three types of approaches, discussed below.
Least indigenised approach
The least indigenised approach in evaluation as characterised by Chilisa (2015) has a dominance of a monolithic Euro-Western evaluation paradigm. These approaches have inherent predetermined standards and criteria and are uncritically transferred into African contexts. These lack substantive appropriate customisation in their implementation. Therefore, this approach lacks depth to qualify as African driven in any manner or approach (Chilisa & Malunga 2012). It has been argued that the ‘logical framework’ or ‘logic model’ – a staple instrument in development capacity building and evaluation – is a typical example of this approach (Carden & Alkin 2012:107), where ‘standard evaluation frames or lenses are not adequately capturing the complexity and realities of the African context, and hence undermining the credibility of practice of evaluation’ (Bellagio Report 2013:47).
Adaptive evaluation approach
The adaptive evaluation approach is moderate and rational in its approach to evaluation. It considers the adaptation of Euro-Western evaluation models, theories and instruments as value adding.
The overarching aim is adaptations rather than uncritical adoption. According to Carden and Alkin (2012), most adaptive methodologies are overwhelmingly still developed by evaluators and theorists in the global north, with a view to application in the global south. Most of these methods have participatory philosophies which Rabie and Cloete (2009:86) describe as ‘lean [ing] towards a more applied social improvement approach to evaluation research with the general aim of development, empowerment and creating shared understanding of the programme between the evaluators, beneficiaries and decision-makers’. Some examples of approaches that might be amenable to this include ‘developmental evaluation’, ‘outcome mapping’, ‘democratic evaluation’ and ‘most significant change’ evaluation approaches. However, the design and development of these are external from Africa. Therefore, the thought and intellectual leadership lie elsewhere, and active application is in Africa.
An Africa-centric evaluation approach would contest this epistemic hegemony. Beyond being bystanders or uncritical recipients of knowledge, African evaluators, beneficiaries and decision makers seek to pursue an explicit and active participatory role that demands a say in the entire evaluation process. This restructuring of epistemic power relations is only possible with active African voices, and our aim in this article is to argue that credible inclusivity of African voices as envisaged in the Made in Africa theoretical framework presents value and credibility for all stakeholders.
Indigenous approach
The indigenous approach to evaluation is far reaching, profound and evolutionary in comparison to the first and second approaches. It can be viewed as a radical approach insofar as it demands deep-rooted and profound indigenous design and development of evaluation. The indigenous paradigm has at its core the ontological, knowledge and value systems that emanate from the cultures, histories and philosophies of those marginalised by colonialism (Cram, Tibbetts & LaFrance 2018; Mbava 2017; Chilisa, Major & Khudu-Petersen 2017:327). Therefore, ‘evaluation paradigms in this context transcend adaptation or adoption of prevailing methods. Rather they encompass evaluations whose methods, design, processes, systems and implementation are indigenous to Africa and its people in all respects’ (Mbava 2017:7). Chilisa and Malunga (2012:32) further emphasise that ‘African-driven evaluation theory and practice can draw from the evolving post-colonial indigenous paradigm to articulate epistemologies and values of an African-driven evaluation’. Carden and Alkin (2012:111) give an example of the African peer review mechanism, an approach of the New Partnership for African Development, as a demonstration of what is possible with an indigenous lens on evaluation ‘because of the significant engagement of African agents and agencies in its design, implementation, critique, and evolution’. These fundamentals and other African knowledge systems call for strong African engagement and leadership in the definition of appropriate and contextually relevant approaches in a quest to further expand the boundaries of knowledge and co-create evaluation theory (Mbava 2017:8). It has been suggested that key stakeholders in African contexts should visibly lead this effort of framing appropriate evaluation approaches (Carden & Alkin 2012). The lekgotla explained in the next section is a good illustration of an African knowledge method that resonates with an indigenous approach.
An African indigenous method – the lekgotla
Chilisa (2015:18) discusses how various African communities apply certain strategies that enable members of the community to engage constructively on key matters affecting the community and to reach consensus. In a Southern African context, a lekgotla is a democratic structure, a meeting where the public or members of a specific group may convene to discuss and resolve a specific issue or agenda. For example, the South African government often refers to ‘cabinet lekgotla’ in reference to prolonged cabinet meetings (RSA 2020). Traditionally a lekgotla is usually a community gathering and deliberation on specific issues to reach consensus. According to Pienaar (2015:58), the lekgotla is based on an ‘indigenous practise of problem resolution’ and is an authentic research methodology leading to authentic research outcomes in African contexts. In a lekgotla, the local community actively engages in the entire research process from initiation to its ultimate resolution. Deeply held belief systems, cultural norms and standards and the community world view influence the research process. For example, a member of the community will bring an issue to the attention of the community. A lekgotla will be convened. The initiator will be given the platform to outline the issue. A community leader will then facilitate the proper framing of this problem amongst members of the community as each community member will take turns in articulating their understanding of the issue at hand. This deliberation will culminate in collective articulation and framing of the problem. Given this coherence and clarity, the community will deliberate on possible options. Some of the questions will relate to how such issues were dealt with in the past, given the set of circumstances at this time, and how the issue should be processed, given the cultural norms, values and attitudes of the community. Given this complexity, at the forefront of such deliberations will be the respected community elders who, through their wisdom, resilience and indigenous knowledge, are relied upon to shape the problem resolution towards sustainable outcome.
In a programme evaluation context, a lekgotla may build ownership, build trust between the evaluator and the community and this empowerment may enable the community ‘to take charge and become the main authors of the research project and the researcher becomes the co-author through the experiences and eyes of the community’ (Pienaar 2015:58). Laher, Fynn and Kramer (2019:398) draw our attention to the importance of demonstrating the appropriateness of research methods that are drawn from indigenous knowledge systems. It is argued that while a research focus group will have similar goals, a lekgotla will have more credibility as this process is managed with indigenous focus and a context-specific manner. Clearly a lekgotla is far more substantive than a typical research focus group which in typical evaluation process will assemble a purposeful sample of people and interview them on a pre-selected thematic topic. In a research focus group, the participant’s involvement is specific and the group has not been involved in the entire evaluation design such as formulating the research question, whereas in a lekgotla ‘the community is responsible for facilitating the research questions and reaching a resolution’ (Laher, Fynn & Kramer 2019:397).
In general, therefore, it seems that there is abundant room for further progress in unearthing various evaluation tools or theories of evaluation emanating from African contexts that can shape evaluation theory and practice. Because the lekgotla ‘underpins the principle of learning from the community, to resolve issues of the community, in the community’s context’ (Pienaar 2015: 60), it is a data collection and analysis methodology that has potential as a useful research method with utility value for researchers and evaluators in a Southern African context. It is possible that variants of lekgotla are practiced elsewhere in Africa beyond Southern Africa with their own terminology. It is evident that other localised methods such as the one illustrated present potential value and there is a need for such methods to be defined and understood in the context of conducting evaluation with sensitivity and leveraging African knowledge systems.
We agree that originating unique evaluation practice and theories through authentic and indigenously African perspectives is the ideal. This long-term goal entails the formulation of a body of knowledge informed by African epistemology, ontology and axiology that engenders participatory, liberating and transformative evaluation practices. The first two approaches are incremental in nature since they are a continuum towards and indigenous approach. This is at best disempowering and self-limiting as as it a adopts an uncritical stance equally within the theory and practice of evaluation. On the extreme end, we are also of the view that the indigenous approach towards an Africa-centric approach, while it is possible, requires a long-term view.
The indigenous paradigm has at its core the ontological, knowledge and value systems that emanate from the cultures, histories and philosophies of those marginalised by colonialism (Chilisa, Major & Khudu-Petersen 2017:327).
To foreground an indigenous African rooted evaluation will demand enhanced intellectual leadership in defining appropriate methodologies and approaches that enhance and push the boundaries of knowledge and contribute towards enhanced theory building. This is a critical contribution that must be made by Africans to frame evaluation methods that are authentic to their lived experiences. The Made in Africa evaluation discourse is firmly on the agenda and gaining momentum, its sustenance and foregrounding will demand a deliberate resolve from African evaluators, given the intrinsic power imbalance between the global north and global south (Mbava 2017:8). Such deliberate and purposeful action is critical, given that ‘we are still facing an uphill task in translating these efforts into widespread practice, especially on the continent, as the evaluation knowledge and practice gatekeepers are still mostly from the North’ (Chilisa & Malunga 2012:33).
An adaptive approach has limitations because the thought leadership, design and development of such evaluation models, theories and instruments largely remain outside of Africa, in this article we argue that insofar as some of these approaches address some of the requirements of the Made in Africa evaluation framework, their adaptation to African monitoring and evaluation conditions may equally prove useful where relevant.
An overview of the realist evaluation approach
In the explanation of how realist evaluation should be operationalised, Pawson and Tilley (1995:23) argue, ‘the success of programs will be highly conditional. Things work if the circumstances are right. Effects occur only if the conditions are right, and they may have to be very particular’. Understanding the factors that generate this change is within the realist framework. Understanding this ‘generative causation’ or ‘generative mechanism’ of a programme is helpful because:
‘Realists conceive of causality in generative terms. Thus, instead of Y simply following X and that being the beginning and end of what can be said, realists consider the causal powers or causal potential that inherent in phenomena and that may be released in some circumstances to produce observable transformations’. (Pawson and Tilley 1995:29)
The reasoning, norms, beliefs and actions of programme participants are the invisible change factors or mechanisms that lead participants to act in ways that result in the observed outcomes. While the implementation context of a programme might be a school, a hospital or a prison, the realist evaluation framework considers a programme’s context in much broader terms. Programme context is an embodiment of ‘prior set of social rules, norms, values and interrelationships’, which influences the extent to which a programme will succeed (Pawson and Tilley 1997:70). Considering an Africa-centric evaluation lens, it is reasonable to consider that programme participants, based on some African contexts, might have African philosophical assumptions about phenomena, specific African worldviews unique to their contextual environments, hold specific traditional belief systems, ways of knowing about their lived realities and ways of doing thing. These underlying thought processes will essentially influence how these programme participants engage with any given intervention. Therefore, these contextual factors will inform observed outcomes. In this context, realist evaluation illustrates and exposes the importance of the value system of the programme participants, not simply those of the evaluator, in the success of the evaluation. This a critical value proposition of the realist evaluation framework insofar as it responds to the Africa-centric evaluation perspective.
The emphasis on the generative mechanism in our view, therefore, makes realist evaluation particularly attractive to Africa-centric evaluators. Realist philosophy considers that an intervention is effective because of the decision-making of programme participants. Pawson and Tilley (2004) specify the main features of the realist evaluation technique. Firstly, it is the theory and hypothesis stage; secondly, the data collection stage; thirdly the data analysis stage and finally the theory testing and refinement stage.
Stage 1: Theory and hypothesis
The idea of this initial stage is to formulate a theoretical proposition about what exactly works in an intervention, for whom does it work and critically why or how does it work. Pawson (2006:25) emphasises that in realist evaluation, three components, namely, context, mechanism and outcome, are critical to facilitate a clear picture of how and why a programme works.
Pawson and Tilley (2004:10–11) argues that the cycle starts by ‘eliciting and formalising the programme theories to be tested in CMO terms and what is involved is bringing the imagination to bear in ‘thinking through’ how a programme works’. The key aim is to have ‘a clear understanding of the basic initial programme theory of change. Then various hypotheses of CMOs for potential testing are elicited through workshops with various stakeholders and programme source documents. These conjured hypotheses should at best meet the purpose and evaluation question to be answered’ (Mbava 2017:65).
In the context of a Made in Africa theoretical framework, this proposition is attractive, given the need for a counterbalance to the enduring tendency for theory and practice of evaluation to be largely centred on and derived from Euro-Western world views and perspectives. Furthermore, the evaluation design and approaches that focus on specific contextual evaluation needs of the continent have potential value.
Stage 2: Data collection
It is claimed that ‘The collection of data and the adopted research methods should be informed by the evaluation question at hand in order to test the theory and effectively answer the evaluation question’ (Mbava 2017:67). According to Pawson and Tilley (2004:11), data are collected ‘that will allow interrogation of these embryonic hypotheses’. The stipulated key outcomes of the programme act as a basis for appropriate data collation on aspects of CMOs. These will include both qualitative and quantitative data. The ‘data collection’ phase has been fully described elsewhere (see Mbava 2017:67). In the context of a Made in Africa theoretical framework, the application of various African data collection methods including folklores, music, dance, oral traditions will have validity in answering the research question (Chilisa 2015:15).
Stage 3: Data analysis
The ‘data analysis’ phase fully described in Mbava (2017:67) and Pawson and Tilley (2004: 11), serves ‘a whole package of CMOC hypotheses to systematic tests, using data sets assembled in stage 2’. The aim is to explore key themes from the collected data. ‘The data analysis aims to test the initial theory against the observed outcome patterns of the programme. These conjectured CMO configurations can be presented in the form of succinct storylines and graphic presentation to facilitate comparison’ (Mbava 2017:67). Cognisant that the lekgotla supports the data collection and analysis methodology stage in a far more authentic and Africa-centred manner, it can bring the evaluation participants in credible ways to jointly interrogate the emerging outcomes.
Stage 4: Theory testing and refinement
According to Pawson and Tilley (2004:11), the last stage involves ‘the assessment and interpretation of the analysis. Have the theories about how the programme worked been supported or refuted by the proceeding analysis? Based on the research findings, the CMO configurations can be revised and be subjected to a further round of theory testing until the results of the analysis explain the observed variations in outcome patterns’. Pawson and Tilley (2004: 11) emphasise that this phase is an ‘iterative and continuous process that aims to ascertain and explain the specifics of programme mechanism’. Having jointly interrogated the emerging outcomes, the evaluation participants in the context of an African lekgotla further engage in confirming or adapting the initial programme theory as informed by the evidence which credibly leverages African knowledge systems.
Towards an adapted approach
The realist evaluation approach has notable strengths for the Africa-centric evaluator. Realist evaluation has the advantage of allowing the evaluator to use theory-based approaches to focus an evaluation on key generative mechanisms that explain why programmes work for some people in some contexts, and not for others. This emphasis on context, relationships and the broader social environment is fundamental to an Africa-centric approach.
Research has pointed to some limitations in the effective implementation of a realist evaluation in some African contexts where monitoring and evaluation systems are nascent. To enable its effective implementation, an adapted realist evaluation cycle reflected in Figure 2 seeks to address these contextual challenges which are discussed subsequently.
Lack of quality baseline monitoring data
Firstly our own research indicated that in some African context, evaluations are not successfully realised because of a lack of useful baseline monitoring data (Mbava & Dahler-Larsen 2019; Mbava & Rabie 2018; Mbava 2017). Similarly, in many other African contexts, such as in North Africa and Central Africa, evaluations are not adequately implemented (Camron, Mishra & Brown 2016:18). This may pose limitations to adequately implementing a realist framework. One sector expert we interviewed summarised this gap in the context of specific types of evaluation such as impact evaluations where it was found that these had some limitations (see Mbava 2017:120). To counter this limitation, we suggest as a first step, the integration and entrenchment of foundational monitoring and evaluation systems and processes to enable and rationalise the forthcoming evaluations. This is depicted in Figure 2.
Differing views on sectoral intervention logic
‘A programme theory is a set of ideas or hypotheses that explain how and why an intervention will work, perhaps with a specification of why it works for a particular group of people in a particular context’ (Mbava & Dahler-Larsen 2019:3). In our research experience, we have found that while programme pathways to change are an inherent assumption in realist evaluation, at the level of practice there is often an evident ‘lack of consensus about the pathways to change and programme mechanism’ – specifically at public policy sectorial level (Mbava & Dahler-Larsen 2019; Mbava & Rabie 2018:124; Mbava 2017). One expert, for example, commented that ‘Realist Evaluation makes assumptions about theories of change. It assumes that we agree on pathways; however, there is no agreement about intervention logic. There is equally no agreement on mechanism’ (see Mbava 2017).
In another exercise, we were contracted to assist in a diagnostic evaluation of some of the most important South African national policies and programmes supporting smallholder farmers since 1994 (DPME 2016). Five evaluation reports conducted by separate evaluation teams under the auspices of the National Evaluation Plan were reviewed during this process. The absence of a clearly defined theory of change was noted by the evaluation teams in all five of these reports – a phenomenon which observers operating within the National Evaluation System indicated was widespread across sectors. Because of these tensions regarding an intervention logic within a specific policy sector, a sectoral lekgotla, which seeks consensus on what that entails, has potential value for prospective evaluation. The second stage of Figure 2 depicts the suggested process to be adopted to achieve better clarity.
To enable its effective implementation and prior to any prospective evaluation, we suggest a prior stage of first embedding foundational systems as depicted in Figure 2. A second stage will be to convene a sectoral lekgotla to agree on that specific sector intervention logic. The lekgotla participants inclusive of programme beneficiaries, programme designers, implementers and policy decision makers will seek agreement on sectoral strategic priorities, desired outcomes, the sectoral intervention logic or theory of change and crucially sectoral outcome indicators. Other key participants in this lekgotla will be Parliament Portfolio Committees as these require the evaluative evidence for their oversight role and holding the executive to account. Research by Rabie (2019) has illustrated the critical role played by Parliament Portfolio Committees in influencing the utilisation of evidence because their buy-in determines the extent of utilisation of evaluation reports.
Considering these findings an adapted realist evaluation cycle that could potentially be implemented in some African monitoring and evaluation contexts is depicted in Figure 2. We argue that these initial stages will go a long way in facilitating the seamless implementation of the realist evaluation cycle. Informed by the existing monitoring and evaluation conditions in a specific context, these two stages may prove necessary in fledgling monitoring and evaluation environments. On the other hand, where systems are resilient and foundational systems are embedded, these two stages may not be a requirement. Therefore, the application of the adapted model is not prescriptive and should rather be informed by practice.
Implications and way forward
We propose that the adaptation of Euro-Western evaluation models, theories and instruments, as a continuum towards indigenising evaluation theory and practice from an African context, may prove useful where relevant. It is suggested that such adaptation is, however, only useful insofar as it addresses some of the requirements of the Made in Africa evaluation framework. The realist evaluation approach provides notable strengths for the Africa-centric evaluator to explain why programmes work for some people in some contexts, and not for others. This emphasis on context, relationships and the broader social environment addresses some of the requirements of the Made in Africa evaluation framework.
An important limitation and challenge is that the actual empirical application of this model has not yet been tested. This will be a priority for our own evaluation practice moving forward, and we encourage other evaluators working in African contexts to take up the challenge of putting this adapted realist evaluation research technique into practice. Such an illustrative application could further strengthen or point to further gaps of the proposed model.
Acknowledgements
Competing interests
The authors have declared that no competing interest exists.
Authors’ contributions
All authors contributed equally to this work.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability statement
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Abrahams, M.A., 2015, ‘A review of the growth of monitoring and evaluation in South Africa: Monitoring and evaluation as a profession, an industry and a governance tool’, African Evaluation Journal 3(1), Art. #142, 8 pages. https://doi.org/10.4102/aej.v3i1.142
Alkin, M.C., 1990, Debates on evaluation, Sage Publications, Newbury Park, CA.
African Evaluation Association, 2007, Special stream statement, viewed n.d., from http://vopetoolkit.ioce.net/sites/default/files/resources/3.8e3%20PUB_Special%20Stream%20Statement.pdf
African Evaluation Association, 2019, 9th African Evaluation Association international conference declaration, viewed n.d., from https://afrea.org/2019conference/afrea-declaration/
Bamberger, M., 2008, ‘Enhancing the utilization of evaluations for evidence-based policy-making’, in Segone, M. (ed.), Bridging the gap: The role of monitoring and evaluation in evidence-based policy-making (120–142). (Evaluation Working Paper #12), UNICEF, New York, NY.
Briggs, J., 2013, ‘Indigenous knowledge: A false dawn for development theory and practice?’ Progress in Development Studies 13(3), 231–243.
Bisgard, J., 2017, Demystifying monitoring and evaluation, viewed n.d., from https://trialogue.co.za/demystifying-monitoring-and-evaluation/#
Carden, F. & Alkin, M.C., 2012, ‘Evaluation roots: An international perspective’, Journal of Multi-Disciplinary Evaluation, 8(17), 102–118.
Cameron, D.B., Mishra, A. & Brown, A.N., 2016, ‘The growth of impact evaluation for international development: how much have we learned?’ Journal of Development Effectiveness, 8(1), 1–21.
Chilisa, B., Major, T.E., Gaotlhobogwe, M. & Mokgolodi, H., 2016, ‘Decolonizing and indigenizing evaluation practice in Africa: Toward African relational evaluation approaches’, Canadian Journal of Program Evaluation, 30(3), 313–328.
Chilisa, B., 2015, A synthesis paper on the Made in Africa evaluation concept: Commissioned by African Evaluation Association (AfrEA), from https://afrea.org/wp-content/uploads/2018/06/MAE2-Final-31st-august.pdf
Chilisa, B. & Malunga, C., 2012, Made in Africa evaluation: Uncovering African roots in evaluation theory and practice, 32–38. In Bellagio Report. 2013. The Bellagio Report. African Thought Leaders Forum on Evaluation and Development: Expanding Leadership in Africa. The Bellagio Centre, from http://www.clear-aa.co.za/wp-content/uploads/2013/09/Bellagio-Report-Vs-Apr-14.pdf
Chilisa, B. & Kawulich, B.B., 2012, ‘Selecting a research approach: Paradigm, methodology and methods’, in Wagner, C., Kawulich, B.B. & Garner, M. (eds.), Doing social research: A global context (pp. 51–61), McGraw Hill, London, UK.
Chouinard, J.A & Hopson, R., 2016, ‘Critical exploration of culture in international development evaluation’, Canadian Journal of Program Evaluation, 30(3), 248–276.
Cloete, F., 2016, ‘Developing an Africa-rooted programme evaluation approach’, African Journal of Public Affairs, 9(4), 55–70.
Cram, F., Tibbetts, K.A. & LaFrance, J., 2018, ‘Editors’ notes: A stepping stone in indigenous evaluation’, Evaluation, 2018: 7–16. doi:10.1002/ev.20327
Dahler-Larsen, P., 2019, Quality: From Plato to performance. Palgrave Macmillan, London, United Kingdom.
Dahler-Larsen, P., 2011, The evaluation society. Stanford University Press, Stanford, CA.
DPME, 2016, Diagnostic evaluation of the government-supported smallholder farmer sector. Department of Planning, Monitoring & Evaluation, viewed n.d., from https://evaluations.dpme.gov.za/evaluations/520
Fournier, D., 2005, ‘Evaluation’, in S. Mathison (ed.), Encyclopaedia of Evaluation, pp 140–141, Sage Publications, Thousand Oaks, CA.
House, E., 1993, Professional evaluation: Social impact and political consequences, Sage Publications, Newbury Park, CA.
Laher, S., Fynn, A. & Kramer, S., 2019, ‘Transforming research methods in the social sciences: Case studies from South Africa’, University of the Witwatersrand, Wits University Press.
Lincoln, Y.S. & Guba, E.G., 1986, ‘But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation’, New Directions for Programme Evaluation, 30, 73–84.
Mamdani, M., 2016, ‘Between the public intellectual and the scholar: Decolonization and some post-independence initiatives in African higher education’, Inter-Asia Cultural Studies, 17(1), 68–83.
Mark, M.M., Henry, G.T. & Julnes, G., 1999, ‘Toward an integrative framework for evaluation practice’. American Journal of Evaluation, 20(2), 177–198.
Mbava, P.N., 2017, ‘The potential value of the realist evaluation method in programme impact evaluations in South Africa’, Unpublished PhD dissertation, Stellenbosch University, Bellville.
Mbava, N., 2019, ‘Shifting the status quo: Africa influencing global evaluation’, in J. Musumba (ed.), Made in Africa Evaluations. Volume 1: Theoretical Approaches, pp. 12–21, African Development Bank Evaluation Matters, Quarter 3.
Mbava, N.P. & Dahler-Larsen, P., 2019, ‘Evaluation in African contexts: The promises of participatory approaches in theory-based evaluations’, African Evaluation Journal 7(1), a383. https://doi.org/10.4102/aej.v7i1.383
Mbava, P.N. & Rabie, B., 2018, ‘Strengthening impact evaluations in the South African National evaluation system’, Administratio Publica 92(26), 75–98.
Mouton, C., 2010, ‘The history of programme evaluation in South Africa’, Unpublished MPhil dissertation, Stellenbosch University, Bellville.
Mouton, C., Rabie, B., Cloete, F. & De Coning, C., 2014, ‘Historical development & practice of evaluation’, in F. Cloete, B. Rabie and C. De Coning (eds.), Evaluation Management in South Africa and Africa, African Sun Media, Stellenbosch, South Africa, pp. 28–78.
Ndlovu-Gatsheni, S.J., 2015, ‘Genealogies of coloniality and implications for Africa’s development’, Africa Development, XL(3), 13–40.
Ofir, Z., 2013, ‘Strengthening evaluation for development’, American Journal of Evaluation, 34, 582–586.
Ofir, Z. & Kumar, S., 2013, Evaluation in developing countries: What makes it different? In S.I. Donaldson, T. Azzam & R.F. Conner (eds.), Emerging practices in international development evaluation, pp. 11–24, Information Age, Charlotte, NC.
Patton, M.Q., 2008, Utilization-focused evaluation, (4th ed.), Sage Publications, Thousand Oaks, CA.
Pawson, R. & Tilley, N., 2004, Realist Evaluation, viewed 01 August 2016, from http://www.communitymatters.com.au/RE_chapter.pdf
Pawson, R., 2006, Evidence-based policy: A realist perspective, Sage Publications, London, UK.
Pawson, R. & Tilley, N., 1997, Realistic evaluation, Sage Publications, London, UK.
Pawson, R. & Tilley, N., 1995, ‘Whither (European) evaluation methodology?’, Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization, 8(3), 20–33.
Pienaar, A.J., 2015, ‘African indigenous methodology in qualitative research: The lekgotla – a holistic approach of data collection and analysis intertwined’, in M. de Chesnay (ed.), Nursing research using data analysis: Qualitative designs and methods in nursing, pp. 57–67, Springer, New York, NY.
Porter, S. & Goldman, I., 2013, ‘A growing demand for monitoring and evaluation in Africa’, African Evaluation Journal 1(1), 10–18.
Rabie, B., 2019, ‘The most influential decision-making criteria in South African Parliamentary Committees’, Administratio Publica 27(4), 26–46.
Rabie, B. & Cloete, F., 2009, ‘A new typology of monitoring and evaluation approaches’, Administratio Publica 17(3), 76–97.
Rossi, P.H., Freeman, H.E. & Lipsey, M.W., 2004, Evaluation: A systematic approach, 7th edition, Sage Publications, Thousand Oaks, CA.
Rossi, P.H. & Freeman, H.E., 1993, Evaluation: A systematic approach (5th ed.), Sage Publications, Newbury Park, CA.
Shadish, W.R. & Luellen, J.K., 2011, History of evaluation, in S. Mathison (ed.), Encyclopedia of evaluation, pp. 184–186, Sage Publications, Thousand Oaks, CA.
Scriven, M., 1997, ‘Truth and objectivity in evaluation’, in E. Chelimsky & W.R. Shadish (eds.), Evaluation for the 21st century: A handbook, pp. 477–501, Sage Publications, Thousand Oaks, CA.
Scriven, M., 1999, ‘The fine line between evaluation and explanation’, Research on Social Work Practice 9(4), 521–524.
Scriven, M., 2003, ‘Michael Scriven on the differences between Evaluation and Social Science Research’, The Evaluation Exchange, IX(4), 7, viewed n.d., from https://archive.globalfrp.org/var/hfrp/storage/original/application/f1be9c61c5a4011b6637bb5d1a3190ed.pdf
SenGupta, S., Hopson, R. & Thompson-Robinson, M., 2004, Cultural competence in evaluation: An overview’, New Directions for Evaluation 102, 5–19.
Stufflebeam, D.L., 2001, ‘Evaluation models’, New Direction for Evaluation 89, 7–98.
Weiss, C.H., 2004, ‘Rooting for evaluation: A cliff notes version of my work’, in M.C. Alkin (ed.). Evaluation roots: Tracing theorists’ views and influences, pp. 154–169, Sage Publications, University of California, LA.
Wildschut, L.P., 2014, Theory-based evaluation, logic modelling and the experience of SA non-governmental organisations (PhD dissertation), Stellenbosch University, Bellville.
|
Hide
Show all
Email this article (Login required)
