|
Background: Participatory and collaborative approaches to evaluation have grown in popularity in recent years, as program contexts increasingly require more culturally responsive and inclusive approaches to addressing complex community, program and organisational needs. This is particularly the case in development evaluation contexts such as Africa. We recently conducted a systematic review and integration of the literature on participatory evaluation that included the review of 121 empirical studies published in peer-reviewed journals and other outlets (Cousins & Chouinard 2012). In that review, only 21 studies derived from development contexts and, of those, only six from Africa.Objectives: In this article, we considered the applicability and relevance of the thematic discussion by Cousins & Chouinard (2012) to the African development context through a close-up look at research in Africa on participatory evaluation. Method: We carefully examined the African studies and, through a conceptual critique, re-examined the prior thematic analysis. Results: We observed that some themes did not give primacy to context and relationships which are essential considerations in the African context. Further, an emphasis on empowerment-oriented outcomes begs attention to societal, cultural and economic considerations, implication for evaluators’ roles and a deeper understanding of power issues. Conclusion: We concluded that our thematic discussion did not resonate well with participatory evaluation in development contexts and that a much more focused and targeted review and integration of research was warranted.
L’évaluation participative pour le développement: examiner les connaissances tirées de la recherche au sein du contexte africainPrésentation: Depuis quelques années, les approches participatives et collaboratives à l’évaluation gagnent en popularité, les contextes des programmes exigeant des approches à la gestion des besoins communautaires, programmatiques et organisationnels de plus en plus adaptées et inclusives sur le plan culturel. Ceci est notamment le cas dans les contextes propres à l’évaluation du développement tels que l’Afrique. Nous avons récemment entrepris une évaluation et une intégration systématique de la littérature dédiée à l’évaluation participative, évaluation qui a inclus la révision de 121 études empiriques publiées dans des journaux évalués par les pairs et autres sources (Cousins & Chouinard, 2012). Dans cette évaluation, 21 études seulement étaient tirées de contextes relatifs au développement, et parmi celles-ci, seulement six étaient tirées du contexte africain. Objectifs: Nous nous sommes intéressés à l’applicabilité et à la pertinence de la discussion thématique de Cousins et Chouinard (2012) au contexte du développement en Afrique, en examinant de plus près les études réalisées en Afrique sur l’évaluation participative. Méthode: Nous nous sommes intéressés de près aux études africaines et, au moyen d’une critique conceptuelle, et avons réexaminé l’analyse thématique préalable. Résultats: Nous avons observé que certains termes n’accordaient pas la primauté au contexte et aux relations qui sont pourtant essentiels dans le contexte africain. De plus, l’accent mis sur des conclusions axées sur l’autonomisation oriente l’attention sur les considérations sociétales, culturelles et économiques, les répercussions sur le rôle des évaluateurs, et une compréhension plus approfondie des questions de pouvoir. Conclusion: Nous avons conclu que notre discussion thématique correspondait mal à l’évaluation participative dans des contextes de développement et qu’une évaluation et une intégration de la recherche davantage concentrée et ciblée étaient justifiées.
Participatory and collaborative approaches to evaluation have grown in popularity in recent years, as program contexts increasingly require more culturally responsive and inclusive approaches to addressing complex community, program and organisational needs. Whilst the participatory approach has taken on myriad meanings over time, with significant blending and mixing of methods across diverse program settings, what distinguishes it from other approaches is the specific focus on the collaborative partnership between evaluators and program community members. It is precisely the relational and dialogic nature of the inquiry process that sets participatory evaluation apart from other approaches, in the active involvement of multiple and often diverse stakeholders, in the engaged positioning of the evaluator, and in the relationships that are created between evaluators and stakeholders. From a participatory perspective, evaluators and stakeholders are inextricably linked together in what Heron (1996) describes as ‘intersubjective space’, which sees evaluators and stakeholders as active co-constructors in the evaluative process.The call for more and better research on evaluation has been sounded many times over the past number of years (e.g., Cousins 2004; Mark 2008; Smith 1993). Recent reviews of research on evaluation capacity building (Labin et al. 2012), evaluation use (Johnson et al. 2009), and cross-cultural evaluation (Chouinard & Cousins 2009) attest to a growing body of empirical research on evaluation. Participatory evaluation is no exception. Recently, we published a major review and integration of research on participatory evaluation over a span of 15 years (Cousins & Chouinard 2012). Through a rigorous search and retrieval process we located a total of 121 studies, most of which were published in peer-reviewed outlets. Participatory evaluation is particularly relevant in Africa and other development contexts for a variety of reasons, but three inter-related considerations are particularly salient. Firstly, participatory evaluation has been shown to be particularly potent in fostering learning (or conceptual use and process use) about programs and the contexts within which they operate. Most would argue that accountability and learning are the two fundamental purposes of evaluation and that there is an imbalance between the two, favouring accountability. In the development context, the case has been made recently that monitoring and evaluation activities most often favour and are driven by the accountability interests of bilateral and multilateral donor agencies over those of national governments (Carden 2009; Hay 2009). To be sure, donors must necessarily be held to account to stakeholders and constituencies for responsible spending and demonstrable impact of development investments. Yet the downside of the argument is that such emphasis can marginalise the learning interests of national governments, civil society organisations (CSOs) and other local actors in development contexts. Secondly, participatory evaluation provides an indirect approach to evaluation capacity building. To the extent that local actors, program community members and stakeholders are involved in the co-production of evaluation knowledge they stand to benefit significantly. Such benefits are often framed as ‘process use’ (e.g., Cousins 2007; Patton 2008) where learning about evaluative thinking and systematic inquiry occurs by virtue of proximity to the evaluation. Finally, in their review of 52 studies on cross-cultural evaluation, Chouinard & Cousins (2009) observed that participatory and collaborative approaches to evaluation were by far the most natural fit and preferred choice for inquiry. Such approaches have the potential to integrate, for example, indigenous ways of knowing into the evaluation knowledge production function. They most certainly help to inform evaluation questions, designs, methods and processes on the basis of local knowledge and understanding. Given the suitability and desirability of participatory and collaborative approaches to evaluation in the development context, we wondered about the applicability and relevance of research into participatory evaluation in the African context. Of the 121 studies that we reviewed, only 21 took place in development contexts and, of those, only a handful (eight) were carried out in Africa. Despite our efforts to be inclusive, the great majority of studies that we tracked down for review were from the global West (mostly North America, some from Europe). In this article, we briefly describe the results of our thematic analysis of the larger body of studies and then take a close-up look at the African studies in the collection. We wanted to know if and how the themes that we identified initially resonate with research in the African context. After describing the contributions of the African-based studies, we then use this as a lens through which to examine the relevance of our thematic analysis. We conclude the article with implications for ongoing research on participatory and collaborative approaches to evaluation in Africa and development contexts generally.
|
Making sense of the research: A thematic discussion
|
|
Our review and integration (Cousins & Chouinard 2012) were guided by a conceptual framework that had evolved over a significant number of years. The framework, which is reproduced in Figure 1, shows that participatory practices (control of decision-making, diversity amongst participating stakeholders and depth of participation) are shaped and informed by a range of antecedent conditions factors and influences: evaluator role, community context, institutional influences and program considerations. Participatory practices in turn affect the production of evaluation knowledge and the usefulness of the evaluation. Note that ‘use’ is defined as both the use of findings and process use, as mentioned above. Downstream, distal outcomes are also specified. An important aspect of the framework is that it is recursive, suggesting a non-linear flow of influence despite a general temporal sequence.
 |
FIGURE 1: Conceptual framework depicting the nature, contextual conditions and consequences of participatory evaluation.
|
|
We used this framework to help guide our analysis and interpretation of the study contributions and we summarised these findings using the primary panels in Figure 1 as an organising structure. At that point we elected to stand apart from the data and to consider the primary contributions. We conducted a thematic analysis and discussion of the results of the empirical studies in the context of relevant theoretical and practical scholarship on participatory evaluation. We identified seven themes: (1) the need for training to enhance participation, (2) evaluator identity, role and ‘positionality’, (3) the relational dimensions of participation, (4) multidimensional contexts, (5) stakeholder selection, (6) multiple dimensions and locations of power, and (7) learning as a foundation for practice and change. We now turn to a brief summary of these themes.
The need for training to enhance participation
Participatory evaluation stands apart from other forms of evaluation in terms of its emphasis on and commitment to an educative or training function that enables active and meaningful stakeholder participation. Whilst a lack of knowledge and research skill can impede participatory practice, the availability of such knowledge will influence the level of participation and who will participate. We found a notable lack of discussion about or study of training in participatory contexts and we acknowledged that contextual, community and stakeholder concerns (access to power, diversity and language) could compromise evaluation capacity building. We argued that, given a dearth of discussion about the processes of inclusion, there is a need to be more self-reflective and move beyond rhetorical framing of participatory evaluation as a social, political and methodological good.
Evaluator identity, role and ‘positionality’
Conceptions of identity, role and ‘positionality’ of evaluators are inextricably linked to how evaluation is conceptualised in a particular program setting, and is shaped not only by the context of the program and community and stakeholder involvement but also by the rationale adopted to advance collaboration. Some would argue that creating the conditions and circumstances to encourage and allow stakeholder participation throughout the process is considered essential to the evaluator’s role. Many roles emerged from the research – trainer, facilitator, mediator, negotiator, critical friend – but tensions were also observed. Salient were tensions in determining balance amongst these various roles and particularly about the balance between being directive and ceding control, with the quality of the participatory experience contingent on the balance of power negotiated between evaluators and stakeholders. Considered fluid and dynamic constructions, evaluator roles are constantly altered and transformed throughout the process as they engage relationally with stakeholders, yet it is equally the case that stakeholder roles change in response to active participation in the research process.
The relational dimensions of participation
As we can glean from the foregoing, participatory evaluation is fundamentally a relational approach grounded in the social relations formed between evaluators and stakeholders. Knowledge is not considered something that is merely collected and accumulated through research and methodological selection, but is jointly constructed through the process of social interaction between evaluators and stakeholders and amongst stakeholders. Knowledge production is therefore dynamic and unfolding, the product of social interaction that is inextricably linked to the relationship between the knower and the known (epistemic participation). On the other hand, also important are the relations amongst people in the inquiry process and the decisions that are made that will ultimately affect them. Thus ‘political participation’ implies that antecedent organisational and institutional priorities will influence the parameters of the process and participation. Epistemic and political components of the relational process dimension are linked together through power and privilege and it is this dynamic that defines the relational dimension of participatory and collaborative approaches to evaluation.
Multidimensional contexts
All evaluations take place in complex multidimensional contexts with underlying social and political conditions; communities are assumed to have a local ecology that influences the dynamics of participation and places it in a specific social, cultural, political and intuitional context. Ecological program and community dynamics thus refer to the multiple and varied relationships within the evaluation context considered at the microcontextual and macrocontextual levels. Relationships can be understood within a specific program and community setting, as well as within the broader sociopolitical context. One of the challenges of understanding influences at the microcontextual level is the tendency to assume communities are homogeneous entities with very uniform compositions and boundaries. The macrocontextual level, by contrast, embeds the local community context within a larger, fundamentally interconnected social system composed of a ‘hierarchy of social forces’ that impinge on the evaluation and act upon the community. Recognising the influence of microcontextual and macrocontextual dimensions is significant because it highlights the myriad challenges involved in developing collaborative relationships, in what are often complex and demanding contexts.
Stakeholder selection
The selection of stakeholders for participation in evaluation is remarkably contentious, yet very little attention to this process emerged in the research. Decisions about who to include in the evaluation relate to the purpose of the collaborative inquiry and to evaluator values. Organisational constraints of time and resources (fiscal and human) play an important role in the decision process. These primary selection decisions shape the inquiry process and help to frame the boundaries of action and the knowledge that is developed. They define the depth of stakeholder involvement, influence data quality assurance, engage relationships and power dimensions, shape the role of the evaluator and constrain the amount of learning that will take place. We can easily see that decisions about inclusion and exclusion can have a profound effect on the evaluation as well as on the community. Thus, who is involved in selecting the participants can ultimately be as important as who is selected. An essential aspect of this theme is a cultural dimension: selecting group or sub-group representatives can be highly contentious, as group categories themselves may be erroneous or problematic.
Multiple dimensions and locations of power
Participatory evaluation is as much a political process as a methodology; most agree that power continues to shape the parameters of relationships and knowledge outcomes created within a participatory context. What is less clear is how power is manifest and how it influences the process and outcome of the evaluation itself. In our analysis of the research we identified three key locations of power. Firstly, relational power frames all social relations within the participatory context. It includes processes of negotiation and decision-making, competing interests, patterns of dominance and dependence, identification of who can or will speak for whom, and rules of inclusion and exclusion. A second location, political power, refers to barriers and biases that preclude people from participating and influence whose voices are heard and whose agendas are included. Political power represents the macrostructures of inequality that include questions of gender, citizenship and class, biases that often shape the contours and parameters of discussion. Finally, discursive power, or the internalised norms and values that guide practice and constrain participation, do more than actually structure ‘reality’ and what is considered real. They actually legitimate, create and help us to make sense of social reality and how power and knowledge are interlaced. Discursive power is less obvious but nonetheless potent in shaping our beliefs, values and norms; it ultimately influences how we think and what we can think about.
Learning as a foundation for practice and change
Participatory evaluation is a deeply educational process and, as research reveals, learning is a common thread from the perspective of both stakeholders and evaluators. Learning takes place at individual, group and organisational levels through process use, capacity building and conceptual use of findings, and it is central to practical, political and philosophical justifications for participatory work. It is through the act of participation, and through relationships that are created and sustained, that learning takes place. An important distinction for consideration is between evaluation considered as a ‘technical undertaking’ or the application of a set of tools, and evaluation as a ‘conceptual function’ intended to generate dialogue and facilitate learning. Learning is highly interactive, social, conversational and dialogic. Dialogue is paramount and provides the opportunity for stakeholders to move beyond technical and instrumental learning to interactions that engage sometimes conflicting values, perspectives and experiences. Thus, through dialogue, participatory evaluation leads to new vantage points and understandings. Privileged is engagement and dialoguing across difference over reaching formal or informal consensus. Participants ultimately grow, thus creating the seeds for learning to take place.Evident in the forerunning discussion are many points of convergence and overlap. In Figure 2 we can see how the themes emerging from the present analysis relate to our overarching conceptual framework (see Figure 1). We would argue that these themes ought to factor into ongoing inquiry about participatory processes.
 |
FIGURE 2: Thematic implications for ongoing inquiry on participatory evaluation.
|
|
Of interest in Figure 2 is that all of the themes touch on considerations of context and antecedent conditions. They help to define the ground from which participatory inquiry emerges or begins. The dynamic complexity of the context within which the intervention (program or policy) is implemented needs to be understood at multiple levels in order to inform decisions to engage with participatory approaches and to shape them. As noted, considerable overlap amongst the themes is evident, but in terms of ongoing inquiry, more needs to be understood about how the various themes interact to shape participatory processes. An important consideration is that participatory processes are themselves dynamic and evolutionary and need to be understood as such. Several of the themes also touch on these processes and will factor into the development and nature of participation over time. Finally, we see that learning in this context is pervasive and by no means limited to the conceptual use of findings. The role of process use is essential in fostering learning about systematic inquiry and in understanding context in deeper, more meaningful ways than would previously be the case. Our discussion reveals not only a considerable level of overlap amongst themes, but also calls attention to the historical, political and social context, interpersonal relations and the role of power, and implications for evaluator roles, responsibilities and practices. We argue that these key contextual and practical dimensions dramatically and fundamentally shape the extent to which participatory evaluation achieves desired and planned outcomes (whether of a practical or transformative nature), and the extent to which unintended outcomes will be positive and appreciated (e.g., with practical or transformative benefits) or unwanted. It may be noted that whilst one could argue that these themes do apply across a broad range of evaluation contexts, from our perspective they are particularly relevant to a diverse range of participatory contexts. But they were derived on the basis of findings from the larger sample of studies in our review and integration, the vast majority of which were implemented in the global West. The applicability and relevance of the themes to the African context remains a question. It seems likely that much of what has been described is pertinent to this context, yet there is probably good justification to be cautious. We point, for example, to the African Evaluation Association’s (AfrEA) initial consideration and then significant adaptation of the Joint Committee Standards for Education Evaluation’s program evaluation standards. The carte blanche adoption of the standards was determined to be unworkable given the relative community-oriented nature of African culture. The African evaluation guidelines (AfrEA 2007) that guide evaluation practised in Africa today are the result of serious transformation, contextualisation and adaptation of the Joint Committee Standards for Program Evaluation. And so we now turn to a close-up examination of research on participatory evaluation conducted in Africa to assess the extent to which and how this research resonates with the emergent thematic discussion described above.
|
Research on participatory evaluation in Africa
|
|
Table 1 provides an excerpt from Cousins & Chouinard (2012), with an eye to describing the eight studies that we located. Francophone African studies were excluded from our sample by virtue of our sampling criteria which focused on English-language studies. Most of the studies were in east and southern Africa with two from west Africa. No studies were implemented in central or northern Africa. Three of the studies focused on the evaluation of health services, two of which were concerned with services offered to HIV/AIDS sufferers. Two of the studies focused on gender issues, examining the empowerment of girls and women. Of the remaining three, one targeted infrastructure development (water supply and sanitation), one examined an agricultural extension program and one looked at services for children in a post-conflict, war-torn context.
|
TABLE 1: Characteristics of studies implemented in Africa.
|
With the exception of one study, all were retrospective or ‘backward-looking’. The study by Parkinson (2009) we classified as longitudinal; it was a case study that took place over one and a half years. The studies used a variety of approaches to participatory inquiry including participatory impact assessment, practical participatory evaluation, transformative participatory evaluation and participatory action research. Six of the eight studies were reflective case narratives, that is, reflective stories about a participatory evaluation experience, implying an element of participant observation. Such narratives, which have become quite popular modes of inquiry in research on evaluation, do not specify methods and are sometimes confused with case studies. It is interesting that six African narratives were written from the point of view of the evaluator(s) or researcher(s) but that almost all were co-constructed by members from other roles (e.g., program directors, NGO staff or external consultants). Cousins & Chouinard (2012) made the argument that collaboratively written narratives are more likely to assure data quality. Most often reflective narratives are limited to the perspective of one person, usually the evaluator. The remaining two studies were case studies, complete with method sections and attention to data quality assurance. We would note that, overall, the heavy reliance on the narrative approach was also common within the larger sample of 121 studies (Cousins & Chouinard 2012). The purposes of the studies varied somewhat and ranged from studies of intervention impact to assessments of the extent to which empowerment was observed to description and analysis of stakeholder involvement in evaluation. Mullinix & Akstsa-Bukachi (1998) compared and critiqued the participatory process against more traditional approaches to evaluation. Table 1 also summarises the main findings of the studies in relation to these purposes. We now turn to an examination of these contributions, particularly in terms of their connection to the foregoing thematic analysis from Cousins & Chouinard (2012).
|
Linkages to the original themes
|
|
Whilst we would argue that, for the most part, the seven themes that we identified previously continue to hold true across all studies, there can be no doubt that these themes must be adapted to the African evaluation context. When looked at from the perspective of the eight African studies, the original themes do not quite capture the depth and complexity of participation, nor the rationale and motivation for its use in this unique context. Specifically, the original themes do not give primacy to the local contextual challenges and relationships with international donor agencies that fundamentally influence the evaluation process in terms of the diversity and range of stakeholders involved, the depth of their involvement, and the delicate balance of stakeholder and evaluator influence on decision-making. At the same time, participation is explicitly intended to empower people and communities, build local and organisational capacity, value local knowledge, use learning to foster social change and address issues of power (Brisolara 1998; Jackson & Kassam 1998). We thus see greater emphasis placed on the cultural, social and economic conditions specific to the African context, a redefined role for the evaluator, an explicit focus on capacity building within the training and learning themes, and stronger linkages between the themes of power, stakeholder selection and relational processes. At this point we envision one additional theme related to the divergent information and program needs between funding agencies and local project teams and beneficiaries. We now turn to this thematic discussion, paying careful attention to what these reconceptualisations may mean to the conceptual framework outlined in Figure 1.
Context
Evaluations must be understood as contextually grounded in the local program setting, as well as interconnected in a broader, more enmeshed social, historical, economic and political climate, what Guzmán (2003:174) has referred to as a ‘hierarchy of social forces’. In the international arena (and in Africa specifically), context exerts tremendous influence on the program and on its evaluation (Fitzpatrick 2012), a finding that becomes all the more significant in culturally and socially diverse communities (Chouinard & Cousins 2009). Whilst the original theme multidimensional contexts of participation did underscore the importance of context to participation and did identify multiple micro and macro relationships of power both locally and more broadly, it did not capture the economic, social and political complexity of African contexts. Consider, for example, the HIV/AIDS epidemic in many African countries, the level of poverty and economic disadvantage, civic strife and post-war conflict zones, the history of reliance on international and humanitarian aid, and the different tribal groups, religions and languages. A number of the African studies described the multiple challenges, constraints and trade-offs involved in conducting participatory evaluations amidst such profound contextual complexity, whether relationally (BiNiba & Green 2005), in terms of power dynamics (Holte-McKenzie, Forde & Theobald 2006), involving local stakeholders (Bradley et al. 2002), training project beneficiaries (Mullnix & Akatsa-Bukachi 1998) or in terms of engaging international donors (Gariba 1998). The point is that context, particularly a context as complex as Africa, fundamentally shapes the contours and dynamics of the participatory process, challenging the range of stakeholders involved, the depth of their involvement, and ultimately the sharing of decision-making between evaluators and stakeholder groups.
Evaluator role
The role an evaluator assumes in a participatory evaluation is dynamic, multiple and engaged, enacted in response to the program community and in helping to shape relationships and build understanding amongst diverse stakeholder groups. As Trickett & Ryerson Espino (2004) observe, the essence of the evaluator’s role is to create the conditions and circumstances necessary to foster and sustain stakeholder participation throughout the evaluation process. As Marsden & Oakley (1990:134) describe, ‘the evaluator is one who can draw together the many diverse threads that link the different actors in the process of project development and weave them into a tapestry of interrelationships’. In the African context, the evaluator has to deal with a very different set of interests, concerns and challenges than in North American contexts (for example), and thus their role will inevitably be fundamentally different. As such, the evaluator’s role in these contexts must in some sense be redefined. Given the often divergent interests and needs of local and donor evaluation requirements, between a focus on both local capacity building and program impacts, the evaluator must transform from the role of judge to the role of educator, facilitator and trainer. As Gariba (1998:67) describes, ‘in this context, the evaluator becomes readily transformed from an investigator to a promoter, and from prosecutor to participant’. Beyond divergent stakeholder interests (specifically at the local and donor levels), consideration should also be given to the influence of power differentials and inter and intra relations between stakeholder groups, the level of capacity building and training required to engage participants in the process and contextual complexity, on the role of the evaluator.
Capacity building
Whilst we noted that with few exceptions training is under-discussed in the empirical literature (Cousins & Chouinard 2012), most of the eight African studies described the need for training, with significant detail about training methods provided by two of the studies (Holte-McKenzie et al. 2006; Mullinix & Akatsa-Bukachi 1998). At a very basic level, the amount of knowledge and skills possessed by stakeholders influences not only the level of participation, but ultimately who does or does not participate in the evaluation (King & Ehbert 2008). Moreover, whilst capacity building was noted in a number of the original 121 studies, in many cases it was identified as an effect or consequence of participation in the process of evaluation (process use), rather than as an explicit motivation for participation. In most of the eight African studies, capacity building was identified as one of the primary rationales for adopting a participatory approach, located at the individual, community or organisational level (e.g., Bradley et al. 2002; Gariba 1998; Holte-McKenzie et al. 2006). Whilst a number of the African studies were cautious in their description of the level of capacity building generated as a result of the process, capacity building and learning were nonetheless given as primary rationales for use, as a way to further local ownership and in providing help in planning and building for the future. In his reflection on a participatory evaluation of a humanitarian education project in Sierra Leone, Maclure (2006) notes that the need to strengthen social capital is paramount, particularly in terms of building local capacity and ownership.
Linkages amongst themes
Approaches to participatory evaluation in development contexts tend to support transformative participatory approaches (Brisolara 1998) that involve diverse program stakeholders (from program beneficiaries to international donors) rather than merely those stakeholders closest to the program (such as program managers) found in more practical participatory contexts. As such, the themes of relational processes and dimensions of voice, stakeholder selection and consequences of participation and locations of power become more interconnected, as the diversity of stakeholders, the differences in power and privilege and the sheer range of program and information needs conflate. The selection of stakeholders, many of whom have little to no experience with research or evaluation, is of central concern, as the emphasis on a ‘bottom-up’ approach and locally defined priorities and perspectives becomes paramount (Cornwall & Jewkes 1995). At the same time, the diversity of stakeholders involved in the evaluation further challenges the co-construction of knowledge amongst participants, an epistemologically critical component of the participatory approach. As a number of the African studies noted, issues of power amongst local and national stakeholder groups and donor agencies ultimately influenced the level of participation achieved (Holte-McKenzie et al. 2006; Parkinson 2009).
Divergent needs between funding agencies and stakeholders
Whilst there is likely quite a range of approaches to evaluation across donor agencies, as well as varying levels of flexibility in terms of accepted methodologies, perception nonetheless persists that participatory approaches to evaluation lack the rigour and overall credibility of impact measurement evaluations. As a result, a number of the African studies (e.g., Gariba 1998; Bi Niba & Green 2005) note an ongoing and constant ‘tug-of-war’ (Bradley et al. 2002) between adopting a participatory approach and measuring impact, also described as a tension between a focus on process and a focus on outcomes. Whilst the debate seems to be about method choice, for many, the issue has little to do with methods and everything to do with power and politics, with who defines the problem and who collects and analyses the data (Cornwall & Jewkes 1995; McGee & Gaventa 2011). As Cornwall & Jewkes (1995:1667) explain, ‘locating the debate about participatory research within the controversies of the qualitative-quantitative divide obscures issues of agency, representation and power which lie at the core of the methodological critiques from which the development of participatory approaches stem’. Despite this ongoing discord, the inevitable result is that many researchers and evaluators struggle to reconcile and realign their participatory approaches to fit with the demands of donor agencies evaluation architecture (BiNiba & Green 2005; Gariba 1998).
|
Implications for ongoing research and evaluation
|
|
Given the foregoing discussion of our findings based on the eight African studies identified, we are sufficiently persuaded that the African context, and more broadly the international development context, is significantly contextually and culturally distinct so as to warrant a more thorough and comprehensive review of participatory approaches to evaluation. As we noted above, our original themes do not resonate with the findings observed in the selected African studies, specifically in terms of the unique social, historical, cultural, political and economic context of African states and in terms of the rationales for use that seem to better fit the agenda of transformative approaches to evaluation. Thus, whilst we have a fairly comprehensive understanding of participatory evaluation across a broad range of program contexts, our understanding of participatory evaluation in an African or international development context is less developed. At this point, our understanding remains limited to slight nuances across our original thematic analysis of participatory evaluation; we still do not know whether our conceptual framework makes sense across different contexts. Previously (Cousins & Chouinard 2012), we noted the influence of the local context on the participatory process, as evaluator’s background and role, the characteristics of the community and organisational and/or funder and program influences all wield incredible influence on the participatory process and consequences (King & Ehbert 2008; Whitmore 1998). Moving forward, we propose an agenda for ongoing research based on the implications for participatory evaluation in an African and international development context.Firstly, we need to explore what it means to conduct participatory evaluation in an African and/or international development context, looking more closely at contextual influences and how they ultimately shape not only evaluation processes, but also the outcomes themselves. Where does the demand for evaluation originate, and what are the rationales for use? How do local conditions (social, cultural, economic, political, historical) influence participation, and how can we exploit existing sociocultural strengths? Secondly, we need to enhance the number of studies for our inquiry into research on participatory evaluation. The eight studies that we selected were located in east and southern Africa; notably absent were studies in central and northern Africa. Moreover, we did not explore research in francophone Africa, a huge omission on our part. We thus envision a more comprehensive study providing a range of evaluation studies across the African continent, so that we can more concretely and thoroughly explore the multiple dimensions of participation and participatory evaluation in Africa. A study of this kind can really help to further our understanding of what it means to conduct participatory evaluation in socially, politically and economically complex environments. Thirdly, given the strong emphasis on the disparity of information needs between international donor agencies and local government and communities, research should focus on strategies that provide a balance between such seemingly disparate needs. In a previous publication (Chouinard 2013), we noted significant tension between participatory approaches to evaluation with the current vision of evaluation that prevails at the national and international levels, between the needs of local communities and larger national and international bodies. Given the fundamental disparities between the two approaches, the question remains whether participatory evaluation can ever adequately serve the needs of both constituencies. Fourthly, whilst accountability remains a rather contentious concept, particularly given its primacy within the rhetoric of evaluation and program management that prevails in the African (or international development) context, the notion of accountability is even less straightforward. Besides ‘upward’ accountabilities to donors and possibly other host government sponsors (as with co-financing agreements), there are also ‘horizontal’ accountabilities between various state institutions, as well as ‘downward’ accountabilities to local actors and project beneficiaries. Given the complexities of conducting participatory evaluations in an African context, we close with a few enduring and provocative questions that merit further thought and ongoing discussion. Can participatory evaluation coexist amidst such strong donor control tendencies, as well as amidst the centralised and authoritarian governance models that persist in a number of countries, and amidst the practice of planning around pre-set targets and strategies unrelated to local contextual demands? What does participatory evaluation look like in the absence of key facilitating conditions? What are the risks and possible benefits of this approach in a potentially unreceptive and undemocratic environment?
Portions of this article are adapted from Cousins and Chouinard (2012). The article benefited from the thoughtful insights of an anonymous peer reviewer.
Competing interests
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
Authors’ contributions
J.A.C. (University of Ottawa) was lead author and responsible for the intellectual direction of the article and the thematic analysis contained herein. J.B.C. (University of Ottawa) contributed to the conceptual framing of the article.
Bagamoyo College of Arts, Tanzania Theatre Centre, Mabala, R. & Allen, K.B., 2002, ‘Participatory action research on HIV/AIDS through a popular theatre approach in Tanzania’, Evaluation and Program Planning 25, 333–339.BiNiba, M. & Green, J.M., 2005, ‘Major factors influencing HIV/AIDS project evaluation’, Evaluation Review 29(4), 313–330. Bradley, J.E., Mayfield, M.V., Mehta, M.P. & Rukonge, A., 2002, ‘Participatory evaluation of reproductive health case quality in developing countries’, Social Sciences & Medicine 55, 269–282. Chouinard, J.A., 2013, ‘The case for participatory evaluation in an era of accountability’, American Journal of Evaluation 34, 237–253. Chouinard, J.A. & Cousins, J.B., 2009, ‘A review and synthesis of current research on cross-cultural evaluation’, American Journal of Evaluation 30(4), 457–494. Cousins, J.B., 2004, ‘Crossing the bridge: Toward understanding use through empirical inquiry’, in M.C. Alkin (ed.), Evaluation roots: Tracing theorists’ views and influences, pp. 319–330, Sage, Thousand Oaks, CA. Cousins, J.B. (ed.), 2007, Process use in theory, research and practice, New Directions for Evaluation No. 116, Jossey Bass, San Francisco. Cousins, J.B. & Chouinard, J.A., 2012, Participatory evaluation up close: An integration of research-based knowledge, Information Age Publishing, Inc., Charlotte, NC. Cornwall, A. & Jewkes, R., 1995, ‘What is participatory research?’, Social Science & Medicine 41(12), 1667–1676. Estrella, M., 2000, ‘An introduction’, in M. Estrella, J. Blauert, D. Campilan, J. Gaventa, J. Gonsalves, I. Guijt et al. (eds.), Learning from change: Issues and experiences in participatory monitoring and evaluation, pp. 1–14, Intermediate Technology Publications, UK. Estrella, M. & Gaventa, J., 1998, Who counts reality? Participatory monitoring and evaluation: A literature review, IDS Working Paper 70, International Development Studies, UK. Fitzpatrick, J.L., 2012, ‘An introduction to context and its role in evaluation practice’, in D.J. Rog, J.L. Fitzpatrick & R.F. Conner (eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation 135, pp. 7–24, Jossey Bass, San Francisco. Gariba, S., 1998, ‘Participatory impact assessment as a tool for change: Lessons from poverty alleviation projects in Africa’, in E.T. Jackson & Y. Kassam (eds.), Knowledged shared: Participatory evaluation in development cooperation, pp. 64–81, Kumarian Press, West Hartford, CN. Guzmán, B.L., 2003, ‘Examining the role of cultural competency in program evaluation: Visions for new millennium evaluators’, in S.I. Donaldson & M. Scriven (eds.), Evaluating social programs and problems: Visions for the new millennium, pp. 167–181, Lawrence Erlbaum Associates, Mahwah, NJ. Holte-McKenzie, M., Forde, S. & Theobald, S., 2006, ‘Development of a participatory monitoring and evaluation strategy’, Evaluation and Program Planning 29, 365–376. Jackson, E.T. & Kassam, Y. (eds.), 1998, Knowledge shared: Participatory evaluation in development cooperation, International Development Research Centre, Ottawa. Johnson, K., Greenseid, L.O., Toal, S.A., King, J.A., Lawrenz, F. & Volkov, B., 2009, ‘Research on evaluation use: A review of the empirical literature from 1986 to 2005’, American Journal of Evaluation 30(3), 377–410. Labin, S., Duffy, J.L., Meyers, D.C., Wandersman, A. & Lesesne, C.A., 2012, ‘A research synthesis of the evaluation capacity building literature’, American Journal of Evaluation 33, 307–338. King, J.E. & Ehlert, J.C., 2008, ‘What we learned from three evaluations that involved stakeholders’, Studies in Educational Evaluation 34, 194–200. Maclure, R., 2006, ‘Pragmatism or transformation? Participatory evaluation of a humanitarian education project in Sierra Leone’, The Canadian Journal of Program Evaluation 21(1), 107–129. Mark, M.M., 2008, ‘Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation, ‘in N. Smith & P.R. Brandon (eds.), Fundamental issues in evaluation, pp. 111–134, Guilford, New York. Marsden, D. & Oakley, P., 1990, ‘Conclusion’, in D. Marsden & P. Oakley (eds.), Evaluating Social Development, pp. 137–156, Oxfam, Oxford. McGee, R. & Gaventa, J., 2011, Shifting power? Assessing the impact of transparency and accountability initiatives, IDS Working Paper No. 383, Institute of Development Studies, UK. Mullinix, B.B. & Akatsa-Bukachi, M., 1998, ‘Participatory evaluation: Offering Kenyan women power and voice’, in E.T. Jackson & Y. Kassam (eds.), Knowledge shared: Participatory evaluation in development cooperation, Kumarian Press, West Hartford, CN. Parkinson, S., 2009, ‘Power and perceptions in participatory monitoring and evaluation’, Evaluation and Program Planning 32, 229–237. Patton, M.Q., 2008, Utilization-focused evaluation, 4th edn., Sage, Thousand Oaks, CA. Smith, N.L., 1993, ‘Improving evaluation theory through the empirical study of evaluation practice’, Evaluation Practice 14(3), 237–242. Trickett, E.J. & Ryerson Espino, S.L., 2004, ‘Collaboration and social inquiry: Multiple meanings of a construct and its role in creating useful and valid knowledge’, American Journal of Community Psychology 34(1/2), 1–69. Whitmore, E., 1998, ‘”We need to rebuild this house”. The role of empowerment evaluation of a Mexican farmers’ cooperative’, in E.T. Jackson & Y. Kassam (eds.), Knowledge shared: Participatory evaluation in development cooperation, pp. 217–230, Kumarian Press, West Hartford, CN.
|
Email this article 
Hide
Show all
