Abstract
Background: This article pays tribute to Dr Sulley Gariba, a pioneer of global and African evaluation, by examining the growing movement of culturally responsive evaluation (CRE) and culturally responsive Indigenous evaluation (CRIE) in the Asia Pacific.
Objectives: The article aims to describe initiatives to increase awareness and uptake of CRE and CRIE as well as good practice examples across the Asia Pacific.
Method: A literature review was conducted guided by evaluation, CRE and CRIE experts.
Results: While Western-derived evaluation practice still dominates the region, particularly in international development, there are increasing examples of CRE and CRIE in the Asia Pacific. The movement to mainstream CRE and CRIE is redressing decades of colonisation and power imbalances in the Asia Pacific and such approaches will in turn, contribute to decolonising policy and practice.
Conclusion: Integrating CRE and CRIE into mainstream evaluation practice in the Asia Pacific will play a vital role in ensuring that evaluation contributes to meaningful, culturally relevant and appropriate improvements in people’s lives.
Contribution: This article contributes to inter-regional awareness of, and learning about, CRE and CRIE by sharing perspectives and good practice from the Asia Pacific.
Keywords: culturally responsive evaluation; Asia; Pacific; Indigenous; world views; culture; decolonisation; localisation.
Introduction
Background
Dr Sulley Gariba was a pioneer for global and African evaluation within the EvalPartners family. He was a strong advocate for Making Evaluation Our Own and Made in Africa Evaluation. This article pays tribute to Dr Gariba by examining the growing movement to improve the cultural relevance of evaluation and in his words, make evaluation our own in the Asia Pacific by raising awareness and increasing the application of culturally responsive evaluation (CRE) and culturally responsive Indigenous evaluation (CRIE).
In this article, culture is broadly defined as ‘the shared living experiences of people’, including groupings based of ethnicity, religion, class, age, gender, sexual orientation, disability, organisations and institutions (Aotearoa New Zealand Evaluation Association [ANZEA] 2011). The intersectionality between culture, evaluation and values has been described this way:
A common thread between culture and evaluation is the concept of values. Culture shapes values, beliefs, and world views. Evaluation is fundamentally an endeavour of determining values, merit and worth. (SenGupta, Hopson & Thompson-Robinson 2004, n.p.)
This article describes examples across the region where localised values inform the design and conduct of evaluation. There are many terms used to describe this type of process including CRE, CRIE, localising evaluation, evaluating using local world views or local wisdom, culturally sensitive evaluation, made in Asia evaluation or made in the Pacific evaluation. In this article, the terms CRE and CRIE are primarily used, but there are also references to the other terms listed here.
Culturally responsive evaluation and CRIE both contest the assumption that Western-derived evaluation methods can be universally applied and argue that evaluations need to ensure the cultural relevance of the methods being used (Chouinard & Cram 2020; Howe 1994). CRIE, which is described by Waapalaneexkweew (Bowman, Mohican/Lunaape) and Dodge-Francis (2018) as a ‘partner to CRE’:
[U]ses traditional knowledge and contemporary Indigenous theory and methods to design and implement an evaluation study, so it is led by and for the benefit of Indigenous people and Tribal nations. (n.p.)
This article describes examples of CRE and CRIE from across the sub-regions of South Asia, Southeast and East Asia and the Pacific in order to demonstrate the diversity of forms that CRE and CRIE can take. Some of the examples recognise pluriversalism or the interconnectedness between Western and/or colonial and community and/or Indigenous cultures and use hybrid approaches that respect and intermingle Western and localised evaluation approaches and methods (Jordan & Hall 2023). In this article, the analysis primarily draws upon the use of CRE in international development practice in the region.
A movement to strengthen and localise evaluation practice by increasing the application of CRE and CRIE in the Asia Pacific is underway. Such a movement strengthens evaluation practice by ensuring that communities involved in an evaluation are empowered to lead and/or authentically participate in the process and that the evaluation is driven by the values and priorities of that community. In doing so, the literature points out that such a movement contributes to decolonisation and self-determination through, ‘valuing, reclaiming, and foregrounding Indigenous voices and epistemologies’ (Darder et al. 2014). In the literature on CRE and CRIE, decolonisation refers to a reversal of:
[T]he changes brought about by colonisation … reclaiming the power and control, the assertion of rights and values … and the ‘breakdown’ of the structures and systems put in place by previous colonizers.
In the international development sector, adoption of CRE and CRIE also redresses postcolonial and other power disparities that have resulted from decades of practice. The literature describes how much of this practice has reinforced Western countries’ positions of power, whereas CRE and CRIE enable significant agency for the communities involved to seek out development benefits that are in their own interests (McGregor 2009; Wilson 2019).
Objective
The purpose of this article was to highlight trends and practical approaches to adopting a Made in Asia Pacific approach to evaluation, drawing upon good practice and innovative examples of CRE and CRIE.
Research methods and design
The Working Group for this research, convened by the Asia Pacific Evaluation Association (APEA) and consisting of evaluation and/or CRE and CRIE experts from across the region, were consulted to help identify peer-reviewed and grey literature on CRE and CRIE practice in the Asia Pacific. A limited literature review was then conducted and data analysed to draw out emergent themes. Good practice examples were chosen so that the three sub-regions of South Asia (Bengali tradition in Bangladesh and India), Southeast and East Asia (Buddhism that is practised across the region) and the Pacific (Talanoa practised in several Pacific Island nations) were represented. However, the choice of good practice examples per sub-region was largely dependent on the data available within the timeframe for the research. The draft article was reviewed by those who contributed data in order to ensure that the good practice examples were represented accurately and respectfully.
Limitations
The examples included in this article were based on the evidence that was able to be collected during the limited research period in 2023 and were not intended to be comprehensive nor representative of work in CRE and CRIE across the region as a whole. For example, efforts to progress CRE and CRIE by government donors funding work in this region were not discussed. In addition, where there was a critique of evaluation or other practices in the article, it was not always possible to include a range of perspectives.
Results
There is a significant history of using Western-derived evaluation techniques in the Global South for international development programmes. Castellano points to the design of research and evaluation as often having resulted in a justification of a colonial agenda and reinforcement of the power imbalance between Western structures and communities in the Global South (2004).
In more recent times, there has been a growing movement of localising research and evaluation in the Asia Pacific region. Initiatives and professional organisations working in evaluation in the region have shared guidance and support with evaluators in order to improve their awareness and application of CRE and CRIE. For example, in 2023, APEA released the Code of Ethics for Evaluation in the Asia Pacific Region. This includes principles that: highlight a need to reflect differences related to ethnicity, gender, religion and other types of diversity among all involved in or affected by an evaluation and that require evaluators to be culturally competent, sensitive and understand the cultural context of the evaluation (Asia-Pacific Evaluation Association 2023). Asia Pacific Evaluation Association has also created a practical guide for communities on the existing knowledge and experiences of indigenous communities and how evaluators have worked with these communities in culturally sensitive ways in the Asia Pacific.
Culturally responsive practice in the Asia Pacific
In this section, good practice examples of research and implementation of CRE and CRIE in the Pacific, Southeast Asia and South Asia are described. These are neither intended to paint a comprehensive picture nor to be representative of the region as a whole.
Pacific
There are 15 independent nations in addition to tens of thousands of islands, islets and atolls in the Pacific region. There is strong practice of identifying and applying research and evaluation that embody the culture, history, traditions and relationships of diverse communities and countries in the region. There is also significant literature on the processes of conceptualising and applying such practice.
One localised approach to research and evaluation that is well known in the Pacific is talanoa. Rooted in a long oratory tradition, talanoa has emerged as approaches to research in countries in the Pacific that involve ‘engaging in dialogue with, or telling stories to each other absent [of] concealment of the inner feelings and experiences that resonate in our hearts and minds’ (Halapua 2008). Talanoa is a concept recognised in many Pacific countries, including Fiji, Samoa, Solomon Islands, Niue, Hawai’i, the Cook Islands and Tonga but in each context within and between these countries it is conceived and applied differently. To an observer unfamiliar with talanoa, it could seem like it is merely a Pacific form of ‘informal open-ended interviews’, but talanoa is actually a multidimensional process involving empathtic understanding and exchange between the participants and the researcher. Vaioleti describes talanoa research as, ‘holistically intermingl[ing the] researchers’ and participants’ emotions, knowledge, experiences, and spirits’ (2006).
Conducting talanoa research and employing it during evaluations involves a deep understanding and practice of empathy between the researcher and the participants (intersubjective empathy), which is conceived as attachment to living kin, deceased ancestors and land, and how a life is lived, which in turn can influence a person’s health or ecological impacts such as the success of a crop. This requires moving away from a Western-derived conceptualisation of empathy, which is subjective and could be viewed as impersonal and reductionist. Subjective empathy is one-directional with the researcher attempting to ‘walk in the shoes’ of the participant, albeit through a passive ‘subject-to-object’ relationship. In literature about talanoa, it is argued that if research methodologies enable intersubjective empathy, then they ‘bring us closer to understanding others’ emotions, feelings and attitudes [and] provide us with a more complex and intimate understanding of cultural and social forms’ (Farrelly & Nabobo-Baba 2014).
‘Our research methodologies must reflect the knowledge making and knowledge sharing of our participants, not the other way around’ (Farrelly & Nabobo-Baba 2014).
The literature talks about how Talanoa research methods are decolonising research and evaluation by reflecting local world views and values throughout the process. Such methods are also subverting the power imbalance between researcher or evaluator and participants that often results using Western research methods by building a shared process involving knowledge, sensation and emotion. When research findings truly reflect the lived experience, values and aspirations of participants, then by extension, policy and practice informed by this research can also be decolonised.
Southeast Asia
The Asia Pacific region is home to 99% of the world’s Buddhists, who mainly live in Southeast and East Asian countries, such as Cambodia, Thailand, Myanmar, China, Vietnam, South Korea, Japan, Sri Lanka, Singapore and Taiwan. Buddhism is based upon the teachings of Siddhartha Gautama, or Buddha, in India 2500 years ago and remains a significant global influence today (Berzin 1996). Research in the region by Dinh, Worth and Haire (2019) sought to identify how a commonly used Western-derived evaluation approach – the most significant change (MSC) technique could be adapted to take into account a Buddhist world view.
Most significant change is a form of participatory monitoring and evaluation that involves the collection of significant change stories emanating from a grassroots level and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff (Davies & Dart 2005).
In one example from this research, in Buddhist teaching, there is the principle of impermanence. This principle describes the world as in a constant state of change in which everything arises, stays in a state of flux and then passes away. Most significant change is a dynamic and adaptive technique, which, in this instance, is quite compatible with a Buddhist world view as it recognises that change is continuous and captures stories of change as points in time.
However, Buddhism incorporates concepts of causality or ‘how things happen, how change occurs, how events relate’ that are quite distinct from those used in most Western-derived evaluation methods (Macy 1978). The Buddhist premise of causation is that cause and effect are interdependent in the past, present and future (Macy 1991). In Buddhist teachings, this means that cause and effect co-arise together, and they are recursive, not unidirectional (Bukkyō Dendō Kyōkai 1966). This means that what is originally considered to be a cause could come to be considered as an effect, and what is originally considered to be an effect could come to be considered as a cause (Russon 2014). The research suggested a modified approach to MSC, involving the use of a story collection template that took into account these Buddhist teachings and could encourage participants to identify multiple causes to an effect and how the effect has simultaneously influenced the cause.
This research is part of a wider body of work on culturally responsive evaluation, which looked at other Southeast Asian world views and how they could be applied to evaluation. In some cases, these world views were largely compatible with Western-derived evaluation approaches and methods and in other cases there were stark contrasts. The research argued that in ensuring an evaluation method or approach is appropriate for a context, firstly the ontologies and epistemologies of the world view(s) of the community participating in the evaluation as well as those of an evaluation method or approach being considered need to be thoroughly understood. Any incongruities between these world views can then be reduced or eliminated by modifying the method or approach. The research suggested that a similar hybrid approach, which recognises pluriversalism, or the interconnectedness between Western/colonial and community/Indigenous cultures (Jordan & Hall 2023) could be used to reflect multiple world views in an evaluation method. It argued that in the complex, globalised and dynamic societies in which evaluators now work, such hybrid methods are very much needed (Dinh, Worth & Haire 2021).
South Asia
A study of the Bengali practice of adda in India, Bangladesh and the Bengali diaspora explored the utility of this practice as a localised approach of participatory evaluation. Adda is defined as ‘a semi-structured and informal congregation of folk who by and large belong to similar world view’ (Pathak 2022). There were three central characteristics of an adda that were described. Firstly, that despite the fact that those who congregate hold by and large the same world view, the adda is paradoxically a space in which views are contested. This contestation occurs through assessment of the actions and behaviours of each other, the efficacy of institutions and interpersonal actions. Secondly, assessment is conveyed through particular forms of expression including, ‘ridiculing, provoking, gossiping, the hilarity of expressions, and assessing with judgmental tenor with an iota of shared empathy’ (Pathak 2022). Thirdly, discussion centres on personal-material benefit rather than ideals or the interrelationship of spiritual or physical dimensions.
The practice of adda is rooted in the ancient Indian philosophy of Lokayata. This philosophy promotes a rationalist, empiricist, material means of verification (or evaluation) based upon what is seen and shown, rather than any spiritual or metaphysical dimensions. What is seen and shown includes both cognition of intellect and reason as well as emotion and sentiment.
In considering adda as a particular world view of participatory assessment, Pathak observes that in conceptions of self, Lokayata merges intellect with emotions and intuition with empirical observation. It also embodies the capacity of the self in making choices and decisions. Thus, the evaluator (self) involved in an assessment of others would be ready to ‘converse and quarrel at once’ as well as to self-evaluate (Pathak 2022).
The study observes that other philosophies in South Asia give rise to similar forms of assessment. Thus, Pathak finds that adda not only provides potential for a participatory evaluation approach that is rooted in Bengali world views and philosophies but also that similar evaluation approaches could be derived from other philosophies in South Asia.
These examples demonstrate that there is good practice and a growing body of literature on CRE and CRIE in the Asia Pacific. There is also a need to keep building this body of literature in order to share experiences in applying CRE and CIRE. This building of an evidence base and sharing of experience is important to strengthen the understanding and capacity of evaluators in the region in CRE and CRIE and also to demonstrate the importance and value of such approaches to commissioners of evaluation.
Conclusion
There is a growing awareness within the evaluation community in the Asia Pacific that Western-derived evaluation approaches cannot be universally applied and may in fact result in inappropriate programming and development practice. Alongside this awareness there is a growing movement in the region of evaluators, researchers, commissioners of evaluation, Voluntary Organizations for Professional Evaluation (VOPEs) and regional and international organisations who are researching, applying and supporting the implementation of CRE and CRIE. This movement continues to build upon the legacy of Dr Sulley Gariba and others in making evaluation our own and improving the cultural responsiveness of evaluation practice.
A significant proportion of evaluations conducted in the Asia Pacific still require and use Western-derived evaluation approaches and methods that do not sufficiently incorporate local world views. Mainstreaming CRE and CRIE in the Asia Pacific will in many cases require a significant cultural shift by evaluation commissioners and many evaluators. Such a shift will redress decades of colonisation and imbalances of power and help to ensure that evaluations in turn decolonise policy and practice in ways that will truly benefit communities. The movement to mainstream CRE and CRIE in the Asia Pacific is underway, and the evaluation community has a responsibility to ensure that this mainstreaming is achieved so that evaluation contributes to meaningful, culturally relevant and appropriate improvements in people’s lives.
Acknowledgements
The author wishes to acknowledge and thank the Dr Sulley Gariba Memorial Lecture Working Group: Randika De Mel, Madhuka Liyanagamage, Stuart Coupe, Nicola Theunissen and Mari Honjo for their guidance, support and review throughout the research for and writing of the article.
Competing interests
The author declares that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Author’s contributions
K.T.D. declares that they are the sole author of this article and responsible for the conceptualisation, methodology, formal analysis, investigation, writing of the original draft, visualisation, project administration, writing, review and editing.
Ethical considerations
This article followed all ethical standards for research without direct contact with human or animal subjects.
Funding information
This research received funding from EvalPartners.
Data availability
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Disclaimer
The views and opinions expressed in this article are those of the author and are the product of professional research. It does not necessarily reflect the official policy or position of any affiliated institution, funder, agency, or that of the publisher. The author is responsible for this article’s results, findings, and content.
References
Aotearoa New Zealand Evaluation Association, 2011, Evaluator competencies, viewed 8 August 2023, from https://anzea.org.nz/about-us/resources/.
Asia-Pacific Evaluation Association, 2023, Code of ethics for evaluation in the Asia Pacific region, viewed from https://asiapacificeval.org/wp-content/uploads/2023/09/Code-of-Ethics-for-the-Asia-Pacific-Region_Sep_23.pdf.
Berzin, A., 1996, Buddhism and its impact on Asia, Asian Monographs 8, Center for Asian Studies, Cairo University, Cairo.
Bukkyō Dendō Kyōkai, 1966, The teaching of Buddha, Bukkyo Dendo Kyokai, Tokyo.
Castellano, M.B., 2004, ‘Ethics of aboriginal research’, Journal of Aboriginal Health 1, 98–114.
Chouinard, J.A. & Cram, F., 2020, Culturally responsive approaches to evaluation, Sage, Thousand Oaks, CA.
Darder, A., Mirón, L.F., Denzin, N.K., Lincoln, Y.S., Guba, E., Olesen, V. et al., 2014, ‘Locating the field: Performing theories of decolonizing inquiry’, in N.K. Denzin, Y.S. Lincoln & L. Tuhiwai-Smith (eds.), Handbook of critical and indigenous methodologies, pp. 21–30, Sage, Thousand Oaks.
Davies, R. & Dart, J., 2005, The ‘most significant change’ (MSC) technique: A guide to its use, s.n., s.l.
Dinh, K., Worth, H. & Haire, B., 2019, ‘Buddhist evaluation: Applying a Buddhist world view to the most significant change technique’, Evaluation 25(4), 477–495. https://doi.org/10.1177/1356389019841654
Dinh, K., Worth, H. & Haire, B., 2021, ‘Culturally responsive evaluation: A theoretical approach relevant to South East and East Asia and guiding principles for Vietnam’, Doctoral thesis, University of New South Wales.
Farrelly, T. & Nabobo-Baba, U., 2014, ‘Talanoa as empathic apprenticeship’, Asia Pacific Viewpoint 55(3), 319–330. https://doi.org/10.1111/apv.12060
Halapua, S., 2008, Talanoa process: The case of Fiji, viewed 15 August 2023, from http://unpan1.un.org/intradoc/groups/public/documents/UN/UNPAN022610.pdf.
Howe, D., 1994, ‘Modernity, postmodernity and social work’, British Journal of Social Work 24(5), 513–532.
Jordan, L.S. & Hall, J.N., 2023, ‘Framing anticolonialism in evaluation: Bridging decolonized methodologies and culturally responsive evaluation’, Journal of MultiDisciplinary Evaluation 19(44), 102–116. https://doi.org/10.56645/jmde.v19i44
Macy, J., 1978, ‘Interdependence: Mutual causality in early Buddhist teachings and general systems theory’, Doctoral thesis, Syracuse University, viewed 9 February 2018, from https://surface.syr.edu/rel_etd/64.
Macy, J., 1991, Mutual causality in Buddhism and general systems theory: The dharma of natural systems, State University of New York Press, Albany.
McGregor, A., 2009, ‘New possibilities? Shifts in post-development theory and practice’, Geography Compass 3(5), 1688–1702. https://doi.org/10.1111/j.1749-8198.2009.00260.x
Pathak, D.N., 2022, ‘Assessment-Adda: A Lokayat approach to another South Asia’, in S. Zaveri & R. Nandi (eds.), Local wisdom matters: Reflections on evaluation theory and practice from South Asia, pp. 80–91, Gender and Equity, Network of South Asia, New Delhi.
Russon, C., 2014, ‘Impact evaluation based on Buddhist principles’, Journal of Multidisciplinary Evaluation 10(23), 73–77. https://doi.org/10.56645/jmde.v10i23.400
SenGupta, S., Hopson, R. & Thompson-Robinson, M., 2004, ‘Cultural competence in evaluation: An overview’, New Directions for Evaluation 102, 5–19. https://doi.org/10.1002/ev.112
Vaioleti, T., 2006, ‘Talanoa research methodology: A developing position on Pacific research’, Waikato Journal of Education 12, 21–34. https://doi.org/10.15663/wje.v12i1.296
Waapalaneexkweew (Bowman, N., Mohican/Lunaape) & Dodge-Francis, C., 2018, ‘Culturally responsive indigenous evaluation and tribal governments: Understanding the relationship’, in F. Cram, K.A. Tibbetts & J. LaFrance (eds.), Indigenous evaluation. New Directions for Evaluation, vol. 159, p. 22.
Wilson, B.R.L., 2019, ‘Scotland, Malawi and the post-development critique: An analysis of power and equality in international development’, Doctoral dissertation, viewed 20 October 2020, from https://theses.gla.ac.uk/74298/.
|
Email this article 
Hide
Show all
