An evaluator can provide eight positive findings; however, if BUT is used as a connector in the sentence before negative evaluation findings are shared, everything up to the word BUT will likely be forgotten. Try to use the word AND if you must use a connector – or, even better, just start a new sentence. (p. 270)
This sentence captures what is unique and useful about Donna Podems’s new book, Being an evaluator: Your practical guide to evaluation. The book is about the experience of being an evaluator. It is unlike a typical evaluation textbook which might have a chapter on each of a range of evaluation approaches, or one chapter per methodology, or focus on one evaluation paradigm. This book takes you on a journey in which you explore the processes of both thinking and feeling while evaluating. It is in two parts: (1) Doing evaluation and thinking evaluatively and (2) Working as an evaluator and exploring evaluation. It takes you through both steps and reflections we evaluators need, by sharing a diversity of approaches while asking questions and offering advice from real experience. In this process, she invites us, the readers, to look honestly at our evaluation practice. It would not be possible to describe all of the book’s content or tone so I’m going to mention some aspects I found particularly useful and delightful.
Words
‘But’ is not the only word she warns about. In essence, her message is ‘words matter’. In her chapter ‘Starting the evaluation journey’ she notes how critical it is to be clear on the problem that the initiative being evaluated is trying to address. She immediately alerts us, however, that it’s possible your client or a stakeholder might find the term ‘problem’ offensive or abrasive in which case she suggests we substitute it, for example calling it a ‘challenge’. She notes how:
something we say in one situation may be appropriate, and in another it may be offensive or just confusing. Changing a word or sentence, or an accent or emphasis on a word in a sentence, may elicit a completely different response. (p. 46)
She follows this with an illustrative video, bringing her message home. Importantly, in the context of African evaluation, she extends the issue beyond ‘words’ to surfacing challenges related to language – that even within English, what words mean differs across countries and continents, something she has learnt well as an American living in South Africa; that body language influences how people receive what you say, as does the tone with which you say it; that cultural meanings are hidden inside words and need to be collectively discussed and named. She offers a tool – ‘Designing a language- and culture-appropriate data collection process and tool: a four-step strategy’ – that guides the evaluator through a process of building a shared terminology with their team and client that is most appropriate to the setting.
She offers us some concepts of her own such as ‘suitcase words’ – arguing that we and our clients use words that can mean very different things, and that ‘an evaluator cannot assess results that lack definition’, hence the need to ‘unpack’ them. Examples she offers include culture, behaviour change, empowerment, capacity building, improved knowledge, sustainability, beneficiaries. She provides guidance in how to do the unpacking by using one or both of the following prompts, as appropriate:
- You mentioned [insert suitcase word]. Please tell me: what is being done, and exactly what do you expect to see happen? To whom?
- Can you describe what [insert suitcase word] means, without using the word in the description? (p. 139)
In particular she notes how obsessing about words can prevent people from really thinking about what they need to know and from making meaning. She describes an experience familiar to many evaluators in which organisations are scrambling to make sense of the different words each of their funders use – is it an output? an outcome? an impact? – and how they try to fit their monitoring and evaluation concepts into conflicting framings of each of their funders. She proposes leaving the labelling until last, rather using a simple process for working with clients to articulate the process of change, without worrying at that moment about which aspects of that process are their activities, and which are actions of others they have influenced. This ensures that what to call these does not get in the way of articulating the client’s thinking and assumptions about their intervention and what it hopes to achieve.
She proposes that one then makes explicit the question of language by discussing it with clients – come to an agreement on how to understand each word for the purposes of this intervention, and how to use it.
Power
At the heart of this particular issue is of course power – and the importance of understanding who holds power in relation to these words, such as funders. What stands out in her overall approach is that she not only names power, but argues that the evaluator must engage, in this case with terminology, ‘to see that power is more evenly distributed’.
Her overall framing of evaluation puts power at the centre. After sharing various understandings of evaluation, she offers her own: ‘There are three fundamental answers to the question “what is evaluation?” Evaluation provides systematic, transparent, and logically collected evidence; it offers a value judgement; and it is political’. She offers ‘nine necessary “who” questions’ regarding who has the power in evaluative process decisions and applies these such that every chapter asks questions about power and how to name it, navigate it, and try for equity – in the culture of an evaluation team, in how the evaluator engages their client, in what methodologies are used and how they are used, in who gets to make meaning of findings. This systematic attention to equity is not unexpected given Podems’s long-standing work on feminist evaluation; but it is very helpful to see these issues named and to get ideas about how to address them at different moments of the evaluation process and especially when dynamics in an evaluation process become difficult. Two of her chapters, ‘Thinking about values’ and ‘Thinking about power, politics, culture, language and context’, surface some of the hidden moments in which values shape the purpose, process and use of evaluation and offer both her own experiences in navigating these, and some exercises to help the reader get to grips with the challenges.
Honesty, humour and help
The book demonstrates Podems’s commitment to equity in the way in which she presents her own challenges in evaluation. In every chapter, she shares experiences she has had where things have gone well, and when they have not. She does not protect herself or set herself up as a know-all. She talks about when her responses have worked well, and when she has been in situations where power dynamics were such that she could not influence the process in ways that would have made the evaluation both more effective and more equitable. I found myself laughing repeatedly as she described familiar and often horrible situations, and the challenges of making one’s way through them. In one example, in her ‘Dirty laundry’ chapter, the client complains about findings, saying they’re not what he had expected, she notes that ‘screaming is one strategy, but is not very professional, so let’s dismiss that one’. She then proceeds to offer some other probably more effective options. At the same time as challenging power and discriminatory approaches in evaluation, she is also pragmatic without letting us off the ethical hook, a necessary skill for those of us being commissioned by others but committed to using evaluation in support of a just society.
The book has a few features in every chapter that help her to draw us in and give us practical help.
Every chapter includes questions that you or I might have asked, and to which she responds, often providing a range of scenarios. Questions run the gamut, for example:
I always work in teams when I conduct an evaluation. I find that sometimes our teams are just rushed out into the field to collect data, and we do not have time to talk about the evaluation approach. Any suggestions? (p. 344)
It seems that if I have a finding that the client is not expecting, it is less likely to be accepted, but if I have a finding the client expects, they question what added value I have brought the process. How do I deal with this challenge? (p. 344)
She includes exercises you can use to gain clarity on issues, thus supporting the actual practice of evaluation.
She ends each chapter with a section entitled ‘Our conversation: Between you and me’ where she asks us questions to help us engage the chapter’s material from where we, as readers, are at and what we need to explore.
Each chapter has an excellent section on where to find further information and debate on the issues, which are organised according to time, for example ‘Have a few minutes?’ ‘Have an hour?’, ‘Have a few hours?’, ‘Have a few hours a night over several weeks?’, so that you can find texts or videos that address your questions within the time you have. Similarly, her chapter on ‘The scholarly side of being an evaluator’ has an invaluable table of materials she has found most accessible for understanding specific paradigms, methods, concepts or processes that allow us as readers to enter in relation to where our own interest or work focus lies.
From my perspective
Given my own background in public health and because my evaluation work focuses on evaluating advocacy for social change, I would have liked to see more fleshing out of some issues.
The book has a section clearly distinguishing research from evaluation, offering three criteria. The first is that valuing findings is not something researchers are asked to do. The second is about purpose since evaluation is intended ‘to improve or judge’ an intervention, while ‘researchers ask questions on behalf of the larger scientific community’. The third is about approaches, noting that evaluation is guided by its own approaches, theories and models. I think this is very helpful as we evaluators are frequently challenged by researchers on method choice or process and need to be clear on why an evaluation approach may differ from a research approach. But it happens that there is a whole field of applied public health research that, usually working with and shaped by the concerns of a community or a health institution, aims to support local identification of problems, and testing of interventions in real time to strengthen them in local context. Such work carries all the markers of evaluation that Podems proposes, and perhaps my concern would be resolved by just calling it evaluation! However, it makes me want to propose that one sees research and evaluation on more of a spectrum than an either/or.
The other area that I’d like to have been given more attention is the growth of outcomes-focused evaluation approaches. Podems does note the perspectives brought by outcome mapping. However, she makes no mention of the outcome harvesting approach which has taken the global social change evaluation community by storm because it challenges the assumption that an intervention’s influence can be predicted and set in stone through indicators, and focuses rather on what changes actually take place, then looking backwards to see if and how a particular intervention contributed towards those changes. In this way, it undercuts many of the damaging power dynamics between funders and grantees, and the judgements from above. Its uptake into the mainstream of evaluation, and increasing recognition by philanthropy, is evidence of a growing focus on questions of values and power which this book centres on. I also noticed, and I’m aware of the irony, since Podems pushes us not to fixate on words, that Podems recognises changes in knowledge or attitudes as outcomes, whereas outcome harvesting wants to see those changes reflected in shifts in behaviour, relationships, actions, activities, agendas, policies or practices, before it recognises them as outcomes. This is a huge challenge for interventions that conduct training or ‘capacity development’ as it requires their evaluation approaches to explore how participants behave well after the training is over. I expect that Podems would bring invaluable insights on this issue, but perhaps that is for a later book.
Finally, the book is heavily reliant on Western literature. It is short on references to African thinking and evaluation work. This is unsurprising given the likely unavailability of the book in bookshops on our continent so that it is essentially talking to an American and perhaps European audience. This is to be expected given, as she so cogently explains, the power dynamics of the evaluation industry and, I would add, of the knowledge-production industry. Podems cannot be held responsible for this, but it would have been great to see more examples in her readings of material that might resonate more closely with people from other parts of the world. In contrast, her actual case material, from her work experience, draws on experiences and locations in different parts of the globe.
A decolonial approach
Notwithstanding this criticism, to me Podems’s book exemplifies attention to some of the key tenets of the African Evaluation Guidelines, such as giving attention to the beliefs- and value-laden nature of both development and evaluation, and respecting the rights, dignity and human value of stakeholders. It also represents a significant contribution to the Made in Africa Evaluation Concept (Chilisa 2015). For example, in its naming of the extractive nature of evaluation practice, its recognition that:
culture, living experience and indigenous knowledge systems must be used to conceptualise the realities to be evaluated and to come up with techniques through which these realities can be known. (Chilisa 2015:20)
But most importantly, it takes us on a journey of surfacing our philosophical and practical assumptions and reflecting on how to do the work in a way that redistributes power.
In conclusion, I think Podems’s book’s content and approach will resonate with evaluators globally. I have already applied a number of ideas gained from reading it. I expect it will be my mentor going forward, offering sage advice, practical suggestions, and lots of humour when the going gets rough.
Reference
Chilisa, B., 2015 (31 August), A synthesis paper on the made in Africa Evaluation Concept, African Evaluation Association, Accra.
|
Hide
Show all
Email this article (Login required)
