Article Information

Authors:
Carla Sutherland1
Barbara Klugman2

Affiliations:
1Gender and Sexuality Law Centre, School of Law, Columbia University, USA

2School of Public Health, University of the Witwatersrand, South Africa

Correspondence to:
Barbara Klugman

Postal address:
70, 7th Street, Parkhurst, Johannesburg, 2193, South Africa

Dates:
Received: 23 June 2013
Accepted: 18 Aug. 2013
Published: 01 Oct. 2013

How to cite this article: Sutherland, C., Klugman, B., 2013, ‘Finding common ground: A participatory approach to evaluation’, African Evaluation Journal 1(1), Art. #39, 10 pages. http://dx.doi.org/10.4102/
aej.v1i1.39

Copyright Notice:
© 2013. The Authors. Licensee: AOSIS OpenJournals.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Finding common ground: A participatory approach to evaluation
In This Original Research...
Open Access
Abstract
Abstrait
Introduction
Background
Methodology
Theoretical framings
Theory of change based on complexity analysis
Application to SOGI grassroots funding
Putting it all together
   • Utilisation-focused evaluation
Implementation
Results
Organisational development
Organisational priorities
Conclusion
Acknowledgements
   • Competing interests
   • Authors’ contributions
References
Footnote
Abstract

Background: This article describes the efforts of a group of donors and activists to collectively develop a national base line on organisations working for human rights in relation to sexual orientation and gender identity (SOGI) in Kenya to develop an ongoing monitoring and evaluation process.

Objectives: The purpose of the base line was to support both activist strategising and ongoing reflection, and more effective donor collaboration and grant making.

Method: Drawing on interviews with key stakeholders, the authors examined the dominant approach to funding and evaluation on social change globally. They analysed the impact of this dominant approach on developing and sustaining a SOGI movement in Kenya. They developed an alternative theory of change and participatory methodology and worked with a range of donors and SOGI organisations to conceptualise and support the collaborative collection of information on four themes: legislation and policy, organisational mapping, political and cultural context, and lived experiences of lesbian, gay, bisexual, transgender and intersex people.

Results: This was a useful process and tool for activists and donors to develop a shared understanding of the current context and capacities influencing efforts to promote SOGI rights. It served as a basis for improved strategising and participants expected it to prove useful for monitoring progress in the longer term.

Conclusion: This theory of change and participatory approach to base line development could be helpful to donors, activists and monitoring and evaluation specialists concerned with supporting social change in the region and globally.

Abstrait

Trouver un terrain d’entente: Une approche participative à l’évaluation

Contexte: Cet article décrit les efforts de donateurs et d’activistes qui développent ensemble une base nationale sur les organisations travaillant en faveur des droits humains en relation avec l’orientation sexuelle et l’identitié de genre (OSIG) au Kenya afin de développer un processus de suivi et d’évaluation continu.

Objectifs: L’objectif de cette base était de soutenir tant l’élaboration de stratégies et la réflexion continue des activistes, qu’une collaboration et un octroi de subventions des donateurs plus efficaces.

Méthode: En se fondant sur des entretiens avec des acteurs clés, les auteurs ont examiné l’approche dominante au financement et à l’évaluation du changement social dans le monde. Ils ont analysé l’impact de cette approche dominante sur le développement et le maintien d’un mouvement OSIG au Kenya. Ils ont élaboré une méthodologie alternative de la théorie du changement et participative et ont travaillé avec une palette de donateurs et d’organisations OSIG afin de conceptualiser et de soutenir la collecte collective d’informations sur quatre thème: la législation et la politique, la cartographie organisationnelle, le contexte politique et culturel, et les expériences vécues par les personnes lesbiennes, gays, bisexuelles, transgenres et intersexuées.

Résultats: Il s’agissait d’un processus et d’un outil utiles pour les activistes et les donateurs afin de développer une compréhension commune du contexte actuel et des capacités influençant les efforts visant à promouvoir les droits OSIG. Il a servi de base à l’amélioration de l’élaboration de stratégies et les participants pensaient qu’il serait utile pour suivre les progrès réalisés à plus long terme.

Conclusion: Cette approche à la théorie du changement et participative pour le développement d’une base de référence pourrait être utile aux donateurs, activistes et spécialistes du suivi et de l’évaluation intéressés par l’appui au changement social dans la région et dans le monde.

Introduction

Programme officers in large grant-making institutions, as well as the evaluators they contract, face a range of challenges. One of the central ones is finding efficient and effective ways to describe their programme strategy and the impact it is having to their senior management and governance structures. Detail and nuance of individual programmes frequently get lost as information is summarised to manageable portions for the highest levels of the institution, particularly the board. A little recognised aspect of this challenge is the struggle programme officers have in linking individual grants to overarching programme goals, and being able to demonstrate how individual grantees are making a specific contribution to achieving these. This is especially so in relation to meaningful social change, which almost by definition requires a large number of organisations and individuals working over a long period of time on a variety of different fronts. Seldom, if ever, is a single donor funding all the components necessary to bring about social change; and seldom, if ever, is a single grantee’s work responsible for significant social change, even if a great many claim that it is so. Despite these limitations, many donors continue to identify major social change, such as significant policy wins, as the intended outcome or impact of their grant-making strategies. All too frequently evaluators, programme officers, and indeed their grantees, are caught on the horns of the dilemma of attempting to demonstrate progress and impact against a goal that has been articulated in a way that makes it almost impossible to achieve.

The purpose of this article is to explore how a particular approach to programme evaluation might be used to address this challenge. It is based on a case study exploring the way in which a group of donors based in the USA, Europe and East Africa worked with a number of their grantees to develop a base line against which to measure progress being made in Kenya towards the broad goal of advancing rights related to sexual orientation and gender identity (SOGI).

Background

Over the last decade, but particularly in the last five years, there has been growing attention amongst international human rights funders to find ways in which to support activists working to address the inequality, violence and discrimination faced by people on the grounds of sexual orientation and gender identity. The Funders for Lesbian, Gay, Bisexual and Transgender Issues (FLAGI) report that between 2007 and 2010 there was a 35% increase in the funds going to advancing the human rights of lesbian, gay, bisexual and transgender (LGBT) people in the global South and East (Funders for LGBTQI Issues 2011). This reflected both an increasing amount of resources longstanding funders had dedicated to this work, as well as the entry of a number of new sources of funding (primarily US and European-based private foundations, but also bi-laterals). In response to this growing interest, as well as giving rise to it, there has been a parallel and not inconsiderable rise in the number of organisations working on these issues in Africa, the Middle East, Asia and elsewhere.

Whilst funds have gone to support a range of work, one of the central themes to emerge from grant-making strategies over this period has been providing support to grassroots organisations. This support primarily has been in the form of small grants (under $10 000) for core support, funds for specific project-based activities, as well as indirect support through technical and capacity building activities. One of the leading foundations pursuing this strategy with regard to advancing LGBT rights in the global South was the Arcus Foundation (Funders for LGBTQI Issues). From 2009 to 2012 the Foundation provided support to 11 ‘intermediaries’ working broadly on human rights and women’s rights to make small grants to grassroots SOGI organisations in Africa, the Middle East and North Africa (MENA) and South East Asia.

At the same time that this approach to SOGI rights funding was developing, within philanthropy more generally there has been a growing dissatisfaction with the ‘let a thousand flowers bloom’ approach to grant making for social change. Particularly within US-based private foundations, increasing attention has been paid to finding ways in which to more clearly articulate specific and measurable outcomes and to develop strategies that more directly pursue these ends over a specified time period. (Guthrie et al. 2005) One of the consequences of this approach is that there has been a growing focus on policy change as it is a visible and measurable signifier of the more amorphous notion of social change (Mackinnon & Amott 2009; Morariu & Brennan 2009). Such donors have viewed policy change as a logical and linear process that can be achieved by grant making to a few partners, working on behalf of a broader movement.

These trends were factors which prompted a great deal of discussion in the development and management of the Arcus Foundation’s international programme. Carla Sutherland, then Director of the International SOGI rights programme at Arcus, hired Barbara Klugman, an independent strategy, evaluation and learning consultant, to work with her to develop a learning and evaluation process that might navigate a middle path between these two opposing viewpoints: to find a process that would embrace the complexity of social change (including policy change) whilst still enabling the international programme to have a set of clearly articulated and measurable outcomes, with a sound set of indicators that would be able to measure progress (or not) towards the achievement of these goals, and that could be easily explained to senior management and the foundation’s board.

In discussion we identified the need to be able to clearly articulate three issues:

1. A starting point so that we could show clearly what needed to change - and why; what resources and opportunities were available to make that change; and what barriers had to be overcome; and an end point (a clear statement of what success would look like and how we would know we had got there);

2. An argument around how we thought changed happened in the specific contexts in which work was being funded, and a rationale for why we thought the way we did: in other words, a theory of change;

3. A process by which we could measure progress from our starting point to our end point, that was grounded in our theory of change, but which would be robust and meaningful enough to show progress over the short term (i.e. during a grant period, which typically was one to two years) the medium term (i.e. matching the Foundation’s strategic planning period of five years), and the long term (i.e. how long it typically takes for policy change and/or social change to take place – which can be anywhere from a few months to ten years, or even longer).

It was this frame that informed the decision to develop a base-line study as the foundation for a participatory evaluation process, which is presented below.

Methodology

The article draws on interviews with donor intermediaries, documents from three workshops organised by the authors, the implementation and findings of the pilot base-line study, and a process of reflection and discussion between the two authors following the conclusion of the pilot study.

The pilot itself involved the following steps:

Interviews were held with 11 donor intermediaries being supported by Arcus Foundation’s international programme in 2010 (Klugman & Kahn 2010). These interviews explored the strategies that intermediaries were pursing with regard to investing the funds they had received from the Arcus Foundation to support grassroots activism to challenge SOGI related violence, discrimination and inequality in Africa, the Middle East and South East Asia.

The interviews informed the planning of the first New York workshop, as it became clear that very few of the intermediaries had any more detailed strategy than providing core support to a range of grassroots organisations to pursue a self-identified programme of action. Very few of the intermediaries could explain how this approach to grant making would bring about the social change that many had identified as the goal of their grant making, such as ‘improved visibility of LGBT groups’; ‘increased acceptance of LGBT issues in the human rights sector’; ‘increased tolerance for LGBT issues in the country’ and so on. Where a rationale for this approach was provided, it tended to focus on the very long-term process of ‘movement building’ both amongst the various LGBT groups, and the LGBT sector and the wider human rights and women’s rights sectors. Hence, this first workshop involved a facilitated process where participants reflected on their individual organisational grant-making strategies, and we explored whether it might be possible to develop a single frame through which to understand both the Arcus strategy (funding a range of intermediaries to make a large number of small grants to grassroots organisations in a number of different countries) and the individual intermediaries’ strategies (how they decided which grants to make, and whether these grants linked to a wider national or thematic strategy).

A follow-up workshop in 2011 pushed this process further, exploring whether it might be possible for the intermediaries to work in collaboration with one another to develop complementary grant-making strategies - informed by a shared understanding of theory of change at the national and sub-regional level. At the second workshop a number of speakers from other sectors, notably the environmental justice movement and the women’s movement, reflected on their experiences of evaluating the impact of strategies that centrally involved small grants to grassroots organisations.

The third workshop in Johannesburg in 2011 was organised with a smaller group of intermediaries and with activists from Kenya to explore collectively a participatory evaluation process that would meet the need of donors to report on the impact of their investments, whilst taking into account (1) activist and intermediaries’ concerns about a donor-driven change agenda; (2) the non-linear process of policy change; and (3) a concern that the evaluation process should not be either too expensive or too time consuming to be useful. At the end of the workshop we had developed a frame and a process for the pilot base-line study and agreed that it should be implemented as soon as possible. Three funders – Arcus, Hivos and Urgent Action Fund – Africa – provided the resources ($10 000) for the study, and a fourth, UHAI, agreed to coordinate the process from Nairobi.

The pilot base-line study was initiated in July 2011, and completed by December 2011. At the start of the process a workshop organised by the Gay and Lesbian Coalition of Kenya (GALCK), and UHAI to share the frame developed in Johannesburg with representatives of Kenyan SOGI organisations where they further refined the proposed questions and method and worked out who would play what roles to get it carried out. Two skilled researchers, Nguru Karugu and Monica Mbaru, who were allies of the movement, were contracted to do the work.

The interim results were presented to a follow-up workshop of activists from Kenya, where participants had a chance to amend the empirical data and provide analysis of the study to the writers. This information was incorporated into the report, which was finalised, printed and widely distributed. Additional resources were found to cover unanticipated costs, so that the final budget was closer to $15 000, as opposed to the original $10 000.

Theoretical framings

As can be seen from above, this was a long and involved process that spanned more than three years. In the future, the process of gathering data to compare against the base line will not be this convoluted as the most time consuming aspects are unlikely to need to be repeated moving forward. However, neither will it ever be a ‘quick’ process, as it is informed by a deep commitment to a participatory evaluation, which is inevitably more time consuming in order for it to be owned by the movement, and used by them for on-going strategising and learning.

Theory of change based on complexity analysis

The most challenging work was developing a theoretical framing for understanding the way in which policy change happens, bearing in mind the specific contexts in which the intermediaries were working. Work was being supported in more than 20 different countries, which were at various different points of development in terms of strategies around LGBT rights. What many of the contexts had in common was that this was (1) nascent work, with newly established organisations just beginning to find their feet; (2) hostile socio-legal contexts with difficulties formally registering organisations and genuine dangers for individual activists; (3) weak or failed states with less than robust and predictable policy making and legislative processes.

The intervention that allowed us to move forward was a presentation that Barbara Klugman made at the first workshop, which identified a theory of change based on complexity analysis (Klugman 2010).

Our starting point in seeking a different approach to evaluation was a concern that the most widely used models, complexity, nuance, and non-linear thinking were excluded. All the participants at the first workshop were deeply resistant to identifying specific outcomes, attached to specific activities. Underpinning this concern was a belief that advocacy for social justice – whether policy change or changes in public norms and actions – cannot be predicted in the short or long term because they takes place in highly complex environments with innumerable actors each operating in their own interests. Participants had not encountered evaluation approaches that were able to capture the complexity of the activism they were supporting; their views were a reaction to deterministic explanations of the process of change (Callaghan 2008).

Early theories of policy change suggested that policy changes were predictable because they were based on rational processes. Policy makers reviewed all possible options and made decisions based on existing evidence (Grindle & Thomas 1991). This happened in an orderly sequence of policy-making steps. However, today it is recognised that there is no inevitable relationship between social problems and efforts to address them because society is highly complex with multiple and unpredictable factors influencing any specific changes (Cilliers 2000, Patton 2012). The model in Figure 1 identifies six different areas that are involved in change – what Kingdon (2002) describes as ‘streams’ – (1) problems, (2) solutions and/or policy or service implementation options, (3) politics, (4) bureaucracy and/or administration, (5) courts and (6) public values and actions1 (Klugman 2000).

FIGURE 1: Model for analysing the advocacy process to influence public perspectives and actions, and for influencing decision makers to make, retain or implement policies.

The rationalist approach of the past assumed that if you had a clear problem you could identify a clear solution that would invariably be taken up by policy makers – a linear perspective on the process of change. However, it is now well recognised that only some problems and some policy options ever get onto the political agenda because ‘political events flow along on their own schedule and according to their own rules, whether or not they are related to problems or proposals’ (Guthrie 2005:20). We also know that even when a law or policy is made, it may never be implemented as intended because bureaucrats and administrators interpret these as they see fit, based on interests which may have little to do with the intentions of the policy makers (Patton 2012, Parsons 1995). Clay and Schaffer describe the relationship between policy and implementation as a ‘chaos of purposes and accidents’(Sutton 1999:32). Changes in policy are also made by courts as they continually reinterpret laws or require legislatures to rewrite them (Rao 2010).

Socially dominant actions as well as culturally dominant norms and beliefs frequently undermine well-intentioned policies and programmes from achieving their expected goals (Friedman & Gordezky 2010). The model incorporates public norms, values and opinions, and indeed prevalent public actions, in recognition that they, too, may undermine or indeed enable the ability of politicians to create a law and bureaucrats to implement. Prejudices amongst some members of the public may undermine the ability of other members of the public to enjoy existing rights. Public opinion may or may not be amenable to change through media influence (Newton 2006).

Compounding this complex dynamic, changes in the political, economic or social context can also open or close windows of opportunity for shifting public opinion or getting issues into political and bureaucratic agendas (Stone, Maxwell & Keating 2001).

What does this theory of change mean for planning strategies for change?

It requires activists to identify problems and build a base of individuals and organisations that recognise these problems and support the activists’ definition of the problem. From a social justice point of view, this means activists need to come from those groups most affected by the issue: those with least power and access to resources; those who directly experience discrimination. At a minimum, activists need to enable such groups to participate in shaping the problem definition.

They need to sift amongst possible policy options or public interventions, and build agreement in as large a constituency as possible on the best or most achievable options for addressing that problem, in the given political and social context. Figure 1 illustrates that the role of activists is to influence problem definition and identify matching ideal or at least viable policy options and to then get these onto the agendas of the politicians, bureaucrats, and other decision makers who determine policies and their implementation, and keep them there in the face of opposition or bureaucratic apathy. At times it may be strategic to use the courts to facilitate this process.

Similarly, they mobilise public opinion in favour of their understanding of the problem and their solution, influencing social and cultural norms over time and using public opinion to sway political and bureaucratic decision makers. Indeed, in some contexts they may be able to shift public discourse, to get specific problems and preferred solutions into public debate in a way that pushes politicians to take action.

To play this role, they need to understand what concerns politicians, and what might draw their interest, and similarly what influences bureaucrats to implement laws and policies. They need to have analysed whether the juridical system and legal framework make litigation a potentially effective strategy. They need to understand current public norms and values and how to influence these. They need to be alert to windows of opportunity for change. This kind of analysis is essential to effective activism.

Another lesson for activists and for donors, from this theory of how change happens, is that achieving change cannot be accomplished by one person or one group. It requires collaboration.

Following this theory of change, whilst the achievement of a social movement’s goals cannot be predicted, it is possible to identify the extent to which an initiative for social change is strengthening over time.

A number of reviews of the literature on successful advocacy initiatives group the outcomes that advocacy campaigns tend to aim for (Coffman 2006; Korwin 2009). Reisman, Gienapp and Stachowiak (2007) call these groups ‘outcome categories’. For the purposes of the kind of organising undertaken by sexual rights activists, these have been organised into eight categories (Klugman 2010). The first four lay the groundwork for effective advocacy:

1. Strengthened organisational capacity including whether groups are registered, have systems of governance and financial management, leadership, strategic and communications capacity, adaptability;

2. Strengthened base of support, that is, the breadth of membership or public figures supporting the issue;

3. Strengthened alliances between organisations working on the same issue, and with organisations and networks on other issues.

These, in turn, draw on:

4. Increased data and analysis from a social justice perspective.

These four outcome categories form the basis for conducting advocacy, sometimes quietly within the corridors of power, and sometimes from the outside, through the mobilisation of constituencies, publications and engagement of the media. Together, achievements in all four outcome areas enable the following outcome category, which is a marker of significant progress in advocacy:

5. The development of consensus around a common definition of the problem and possible policy options by an ever widening constituency of people (both of which will also evolve over time with new insights, data and constituencies informing them).

These, in turn, form the basis for the advocacy movement as a whole, comprising individuals, organisations and alliances that are continually adapting to changes in context in order to ensure the ‘readiness’ of their organisational capacity, messages and strategies. They enable effective engagement in the policy process, which falls within the sixth outcome category:

6. Increased visibility of the issue in policy processes resulting in positive policy outcomes, including maintaining gains, and maintaining pressure through on-going monitoring of the implementation of policy. A litigation process and judicial finding would also fit within this outcome.

Ultimate impacts, usually beyond the timeframe of any grant or set of grants, would be:

1. Shifts in social norms (such as decreased discrimination against a specific group or increased belief that the state should provide high quality education). That said, along the way, one may start to see shifts in public understanding and visibility of the issues, as the problem definition or potential solutions gain social acceptance over time; and

2. Shifts in population-level impact indicators (such as decreased violence against women, suicides of gay youth, or increased educational achievement amongst groups with historically poor achievement).

Application to SOGI grassroots funding

This theoretical framing allowed us to imagine a way in which to both analyse where a ‘movement’ was (i.e. a collection of organisations working towards a common goal) and hence what were realistic changes over the short term (i.e. within the frame of a grant period, one to three years) to medium term (i.e. within the frame of a funders’ strategic plan, five to ten years). Further, if the model we developed was flexible enough we would be able to measure realistic progress, against a theory of change, that would enable funders to see if progress was being made, and individual organisations and activists to see if their strategies were being effective.

What we developed was a very simple model against which to map individual organisations working on a common thematic area, in a common geographical space. Collectively, the map could give a rough indication where the ‘movement’ as a whole was, and if repeated, an indication of whether or not progress was being made, both by individual organisations and the movement more widely. Secondly, if mapped against the theory of change outlined above, we would also be able to realistically assess the kind of outcomes that could reasonably be expected so that much more precise goals could be set for individual grants, as well as the programme strategy as a whole.

The simple model we developed is presented graphically in Figure 2.

FIGURE 2: Mapping organisations working on similar issues in common geographic space.

Putting it all together

Utilisation-focused evaluation
Clearly all of this could have been more quickly if we had been prepared to do much of the work without involving other donors or grantees. Our principled commitment to participatory evaluation was based on the assumption that evaluation would only be worth if it was useful. We recognised that whilst donors and organisations they support may want evaluations carried out to prove that promised work has been performed and intended goals reached, we were concerned with building evaluation into the on-going process of work. We thought this was important in order to support those doing the work in assessing how they are doing, and strengthening their work over time so that they are more likely to achieve their expected and hoped for outcomes.

Quinn Patton’s (2012) work on utilisation-focused evaluation argues that it is essential to involve ‘primary intended users’ (PIUs) in shaping the evaluation. PIUs are simply those individuals and organisations who need to find the outcome of the evaluation meaningful. In our case, this was both activists and the donors funding their work. Patton suggests meaningful participation of PIUs needs to include ‘making design and methods decisions, and interpret(ing) results to assure that the evaluation is useful, meaningful, relevant, and credible’ (Patton 2012:67). Patton notes that such evaluations have ‘process use’, meaning that the doing of the evaluation will enable learning and may well shift people’s actions well before there are evaluation findings.

For the pilot development of our base line, these issues were managed by creating key decision-making moments that explicitly involved both donors and activists. At the Johannesburg workshop, we had participation from four key funders supporting work in Kenya. In addition, there was an equal number of activists who were invited primarily because they were all in Johannesburg attending the first African Same-Sex Sexualities conference. The key decisions collectively reached at the workshop were:

1. A basic frame and process, including engaging with a wide range of activists at the beginning and the end of the study;

2. A first listing of outcome categories that framed the base line, which included a discussion around how policy change happened in Kenya;

3. Ideas for how information could be gathered, and who should do the data gathering, including drawing on information that organisations already had, as well as identifying where consultants would be helpful. It was agreed that all the consultants should be based (living and working) in East Africa;

4. A commitment that the evaluation could not use funds disproportionate to the funding levels of organisations in the field; and

5. Who would be the lead organisation for the process, and what their responsibilities would be.

In terms of the substantive content of the base line, and again drawing on our theory of change, at the first meeting of donors and activists in Johannesburg, we agreed to collect data on the following four areas:

Legislation and policy: We agreed that the base line needed to provide a comprehensive understanding of the laws and policies that discriminated against or failed to protect people based on their SOGI. The second area we were interested in was how laws and policies were applied in ‘real life’, in part because activists were conscious that whilst there was a major focus on decriminalisation amongst the international NGOs and many funders, there were other laws that were much more readily applied against them, such as loitering and solicitation. In terms of thinking through how best to collect this data we agreed on two strategies: first, the national coalition organisation (i.e. that had as members organisations rather than individuals) said that they had been collecting this information in the form of newspaper reports and reports filed by their organisational members. Despite having collected this information over more than two years, the coalition had not found any particular use for it. This was the first of many examples of organisations having valuable records that they had either not used or under-utilised both in strategising processes for activists, and in evaluation data collecting by funders. Second, we agreed that we would include a question about experiences of discrimination in the proposed questionnaire that we decided to undertake (discussed below).

Organisational mapping: We thought it was important to provide a ‘snapshot’ of the different organisations working on SOGI issues in Kenya. Information related to this that we thought important was: (1) organisational priorities and activities; (2) where it was based and its geographic reach; (3) number of members, and whether it had specific types of members (e.g. transgender, men who have sex with men [MSM], lesbians, etc.); and (4) whether it was registered. We quickly realised that this was information that donors had readily to hand, and that a fairly comprehensive picture would emerge if we pooled our information. Hence, no new data would need to be collected (or paid for).

Political and cultural context: We agreed that identifying some proxy measures for the hostility or openness of wider society and key decision makers to advancing LGBT rights was important to getting a clear picture of the context in which activists were working. Progress towards policy change was likely to be accelerated in more open contexts, and being able to demonstrate movement towards this would be useful to understand whether the work was being effective. Whilst we had a wide-ranging discussion the two measures we agreed on were (1) media reporting and (2) leading activists in civil society human rights organisations; leaders from the major political parties and religious leaders from a range of denominations. In terms of media reporting, initially we identified a number of organisations that did newspaper reviews and clippings, and agreed that approaching one of them to do a base line of coverage of LGBT issues, both in terms of frequency and ‘tone’ would be the easiest way in which to do this. Concern was expressed by activists that newspapers were not the best litmus test as the most interesting and insightful discussions were happening on radio talk shows and television. In response we agreed on a compromise strategy of employing a media expert to write an analytical report, identifying key trends with examples of the kind of writing or discussion that reflected the dominant discourse. In terms of leadership attitudes, we agreed that activist volunteers would seek to interview around ten key public opinion leaders drawn from the various sectors identified and use a pre-identified set of questions to engage all of them. When people refused to be interviewed, that fact, and the reasons why, would be regarded as the outcome of that interview. If we had this as a base line we would be able to measure progress over time as a deeply stigmatised issue that was either not discussed or discussed only in disparaging terms became more visible and more acceptable.

Lived experiences: The final category for information gathering came out of a realisation that in the end the change that was being sought was ultimately to improve the lived experiences of LGBT people. To this end we thought it important to try and capture a snapshot of the kind of issues, concerns and experiences that LGBT people had in Kenya. Although not comprehensive, we thought a rapid way in which to collect this data would be through a simple questionnaire that each of the participating organisations would conduct with five of their members. The organisations would be requested to pass out the questionnaire to their members. We thought that this information might be useful to activists in order to see the extent to which they were working on the most pressing issues for their members; and for donors to see to what extent the issues they had prioritised for funding were connected to the lived experiences and priorities of LGBT people. We agreed that this was important information and that a qualified and experienced consultant would need to be found to develop, distribute and analyse the questionnaire and its results.

Implementation

The information collecting process took place over a two-month period. The tight time line was partially informed by the fact that a major regional workshop was happening at that time, and it seemed a perfect opportunity to present the results both to donors and activists attending it. A report-back process took place in a closed activists’ forum. There discussion was robust with a number of findings being challenged and errors corrected. Perhaps the most valuable outcome of the workshop was that for the first time activists began to identify common goals that they recognised would be more effective if they worked on collectively. The background to the study and the initial results of it were also presented to a closed meetings of donors by the lead consultants and some of the donors who had attended the Johannesburg workshop. There was support for the initiative and a number of donors expressed interest in supporting similar work elsewhere.

The final report, Imagined futures, lived realities (Karugu & Mbaru 2011), was finalised three months later and widely distributed to activists and donors.

Results

Although imperfectly implemented when measured against the agreed frame from the Johannesburg workshop, the results of the pilot base-line study were both interesting and helpful. Discussed below are two key findings that have been helpful in conversations with donors, between programme officers and senior management in donor organisations, and amongst activists in Kenya.

Organisational development

We mapped the information that we collected on the organisations that were working on SOGI-related issues against the model of organisational development that we had developed. In addition to being able to place the organisations on the chart, using a simple colour coding process, we were also able to visually represent (1) the type of membership (general, MSM, transgender, lesbian); (2) the geographic reach (local or national); and (3) the focus of work (advocacy, service, arts). Figure 3 provided us with the first summarised understanding of the state of the SOGI movement in Kenya in 2011.

FIGURE 3: Chart of organisations working on SOGI-related issues in Kenya (2011).

In terms of organisational development, the chart clearly shows that the movement is very nascent. Of the 18 organisations mapped on the chart, a third are at the very earliest stages of largely informal groups undertaking social activities, with an additional five just beginning to find their feet in terms of attracting formal membership and developing very basic organisational programmes. What is important to note also, however, is that there are at least two organisaisations (Gay Kenya and GALCK) that have much more robust and mature structures, with the potential capacity to provide leadership at a national level.

This view is reinforced when looking at the geographical focus and reach of the organisaisations. Four of the organisaisations have a national reach, which our theory of change suggested was essential for substantial social change to take place. As would be expected, the newer organisaisations (clustered at the left hand side of the chart) have a more local focus. However, what is also important to note is that most of these organisations are based outside of the capital, Nairobi. This suggests a strengthening of the movement, as the issue was moving beyond urbanised centres to include organised activism in other (including rural) areas.

Finally, in terms of membership and focus, the chart shows an important level of diversity and specialisation. As would be expected, the strongest national organisations had a more general membership base with an advocacy focus. The one trans-gender organisation had a focus that extended nationally, and there were three lesbian-focused organisations, one outside of Nairobi. Just over a third (seven) of the organisations were focused specifically on MSM and on HIV/AIDS related work (both service and advocacy).

A chart like this can be helpful to programme officers attempting to refine their grant-making strategies to support the advancement of SOGI-related rights in Kenya. Against the theory of change underpinning the chart and the analysis, two distinct grant-making strategies emerge. The first is to provide specific support to assist emerging organisations to move from being small, largely informal groups of friends and acquaintances to ones that can undertake outreach efforts such as advocacy, public education or service to members. It is unlikely that all the groups will be able – or indeed willing – to move in this direction. However, a grant-making strategy designed to strengthen and encourage progress to social change needs to be providing resources to those groups that are able to develop the capacity to press forward that agenda. The kind of work that might be supported to encourage this will include continuing small grant support, but also some collective workshops to strengthen the capacity of activists to build and maintain organisations as well as to deepen strategising and advocacy skills.

The need to provide support, beyond direct resources, feeds into the second strategic approach that the chart suggests, namely providing support to the stronger national organisations to enable them to play this kind of leadership role within the country. Again, organisations may not want to take on this role but if substantial policy or other social change is to occur, it is needed. At times, this need is masked by the strong presence of international NGOs stepping into that space but that is neither ideal, nor sustainable in the longer term. At the very least, what the chart enables programme officers to do is to engage with grantees (and potential grantees) to explain what work they are looking to support and why. The challenge to a good programme officer is to listen carefully to the responses and if there is no organisation willing or able to take on that role to either adapt the theory of change (in response to grantees’ arguments about how change happens) and/or seek to support an interim process that will enable at least one or two organisations to play that role in the future. This underscores the importance of undertaking a highly participatory approach to the evaluation process, as it is much more likely that this kind of analysis will be shared by grantees if they have been party to gathering and analysing the information.

Finally, what the chart has the potential to do for activists, programme officers and evaluators, is to enable them to show measurable progress in response to a particular grant making strategy. To give a sense of how this might work, Figure 4 shows a similar chart for 2006, that is, five years earlier, based on information on which groups the Ford Foundation, Hivos and UHAI were supporting at that time. Hence, whilst not covering all potential groups, it does provide a brush stroke picture that is different in important ways from the 2011 chart.

What this earlier mapping demonstrates is that over a five-year period the SOGI movement in Kenya had considerably strengthened. Not only was there a significant growth in the number of organisations visibly working on these issues, but the organisations that were already in existence had become stronger and better able to tackle the social change they were seeking. In addition, it is noticeable how few of the organisations were based outside of Nairobi, and how little diversity there was in terms of membership. The two charts together provide a strong visual summary of progress being made within the country, and suggest that the grant-making strategy underpinning it was successful. Certainly, it shifts the conversation from one of ‘movement building’, which is very hard to report on, particularly when the only outcome that matters is ‘policy change’ or another proxy of social change that is likely to be sometime in the far distant future.

Organisational priorities

For activists, one of the most interesting outcomes of the pilot base-line study was the gap between their organisational priorities and what their own members identified as their lived experiences. In discussing this dissonance, concern was expressed that organisational priorities were reflective of what donors were willing to fund rather than what was actually needed. Below is a table outlining the responses from the members’ survey on their lived experiences, as well as a compilation of priorities identified by the participating organisations. Again, empirical evidence presented in this manner facilitates discussions amongst activist organisations about their priority strategies, as well as between grantees and their donors around what work needs to be prioritised, and why.

Conclusion

The purpose of this article was to describe a particular approach to programme evaluation, which we developed in response to the challenge of dealing with the complexity of supporting social change around advancing rights related to SOGI in Kenya. We suggested that the starting point to this approach was developing a specific theory of change that, rather than flattening and simplifying the process of social change, was grounded in complexity analysis. We also argued that developing a highly participatory approach that included both a number of donors working on the same issues, as well as a cohort of their grantees was essential to its success. We described possible ways in which evaluators might present information that would be useful to both programme officers in funding institutions, as well as activists in organisations on the ground. As with all pilot studies, this took much longer to conceptualise and implement than we hope it will be to reproduce in the future. The real test of its usefulness will only become fully apparent when a follow-up study, framed in the same way, is conducted at some time in the future. It is anticipated that such a study will happen in 2014 under the leadership of UHAI.

The broader value of this kind of approach is to ensure that donor programming and activist agendas are grounded on the actual context and dynamics in a specific country, rather than the hopes of donor boards or programme officers on other continents. This is a key role for African evaluators, and offers them the challenge of developing practices grounded in the values of participation and collective learning.

FIGURE 4: Chart of organisations working on SOGI-related issues in Kenya (2006).

TABLE 1: Lived experiences vs. organisational priorities, Kenya, 2011.

Acknowledgements

The authors would like to thank Happy Mwende Kinyili (UHAI) and Paul Jansen (HIVOS) for giving feedback on an earlier draft of this article.

Competing interests
The authors declare that they have no financial or personal relationship(s) which may have inappropriately influenced them in writing this article.

Authors’ contributions
C.S. (Columbia University) was responsible for conceptualising the idea of undertaking a pilot base line in Kenya, bringing together donors and their grantee activists. B.K. (University of the Witwatersrand) was responsible for conceptualising the overall approach to evaluating social justice movements and the content areas of the base line. They jointly wrote the manuscript.

References

Callaghan, G., 2008, ‘Evaluation and negotiated order: developing the application of complexity theory’, Evaluation 14 (4), 399–411.

Cilliers, P., 2000, ‘What can we learn from a theory of complexity?’, Emergence 2(1), 23–33.

Coffman, J., 2007, ‘Using the advocacy and policy change composite logic model to guide evaluation planning’, Harvard Family Research Project, Boston.

Friedman, M. & Gordezky, R., 2010, A holistic approach to gender equality and social justice, Gender at Work, Johannesburg.

Funders for LGBTQI Issues , 2011, A global gaze: Lesbian, gay bisexual, transgender and intersex grantmaking in the global south and east, Funders for LGBTQI Issues, New York.

Grindle, M. & Thomas, J., 1991, Public choices and policy change, Johns Hopkins University Press, Baltimore.

Guthrie, K., Louie, J., David, T. & Foster, C., 2005, The challenges of assessing policy and advocacy activities: strategies for a prospective evaluation approach, The California Endowment, Oakland.

Karugu, N. & Mbaru, M., 2011, Lived realities, imagined futures: Baseline study on LGBTI organising in Kenya, UHAI, Nairobi.

Kingdon, J., 2002, Agendas, alternatives, and public policy, Longman, New York.

Klugman, B., 2000, ‘Empowering women through the policy process: The making of health policy in South Africa’, in H. Presser & G. Sen (eds.), Women’s empowerment and demographic processes: Moving beyond Cairo, p. 95–118, Oxford University Press, Oxford.

Klugman, B., 2010, ‘Is the policy win all? A framework for effective social justice advocacy’, Foundation Review 2(3), 93–107.

Klugman, B & Kahn, M., 2010, ‘Approaches to human rights evaluation: Themes emerging from interviews with donors, in preparation for the International Human Rights Funders’ Group Evaluation Institute, July 2010’, unpublished.

Korwin, L., 2009, The catalyst fund theory of change, Korwin Consulting for the Tides Foundation, San Francisco.

Mackinnon, A. & Amott, N., 2006, Mapping a change: Using a theory of change to guide planning and evaluation, Grantcraft, New York.

Morariu, J. & Brennan, K., 2009, ‘Effective advocacy evaluation: The role of funders’, Foundation Review 1(3), 100–108.

Newton, K., 2006, ‘May the weak force be with you: The power of the mass media in modern politics’, European Journal of Political Research 45, 209–234.

Parsons, W., 1995, Public policy: An introduction to the theory and practice of policy analysis, Edward Elgar, Cheltenham.

Patton, M. Q., 2012, Essentials of utilisation-focused evaluation, Sage, Los Angeles.

Rao, M., 2010, Gender justice: Comparative perspectives and lessons from around the world, Women’s Link Worldwide, Bogota.

Rao, M & Klugman, B., 2013, Courts as a site for social change, Women’s Link Worldwide, Madrid, forthcoming.

Reisman, J., Gienapp, A. & Stachowiak, S., 2007, A guide to measuring advocacy and policy, Organisational Research Services, Seattle.

Stone, D., Maxwell, S. & Keating, M., 2001, ‘Bridging research and policy’. Background paper presented for An International Workshop Funded by the UK Department for International Development, July 16-17 2001, Radcliffe House, Warwick University, Warwick.

Sutton, R. 1999, ‘The policy process: An overview’, Working Paper 118, Overseas Development Institute, London.

Footnote

1. Whilst Kingdon identified the three streams of problems, solutions and politics, the work described here engages other terrains that have been incorporated into this model.


Crossref Citations

No related citations found.