About the Author(s)


Michael R. Ward Email symbol
Environmental Learning Research Centre (ELRC), Faculty of Education, Rhodes University, Makhanda, South Africa

Eureta Rosenberg symbol
Environmental Learning Research Centre (ELRC), Faculty of Education, Rhodes University, Makhanda, South Africa

Citation


Ward, M.R. & Rosenberg, E., 2020, ‘Revealing mechanisms in a transdisciplinary Community Reforestation Research Programme’, African Evaluation Journal 8(1), a467. https://doi.org/10.4102/aej.v8i1.467

Research Project Registration:

Project number: KZN SS Research Programme Process Evaluation

Note: Special Collection: SAMEA 7th Biennial Conference 2019.

Original Research

Revealing mechanisms in a transdisciplinary Community Reforestation Research Programme

Michael R. Ward, Eureta Rosenberg

Received: 18 Feb. 2020; Accepted: 27 July 2020; Published: 23 Oct. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: The Community Reforestation Research Programme is a joint initiative between eThekwini Municipality and the University of KwaZulu-Natal in South Africa. Addressing complex sustainable development issues requires collaboration across disciplines and between researchers, practitioners and policy-makers. Understanding how this occurs has the potential to inform and improve transitions to sustainability.

Objectives: A summative evaluation of one project within the broader programme was commissioned to develop in-depth understandings into the underlying systemic influences or mechanisms that supported or hindered transdisciplinary research and practices.

Method: A realist evaluation method informed the analysis of questionnaires, documents, interviews, focus group discussions and participation in programme reporting and planning processes. This approach supported the development of a contextual profile within which underlying mechanisms were identified. These mechanisms were considered in relation to the outcomes of the project.

Results: The main mechanisms identified in the evaluation process related to orientations to research, orientations to education, orientations to value creation, orientations to environmental management and orientations to organisational leadership. These orientations either enabled or hindered particular outcomes in the research on, and environmental management of, community reforestation in the context of climate change. The mechanisms also had significant implications for understanding ‘what worked well for whom in what circumstances and how’.

Conclusion: Transdisciplinary research and practice across institutional boundaries are enabled or hindered by underlying mechanisms. By identifying and understanding these mechanisms, insights were developed that have the potential to enhance transdisciplinary sustainability initiatives at the local level.

Keywords: Realist evaluation; Evaluation methods; Transdisciplinary research; Community based; Sustainable development.

Introduction

Addressing complex sustainable development issues requires collaboration across disciplines and between researchers, practitioners and policy-makers. This article reports on a summative evaluation of a transdisciplinary research initiative that sought to support this kind of collaboration within a community reforestation project in an urban setting. In addition to reporting on the outcomes of the project, the evaluation sought to understand how and why the transdisciplinary research project produced the outcomes that it did. This required an identification of underlying mechanisms. As these mechanisms are by their very nature ‘not visible’, this evaluation draws on realist evaluation literature, supported by reference to critical realist philosophy to identify the underlying mechanisms. Attention was given to the implications of a depth ontology and the use of retroduction (both of which we describe in further detail below) to understand the outcomes of the project in terms of contextual structures and mechanisms that enabled or hindered the achievement of the project outcomes.

Setting

The Community Reforestation Research Project (CRRP) was a joint initiative between eThekwini Municipality and the University of KwaZulu-Natal (UKZN) in South Africa. This transdisciplinary research project fell under the broader Durban Research Action Partnership (D’RAP), which is an ongoing programme of collaboration between the two institutions. The purpose of the CRRP was to develop research capacity and expertise that enhances the delivery of environmental management in an urban context.

Key focus

The CRRP provides a useful case for examining the underlying assumptions or mechanisms that enable or hinder transdisciplinary collaboration both within universities and between universities and key stakeholders including local government. By using a realist evaluation approach (Pawson & Tilley 1997), further deepened by critical realism (Bhaskar 2008; Bhaskar & Hartwig 2016), this article contributes to an understanding of the transdisciplinary research component of the broader Community Reforestation Programme. In so doing, it contributes to the development of suitable methodology for evaluating complex research and learning programmes.

Background

Responding to the global, national and local environmental, social and economic issues at the municipal level requires individual competence and institutional capacity at multiple levels. Within eThekwini, skills shortages and the resultant difficulties in filling vacant municipal posts in the then Environmental Department in the early 2000s catalysed endeavours to build environmental capacity within the municipality (Roberts et al. 2012; Rouget & Cockburn 2014). By 2010, these capacity development endeavours had coalesced around an internship programme and a research partnership between eThekwini Municipality, particularly the Environmental Planning and Climate Protection Department (EPCPD) and the UKZN, particularly the School of Biological and Conservation Sciences. This partnership later became known as the D’RAP and acted as an umbrella under which a number of projects were developed and implemented. The National Green Fund supported a community reforestation project with 10% of the project funding allocated to ‘conducting a collaborative research project in the area of Reforestation within the Municipality’ (UKZN & eThekwini Municipality 2013). The intention of the CRRP was to ‘promote the knowledge base in the delivery of environmental management in an urban context, and to develop research capacity and expertise in this field’ (UKZN & eThekwini Municipality 2013).

Trends

Bridging the responsibilities and aspirations of practitioners and academic institutions to develop the capacity to address complex issues through collaboration and across disciplines takes multiple forms (Lang et al. 2012). Cockburn et al. (2016) identified transdisciplinary research as the approach informing, and referenced by, the participants within the Community Reforestation Research Programme.

Scholz and Steiner (2015a, 2015b) have noted that ‘transdisciplinarity is a demanding approach, particularly if applied factually in a real-world context’. They (Scholz & Steiner 2015b:658) go on to note that ‘the evaluation of transdisciplinary processes is a special methodological challenge’ and that ‘there are relatively few thorough evaluation methods’. Given the diversity of contexts, the complexity of the issues and the impact that these would have both on expected outcomes and the interpretation of value or success of the outcomes, there is a need to develop appropriate evaluation methodology for transdisciplinary research.

Objectives

A realist approach to science (Bhaskar 2008) argues that observational evidence alone cannot establish causal uniformities between variables (Dalkin et al. 2015). Rather, it is necessary to understand the context within which particular outcomes are (or are not) evident and to establish what goes on in the system that connects context, inputs and outcomes. Pawson and Tilley (1997) provided a useful metaphor to illustrate the significance of this point:

We can never understand how a clock works by examining only its face and the movement of its hands; rather we examine the clockworks, and so a proper understanding requires us to master the construction of the balanced spring or the oscillation of caesium atoms. (p. 65)

Similarly, evaluations often focus on which outcomes have or have not been achieved. They thus answer a rather simple question of ‘what worked’. The objective of realist evaluation is to develop an in-depth understanding of the underlying mechanisms that support or hinder change. The evaluative question thus expands to ‘what worked well for whom in the context of this programme and why?’ (Pawson & Tilley 1997). The significance of this approach is that, rather than moving directly from the identification of the obvious (visible) problems to experimenting with and taking new kinds of actions, deeper, more significant change requires emphasis on the underlying mechanisms. Once these mechanisms are made explicit, new ways of bringing about change may become apparent that, in turn, can lead to more sophisticated structural change that addresses underlying systems and not just visible outcomes (Virkkunen & Newnham 2013).

Contribution to the field

This article explores the identification of the mechanisms that enabled or hindered transdisciplinary research in the context of the CRRP. In doing so, it makes these mechanisms more explicit and provides programme developers insights with which to strengthen their programme theory and the design of future programmes. It provides academics in this field with theoretical insight into transformation processes. In addition, it proposes a suitable methodology for evaluating complex programmes. As Danermark et al. (2002) have noted:

The most productive contribution to social practice that social science can make … is the examination of social structures, their powers and liabilities, mechanisms and tendencies, so that people, groups and organisations may consider them in their interaction and so – if they wish – strive to change or eliminate existing social structures and to establish new ones. (p. 33)

Literature review

Despite a substantial body of literature on transdisciplinary research (Bieluch et al. 2017; Cundill, Roux & Parker 2015; Polk 2014; Scholz & Steiner 2015a, 2015b; Steelman et al. 2015; Swilling 2014; Wiek et al. 2012), few of the authors discuss evaluation, those who do (e.g. Scholz & Steiner 2015b) acknowledge that there is a need to develop a broader suite of evaluative methods in this field.

In approaching an evaluation of a transdisciplinary research project, it is useful to make a distinction between the logical models that provide operational details about how the project is carried out and the project theory that deals with the mechanisms that intervene between the delivery of programme services and the occurrence of outcomes. According to this perspective, mechanisms are ‘… underlying entities, processes, or structures which operate in particular contexts to generate the outcomes of interest’ (Astbury & Leeuw 2010:368).

Astbury and Leeuw (2010) provided a review of the use of the term ‘mechanism’ in evaluation literature. They suggested that ‘a detailed treatment of the concept of “mechanism” did not appear in the (evaluation) literature until the publication of the book Realist Evaluation by Pawson and Tilley (1997)’ (Astbury & Leeuw 2010:366). They also note that the approach of Pawson and Tilley is distinctive because it is strongly based on the work of early realist philosophy of science developed by Bhaskar in 1975. Central to this approach is the idea that ‘it is not enough to simply cite programmes as a cause of outcomes – the mechanisms connecting causes and their effects must also be identified’ (Astbury & Leeuw 2010:366). Despite this centrality of the idea of mechanisms for explaining how and why a programme may function as it does to produce the outcomes, there is very little practical guidance on how to identify mechanisms (Dalkin et al. 2015). To address this challenge, some authors have returned to the critical realist philosophy of Bhaskar and sought to elaborate the understanding of mechanisms from a broader theoretical perspective (De Souza 2013; Jennings 2015). This article uses these perspectives and particularly the method of retroduction to identify mechanisms through a consideration of the outcomes of a project.

Research method and design

Materials

This study took place within the broader D’RAP programme and was thus well supported by research reports on the CRRP (Bertolli, Teixeira-Leite & Macfarlane 2013; Douwes 2017; Douwes 2015; Egoh 2015; Greater Capital 2011; Macfarlane, Harvey & Hamer 2011; Taylor & Manqele 2016; The Cirrus Group n.d.). One paper (Cockburn et al. 2016) elaborated on the transdisciplinary research focus of the CRRP, whilst another paper (Taylor et al. 2016) summarised internal reviews of the CRRP. In addition, there was project contractual documentation (UKZN & eThekwini Municipality 2013). Access to the manager responsible for the programme within the municipality, the project lead within the university, the co-ordinator within the university, lecturers (five principle investigators [PIs]) and students (13 masters and 3 PhD students) was available throughout the evaluation process. In addition, the evaluator participated in two key meetings involving national, provincial and local stakeholders. The final draft was circulated to key stakeholders in the municipality and the university, and report-back sessions were held with both institutions. All these interactions provided a wealth of material that was analysed both to develop a thick description of the outcomes and for delving below the visible surface to identify underlying mechanisms associated with the outcomes of interest to the CRRP.

Study site

It is important to make a distinction between the Community Reforestation Programme that involved community members adjacent to the Buffelsdraai Waste Treatment facility and the Community Reforestation Research Programme. The latter programme, although involving students in research at the Buffelsdraai site, was focused on the development of transdisciplinary research capacity and new knowledge across the EPCPD in the municipality and the College of Agriculture, Engineering and Science at the UKZN. The evaluation thus focused on the institutional interaction within and between the participating university and municipal entities.

Design

As mentioned, the evaluation was guided by a realist approach to science (Bhaskar 2008; Bhaskar & Hartwig 2016; Lotz-Sisitka & Price 2016) and, specifically, a realist evaluation methodology (Pawson & Tilley 2004):

Realist evaluation is about theory testing and refinement. Context-mechanism-outcome pattern configurations (CMOCs) comprise models indicating how programmes activate mechanisms amongst whom and in what conditions, to bring about alterations in behavioural or event or state regularities. (p. 9)

As a summative evaluation of the CRRP, with a view to informing ongoing work within the broader Durban Action Research Programme, the design of the evaluation sought to contribute to theory building. The evaluation was faced with a significant challenge in that the mechanisms for change remained largely implicit in the programme documentation and implementation processes. Thus, whilst there was substantial information on the context of the research programme and project reporting had been thorough in terms of outcomes, the actual mechanisms through which change was meant to be affected remained virtually invisible.

Identifying the particular mechanisms thus required the application of (Jennings 2015):

[C]reative interpretation and judgement to social and behavioural theory to create plausible explanations drawn from propositions that certain mechanisms must exist, have been triggered in a certain way, and must be at work in producing outcomes. (p. 52)

It is here that the depth ontology developed by Bhaskar was useful for understanding and informing this realist evaluation. Danermark et al. (2002) argued that reality is enacted in three ontological domains: the empirical, the actual and the real:

The empirical domain consists of what we experience, directly or indirectly. It is separated from the actual domain where events happen whether we experience them or not. What happens in the world is not the same as that which is observed. But this domain is in its turn separated from the real domain. In this domain there is also that which can produce events in the world, that which metaphorically can be called mechanisms. (p. 56)

The implication of this stratified conception of reality is that reality cannot be reduced to observation of phenomena at the empirical level. In evaluation, just identifying and reporting on measurable outcomes is an example of such a shallow epistemology. To acquire useable or useful knowledge, it is necessary to know the mechanisms that produce the empirical events and, as in the case of the CRRP, these are seldom directly visible. Put differently, the challenge for the evaluator is to transcend the observation or experiences occurring in the empirical domain and the events in the actual domain, recognising that these are triggered by, and arise from, mechanisms in the real domain.

Bhaskar developed a form of inference known as retroduction that enables the researcher to move from observation made in the empirical domain to the prerequisite condition necessary for the structures and mechanisms in the actual and real domain to create the observed or experienced outcomes. Retroduction allows a researcher or evaluator to move from a description and analysis of concrete phenomena (e.g. number of papers produced and even interview statements) to reconstruct the basic conditions for these phenomena to be what they are. In the CRRP evaluation, existing evaluation reports, project reports, interviews, questionnaires and relevant associated literature on transdisciplinary research and universities’ engagement on sustainable development were used to support a process of initial review and retroduction.

Procedure

The depth ontology developed by Bhaskar recognises that generative mechanisms are not observable and thus requires that research and evaluation design bring these mechanisms into view. A number of authors (Bhaskar & Hartwig 2016; Blaikie 2000; eds. Danermark et al. 2002) have developed procedures that support this process and although there are some differences in how the process is structured, there is also substantial overlap between the approaches. All three approaches can be described as ‘a movement from the concrete to the abstract and back to the concrete’ (eds. Danermark et al. 2002:108). All three sets of authors are also clear that the models or approaches should be seen as guidelines and not as templates to be followed to the letter. In addition, it is acknowledged that at times it will be necessary to move iteratively back and forth between the stages in the approach and that ‘there may also be reasons for concentrating on a certain stage(s) and touching on others more lightly’ (eds. Danermark et al. 2002:108). Using the approaches referenced above, the following procedure was followed.

Stage 1: Description of important aspects of the context within which the programme is situated. This was performed through meetings with key programme managers at the university and the municipality. It was also informed by document analysis of organisational profiles of the key institutions and the contexts within which they operated.

Stage 2: Analytical resolution involved the separation or dissolving (analysis) of the context to enable a focus on those aspects of the context that were most significant to the study. In this case, the focus on transdisciplinary research led to a focus on the institutional relationships, the notion of research and the potential for the research to support better management practices in the context of sustainable development.

Stage 3: Theoretical redescription was closely linked to the description of the programme in the programme proposal and subsequent reports that revealed (hinted at) theories of change associated with particular structures and relations. Based on this initial redescription and emerging from the interviews, a first round of coding of document analysis, interview or questionnaires and focus groups discussions was developed.

Stage 4: A process of retroduction began with a second round of coding that sought to identify the basic prerequisites or conditions for social relationships, people’s actions, reasoning and knowledge as observed, reported on and expressed in the interview processes. The coding process identified recurrent phrases or themes as well as interesting outliers or exceptions that were relevant to the subject of transdisciplinary research. It also sought to identify absences or silences where an important aspect of transdisciplinary research highlighted in the literature was absent from the data being examined. Finally, related research was examined for particular themes and this was compared with the themes emerging in the case data. Key to this process was what Danermark et al. (eds. 2002:109) described as ‘counterfactual thinking’. In this process, the evaluator uses their ‘stored experience and knowledge of social reality, as well as (their) ability to abstract and to think about what is not, but what might be’ to identify possible underlying mechanisms. Within this process, inferences are made about the ‘demi regularities’ (Pawson & Tilley 2004) between the observed social phenomena (including feelings, perceptions, understandings and meanings) as detectable expressions of outcomes and outcomes in the empirical domain, on the one hand, and mechanisms that would exist in the real domain, on the other hand. The evaluator works through a range of transfactual questions and associated inferences to identify the most plausible mechanisms.

Stage 5: The next stage would be a process of concretisation and contextualisation that examines how the mechanisms identified manifest themselves in different contexts. Although this stage was touched upon in discussions in the workshop looking at the way forward for the Durban Action Research Partnership, it was not a major focus of this evaluation, which was undertaken at the end of the CRRP.

Ethical considerations

The evaluation of the project received ethical clearance through the university. As all data were anonymised except for the names of institutions already identified within documents in the public domain, it was not required that additional ethical clearance be obtained. In addition, each respondent was notified in writing that their responses would be anonymous, that they were under no obligation to participate in the questionnaires and group discussion and that they could withdraw at any time. Signed copies of this agreement were obtained from each participant. Before release, the evaluation was shared with the project managers in both participating organisations and they were invited to comment on the draft. Finally, the report was presented to senior staff in both organisations with the opportunity to raise any concerns that would have been addressed before finalisation of the report had any arisen.

Results

The evaluation used the tasks mentioned in the Memorandum of Agreement (UKZN & eThekwini Municipality 2013) as the basis for structuring the study. The first level of results is thus briefly reported in relation to a selection of the tasks and, more specifically, the outcomes associated with those tasks. A second level of results, relating to the mechanisms identified across the project, is then reported on. The selected quotes below were identified through the coding process with the specific quotes representing themes and insights that emerged as the analysis proceeded.

A key task within the CRRP was to support evidence-based best practice to guide the Reforestation programme in terms of current and future management practices. Whilst previous evaluative work within the project highlighted what outcomes had been achieved (e.g. number of studies in relevant areas), interview data on the outcomes allowed for a more nuanced understanding of what worked for whom. One of the managers within the municipality noted:

‘At the start of the project there was a real disconnect between ourselves and the Principal Investigators. The Principal Investigators have their own research and professional interests and they bring students into their research programmes to further these interests … But we need real world solutions… Slowly over time the researchers have come to realise that the research is interesting and relevant and there is closer alignment between the needs of the municipality and the university researchers’. (Municipal manager 1, male, 2016)

This alignment of research interests is also reflected by the ‘PI’ or research supervisors as evidenced in the following quote:

‘The close interaction between the municipality stakeholders and the university has helped to bridge the gap between research and the local government’. (Principal investigator 1, female, 2016)

However, it was less evident that the researchers (students) had resolved this issue. During the focus group discussion, a number of suggestions were made on how to align the research with the municipal needs. One of the researchers commented:

‘Our challenge is the synergy between the university and the municipality. Their approach is more policy/management driven and the challenge we face is most of us are research driven …’. (Researcher 1, female, 2016)

The comments by the student researchers related to the relevance of their research suggested an ongoing tension between the research programmes of PIs, the eThekwini Municipality and the researchers’ research topics. As will become evident in the discussion on mechanisms, this tension may be the result of deep underlying differences related to research and educational orientations.

A second cluster of tasks focused on human capital development and the building of partnerships and institutional capacity. Despite frequent references in project reporting to quantitative metrics such as number of masters and PhD students, number of meetings or number of publications, the interviews revealed a desire to engage with qualitative dimensions of the project. These included closer collaboration between the university and eThekwini Municipality; the creation of high-quality research opportunities for PIs and their students; and closer working relationships between the different academic disciplines within UKZN.

The researchers made specific suggestions on how the interaction between themselves and eThekwini Municipality could be enhanced. The following quote from the focus groups is illustrative of these suggestions:

‘Job shadowing would allow students or graduates to spend some time with professionals in an organisation so that when they are doing their research it would be a bit more relevant to the needs and practices within the organisation …’. (Researcher 2, female, 2016)

These ‘authentic’ learning experiences in work contexts with professionals in the field were thus considered important in terms of enhancing the project outcomes both in terms of capacity development and in terms of linking the research to management issues faced by the municipality.

Another outcome that was key to this project’s success was closer working relationships between the different academic disciplines within UKZN. As one of the PIs noted:

‘The working together of different disciplines of research in human sociology, environmental practices, sciences (biology, geography, chemistry etc.) is a rare occurrence and is difficult to bring together cohesively as one project. However, this project is succeeding quite nicely and should be a benchmark for other projects’. (Principal investigator 2, female, 2016)

Whilst there are other mentions of this ‘cross discipline … synergy’ (Annual report:26), many of the comments received during interviews suggested that the D’RAP and within it, the CRRP provided more scope for transdisciplinary collaboration within UKZN than was being explored.

Within the focus group discussion, one of the researchers expressed the following perspective:

‘I love the idea of this transdisciplinary research but I always wonder how we can do it. We need our supervisors to be willing to work together for us to work together. So for me one of the biggest things for us as students is that if the supervisors do not work together then … [implies that they as researchers do not work together]’. (Researcher 3, female, 2016)

The researchers also raised additional challenges to working across disciplines. They were, for example, clear that their first priority was ‘to get out our dissertation’ and that it is ‘simpler to just work separately than it is to work together’. The researchers concluded that this ‘needs more communication on how we work together’ and that the supervisors need to take a more proactive role in this regard.

Another important output for the CRRP is the number of graduate students. The outcome related to this output is enhanced capacity in the field of environmental management. This was one of the original motivations for the broader D’RAP and remains an important outcome for both the municipality and the university. For the municipality, there is a range of benefits related to the CRRP including producing scarce skills for the municipality and sustainable development more broadly. For the PIs and UKZN as a whole, the researchers support the research programmes of the PIs, contributed to graduation numbers and the production of academic articles – all important contributors to the economy of the university. For the researchers, the CRRP provided an important opportunity to pursue research that is relevant to the public and private sectors.

Also linked to the programme is the provision of high-quality research opportunities. A key aspect here was the provision of bursaries to cover some of the researchers’ costs. When asked why they had become involved in the CRRP, a number of the students noted the importance of the funding. As one researcher noted, ‘I suffered during Honours so I told myself that if I could get a scholarship then I would do a Masters’. Nearly all the researchers, however, went on to say that the CRRP also attracted them because of the relevance of the research to pressing issues in South Africa.

Following the description of the context and the coding of previous project, plans, evaluations, interviews or questionnaires and focus group discussions were the process of identifying the underlying mechanisms. This required moving from the observations and perception of events and outcomes to arriving at what critical realists call ‘transfactual conditions’. The term ‘conditions’ is used to signify the circumstances without which something (X) cannot exist. This gives rise to the question ‘What conditions must exist for X to be what X is?’ These conditions are transfactual in so far as they go beyond what can be observed or spontaneously experienced (empirical and actual domains). The challenge is thus to identify the social structures, relations and reasoning that have effects on X and are thus real in so far as they have these effects but transfactual in so far as they move beyond the observable or empirical. This retroductive process is informed by the question: ‘What conditions exists for a particular outcome (positive or negative; intended or unintended) to exist in a particular way?’ Or put another way, ‘What underlying mechanisms are related to a set of outcomes?’

The retroductive process has been informed by a careful consideration of the way in which different respondents reflected on the project outcomes and by reference to published literature related to university or government interactions, transdisciplinarity and the CRRP. The following mechanisms were identified through this process and are offered as tentative suggestions for better understanding the CRRP and the potential for enhancing the achievement, effectiveness and impact of transdisciplinary projects in similar contexts.

Orientations to research

Different orientations to research are evident within the CRRP. Particularly evident are tensions between issues such as disciplinary rigour, research programmes and publication targets within the university structures, issues related to applicability of research to management within eThekwini and the social relevance of the research to communities. This tension is not specific to this project and the underlying mechanism should be understood as a structural challenge currently faced by universities as they grapple with deeply structural issues that privilege disciplinary approaches to teaching and research and reward the publishing of academic papers as an output of academic activity.

Orientations to education

A number of tensions related to orientations to education were evident within this programme. At the centre of this tension, there is a well-documented discrepancy between an approach to teaching and learning as an induction into an existing and clearly defined universal body of knowledge, on the one hand, and, on the other hand, an approach to teaching and learning as the critical co-creation and exploration of open and emergent knowledge systems in context. Many of the comments from the researchers suggested that they were torn between getting their degrees completed within the stipulated period by keeping their focus narrow and easily defined within the departments’ and supervisors’ disciplinary scope, and engaging with more meaningful learning experiences in the project area that took into account the interconnections between research, policy, implementation and community engagement. Meeting as a university group and with a broader group that included managers and staff from eThekwini municipality provided an opportunity to address some of these tensions.

Orientations to value creation

Both universities and local government institutions are under increasing financial pressure. This can lead to a focus on cost, financial benefits and a narrow view of return on investments. Universities, for example, have strong structural incentives to graduate students as efficiently, publish academic papers and orientate their academic programmes towards research and development that has economic value. At the same time, universities are seeking to create value by building relationships with local government and communities based on increased relevance, more inclusive access to the knowledge outputs and direct engagement through students, researchers and staff. Local government institutions are looking to enhance service delivery and at the same time respond to issues of social inequality, low economic growth and environmental challenges. In this context, the programme has provided useful funding to the university and particularly postgraduate students. It has also provided a way for the municipality to enhance its scientific knowledge to support management, its ability to motivate for and secure additional funding and potentially to reduce recruitment costs.

Orientations to environmental management

There is a tension between traditional approaches to environmental management that tend to focus on establishing objective facts and causality and using this knowledge to manage and control ecosystems for the benefit of the people, on the one hand, and, on the other hand, a more open-ended and adaptive approach to environmental management that emphasises the co-production of knowledge and co-management through processes of social and expansive learning in a particular context. The former approach is more evident in the approach of the university respondents, whilst the latter was more evident in the municipality.

Discussion

Returning to the guiding question of realist evaluation namely ‘what works for whom in what circumstances and how?’, it is now possible to recognise that the underlying mechanisms have important implications both for what outcomes are achieved and how these outcomes are perceived by different role players.

As mentioned previously, the number of papers published is included as a key deliverable in the project planning documentation and as a key output in the project reporting. Both the university and municipal participants in the project stressed the importance of publishing transdisciplinary papers as an output of this project but (for different reasons) noted the challenges faced in finding time to focus on the production of academic papers. The underlying mechanisms identified relate to the privileging of disciplinary approaches to research and the financial pressures on both universities and students to finish research within tight time frames. In response to these pressures, the project hired a researcher or writer to integrate insights from the various studies into publishable papers and more accessible reports. Although an innovative and productive response to the pressure to produce transdisciplinary papers, this represents a relatively superficial response to the underlying mechanisms related to research that require attention for more meaningful transdisciplinary engagement both within universities and between universities and local government.

In addition to the publication of papers, all the people interviewed expressed a desire for more engaged research. Many of the researches commented that not much has happened until the dissertation is produced and the papers are published. This ignores the possibility for a more action research or even interventionist research design that seeks to bring about change through the research process itself. The paper on transdisciplinarity by Cockburn et al. (2016) reveals that much work has been performed to open possibilities for this kind of research orientation within the programme. However, many of the comments (particularly related to community engagement) suggest that there are still underlying mechanisms particularly related to orientations to research, education and environmental management hindering this outcome.

As orientations to environmental management recognise both the need for scientific and technical knowledge as well as the co-production of knowledge and adaptive management with a wider range of stakeholders, new possibilities for research and practitioner configurations emerge. The CRRP represents an innovative response to this possibility and the participants in the programme expressed a desire to explore these possibilities further. The embedding of researchers within eThekwini Municipality represents one possibility, whilst formal engagement sessions through schools and local community structures were mentioned as another.

The monitoring and evaluation of outcomes within the CRRP had produced a careful record of the outcomes that were being achieved by the project. There was also evidence within the evaluations of a desire to, for example, publish more. However, by adopting a critical realist approach to the evaluative work, the underlying mechanisms related to orientations to research, education, value creation and environmental management, amongst others, revealed more systemic and structural challenges and opportunities related to the project. This opens the possibility for a deeper level of engagement to bring about change in the mechanisms that prevent or obstruct particular outcomes.

Conclusion

The building of individual and institutional capacity in the environmental sector, and more specifically within eThekwini Municipality and UKZN, was a key objective of the CRRP. A large number of outcomes were identified including: closer collaboration between the University and eThekwini Municipality; the creation of high-quality research opportunities for PIs and their students; closer working relationships between the different academic disciplines within UKZN; and the completion of masters and PhD studies. In the more recent reports and published papers, this array of outcomes is increasingly being understood under the broad banner of transdisciplinary research (Cockburn et al. 2016). This literature has articulated many of the successes achieved by the CRRP in terms of laying the foundation for transdisciplinary research and practice. However, as many of the people interviewed noted, there are institutional and systemic challenges to building transdisciplinary capacity at the individual, organisational and partnership levels. These challenges become easier to understand once the underlying mechanisms are made more explicit.

Moving beyond measurable outputs and outcomes to an understanding of the underlying mechanisms that enable or hinder these outcomes requires evaluation methods that are capable of accessing hidden casual influences. The realist evaluation method first developed by Pawson and Tilley opens up possibilities for understanding ‘how’ and ‘why’ particular outputs and outcomes were achieved and, more specifically, for whom they were achieved within the CRRP. The depth ontology developed by Bhaskar challenged the evaluator to identify what mechanisms must exist for observations and experiences occurring in the empirical domain and events in the actual domain to be as they are. A process of inference known as retroduction allowed the evaluator to move from a description and analysis of concrete phenomena including interview data to understanding the basic conditions for these phenomena to be what they are. By working back and forth across the six stages outlined in the procedure section of this article (with a particular emphasis on the use of a range of transfactual questions and related literature to support a process of analysis and coding), the evaluator identified a number of mechanisms. These mechanisms are linked to particular orientations to research, education, value creation and environmental management. This process of making the mechanisms within transdisciplinary research more explicit supports new ways of bringing about the structural changes required at the interface between municipal management and academic research to address complex sustainable development challenges.

Acknowledgements

The authors acknowledge the clarifying discussions with Prof. Mathieu Rouget and Chantal Janks (née Taylor) as they conceptualised and worked on the evaluation. The authors would also like to thank the many people who gave so freely of their time to participate in the interviews, groups discussions and questionnaires. These include Errol Douwes, Angelique Lazarus, Lutendo Mugwedi, Kusasalethu Sithole, Vishalan Pillay, Karabo Motau, S’vumile Mbatha, Sibusisiwe Mnembe, Rerani Ramaano, Rashieda Davids, Hloniphani Mthunzi, Prof. Mathieu Rouget, Chantal Janks, Dr Brenda Moodley and Suveshnee Munien.

Competing interests

Mike Ward conducted the original evaluation on which this article is based as a paid consultant to the University of KwaZulu-Natal. The contract ended in 2019. Eureta Rosenberg has no conflict of interest. This article is based on an independent external evaluation that was commissioned and paid for by the Community Reforestation Research Programme.

Authors’ contributions

M.R.W. and E.R. contributed equally to this research article.

Ethical considerations

Ethical consideration of this study was approved by the University of KwaZulu-Natal – HSS/0724/013.

Funding information

The original evaluation received funding from the Durban Research and Action Partnership (D’RAP). No funding was received for the development of this article.

Data availability statement

Data sharing is not applicable to this article as no new data were created or analysed in this study.

Disclaimer

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

Reference

Astbury, B. & Leeuw, F.L., 2010, ‘Unpacking black boxes: Mechanisms and theory building in evaluation’, American Journal of Evaluation 31(3), 363–381. https://doi.org/10.1177/1098214010371972

Bertolli, L., Teixeira-Leite, A. & Macfarlane, D., 2013, Buffelsdraai reforestation programme: Monitoring report of planting activities, report, Eco-Pulse Environmental Consulting Services for Wildla, eThekwini Municipality, Durban.

Bhaskar, R., 2008, Dialectic: The pulse of freedom, classical texts in critical realism, Routledge, London.

Bhaskar, R. & Hartwig, M., 2016, Enlightened common sense: The philosophy of critical realism, Ontological explorations, Routledge, New York, NY.

Bieluch, K.H., Bell, K.P., Teisl, M.F., Lindenfeld, L.A., Leahy, J. & Silka, L., 2017, ‘Transdisciplinary research partnerships in sustainability science: An examination of stakeholder participation preferences’, Sustainability Science 12, 87–104. https://doi.org/10.1007/s11625-016-0360-x

Blaikie, N.W.H., 2000, Designing social research: The logic of anticipation, Polity Press, Cambridge, MA.

Cockburn, J., Rouget, M., Slotow, R., Roberts, D., Boon, R., Douwes, E. et al., 2016, ‘How to build science-action partnerships for local land-use planning and management: Lessons from Durban, South Africa’, Ecology and Society 21(1), 28. https://doi.org/10.5751/ES-08109-210128

Cundill, G., Roux, D.J. & Parker, J.N., 2015, ‘Nurturing communities of practice for transdisciplinary research’, Ecology and Society 20(2), 22. https://doi.org/10.5751/ES-07580-200222

Dalkin, S.M., Greenhalgh, J., Jones, D., Cunningham, B. & Lhussier, M., 2015, ‘What’s in a mechanism? Development of a key concept in realist evaluation’, Implementation Science 10, 49. https://doi.org/10.1186/s13012-015-0237-x

Danermark, B., Ekstrom, M., Jakobsen, L. & Karlsson, J. (eds.), 2002, Explaining society: Critical realism in the social sciences, critical realism-interventions, Routledge, London.

De Souza, D.E., 2013, ‘Elaborating the context-mechanism-outcome configuration (CMOc) in realist evaluation: A critical realist perspective’, Evaluation 19(2), 141–154. https://doi.org/10.1177/1356389013485194

Douwes, E., 2017, Researching reforestation in Durban, eThekwini Municipality, Durban.

Douwes, E., Roy, K.E., Diederichs-Mander, N., Mavundla, K. & Roberts, D., 2015, The Buffelsdraai Landfill Site Community Reforestation Project: Leading the way in Community Ecosystem-Based Adaptation to Climate Change, eThekwini Municipality, Durban.

Egoh, B., 2015, Buffelsdraai reforestation project: Monitoring framework for biodiversity and ecosystem services, eThekwini Municipality, Durban.

Greater Capital, 2011, Final report to wildlands conservation trust: Social assessment of the Buffelsdraai landfill site community reforestation project, Unpublished report, eThekwini Municipality, Durban.

Jennings, P.L., 2015, ‘Critical realism: An alternative perspective on evaluation methodology’, PhD thesis, University of Warwick.

Lang, D.J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P. et al., 2012, ‘Transdisciplinary research in sustainability science: Practice, principles, and challenges’, Sustainability Science 7, 25–43. https://doi.org/10.1007/s11625-011-0149-x

Lotz-Sisitka, H. & Price, L., 2016, Critical realism, environmental learning, and social-ecological change, Routledge, London.

Macfarlane, D., Harvey, J. & Hamer, M., 2011, Biodiversity assessment of the Buffelsdraai Landfill Site Community Reforestation Project. Report No. EP 08-01, eThekwini Municipality, Durban.

Pawson, R. & Tilley, N., 1997, Realistic evaluation, Sage, London.

Pawson, R. & Tilley, N., 2004, Realist evaluation, Sage, London.

Polk, M., 2014, ‘Achieving the promise of transdisciplinarity: A critical exploration of the relationship between transdisciplinary research and societal problem solving’, Sustainability 9, 439–451. https://doi.org/10.1007/s11625-014-0247-7

Roberts, D., Boon, R., Diederichs, N., Douwes, E. Govender, N., McInnes, A. et al., 2012, ‘Exploring ecosystem based adaptation in Durban, South Africa: “Learning-by-doing” at the local government coal face’, Environment and Urban ization 24(1), 167–195.

Rouget, M. & Cockburn, J., 2014, KwaZulu-Natal Sandstone Sourveld research programme: Close-out report phase 1 July 2011–2014, University of KwaZulu-Natal, Pietermaritzburg.

Scholz, R.W. & Steiner, G., 2015a, ‘The real type and ideal type of transdisciplinary processes: Part II–what constraints and obstacles do we meet in practice?’, Sustainability Science 10(4), 653–671. https://doi.org/10.1007/s11625-015-0327-3

Scholz, R.W. & Steiner, G., 2015b, ‘The real type and ideal type of transdisciplinary processes: Part I–theoretical foundations’, Sustainability Science 10, 527–544. https://doi.org/10.1007/s11625-015-0326-4

Steelman, T., Nichols, E.G., James, A., Bradford, L., Ebersöhn, L., Scherman, V. et al., 2015, ‘Practicing the science of sustainability: The challenges of transdisciplinarity in a developing world context’, Sustainability Science 10, 581–599. https://doi.org/10.1007/s11625-015-0334-4

Swilling, M., 2014, ‘Rethinking the science-policy interface in South Africa: Experiments in knowledge co-production’, South African Journal of Science 110(5/6), 1–7. https://doi.org/10.1590/sajs.2014/20130265

Taylor, C., Cockburn, J., Rouget, M., Ray-Mukherjee, J., Mukherjee, S., Slotow, R. et al., 2016, ‘Evaluating the outcomes and processes of a research-action partnership: The need for continuous, reflective evaluation’, Bothalia 46(2), a2154. https://doi.org/10.4102/abc.v46i2.2154

The Cirrus Group, n.d., Buffelsdraai landfill site community reforestation project: Community, climate and biodiversity standard project design document, Environmental Planning and Climate Protection Department, eThekwini Municipality, Durban.

University of KwaZulu-Natal & eThekwini Municipality, 2013, Memorandum of agreement regarding collaborative research project between the University of KwaZulu-Natal and the eThekwini municipality in the area of reforestation within the municipality, Environmental Planning and Climate Protection Department, eThekwini Municipality, Durban.

Virkkunen, J. & Newnham, D.S., 2013, The change laboratory: A tool for collaborative development of work and education, Sense Publishers, Rotterdam.

Wiek, A., Farioli, F., Fukushi, K. & Yarime, M., 2012, ‘Sustainability science: Bridging the gap between science and society’, Sustainability Science 7, 1–4. https://doi.org/10.1007/s11625-011-0154-0



Crossref Citations

No related citations found.