A study which has provided a useful framing for graduate-level competency needs is a learning needs assessment for the green economy policy-practice context in South Africa (PAGE 2016) conducted for Partnership for Action on the Green Economy (PAGE) under the auspices of the United Nations Institute for Training and Research (UNITAR), the International Labour Organisation (ILO), DEFF (then Department of Environment Affairs) and others. The assessment determined what skills policy practitioners need to advance a green economy, and what educational interventions would be suitable to produce such skills. This was the second such assessment undertaken through PAGE, the first being in Ghana. The South African study featured a mixed methods approach consisting of focus group discussions with experts and stakeholders; online questionnaires; document analysis for a policy review; mini-case studies based on document analysis and telephonic interviews; an education provider audit; and face-to-face in-depth interviews with practitioners and educators. It resulted in a range of competencies identified and clustered (PAGE 2016).

Two outcomes of the learning needs assessment are relevant to the current article. The first is the value of the conceptual framework that was used to determine learning needs. This framework conceptualised learning needs as ‘competencies’ (as the plural to competence), which was taken to mean a complex of intertwined knowledge, values/dispositions and application skills. This framing was in turn adapted from competency research in the sustainability sciences (Wiek, Withycombe & Redman 2011). The range of competencies described by Wiek et al. was synthesised by using the framework of Scharmer (2009), who identified that leaders in challenging contexts need a combination of technical, relational and transformational knowledge. In the Green Economy Learning Assessment (GELA) Scharmer’s framework was re-articulated as technical, relational and transformational competence, to denote the need for a combination of knowledge, skills and values, and then tested with experts and stakeholders (Figure 2).

The feedback from respondents in the GELA was that the framework was challenging to use as it required much careful thought, but that it reflected well the range of learning needs these experts have been encountering in the field. The technical–relational–transformational (T-R-T) framework was then used to survey learning needs through online questionnaires and interviews. Respondents were asked: What technical competencies do policy-practitioners need in your field? What relational competencies? What transformational competencies? Definitions were provided and examples given. The study yielded a wealth of information which was used to map out clusters of learning needs and produce the GELA report (PAGE 2016). The researchers went on to use it to reflect on educational programmes in university contexts (Rosenberg, Lotz-Sisitka & Ramsarup 2018).

In this article, the same T-R-T framework is used to reflect on learning needs observed in the field of evaluation practice in which the authors work. Before this field is discussed, it is important to consider whether this framework, developed in a green economy policy-practice context, is relevant for the field of M&E. This article proposes that it is, based on the following. Firstly, the field and practice of M&E is contending with a changing world, characterised by complex social–ecological challenges noted earlier (and described, for example, by Patton 2010 and Funnell & Rogers 2011). A framing emerging out of the sustainability sciences (spanning multi- and transdisciplinary contexts) could therefore be appropriate to be further tested for its relevance. Secondly (and this is the second GELA finding relevant to this article), the GELA identified as one of the learning needs for the green economy context, the need for evaluation competence, further qualified as reflexive, transdisciplinary and supporting social learning (PAGE 2016). The finding suggests that there is a basis for applying the framework to try and describe what expertise evaluation practitioners need, even if only in this field (see the section on Limitations of the study).

The next part of the article describes the sectors or field of practice in which the participatory observations had been made, and to which the T-R-T framework was applied. Monitoring and evaluation takes place in a variety of contexts: basic education, higher education, public health, and more. In this case, the field of practice can broadly be described as environment and society. This field is multi- and transdisciplinary in nature, and draws on complex systems thinking, social and organisational learning, adaptive management and ecological economics, among other theoretical trajectories. It involves social, ecological, organisational and educational sciences, and the confluences between them. Examples are climate change adaptation and resilience programmes, social–ecological landscape and river catchment programmes with social learning components, organisational learning in conservation agencies and protected areas, and education for sustainable development (ESD) and capacity-building. Despite their diversity, these are all contexts in which societal change, social learning and new practices are key objectives, underpinned by concerns about economic and broader social justice in the face of development failures, climate change and sustainability issues in general. Although such programmes are rolling out across the globe, the projects reviewed are from Southern Africa. Funding is typically from national government, international donors or corporates. The approaches to M&E in the reviewed projects varied, but most were developmental in nature (sensu Patton 2010), mostly formative and also summative, often participatory, and involved evaluators with both internal and external roles. The M&E systems observed seldom involved only a one-off event in the life of the programme, or only external evaluators. Furthermore, the M&E approaches used typically required both M&E as an ‘interconnected system’ where ongoing monitoring provides the data for periodic evaluative activities (cf. Abrahams 2015:1 footnote). Finally, the programmes evaluated generally had an explicit transformation agenda. The findings that follow should be interpreted in this light.

There is not a one-to-one correspondence between the bullets in the two columns; several of the listed competencies may have been observed in one project, and most competency needs were observed in more than one project (Table 1). Competency needs were identified through two kinds of observations during the authors’ participation in the design and implementation of M&E systems in the field, either that: ‘We are able to do this because we have this competency (in the M&E team)’ or ‘We are unable to do this because we lack this competency’. The years listed indicate the periods during which observations were made, rather than the duration of the project or its M&E.

In Table 2 the observations in Table 1, column 2, are framed as evaluator competencies and grouped using the T-R-T framing.

What follows is an explication of findings in Tables 1 and 2.

The authors’ observations confirmed the importance of technical competencies, which is well established and documented elsewhere, for example in evaluator competency lists drawn up by SAMEA, the American Evaluation Association (AEA 2018) and the Canadian Evaluation Society (CES 2018). Evaluators are technically competent when they can design and implement standard M&E procedures and practices that are tried and tested. In South Africa the website of the Department of Planning, Monitoring and Evaluation (DPME), for example, provides guidelines and templates for diverse standard evaluation types and procedures that comprise technicalities with which evaluators should be familiar. Observations also confirmed the importance of newer skills such as working with programme theory (popularised as ‘theories of change’) and conducting developmental evaluations. These were particularly important in the complex contexts of many of the programmes in Table 1, column 1, where M&E was required to support organisations in continuously learning from implementation (developmental evaluation) and in the process, refining how they understood their intervention and how it would result in the desired changes. This work, while technical in nature, also had a strong relational dimension (see below).

The need for relational competencies (row 2, column 2 in Table 1) was evident whenever evaluators applied their technical skills in the field, where they encountered a range of challenges requiring ‘relational’ responses. The authors observed that some evaluators were able to build trust, confidence, rapport and enthusiasm for M&E, whereas others gave up, or expressed a concern that engaging with programme staff would compromise objectivity and credibility. Evaluators with relational competence were asked to return and do more evaluations, as they were regarded as helpful contributors to a shared transformational endeavour. This achievement, however, cannot come at the cost of evaluator credibility. Funders and other stakeholders need to experience evaluations as credible. Thus, relational competence is not useful in the absence of technical competence to produce valid findings. Both were needed.

The combination of technical and relational competence was also evident in ‘theory of change’ work in which the authors participated. The evaluator had to lead implementers in articulating their programme theory; this required not only a deep understanding of the process and its purposes, as well as the stages through which to do it (technical competence), but also a sensitivity to the context and ability to draw often implicit insights from implementers (relational competence).

Findings regarding transformational competence are listed in the last row in Table 1. These relate to the evaluator’s ability to ‘come up with something new’, either findings or methods, and the ability to see the need for something new (anticipatory competence).

Based on feedback from implementers (e.g. in the RESILIM-O programme) it was clear that evaluation findings must provide new insights, if they are to be useful to implementers with an interest in learning from evaluation, rather than merely ‘ticking the box’ for the funder. In all the projects reviewed, that substantive knowledge of the field in which M&E was being done, proved essential for developing such new insights.

Substantive knowledge of the field or domain in which the evaluation took place also seemed to be required for evaluators to establish trust and credibility with programme implementers; to guide the context-sensitive development of theories of change and indicators; to design case study instruments; and to analyse findings with some depth.

A case vignette illustrates the role of domain knowledge. In the Tsitsa Project, programme staff invited the M&E officer along on a field trip of the ‘Wisdom Trust’, a group of experts in the transdisciplinary social–ecological sciences that form the substantive domain of this programme. It was seen as a valuable opportunity for the officer to learn about the ideas that shaped the programme, to support her in developing the indicators for monitoring and for conceptualising evaluative case studies. The field trip, however, proved less than helpful in this regard: the experts’ discussions assumed a lot of background knowledge which the M&E officer lacked, generally preventing her from gaining the requisite insights.

It was also noticeable, in the transdisciplinary contexts of the Tsitsa and RESILIM-O, that evaluators must be able to flexibly use a variety of data sources and ‘knowings’. Knowledge sources ranged from scientists to government officials to small-scale farmers, and opinions often varied among respondents. How do evaluators come to credible conclusions? How do they effectively combine quantitative information on numbers of people trained or numbers of hectares affected by the programme, with qualitative data on the nature of training or the nature of the changes effected on the land? How do they do so across scales? Implementers in the Tsitsa, RESILIM-O and SANParks projects, who used complexity-sensitive evaluation approaches for ‘on-the-ground’ learning, while also reporting upwards to funders or line departments, expressed difficulty in bringing different M&E approaches together for sense-making and learning across system levels. For the evaluators to do this, they needed knowledge and skills that could be called transformational competence as it not only transforms knowledge forms into new syntheses, but also transforms standard M&E methods focussed more on accountability, to hybrid methods that meet both accountability and learning needs.

Table 2 does not show a comprehensive list of evaluator competencies. It however indicates an expanding range of competencies that evaluators seem to need in a changing world.

This article does not focus strongly on technical competencies, but this does not imply that they are unimportant, or in good supply. Podems et al. (2014) referred to a DPME survey which found technical competence (particularly evaluation competencies) to be in short supply in South Africa. Technical competence has however generally been well described, and the methods for developing them may be more well developed (see discussion on Table 3).

Relational competence is also described in various forms in the literature. The competence lists drawn up by SAMEA, the AEA and the CES, for example, all make reference to interpersonal skills, communication skills and cultural sensitivity. The relational competency needs observed in the field included but also exceeded interpersonal skills; it involved an ability to engage not only individuals, but to relate the engagement with individuals to a wider context, to understand and explain the programme and its M&E in the wider context, and to create a context in which stakeholders are motivated to collaborate and contribute. Evaluators drew on relational competence to work with programme staff to build and strengthen the culture of learning in organisations. The evaluator had to ‘read’ the multi-layered organisational context and relationally respond to it. Garcia (2016) conducted an in-depth study on interpersonal competence and situational awareness. The relational competence described here perhaps comprises a combination of interpersonal skills and situational awareness.

Although the competencies described above are generally represented in SAMEA’s and other competency frameworks, the framing proposed here creates room for transformational competence. This includes the ability to gain new insights from evaluation findings, as well as the ability to see the need for and develop alternative evaluation methods and processes. This competence includes a commitment to credible empirical findings rather than either empiricism or relativity. The authors propose that it requires a grasp of depth ontology (sensu Bhaskar’s critical realism, see ed. Bhaskar 2010) that enables integration across multiple layers of reality, from personal meanings and social constructs to physical realities, and to use a variety of M&E methods coherently. These transformational competencies are not represented in evaluator competency lists from South Africa, the USA or Canada.

A possible reason for the absence of transformational competencies from these lists may be the lack of a strong focus on, or the lack of evaluator experience with, developmental evaluation approaches. Developmental evaluation, as a response to complexity (Patton 2010), brings M&E much more closely into organisational management, planning and governance so that the evaluator is a member of the ‘core’ team, and M&E is more integrated into the functioning of the organisation (Dozois, Langlois & Blanchet-Cohen 2010). This requires more emphasis on relational and transformational skills, which need to be applied on an ongoing basis. The evaluator must be able to play the role of ‘critical friend’ and raise difficult questions without jeopardising the relationship. There is also an increased need for leadership and advocacy by the evaluator, to build capacity for evaluative thinking over time, create a sense of ownership and shift the responsibility for making meaning from the evaluator to the whole team (Dozois et al. 2010).

Monitoring and evaluation methods and approaches that respond to complexity, and generate multiple data types, are not unanticipated in the literature. However, they do not currently feature strongly in short courses or higher education programmes for evaluators, and even when they do, much is still to be worked out in practice. An ability to see and understand the need for complexity-sensitive M&E, and in particular, the ability to design, implement and adapt such approaches in the field, emerged as a very significant evaluator competence required at this time, alongside the technical and relational competences that are also essential but perhaps more established.

The professional judgement or discernment that is required to effectively conduct M&E in the absence of blueprints, in complex contexts, fall in a category that Freidson (2001) called a professional logic (distinct from the logics of the market and bureaucracy). This links to Podems’ contention that evaluation is an art. Although the authors would suggest that it is also a science, the implications of the metaphor of art, which implies high levels of skill and judgement on the part of the professional evaluator, certainly seem apt.

This might suggest that the findings shared in this article pertain mostly to the senior evaluator, or evaluation leader, but the authors have also observed the need for transformational and relational skills for the early career evaluators in an evaluation team, given that they are often the interface with the programme implementers.

The clustering of competencies in groups, here labelled as technical, relational and transformational, is significant, and so is the intertwining of knowledge, values and skills, as competencies. This should be compared to lists of competencies, or lists of knowledge, skills and values, described by Wiek et al. (2011) as ‘laundry lists’. There are several reasons to be cautious about ‘laundry lists’ or discrete bits of knowledge, skills and values. They lend themselves to narrow interpretations of how evaluations should be done, and of who should be selected to do evaluations. Podems (2014) commented on the gatekeeping role of competency lists, when they become bureaucratised, that produce the risk of keeping out not only some evaluators, but also different perspectives on evaluation. The competency approach in general has been criticised by educators who point to the holistic way in which professionals (such as educators) use their knowledge, values and skills in their professional practice, and that this is lost when atomised. Wiek et al. (2011), too, were concerned about the limited value of the competency list approach and hence proposed the ‘clustering’ approach, which was followed in the GELA and in this article. It is the view of the authors that the ‘cluster’ better reflects the way in which knowledge and skills from different domains, and dispositions, are deployed by evaluators. Above all, the combination of competencies was vital.

In simple contexts, with few variables that can be easily manipulated, a small set of technical skills, interpersonal skills like effective communications and a bureaucratic logic may suffice. In the changing world in which development contexts present as complex, open systems with multiple interacting variables and emergence (as in climate change programmes or pandemic responses) more is needed to ensure that M&E is able to support learning and a world that is changing for the better.

In complex contexts learning from M&E is critical, and theory-driven and developmental evaluations are prominent as programme developers and donors alike search for new ways to undertake M&E. In these contexts, evaluators need not only standard technical M&E competencies as well as relational competencies, but also transformational competencies.

Transformational competencies, generally not included in existing evaluator competency lists from South Africa and abroad, refer to the abilities that evaluators in complex contexts need to work effectively across knowledge boundaries (e.g. M&E expertise combined with knowledge of the substantive domain of the programme being evaluated); to design new but nonetheless credible evaluation methods where standard M&E approaches seem inadequate (e.g. they are not complexity-sensitive); to use multiple data sources and ‘ways of knowing’ with a sound understanding of ontology to draw credible conclusions; and to reflect and adapt on an ongoing basis. That is, they require evaluator reflexivity.

The reflections shared here, and the framework on which they were based, broadened the authors’ understanding of the competencies evaluators need, necessitated by the dynamically shifting context in which we work, where long-standing assumptions about how we need to develop and how we need to evaluate development, are being challenged by awareness of ‘the shadow side of modernity’, which includes the ways in which the sciences – social, natural and economic as well as evaluation – have either contributed to social–ecological problems, or failed to address them.

Technical–relational–transformational competencies was found to be a useful framing for understanding and discussing evaluator competencies, as it allows for a good grasp of the complex, interconnected knowledge, skills and dispositions and full range of competencies that evaluators need. This in turn allows educators to think of the design of programmes that may usefully include opportunities to explore complex contexts and try out different approaches, including engagements with real-life challenges and with other role players.

The findings should be considered in context. The T-R-T framework and identified competencies are perhaps most Would it be possible to space these words slightly further apart, please? transdisciplinary contexts. Its relevance in contexts like Public Health and Basic Education could be further explored.

A limitation of the competency framing applied here is that it individualises practice, which in the field is seldom the remit of isolated individuals alone. Although competency frameworks focus on what individuals should be able to do, most complex tasks – such as designing and undertaking a big evaluation – are undertaken by people in functional groups. Space does not allow for elaboration of this point here. The tendency of interpreting or presenting competency frameworks as a list of discrete skills residing in individuals is associated with assessment and accreditation practices, but need not shape the way in which educators design and deliver courses. That is, courses could create opportunities for learners to work together, support and complement each other as they would do in professional practice, learn from and with each other, and in the process develop the required sensitivities and skills that form part of relational competence, and the anticipation and discernment that contribute to transformational competence.