Before launching into the methodological aspects of our investigation, we need to address a few caveats. Firstly, programme evaluability is a dated concept (notions of evaluability emerged in the early 1970s), with inconsistent operationalisations articulated in evaluability checklists and guidance material (Davies 2013). These operationalisations are often unqualified (i.e. there are no explicit weightings tied to the different evaluability criteria articulated, conveying the mistaken assumption that they are all of equal importance). However, there appears to be some consensus amongst international development agencies on the general meaning of the term ‘evaluability’. The following working definition from the Organisation for Economic Cooperation and Development’s (OECD) Development Assistance Committee (DAC) is widely quoted and used: ‘the extent to which an activity or project can be evaluated in a reliable and credible fashion’ (OECD-DAC 2010:21). Embedded in this definition is the notion of feasibility and the sentiment that we can ‘literally evaluate anything, at least in some way, at some level, and at a certain cost’ (Finckenauer, Margaryan & Sullivan 2005:266), but not necessarily in a reliable and credible manner. This definition can be further expanded to capture the evolving purposes of EAs since the inception of the method in the late 1970s – from determining programme readiness for summative evaluations (Wholey 1979) to a much broader scope, which includes ensuring that relevant and technically feasible evaluations are conducted, maximising evaluation utility, building evaluation capacity and determining the feasibility of implementing the desired evaluation design (Davies 2013; Leviton et al. 2010). This is a difficult undertaking given that there is no consensus on what EAs should achieve. One could argue, however, that a comprehensive definition of evaluability should, at the very least, tap into some (if not all) of the utility, feasibility, propriety, accuracy and accountability standards of the Joint Committee on Standards for Educational Evaluation (JCSEE) (Yarborough et al. 2019), and have at its core a recognition of the desired type of evaluation.

Secondly, EA process models that prescribe the procedural steps that underlie EAs are often discussed in isolation of the operationalisations or evaluability criteria embedded in evaluability checklists. The development of EA process models appears to be within the remit of evaluation theorists, whilst international development agencies such as the United Nations Fund for Women (UNIFEM) concentrate on the development of evaluability checklists applicable to their programmes. For instance, the UNIFEM checklist consists of 17 questions framed around evaluability parameters of programme design, availability of information and conduciveness of the context. Existing EA models represent adaptations of Wholey’s (1979) original eight-step model, which he later refined into six-step model (Wholey 2004), articulating the following iterative process: (1) involving intended users of the evaluation, (2) clarifying the intended programme, (3) exploring programme reality, (4) reaching agreement on any required programme changes, (5) exploring alternative evaluation designs and (6) agreeing on evaluation priorities and intended uses of information. Examples of adaptations include Kaufman-Levy and Poulin’s (2003) five-task model for EAs, Smith’s (1981) 10-step EA model and Thurston and Potvin’s (2003) seven-step framework. Despite the proliferation of EA models, with reasonably well-defined procedural steps, there is no agreed-upon operationalisation of what a well-executed EA process should look like (Trevisan 2007; Watts & Washington 2016).

This brings us to our third caveat: evaluability is not an absolute condition. Rather, it occurs along a continuum from more to less evaluable. A categorical judgement of evaluability is not only restrictive, but also at odds with many evaluation approaches and the broader scope of EAs nowadays (Davies 2013).

The empirical literature on programme evaluability is sparse and little is known about how evaluators operationalise prescriptive articulations of evaluability. The purpose of this article is to refocus our attention on the concept and its empirical operationalisation(s). Our stance is that we need sharper insight into the ‘what’ of EA, from evaluators’ perspectives, before deliberating on issues relating to the EA method or process (i.e. the ‘how’ of EAs) – most of which stem from rationalistic assumptions that underlie the EA process (Dahler-Larsen 2012). As such, our study aims to explore inductively whether or not evaluators have a common perspective towards evaluability, despite the ambiguous articulation of the concept of evaluability in the literature. More specifically, it addresses the following research question:

We draw on Kundin’s (2010) framework for studying evaluators’ practice decisions. The framework isolates three key elements that might shape an evaluator’s decision-making process: situation awareness, practical reasoning and reflection in action.

What we are interested in capturing in our study is working logic – a component of practical reasoning, which goes hand in hand with general logic. The first type of logic ‘specifies the game and the rules of the game that one is playing when conducting an evaluation in any field’ (Fournier 1995:17).

The second type of logic, also referred to as logic in use or reconstructed logic, represents the different operationalisations of general logic. In other words, they represent the variations in application of general logic in practice.

In our view, EA is a systematic ‘game’ with abstract ‘rules’ and room for variations in application. These variations can be isolated empirically by reconstructing evaluators’ working logic of both the EA method or process (i.e. the procedural steps that underlie an EA) and the evaluability criteria to be prioritised in an EA. We are interested in the latter.

We take the study one step further by contrasting the evaluability perspectives of four different evaluator cohorts practising in different contexts. Theorists have long recognised the role of context in shaping evaluation practice (e.g. Stake 1990). Despite the growing emphasis on the role of context in evaluation, there is no unified understanding of what context means and how exactly it influences evaluation practice (Dahler-Larsen & Schwandt 2012). We are interested in the practice of evaluation in different countries. As such, a secondary aim of our study is to explore comparatively how evaluators from selected countries operationalise programme evaluability.

We suspect that evaluators who practise in middle-income countries with emerging evaluation cultures might hold different evaluability perspectives than those who practise in upper-income countries with mature evaluation cultures. It is imperative that we subject this assumption to empirical testing and deliberate on its implications as evaluation is becoming ‘more global and more transnational’ and ‘problems and programs that we are called upon to evaluate today often extend beyond the boundaries of any one nation, any one continent, or even one hemisphere’ (Chelimsky & Shadish 1997:xii). The practice contexts (countries) we investigate in our study are described in more detail under the heading ‘Participants’.

A descriptive design was used in this study as we were primarily concerned with the collection and description of cross-sectional data relating to participants’ perspectives of programme evaluability.

Our focus was to analyse evaluators’ subjective operationalisations of programme evaluability. We did so by applying the Q-sort method (Stephenson 1935), a procedure that facilitates the systematic study of participant subjectivity and the different accounts that people construct (Cross 2005). Whilst Q studies have often been criticised as small-sample investigations of unknown reliability, such scepticisms have been discredited by authors such as Eghbalighazijahani, Hine and Kashyap (2013) and Brown (1993). We also drew on the literature to inform our decisions regarding the relative sizes of the P set and the Q set, and the condition of instruction, and as such are convinced that the Q-sort method, as applied in our study, is sufficiently robust for examining subjective operationalisations of programme evaluability.

In a Q study, participants are instructed to sort a set of randomly ordered statements relating to a specific topic into a subjectively meaningful pattern based on their individual preference or judgement – a procedure called Q sorting. The individual rankings are then subjected to a Q factor analysis. A defining feature of the Q-sort method is that statements relating to the same domain are not analysed individually but in the context of other equally relevant statements.

We chose to use the Q-sort method in this study as it is particularly well suited to studying the phenomena ‘in which there are numerous ideals present in a reality where only a limited number of ends or means can be realistically pursued’ (Thompson 1998:1). This is particularly evident in an evaluation context, where practical realities constrain evaluators to prioritise a set of evaluability criteria at the expense of others. The Q-sort method would allow us to simulate prioritisation patterns of participating evaluators and derive evaluability perspectives that embody variations in working logic.

We firstly examined the existing opinions and perspectives around the topic of evaluability by means of an exhaustive literature review (Boodhoo 2016). In Q methodology, this step refers to the definition of the concourse. In a second step – the development of the Q set – we extracted and synthesised the evaluability criteria most commonly cited in the concourse and categorised them under three dimensions: (1) programme characteristics or structural features, (2) methodological or logistical requirements and (3) stakeholder characteristics (Appendix 1). The seminal writings of Wholey (1979), Schmidt, Scanlon and Bell (1979) and Nay and Kay (1982), as well as the more recent works of Davies (2013) and Dahler-Larsen (2012), laid the foundation of this exercise because of their systematic and comprehensive coverage of ‘evaluable models’ and evaluability parameters (unlike other recent or previous EA-related publications, which tend to focus on the application of EA process models or evaluability criteria established in seminal writings).

We then standardised the formulation of the Q set by drawing on conceptual definitions specified by Rossi, Lipsey and Freeman (2004) and made further refinements based on common usage and pragmatism. This process culminated in a set of refined, standardised evaluability criteria, formulated in terms of 19 Q statements (see Table 1), which could be classified into one of the five ranking categories (ranging from not at all important to essential), according to their assigned importance.

In the Q-sort method, participants (referred to as the P set) are purposively selected and expected to possess clear and varied viewpoints on the topic under investigation – in our case, evaluators were recruited from Brazil, South Africa (SA), the United Kingdom (UK) and the United States of America (USA). These four practice contexts were selected based on their current socio-economic standing and the maturity of their evaluation cultures. We also capitalised on the relative ease of accessing our target population through the membership database of four professional associations (the Brazilian Monitoring and Evaluation Network, the South African Monitoring and Evaluation Association, the UK Evaluation Society and the American Evaluation Association) and our professional networks in each country.

Each country of interest could be categorised as a high-income or middle-income country (using the 2018 World Bank country classification system), with emerging or mature evaluation cultures (following an in-depth qualitative analysis of the historical trajectory of programme evaluation in each country, including its emergence and development). We recognise the inherent difficulty in ranking countries based on the maturity of their evaluation culture – a broad concept, with no perfect operationalisation (Dahler-Larsen & Boodhoo 2019; Jacob, Speer & Furubo 2015). We simply contend that Brazil and SA and the UK and USA could be considered, in broad terms, as comparable practice contexts. For example, Brazil and SA (both classified as developing countries) have similar historical, social and economic trajectories. Programme evaluation also emerged around the same time in both countries (in the 1990s), as part of their re-democratisation process (Abrahams 2015; Henriques et al. 2010). The firm and uniform institutionalisation of evaluation is, however, still underway in these two countries (Goldman et al. 2018; OECD 2017).

The USA and the UK, on the other hand, belong to a cohort of developed countries considered as pioneers in the development of public policy evaluation, with an evaluation tradition that dates back to the early 1960s (Gray & Jenkins 2002; Rist & Paliokas 2002). The USA and UK have also been characterised as having a high degree of evaluation culture maturity (Jacob et al. 2015).

Participants from the USA (n = 86) were highly experienced evaluators, with 28.7% having between 11 and 15 years of evaluation experience, and 30.9% holding a PhD degree in evaluation. Participants from the UK (n = 26) were also experienced evaluators, with 36.7% having between 6 and 10 years of evaluation experience. Most of these participants (63.3%), however, had not received any formal training in evaluation. The Brazilian and South African cohorts (n = 79; n = 38) were the least experienced, with 40.7% and 33.3%, respectively, having between 1 and 5 years of evaluation experience. Most of these evaluators were either self-educated or had completed a short course certificate in evaluation.

Our small sample size was not considered problematic, given our study aims and method of choice. The Q method attempts to isolate subjective structures in the data and the extent to which these are similar or dissimilar, rather than calculating the percentage of the sample or population that adheres to them (Eghbalighazijahani et al. 2013). The issue of generalisability is therefore not relevant here. In fact, there is no clear-cut rule on the minimum number of participants to include in a P set (Dziopa & Ahern 2011).

In addition, the focus of the method is not on the constructors (i.e. the participants) but on the accounts that they construct. As would be the case with any other topic, we expect only a limited number of distinct perspectives to exist on evaluability, and we are confident that our carefully selected P set is adequate to reveal these perspectives.

We collected our data over a period of 2 months following ethical clearance from the Ethics in Research Committee of the Faculty of Commerce, University of Cape Town, and the translation of our study materials into Portuguese for the Brazil cohort. We used an online data collection platform, given our geographically dispersed sample. Results of self-administered, electronic-based Q sorts have been found to be consistent with traditional methods of administration (Reber, Kaufman & Krop 2000).

We used a free-sort condition of instruction, whereby participants were instructed to distribute iteratively randomly ordered Q statements across five ranking categories (representing different gradients of importance), until all 19 Q statements had been sorted. We opted against a forced-choice condition of instruction (which ‘forces’ the placement of Q items in a fixed and specified distribution), in light of the reported difficulty experienced by participants in a small pilot study we ran prior to the four-country study. Whilst a free-sort condition of instruction will inevitably yield variations in the ‘shapes’ of item distributions, this is not expected to affect our overall factor analytic solution (Watts & Stenner 2012).

After completing the Q-sort, participants had to respond to 12 items relating to their current involvement in evaluation, employment setting, highest academic qualification, type of training in evaluation, level of experience in conducting different types of evaluations, practice context, as well as the number of evaluations that they conducted in the last 5 years.

We applied Q factor analysis (with principal component analysis as a method of factor extraction) across our four data sets to identify (1) dominant evaluability perspectives that may be unique to each cohort of evaluators and (2) the specific criteria that characterise those perspectives. What distinguishes Q factor analysis from conventional factor analysis is not the mathematics of the factor analytic process but the organisation of the raw data matrix, with rows representing Q statements and columns representing respondents’ Q sorts.

We did not identify any systematic sorting pattern that warranted the exclusion of specific Q sorts for the US, UK and South African evaluator cohorts. Three problematic cases were, however, identified amongst Brazilian respondents and thus deleted from subsequent analyses: one respondent allocated all 19 Q statements to the not at all important category, and two respondents used only the first two categories not at all important and quite unimportant to distribute the Q statements (distribution ratio of 18:1 and 17:2, respectively).

We first confirmed the factorability of the inter-correlation matrixes – all factored entities had a correlation of at least 0.3 with multiple other entities, and all communalities were well above 0.5. Only factors with eigenvalues greater than 1, and positioned to the left of the inflexion point in the scree plot, whilst cumulatively explaining at least 60% of the total variance, were retained for Varimax factor rotation.

We deleted the following cases after examining the initial rotated factor matrixes: (1) 32 respondents with no significant factor loadings on any factors (we applied a 0.50 cut-off point), (2) 11 respondents with multiple significant cross loadings and (3) 13 respondents who were not adequately accounted for by the factor solution (i.e. respondents with significant factor loadings but with communalities < 0.50). We re-ran the analysis and deleted problematic cases iteratively until a satisfactory factor solution with pure factor loadings (indicating well-defined perspectives) and high factor reliability was obtained for each evaluator cohort. The reliability of a factor is determined by the number of respondents that define it. According to Brown (1993), five respondents are sufficient to obtain a clear reading of the perspective embodied in a given factor. Any additional respondents would only marginally clarify the picture.

We then examined the factor scores, which indicate the importance of each Q statement in defining a given rotated factor. In line with Thompson (1998), we used a cut-off score of -1 to identify the most important and least important evaluability criteria associated with a particular factor.

As a final step, we interpreted the different factors or viewpoints that emerged from the Q analysis. In the absence of a set strategy for factor interpretation in the Q method literature, we applied the interpretative framework proposed by Watts and Stenner (2012) to arrive at a systematic and holistic interpretation of each factor. We developed a ‘crib sheet’ to organise the Q statements based on the size and rank order of their associated factor scores. The following distinctions were made: (1) statements with the highest rankings in each factor array, (2) statements with the lowest rankings in each factor array, (3) statements with higher rankings in a given factor array compared to other factor arrays and (4) statements with lower rankings in a given factor array compared to other factor arrays. We considered the positioning of each statement and applied the logic of abduction to generate iteratively the overall story underlying the various statement rankings and derive preliminary hypotheses that could account for a particular item configuration or factor array.

We also selected an appropriate label to represent each factor by examining the Q statements that distinguished them. Whilst factor labelling is not a methodological requirement, it conveys in a parsimonious manner what distinguishes factors from one another. Statements with positive factor scores (i.e. those characterised as essential) were given more weight in factor labelling. Whilst this approach does not in any way capture the complexity of a given viewpoint, we strove to label each perspective in a manner that best integrates all the distinguishing statements associated with it.

The final interpretation of each factor is presented in narrative form. The relevant statements were linked together to create a unified account of the viewpoint embodied in each of the factors identified.

Ethical clearance was obtained from the Ethics in Research Committee, Faculty of Commerce, University of Cape Town on 17 March 2014, no clearance number was issued at the time.

Factor 1 reflects a perspective that favours an explicit change logic and implementation fidelity but minimises the importance of logistical imperatives and stakeholder collaboration, authority and transparency. It would seem that the underlying focus is on opening the black box of evaluation and making the underlying assumptions and implementation of the programme activities clear. This perspective could be construed as the essence of a theory-driven evaluation approach and was labelled as such. Twenty-two respondents in our final USP set (n = 56) were most associated with factor 1.

Factor 2 seems to reflect evaluators’ concern with mechanisms that support the utilisation of results by intended users: stakeholder transparency, authority and consensus. We therefore labelled this perspective as utilisation-focused. Twenty-two respondents were most associated with factor 2.

The item configuration of both factors 3 and 4 is not related to any explicit notion of evaluation practice, thus making the characterisation of these perspectives problematic. Both factors also explained a relatively low proportion of variance (8.9% and 8.8%, respectively) and were defined by a small number of respondents (n = 6).

The first factor extracted for the UK cohort reflects a perspective that minimises the importance of stakeholder collaboration, authority and transparency, and certain logistical imperatives. Here, the underlying focus is on the ability to measure implementation fidelity and explain why a programme worked or did not work. This perspective is similar to the first factor identified for the US cohort. As such, we decided to label it as theory-driven. Eight respondents from our final UKP set (n = 13) were most associated with this perspective.

The second factor reflects a perspective that emphasises the need for (1) a logic model that articulates realistic and measurable outcomes and (2) stakeholders’ transparency and authority. An explicit programme delivery plan and certain logistical requirements are not of high priority. There seems to be a dual focus on programme theory, and the necessary conditions for utilisation of evaluation findings. This factor reflects a combined theory-driven and utilisation-focused perspective and was labelled as such. Five respondents were most associated with this perspective.

Fourteen respondents (out of the 36 in the Brazil P set) were most associated with the first factor. This perspective emphasises the need for an explicit and plausible theory of change, which operationalises clearly specified and agreed-upon programme goals. Evidence that the programme has been implemented with fidelity is also of high priority. It would seem that the underlying focus is on mechanisms that support the change process or programme success and the ability to explain why the programme worked or did not work. This bottom-up approach mirrors a theory-driven approach to evaluation. We decided to label this perspective as theory-driven.

Nine respondents were most associated with the second factor. Evaluators who shared this perspective prioritised stakeholder transparency, authority and consensus and fidelity of implementation. Issues pertaining to data collection, evaluation design and programme theory were assigned less importance. Given that the overall focus is on mechanisms that support the utilisation of findings, we labelled this perspective as utilisation-focused.

For factor 3, we could not meaningfully reconcile the entire item configuration in terms of evaluation practice and therefore decided not to retain it.

For factor 4, the emphasis was on the specification and proper implementation of the service delivery plan, availability and accessibility of data and sufficient budget to conduct the evaluation. Data quality, plausibility of the change logic and consensus on programme goals are of lower priority for the six evaluators who shared this perspective. The underlying focus appears to be on the minimum requirements to measure implementation fidelity and it was labelled as implementation-focused.

Ten respondents (out of 16 in the South African P set) were most associated with the first factor. Evaluators who shared this perspective seemed to prioritise (1) an explicit and plausible theory, articulating realistic and measurable outcomes for a specific target population; and (2) data for measuring implementation fidelity. Taken together, the underlying focus appears to be on opening the black box of evaluation, and the ability to explain why the programme worked or did not work. This perspective resonates with a theory-driven approach to evaluation and was labelled as such.

Six respondents were most associated with the second factor. Evaluators who shared this perspective prioritised stakeholder transparency, authority, consensus and collaboration. Issues pertaining to evaluation design, data collection and evaluation time frame are of less priority. Given that the overall focus is on mechanisms that support the utilisation of findings, we labelled this perspective as utilisation-focused.

Here, we deliberate on four questions that stem logically from the first finding of this study and present our preliminary thoughts.

Our second finding focused on the evaluability perspectives that were shared by all or most evaluator cohorts. We frame our discussion around the following three questions.