Successful evaluation capacity development (ECD) at regional, national and institutional levels has been built on a sound understanding of the opportunities and constraints in establishing and sustaining a monitoring and evaluation system. Diagnostics are one of the tools that ECD agents can use to better understand the nature of the ECD environment. Conventional diagnostics have typically focused on issues related to technical capacity and the ‘bridging of the gap’ between evaluation supply and demand. In so doing, they risk overlooking the more subtle organisational and environmental factors that lie outside the conventional diagnostic lens.
As a result of programming and dialogue carried out by the Centre for Learning on Evaluation and Results Anglophone Africa engaging with government planners, evaluators, civil society groups and voluntary organisations, the author has developed a modified diagnostic tool that extends the scope of conventional analysis.
This article outlines the six-sphere framework that can be used to extend the scope of such diagnostics to include considerations of the political environment, trust and collaboration between key stakeholders and the principles and values that underpin the whole system. The framework employs a graphic device that allows the capture and organisation of structural knowledge relating to the ECD environment.
The article describes the framework in relation to other organisational development tools and gives some examples of how it can be used to make sense of the ECD environment. It highlights the potential of the framework to contribute to a more nuanced understanding of the ECD environment using a structured diagnostic approach and to move beyond conventional supply and demand models.
There is increasing evidence of a growing demand in Africa for evidence from monitoring and evaluation (M&E) systems (Porter & Goldman
The organisational development (OD) sector is replete with a range of assessment tools, many developed over years by respected applied behavioural scientists such as Burke and Litwin (
The application of graphic devices to the capture and presentation of structural knowledge has been an area of interest within the education sector for some years. Tony Buzan’s (
Mind mapping sits within a broader field of graphic techniques that attempt to represent knowledge that include spider mapping, pattern notes, concept maps, schematising and others (Beissner
Structural knowledge is defined by Beissner (
But what are the benefits of imparting structure to knowledge within a particular field? Beissner (
Graphic techniques to structure knowledge that have emerged over the last 40 years are wide and various; however, they may be categorised into one of two recognisable groupings (Eppler
So once the representations of structural knowledge have been constructed, what is their added value? Beissner (
While the interest in graphic techniques to represent structured knowledge has been driven principally by an interest to enhance understanding among learners, there is potential for the use of this technique to construct knowledge around the complex dynamics that exist within the environment for ECD.
Potential users of evaluation come to recognise that they can affect policy processes to their benefit through using evaluation and create demand. If managers and conductors of evaluation have the capacity, political understanding and funds, then they respond to the demand from users. If commissioning and use of evaluation becomes widespread, then virtuous cycles of evaluation capacity development take place, leading to more institutionalised evidence-based practice. (CLEAR AA
Conventional approach to M&E readiness assessment.
On reflection, the limitation of both the diagnostic and the ECD theory of change described above is that is assumes that:
There are sufficient surplus resources available within the institution or state to build M&E systems without compromising much-needed frontline services. States (and institutions) have adequately established democratic systems that have within them political space for M&E champions. Within these political spaces there is room for champions to challenge vested interests. There is sufficient trust and collaboration between institutions, government and key ECD stakeholders. All stakeholders hold a common vision for development (to which the use of evaluation as evidence of performance is seen to contribute).
While these may be reasonable assumptions in the majority of countries that are members of the Organisation for Economic Cooperation and Development (OECD), we might legitimately question the degree to which they hold across the diverse range of institutions and states that make up the African continent. Clearly, a more nuanced diagnostic framework is needed to guide us through the complexities that are the reality within non-OECD states.
Given the diversity of ECD contexts in which CLEAR AA is engaged within Africa, the
The six-sphere framework.
The framework is arranged as a hierarchy of thematic spheres that move progressively (from surface to core) through logistical, technical, contextual, relational, political and ideological domains. The presentation of spheres as ‘nested’ is intended to convey:
The explicit and immediate nature of factors that fall into the external logistical and technical spheres. The tacit and ‘deep-rooted’ nature of factors that fall into the central political and ideological spheres. A hierarchy of influence that flows from core to periphery.
While inner spheres are placed ‘internal’ to those at the periphery, the graphic is not intended to imply a relationship in terms of ‘sets’ and ‘sub-sets’ such as one would expect in a Venn diagram. The nature of each domain and how it relates to the ECD landscape is described below.
The logistical sphere encompasses considerations related to time and resources (available or not available) to generate monitoring and evaluation data and to move it along the demand and supply chain. On the supply side this includes a consideration of time and resources made available to evaluators to generate evaluations; on the demand side, time and resources made available to decision-makers who identify and use evaluations. The logistical sphere also encompasses a consideration of factors that impact the timeliness of information flowing through the system. In terms of ECD interventions the logistical sphere is the realm in which agents leverage funds, set timetables and schedules for change, promote access to information and raise awareness.
The logistical sphere encompasses considerations related to the evaluation technical capacity of individuals and departments both on the demand and supply side. On the supply side this may relate directly to the capacity of evaluators within the system in question and on the demand side, the capacity of decision-makers to interpret and use evaluation evidence.
Linked to the technical capacity of agents within the system is a consideration of the quality of M&E products that flow within it. Naturally, not all evaluation evidence is of equal quality; South Africa’s Department of Planning, Monitoring and Evaluation (DPME) has created a repository of all evaluations of government programmes carried out since 2006, which, in addition to making available a range of 83 recent evaluations, grades them in accordance with a specified quality assurance tool (DPME
Further to a consideration of the technical capacity of agents and the quality of evaluation products is a consideration of the
The contextual sphere encompasses considerations related to structures, linkages, networks and culture. These are described in a little more detail below.
At an institutional level, considerations of structure encompass the organisational arrangements and hierarchies within the institution that facilitate the production and use of M&E data. This may include reflection on the positioning of M&E units within the institution in question or at a broader level the positioning of M&E coordinating units or departments within the executive. Given that the positioning of M&E units or coordinating departments is on the one hand important in determining their perceived independence and objectivity, and on the other their clout and influence (Kusek & Rist
Aligned with a consideration of structure is a consideration of linkages and networks. Institutional arrangements between the coordinating bodies for M&E and line ministries or departments is a key factor in determining relationships between them and subsequently the ease with which evaluation evidence can flow. So too are linkages between government departments as users of evaluation evidence and the agencies that produce the data that feed them. A consideration of the linkages between government and civil society as a potential catalyst for demand for evaluation evidence is also encompassed within the contextual sphere, as is the extent to which media (including social media) connects citizens with government.
The contextual sphere also encompasses what is often referred to as the ‘culture of evaluation’ (Gorgens & Kusek
While structure, linkages and culture make up the contextual sphere, at a deeper level the contextual sphere is where incentives and disincentives may reside for the use of evaluations emerging from the political economy.
The relational sphere encompasses considerations related to levels of trust and collaboration between evaluation system stakeholders. If evaluation evidence is to flow efficiently throughout the system from producer to user, then it is important that there is collaboration and trust not only between producers and users but also between those who act as transmission points or gatekeepers along the chain. A lack of trust among stakeholders is likely to lead to evaluation evidence that is diverted, delayed or even discarded within the system. A chronic shortage of trust between stakeholders may lead to environments that are hostile towards the use of evidence and lead to perceptions that evaluations are tools designed to punish poor performance. It is interesting to note that while trust and collaboration by nature take time to form, they can rapidly be eroded.
The political sphere encompasses considerations around the extent to which leadership and vision for change exist at critical parts of the system. If a government-wide system is being considered then leadership within the higher levels of the executive and even the presidency may be necessary (as well as support within political parties). If an institution-wide system is being considered then consideration of the political sphere encompasses leadership among senior managers and directors. It is worth noting that while political leadership and vision may be necessary to support constructive change in terms of M&E, the establishing of such systems may run counter to significant political elements that may in turn challenge or even counter such efforts.
The ideological sphere encompasses considerations related to the extent to which the use of evaluation as evidence resonates with core principles and values at the heart of the institution or, at a wider level, the ideological underpinnings of the state. This might be reflected for example in the extent to which there is a demonstrated commitment to principles of transparency and accountability or deeply held values underpinning equitable development. In some cases, these underlying principles may not be explicit and might require some work to unearth them. This in itself may be regarded as an indication of the state of health of the ideological sphere. In others, the underlying principles may run counter to the use of evidence together with the transparency and accountability that comes ‘bundled’ with it.
As is apparent, readiness assessments such as the format outlined by Kusek and Rist (
In addition to the broadened scope of factors that are included for consideration, the proposed framework also offers some insights that emerge from the hierarchical arrangement of its dimensions. In this sense, the framework would also seem to qualify as a ‘theoretical thinking tool’ (Jerneck
At the core is an ideological belief. If this belief incorporates the principles of equitable development, transparency and accountability then in terms of establishing an M&E system, each subsequent level (being contingent on the previous) leads to a virtuous cascade, that is, if there is a ‘belief in equitable development and the value of accountability and transparency’ then there will likely be ‘political vision for M&E’; if there is ‘political vision for M&E’ there will likely be ‘trust and collaboration among M&E stakeholders’, if there is ‘trust and collaboration among M&E stakeholders’ then there will likely be interest in ‘building technical capacity’ and so on. Of course, if principles of transparency and accountability are not enshrined at the core of the system the opposite dynamic may be evident.
Levels of complexity in terms of what is needed to leverage change increase as we move inwards. At the periphery, we can think of leveraging resources and providing technical expertise as a means of tackling ECD logistical and technical challenges. Towards the centre however it becomes less clear as to how we shift political will and alter core beliefs, particularly if these do not resonate with values that underpin the use of evaluation as evidence. This suggests that developmental change takes place slowly at the centre (ideology and political vision) but may take place more rapidly as we move outwards (technical capacity and resources).
Surface domains (logistical and technical) are relatively straightforward to address through conventional interventions (e.g. allocating resources and capacity building). However, as we move inwards through contextual, social, political and ideological domains these become increasingly challenging to influence. At the core (ideological) we may have no direct influence over change at all; ECD interventions that seek to change institutions or states at these levels may be challenged themselves.
A further reflection on the framework suggests that the periphery (the logistical and technical spheres) is where evidence is generated, but that the centre (ideological, political and relational spheres) is key in determining whether evaluation evidence is used. In moving from the periphery to the centre M&E information is transformed progressively from data through to evidence and ultimately judgement. In a sense, the inner layers are essentially where the political economy operates while the outer layers are where evaluators operate.
Considering the framework as outlined in
Typical interventions here focus on allocation of resources, technical capacity building and developing the institutional context (Tarsilla
The framework also suggests that while ECD in developmental states may have as a starting point to allocate resources and build M&E technical capacity, unless there is accompanying support from the deeper levels, they are unlikely to be successful.
One of the attributes of the six-sphere framework as a visual tool is that the hierarchical arrangement of layers can generate insights through exploring relationships between ideas that are situated on different levels. Some examples of this process of integration (Beissner
Using the framework as a model, evidence can be thought of as being produced at the periphery whereas demand to use it emanates for the centre. As evidence is drawn from the periphery it is translated from raw data through the filters of institutional culture, political viewpoints and finally deeply held principles and beliefs before it is used to make judgement.
The extent to which an evaluation will be regarded as professional indicates a requirement to be both competent and objective (technical and contextual spheres). The extent to which it will be welcome may be additionally shaped by how easily it is aligned with political agendas. To be welcomed (and used) it therefore must resonate with both political agendas and ideological standpoints at the centre of the system (political and ideological spheres).
This visual analogy would seem to resonate in a very real sense with our experience that even though evaluations may be technically sound, the extent to which they spotlight good or bad performance inevitably determines the reception they have among senior decision-makers, policymakers and politicians. This journey across the relational and political spheres, where trust and collaboration between stakeholders and alignment with political agendas are major factors, means that the journey may sometimes be smooth but is more likely than not to be turbulent.
As well as providing some insights into the nature of institutions and how they might be expected to react to moves to systematise M&E, the framework also suggests a series of guiding questions that can be used within an M&E diagnostic. A selection of these is reproduced in
Diagnostic questions generated using the six-sphere framework.
Sphere | Key questions |
---|---|
Logistical | Are sufficient |
Technical | Does |
Contextual | Are structural or cultural aspects of the institutional |
Relational | To what extent is there the necessary trust and commitment and collaboration among key stakeholders (individuals, units, departments) that will allow M&E systems to work optimally? |
Political | To what extent is there leadership and a common |
Ideological | To what extent is the institution founded on shared values and principles that support the accountability and transparency that M&E systems bring? |
Used as an initial round of questions the framework has been used within CLEAR AA as a valuable structure from which to
Conversations and dialogues around ECD serve to highlight the contextual diversity and complexity of the sector. At recent events organised by CLEAR AA in Cotonou and Johannesburg in 2015, delegates from across 11 national governments, from both Anglophone and Francophone countries, identified variously resource allocation, knowledge production and communication, institutionalisation of evaluation, the policy environment, leadership, public sector attitudes, alignment with national plans and priorities, the engendering of a culture of evaluation, technical capacity building and the identification and supporting of champions as being among a range of focal areas that require attention if evaluation is to become sustainably rooted within state governance systems (CLEAR AA
As an illustration, S. Gariba, interviewed by K. Crawley, 12 October 2015, discusses the challenges of leveraging greater budget allocation to evaluative processes within government departments. Budgets are under the control of senior government policymakers who, if sufficiently convinced of the merits of using evaluation as evidence for decision-making, will likely advocate for allocation of resources accordingly. He proposes that the reluctance of such policymakers to advocate for greater use of evaluation may not necessarily stem from the fact that they do not perceive it as useful, but rather that they see evaluation as a highly technical field about which they know very little. As Gariba points out, the potential for a senior member of government to embarrass themselves by failing to convincingly elucidate on an approach that they are championing may outweigh wider potential benefits to the system. Yet senior policymakers are not normally among those who see fit to attend technical training courses. One solution to this hypothetical conundrum has been DPME’s initiative to provide technical orientation for senior policymakers using ECD training with the intention
The six-sphere framework in use.
While the narrative of the story above is fascinating, the challenge for proponents of ECD is to structure understanding and insights that emerge from such narratives in a manner that allows a coherent strategy to be constructed. Using the six-sphere framework to structurally reconstruct knowledge is potentially a powerful way to make sense of what we are hearing and to identify pathways for change in future.
Factors supporting or constraining the use of M&E evidence in decision-making are wide and various. Conventional approaches to ECD tend to focus on allocation of resources, building technical capacity and identifying champions to help ‘bridge the gap’ (ed. Segone
The tool has been shown to highlight linkages between factors across dimensional levels and to help to make sense of what is otherwise a complex tangle of issues. Participative use of the tool allows the uncovering and exploration of issues that may otherwise remain hidden.
The article outlines a framework that may be used as both a diagnostic tool to build understanding of the health of monitoring and evaluation systems and a framework to design evaluation capacity development interventions. The tool has been developed by the author and used to inform the work of the Centre for Learning on Evaluation and Results Anglophone Africa.
The author declares that he has no financial or personal relationship(s) that may have inappropriately influenced him in writing this article.
UCT in collaboration with DPME runs regular executive courses in evidence-based policy making and implementation for senior policymakers.