Abstract
Background: This article emphasizes the importance of reflecting on the methods employed when designing diagnostic tools for monitoring and evaluation (M&E) systems. It sheds light on a broader debate about how we understand and assess M&E systems within their political and organisational contexts.
Objectives: The article looks at what divergent purposes of M&E mean for how M&E systems are assessed, and how context-appropriate diagnostic studies can be designed.
Method: The article draws on two different approaches: a survey that looks at the technical components of an M&E system and a complexity framework that analyses the way a system functions in a broader political and organisational context. The foundation is provided by survey and interview data from over 70 officials from across the City of Johannesburg’s administration.
Results: The study revealed great diversity as to respondents’ understanding of what M&E structures and processes should do and achieve within the city, ranging from a management function closely linked to auditing and oversight responsibilities to a governance role that is more linked to learning and planning. Limitations in M&E capacity and/or performance were linked to contested political and bureaucratic structures.
Conclusion: The mixed method approach to diagnostics proposed in this article contributes to the call in the ‘Made in Africa’ debate for more contextualised methods and tools around the practice and the assessment of M&E. The article proposes the development of a synthetic tool that covers both M&E technical components and capacity on one hand, and an analysis of how these are embedded in a political and organisational context on the other.
Introduction
The Centre for Learning on Evaluation and Results – Anglophone Africa (CLEAR) recently conducted a diagnostic of monitoring and evaluation (M&E) systems and capacity across the City of Johannesburg’s departments and entities (CLEAR 2016). The process has illustrated that there is a multitude of perspectives on the purpose and role of M&E in the city. Some departments locate M&E as a management tool to achieve predetermined objectives efficiently and accountably, while others see M&E as a strategic governance tool to guide the alignment of programmes towards certain outcomes. Since there is not capacity to play both roles, the absence of an agreement on M&E’s purpose and role for the city’s planning and programme management means that it has been difficult not only to enhance capacity across the city’s departments and entities, but also that developing diagnostic tools for measuring M&E systems and related capacity requirements has been challenging.
This article looks specifically at what these divergent purposes of M&E mean for how M&E systems are assessed, and how context-appropriate diagnostic studies can be designed for such studies in the future. Instead of presenting detailed results of the survey, which are explored elsewhere (Smith & Ndhlovu, in this issue), this article looks at the design of the diagnostic studies themselves, and how these, applied in the context of the City of Johannesburg, have influenced the results found. Taking a methods-focused approach to the diagnostic study made it clear that if M&E practitioners have different perspectives on what M&E can and should contribute to an organisation’s performance, this can lead to a mismatch around what constitutes an appropriate and effective M&E system. This in turn means that building matching capacity becomes a challenge, particularly in a heterogeneous and decentralised organisation, such as the City of Johannesburg administration.
CLEAR has assisted a range of partners in southern Africa through diagnostics of their existing M&E systems over the last two years, with the aim of identifying areas for appropriate capacity building. The City of Johannesburg diagnostic is the fifth such process implemented. A synthesis research piece on these tools is an emergent piece of research that will be developed by CLEAR in the coming year, but the results of some of the diagnostic studies are available on CLEAR’s website. Each of these five was custom designed, and there is considerable variance among the five processes due to the different levels of analysis, purposes, priorities and administrative arrangements around the M&E systems in question. As a result, however, with each subsequent piece of work, we have improved our understanding of how the design of the diagnostic study impacts on the assessment outcomes.
Building on that experience and drawing on the specific involvement in the fifth of these reviews, that is, the diagnostic in the City of Johannesburg, this article argues that having an iterative diagnostic process employing mixed methods allows for multiple approaches to complement each other, giving a more holistic picture of the M&E system and required capacity in the organisation under review. We assert that assessments of M&E systems in an African bureaucratic context need to consider both technical aspects and contextual factors related to the political and organisational environments within which the system is operating.
This is in keeping with the emergent ‘Made in Africa’ themed research from regional evaluators at the African Evaluation Association (AfrEA), the South African Monitoring and Evaluation Association (SAMEA) and other relevant platforms (Abrahams 2015). While members of the M&E community in sub-Saharan Africa are increasingly arguing that there is something unique in the context of M&E in the region, the debate is only just emerging about how this context can best be taken into account through regionally relevant methods and approaches.
The article starts by describing the diagnostic process and approaches used in the City of Johannesburg. It then outlines the scope and aim of each tool before discussing in some detail how both describe certain aspects of M&E systems, and how they address issues of capacity within the system. Finally, we explore how each different approach implies a specific purpose for M&E, and discuss what these diverse views mean for the diagnostic study as a whole – and how they can complement each other.
Methodology
Diagnostic studies are designed to understand areas of strength and weakness in a system. They generally involve applying a particular diagnostic tool that inspects and describes various elements of a system, culminating in an analysis that identifies the strong and weak aspects of that system. The concept of a diagnostic study comes from the medical field, but is now applied widely to a range of organisational and other systems. A diagnostic approach is particularly appropriate for systems that are poorly documented or described, as the process involves a large component of information gathering. As such, it is now common practice in the assessment of M&E systems to begin with a diagnostic study, so as to be able to describe the components of the system and understand their strengths and weaknesses. While this seems straightforward on the surface, as we begin to look at the application of different approaches to diagnostic studies, we will see that developing an appropriate diagnostic approach for an M&E system is actually an act of determining and designing the role M&E should play in an organisation. The more complex the organisation and environment within which the M&E system is located, the more it is necessary to bring different approaches together.
When CLEAR and the City of Johannesburg entered into a partnership in 2016 to strengthen M&E capacity within the municipality, a diagnostic study was carried out as part of this process. The first step was to gather basic information about the M&E system through an online survey of key role players within the system itself. This then informed a series of workshops, which would build a theory of change for the enhancement of the city’s M&E processes and structures. Simultaneously, interviews with key stakeholders were carried out to get a deeper understanding of the context and meaning of the survey information and workshop results. Finally, on the basis of all of these activities, a diagnostic report was drafted, directly informing a subsequent training and technical assistance programme. This article is being written as training is being rolled out. It aims to highlight the importance of including in the diagnostic process both a technical and a wider contextual lens, the combination of which will outline both the technical needs for enhancing the system’s functioning but also the bottlenecks emanating from political or organisational factors.
As part of a mixed methods approach to understand the M&E systems within the City of Johannesburg, we used two tools. The first was an online survey, which was distributed to 80 key individuals across the municipality who had a leading role in the municipality’s M&E structures. The tool was adapted from the 12 components of an M&E system as developed by Görgens-Albino and Kusek (2009). It aimed to better describe the existing components of the M&E system across the City of Johannesburg administration, such as knowing which departments within the municipality had dedicated M&E units, the nature of individuals’ existing M&E capacity within departments and entities, and what duties departmental M&E units fulfilled. The categories in the survey included:
- Demographic information of individuals and their role within the metro
- Objectives of M&E system
- Organisational structures and functions
- Data and Information Management
- Leadership and capacity on M&E
- Linkages to organisational performance and use
In essence, this step sought to collect data that is technical in nature, from those in the know about these aspects of the current M&E practice across the city. This is an important part of gaining a strong understanding of M&E practice in the city. It is all the more important in an environment where M&E systems are decentralised, with different practices having emerged over the last five years across different departments. With a high response rate of 70% and a 100% coverage of the departments and entities targeted, the survey itself presents a comprehensive view of the city’s M&E systems from a broad range of organisational perspectives.
After the online survey was complete and workshops were held to reflect on the results, to draft a diagnostic report on the M&E system, CLEAR used the five dimensions of complexity framework proposed by Bamberger et al. (2015) to analyse all emerging data. Data sources included the online survey discussed in this article, but also included interviews, document reviews and other data sources. The five dimensions in the tool are:
- Embeddedness and nature of the system
- Causality and change
- Evaluation
- Intervention
- Institutions and stakeholders
The following sections reflect on the different nature of these two tools used in the diagnostic study and their inherently different, albeit mutually illuminating, approaches to reflecting on M&E structures and processes.
Findings
While the full findings from the diagnostic study are discussed in detail elsewhere (Smith and Ndhlovu, in this issue), this section discusses specifically how the different tool designs have led us to a more nuanced and more comprehensive understanding of the M&E systems used within the City of Johannesburg. The online survey, with its technical focus, provided data with regard to technical aspects currently in place or aspired to. The complexity approach proposed by Bamberger et al. (2015) moves beyond the technical focus and sheds light on the broader political and organisational context, within which the technical aspects are situated and are meant to operate. With a view to emphasising each tool’s contribution to understanding the city’s M&E systems, this section outlines the key findings we obtained through each of these tools.
Among other facts (see Smith and Ndhlovu, in this issue), the online survey revealed that great diversity exists among the respondents as to their understanding of what M&E structures and processes should do and achieve within the city. The purpose of M&E, as reported by respondents, ranges from a management function closely linked to auditing and oversight responsibilities in some departments and entities, to a governance role that is more linked to learning and planning in others. This varied understanding, in combination with the city’s decentralised M&E system and individual departments’ M&E units’ relative autonomy, contributes to fragmented systems and great variance in capacity within these. There is a spectrum across departments and entities, from robust data systems for reporting that are used for decision-making, to emergent evaluation systems that, however, lack strong monitoring structures to allow them to operate effectively. Similarly, some M&E units in departments and entities lack plans or budgets, while other departments or entities, despite having plans and budgets in place, lack a dedicated unit to carry them out. As Osborne and Gaebler (1992) discuss, it is not clear whether the M&E system is there as a mechanism for rowing, or for steering. Parts of both are in place, but not a full system to do either effectively.
This diverse landscape is not surprising given the rapid introduction of M&E competencies to the South African public service, the size and complexity of the overall City of Johannesburg’s metro system and the speed of change of the actual municipal context itself – through rapid in-migration, dynamic political change and more (Heller & Evans 2010; Parnell & Robinson 2012). It does, however, render tools that simply describe technical aspects of M&E functions insufficient. In a more rationalised bureaucratic system that may characterise metro areas in regions of the northern and western hemisphere where many diagnostic tools were designed, this description may come with an implicit assumption that these functions and the systems to execute them are fully defined, shared and operational.
An advantage of a technical focus is that it can bring forward much-needed descriptive information in an often poorly understood system. In our case, this tool helped shed light on the divergent expectations around M&E. For example, some departments had quite strong data systems, but relatively weak structures to encourage the use of results. Other departments reported quite elaborate processes in the absence, however, of a dedicated M&E unit.
Integrating, in the second step of our process, the survey results with additional interview data and emerging voices from the subsequent workshops meant that we began to take a broader, contextual perspective. Officials suggested that by exclusively analysing technical strengths and weaknesses, we would be risking not getting to the root causes of some of the shortcomings emerging from the online survey. Reports emphasised that these would often be found in the city’s contested political and bureaucratic environment. These dynamics were brought out by using the complexity tool as an analytical framework. This helped map various stakeholders that operate within the M&E system, with an understanding that the implementers of M&E processes are not always the primary custodians of the system, and there are often other role players that play an important role in supporting and enabling effective use of the system. Furthermore, the data emerging during this phase helped articulate how change is seen to take place within the municipality, and what the role of M&E is considered to be within this landscape.
Officials’ reports speak directly to the assumption underlying both commonly applied diagnostic approaches: that if only we get the technical processes right and engage high-quality staff, M&E processes will ultimately function well. However, this assumption means that political and organisational factors are largely excluded in the assessment of M&E systems, leaving the ‘steering’ role to other structures. That does not do justice to M&E’s role in a space of decision-making, where political support to the establishment and conscientious use of M&E structures is key in making M&E data relevant for policy and programme design.
By combining data emerging from the online survey tool with that emerging through the complexity tool, we hence begin to see both how the M&E system is located within the municipality and what structures and capacity exist. Both of those are critical pieces of information to understand what constraints there are to effective use of evidence within the municipality, as well as how best to direct capacity building interventions.
Discussion
In related work over the last 18 months, CLEAR has found, particularly through engagement with parliamentarians (Crawley 2016), that behind logistical and technical limitations that are manifested, there are often political and value-based root causes. Diagnostic designs ought to recognise that technical deficiencies that are uncovered during the diagnostic process may well be rooted, at best, in the lack of adequate political will and support, or indeed in deliberate resistance from political stakeholders. This is a particularly important consideration in the context of southern Africa, where some M&E structures have been established without the corresponding capacity (Cingolani, Thomsson & de Crombrugghe 2015; Van der Meer, Raadschelders & Toonen 2015).
In our recent experience, government clients are often supportive of evaluations uncovering ‘hard truths’, and there is little political disincentive from airing critical findings. However, there is also a significant lack of available empirical information about either technical capacity or the political landscape. For the assessor, gathering this information during the diagnostic process is an important starting point. The online survey played this information gathering function well, but was insufficient to establish whether the M&E system could play a ‘steering’ role, which is more context dependent. For a more nuanced analysis on aspects like political context and organisational values to be included in a diagnostic assessment, it requires a different set of skills and approaches for the assessor, in addition to various approaches to data collection and analysis, and different relationships with the client. This needs to be explored in more detail in the future, and research around it has just begun within CLEAR. The first step, through this article, is of analysing the strengths and weaknesses of various approaches to collecting relevant data.
On one hand, having a complexity framework for analysis provided a helpful lens for understanding the steering role of the municipality’s systems, trying to bring about change in an already rapidly changing environment (Craig et al. 2008). On the other hand, however, it de-emphasised the actual technical components of an M&E system, and the ability of this system to deliver on specific functions. As a result, this framework could not stand alone. Without the survey, which was designed to scope the technical components and capacity of the M&E system, the actual elements and capacity may have been overlooked. These can constrain and shape the role of the M&E system, and influence everything from use of evidence to management culture within the municipality. In other words, looking at political and organisational context only would have easily neglected the fact that there were indeed technical and capacity aspects to take into consideration when aiming at strengthening the role that M&E can play within the city.
Due to the critical role of systems capacity in the region, we do not advocate for taking an exclusively contextual approach to a diagnostic of M&E systems. In fact, stemming from a developmental evaluation approach (Patton 2011, 2015), clients often want a depoliticised analysis of the components of an M&E system. This is an argument in favour of taking a more descriptive and technical approach to a diagnostic study. From the point of view of the user, it is then easier to accept the outcomes of the study as an advocacy tool internally. A more political analysis that undoubtedly exposes, even if constructively and with supportive intentions, a political or organisational environment in which leadership steer away from evidence-informed decision-making, can create even more resistance to engaging with M&E processes. At the same time, focusing only on the technical dimensions of M&E can alienate people within the M&E system itself, who are very aware that shortcomings in resourcing, capacity and systems often have roots in executives’ or political principals’ non-commitment. This reflects a problem in adopting an overly technically focused approach to diagnostics of M&E systems and processes; different users have different interests, and the approaches chosen ought to mediate these diverse needs.
Our experience in this diagnostic of triangulating approaches that collect both technical and contextual data implies specific methodological suggestions. One is the need for a robust mixed methods approach. No individual tool or engagement approach would, in isolation, have been sufficient to understand the context, systems and needs of the metro’s M&E system as a whole. Rather, it is only through our long-term engagement with a range of different stakeholders, towards a range of different objectives, that a more comprehensive view is emerging. In fact, by applying two different tools that take very different approaches to understanding and measuring M&E systems, in some ways, the process of carrying out the diagnostic study allowed the CLEAR team to speak to and mediate differing views within the municipality: one that assumes that M&E ought to be a largely technical function, versus one that recognises that it could play a more expansive role in governance.
Conclusion
This article reflected on different ways of designing diagnostic tools for the assessment of M&E systems in public sector organisations by examining two tools recently employed in a diagnostic study conducted across the City of Johannesburg administration. Taking into consideration the decentralised design of the city’s M&E system across its departments and entities, with a vast array of different practices and understandings of what M&E can and should contribute to decision-making processes, we demonstrated the value of drawing on a basket of tools in the design of diagnostics. This is particularly relevant in a political and bureaucratic context where no shared understanding of the purpose and usability of M&E exists – such as in many public sector environments across sub-Saharan Africa.
The greater the role of context in a diagnostic assessment, the more space there is to debate the political and organisational elements of an M&E system – the steering role – which are central to understanding how a technical process will ultimately operate and what contribution it can make to decision-making. At the same time, however, this debate cannot happen in isolation of a sufficient understanding of technical capacity and processes. As our understanding on this develops, it will be important to shape our future research across the region. Across African public sector institutions, there are variations in the extent to which capacity constraints are technical and internal, versus contextual and environmental. Maintaining a dynamic understanding of the connectivity between the two should remain fundamental to informing our understanding of M&E systems in the region.
An awareness of the implications of each approach, as well as the implicit assumptions linked to each tool, is a first step to making an informed choice about which approach to use. However, there are still two clear needs implied by the research process so far. The first is that in order to sufficiently understand the effectiveness of an M&E system, there needs to be a broadly shared view on the purpose of such a system. While this will be an important output of the collaborative initiative between CLEAR and the City of Johannesburg, ensuring that the consensus that will be developed is inculcated and reinforced in organisational culture and practice is a long-term task.
Secondly, there may be space for developing a synthetic tool that includes both an adequate description of the system’s technical components and capacity on one hand, and an account of how these components are embedded in a political and organisational context on the other. As CLEAR continues its diagnostic work of complex M&E systems in a variety of settings across southern Africa, we will trial integrated tools, and document the process – thereby continuously contributing to the ‘Made in Africa’ debate for more contextualised methods and tools around the practice and the study of M&E.
Acknowledgements
This work is based on ongoing technical assistance cooperation between CLEAR-AA and the City of Johannesburg, the design and implementation of which rests on the contributions of a range of CLEAR-AA colleagues. We also acknowledge the enormously helpful comments and advice received from the peer reviewers.
Competing interests
The authors declare that they have no financial or personal relationship that may have inappropriately influenced them in writing this article.
Authors’ contributions
C.B.M. and M.T.K. contributed equally to this article in terms of the conceptual background work, the analysis and the final argument and conclusions. M.T.K. was also involved in the technical assistance design and implementation through which the data presented was gathered.
References
Abrahams, M.A., 2015, ‘A review of the growth of monitoring and evaluation in South Africa: Monitoring and evaluation as a profession, an industry and a governance tool’, African Evaluation Journal 3(1), 1–8. https://doi.org/10.4102/aej.v3i1.142
Bamberger, M., Vaessen, M.J. & Raimondo, E. (eds.), 2015, Dealing with complexity in development evaluation: A practical approach, Sage, Los Angeles, CA.
Centre for Learning on Evaluation and Results (CLEAR), University of the Witwatersrand, 2016, City of Johannesburg Monitoring and Evaluation Assessment Survey, Johannesburg, Unpublished raw data.
Cingolani, L., Thomsson, K. & de Crombrugghe, D., 2015, ‘Minding Weber more than ever? The impacts of state capacity and bureaucratic autonomy on development goals’, World Development 72, 191–207. https://doi.org/10.1016/j.worlddev.2015.02.016
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., Petticrew, M. et al., 2008, ‘Developing and evaluating complex interventions: The new Medical Research Council guidance’, BMJ 337, a1655. https://doi.org/10.1136/bmj.a1655
Crawley, K., 2016, ‘Strengthened oversight capacity within African legislatures’, Paper Presented to the African Evidence Network Conference, 21–23 September 2016, viewed 1 February 2017, from http://www.africaevidencenetwork.org/strengthened-oversight-capacity-within-african-legislatures/
Görgens-Albino, M. & Kusek, J.Z., 2009, Making monitoring and evaluation systems work: A capacity development toolkit, World Bank, Washington, DC.
Heller, P. & Evans, P., 2010, ‘Taking Tilly south: Durable inequalities, democratic contestation, and citizenship in the Southern Metropolis’, Theory and Society 39(3–4), 433–450. https://doi.org/10.1007/s11186-010-9115-3
Osborne, D. & Gaebler, T., 1992, Reinventing government: How the entrepreneurial spirit is transforming the public sector, Addison-Wesley, Boston MA.
Parnell, S. & Robinson, J., 2012, ‘(Re)theorizing cities from the Global South: Looking beyond neoliberalism’, Urban Geography 33(4), 593–617. https://doi.org/10.2747/0272-3638.33.4.593
Patton, M.Q., 2011, Developmental evaluation: Applying complexity concepts to enhance innovation and use, Guilford Press, New York, NY.
Patton, M.Q., 2015, ‘The sociological roots of utilization-focused evaluation’, The American Sociologist 46(4), 457–462. https://doi.org/10.1007/s12108-015-9275-8
Van der Meer, F.M., Raadschelders, J. & Toonen, T. (eds.), 2015, Comparative civil service systems in the 21st century, Springer, New York, NY.
|
Hide
Show all
Email this article (Login required)
