Abstract
Background: The City of Johannesburg (COJ) provides services to approximately 5 million people; yet the elements of monitoring and evaluation remain a missing link in the design and implementation of programmes. This was the case even after the introduction of the monitoring and evaluation framework in 2012. This case study is filling an empirical gap.
Objectives: The aim of this study was to understand the policies, practices and use of monitoring and evaluation (M&E) in tracking the performance of the City towards meeting its long-term developmental plans.
Method: A mixed methods approach was used to gather quantitative data from 54 senior M&E officials. This was complemented with qualitative data drawn from in-depth interviews and focus group discussions conducted during three workshops with M&E officials.
Results: The study reveal a number of weaknesses: poor integration of M&E practices in planning, budgeting, service delivery and policy development oversight. The inter-governmental institutional environment and various committees and utility boards has resulted in extensive resources being devoted to compliance reporting. Consequently, the foundations for building an evaluation system have been neglected.
Conclusion: The five-dimension complexity model was found to be a useful organising framework for effectively evaluating the city’s M&E capacity. These findings form the first phase of an intervention that will inform the second phase targeted at building the foundations for a city-wide evaluation system.
Introduction
With the third World Urban Forum having recently concluded (October 2016), one of the key message emerging was the rise of inequality in cities contributing to increasing racial segregation between high-income and low-income households (OECD 2016). Striving for inclusive growth is not new to the City of Johannesburg, being part of a country with a Gini coefficient of between 0.66 and 0.70 (World Bank 2016). In October 2011, the City of Johannesburg (COJ) launched its long-term growth and development strategy called Joburg 2040 (City of Johannesburg 2011). The strategy was a culmination of two years of research and an extensive public outreach process from which a vision of the future City of Johannesburg emerged. Arguably one of the most important tenets of Joburg 2040 was the focus on redressing inequality through sustained investment in communities (City of Johannesburg 2011). Despite considerable progress in the implementation of Joburg 2040 over the past four years, there remains a challenge in tracking what progress has been made towards achieving the city’s long-term outcomes.
In 2012, the City developed a monitoring and evaluation (M&E) framework through which progress towards Joburg 2040 could be systematically measured. An important goal of the framework was the strengthening of the capacity of officials to undertake M&E (City of Johannesburg 2012). Four years later, the framework has not been fully implemented, reinforcing the challenges in understanding how short-term and medium-term interventions are contributing to the city’s long-term development plan. The COJ approached the Centre for Learning on Evaluation and Results – Anglophone Africa (CLEAR AA) to conduct a diagnostic study of existing M&E capacity in the city and the bottlenecks in implementing the existing framework as the first phase of a longer term evaluation capacity development (ECD) intervention. In carrying this out, some important insights have emerged regarding the reasons for a lack of implementation and capacity to achieve the goals identified in Joburg 2040.
The effort to improve M&E capacity in the COJ occurs in the context of a continent where there is a dynamic space with regard to the construction of national evaluation systems (NES). Benin started in 2007 while Uganda and South Africa both started in 2011. On the South African side, the construction of its NES has been led by the Department of Planning, Monitoring and Evaluation (DPME) and has grown rapidly, beginning with the National Evaluation Framework in 2011. This was followed with a suite of guidelines to assist the civil service to work within a common set of standards in implementing an evaluation. This technical support provided by DPME was complemented, with some assistance by CLEAR AA, through a rollout of in-service training to national government departments seeking assistance on how to follow these new guidelines. Provinces are also required to adhere to the new policies set out by the national evaluation policy (NEP; implemented in 2012), but have struggled to do so as the capacity building dimensions for doing this have been limited. Local government is now beginning to surface its own internal M&E capacity building needs and this article is the subject of CLEAR AA’s response to this need from one of the country’s most significant cities.
Apart from the effects of specific evaluations, a significant effect of the NES has been elevating the status of evaluation as an essential element of government work. The NES ‘forces’ the placement of evaluation on the agenda of government departments, responding to the request from DPME to government departments to submit proposals for evaluations in the NES in which they are co-financed by DPME and the department (World Bank 2016). Nevertheless, there is a limited track record on the impact of this evaluation system on programme performance as the implementation plans that get approved at a cabinet level, following each evaluation that goes through the NES, have been slow to take effect.
Bringing this evaluation landscape perspective to the local level, this article highlights evaluation capacity as one of the critical skills lacking in large metropolitan municipalities, such as Johannesburg. The Sector Skills Plan (SSP, 2016–2017) for the Local Government Sector and Training Authority (LGSETA) identified monitoring, modelling and evaluation as critical skills in short supply (LGSETA 2017). Service delivery demonstrations by citizens from many local authorities in South Africa suggest that the government is caught unaware and this partly be a result of its inability to use M&E tools to track results of interventions aimed at improving the lives of its residents. The merit of investing capacity building in M&E is that it can contribute to better programming and policy formulation. The model developed in the COJ, and highlighted in this article, can be adopted by other metropolitan municipalities in the country.
As the voice of local government is beginning to emerge regarding their M&E capacity building needs, this article presents an opportune moment to reflect on different methods for understanding the state of existing capacity of city officials to track and reflect on the implementation of intermediate and long-term development plans amidst rapidly changing contexts. National governments in the region have been evolving over the past few decades in developing NESs to help track performance in the implementation of national development plans (Basheka & Byamugisha 2015; Goldman, Rabie & Abrahams 2015). This practice is only beginning to emerge at the subnational level and is still largely focused on trying to put monitoring systems in place.
In the effort to construct or strengthen nascent evaluation systems at the city level, the authors believe that the growing body of literature on complexity theory presents a variety of conceptual frameworks for thinking about how to bring evaluative thinking into the complexity of urban systems (Barnes, Matka & Sullivan 2003; Byrne 2013; Martens 2011; Walton 2014). These theories are particularly useful at the subnational level when the interface between policy and implementation iterates so rapidly that it is often difficult to dissect the institutional enablers or constraints. Such is the case with the COJ.
The first part of the article outlines the complexity theoretical framework used to analyse the city’s existing M&E systems and practice. The second part speaks to the main findings from the diagnostic. The conclusion addresses learnings from the approach taken.
Complexity theory is becoming increasingly useful for the ECD community in building M&E capacity for organisational systems change. Some of the key concepts of complexity theory, relevant to the subnational context, are deliberative versus emergent processes of change, the formation of shared meaning, the participation of multiple agents, the management of tensions, the distribution of intelligence, the co-evolutionary interaction processes in the design process and use of complexity policy evaluation (Marra 2011). In essence, the focus of complexity theory can be used to investigate the relationships between different parts of a system and how they interact. In scientific discourse, complexity is also seen as a system with multiple elements or entities adapting or responding to the patterns they create.
In developing a diagnostic assessment for the COJ’s readiness to implement its existing M&E framework, we have drawn on Bamberger, Vasessen and Raimondo’s (2015) definition of general complexity as a useful framework for understanding dimensions that make up the governance of the City of Johannesburg as a whole, and why it is struggling to monitor its progress in reducing inequality through inclusive growth according to its Joburg 2040 long-term development outcomes. Bamberger et al.’s general complexity definition, taken from Morin and Montuori (2008), Byrne (2013) and Byrne and Callaghan (2014), is reliant on the concept of multilevel emergence in which complex systems have distinct properties of their own that have to be addressed in and of themselves in order to understand how they relate as a whole.
This resonates with the dimensions of complexity (Bamberger et al. 2015), which needs careful mapping and comparison of context-dependent causation, such as different departmental experiences, to make sense of complexity in a given geographic and temporal setting. We have used this general definition as well as the five-dimension model to present our analytical findings from a diagnostic assessment conducted on the COJ’s readiness to implement its M&E framework. The model, in our view, captures the fluidity associated with the moving components or dimensions of the puzzle that make up the governance of a city, which defines its reporting and accountability structures and its ability to learn and respond accordingly.
The five dimensions of the complexity framework are presented in Figure 1. The first dimension looks at the key intervention for change in terms of what the programme (in this case the CLEAR ECD model) is trying to achieve and how. The second dimension, ‘Embeddedness’, looks at the context within which a programme (in this case the M&E framework) is ensconced, which is related to the culture of the organisation (values and beliefs, the history that has shaped the existing legislation, the socio-economic conditions that enable or constrain the implementation of this legislation and other factors). The third dimension addresses interactions among the different stakeholders and agencies or units within the organisation under consideration. In this case, these are the main power dynamics with key groupings within the city that structure the purpose of the reporting and the disjuncture in how this reporting relates to monitoring the city’s performance in meeting its Growth and Development Strategy (GDS) 2040 objectives. The fourth dimension relates to ‘the nature of the processes of change and causality’ or rather how change is captured. This dimension reveals the causal relation between programme inputs and its intended outcomes. Lastly, the fifth dimension, ‘the nature of the evaluation process’, focuses on the institution’s data availability and capacity to process and analyse it.
 |
FIGURE 1: The five dimensions of the complexity framework. |
|
Results from the diagnostic study are presented in this section in accordance with the five dimensions of the complexity framework. Areas of concern and factors that hinder achievement of the broader goal and outcomes of the city are identified.
Intervention: CLEAR AA’s methodology for carrying out a monitoring and evaluation diagnostic on the city
When the COJ decided to hone in on M&E as a lever to improve overall performance management in the city by approving its M&E framework in 2012, it set in motion a large-scale organisational change process, which was to be given effect through the group-wide implementation of the adopted framework. The COJ approach was to integrate efforts across the entire M&E value chain, starting from the planning process related to the timeframes of integrated development plans (five years) and eventually closing the loop leading to the next planning cycle with an honest reflection based on the evidence gathered regarding what works, and why it works. Four years after the adoption of the M&E framework, there has been limited implementation of the monitoring dimension of the framework and little evidence of any systematic approaches to evaluation. This is due to several reasons: firstly, the M&E framework suffered from the outset from being more theoretical than applied and was not customised for the COJ operational context. Secondly, the M&E framework was not evenly implemented across all the organisational units that comprise the City. This could be due to the need to continuously develop strong support programmes for implementation of the M&E framework, including the design of critical processes (e.g. suitable methods) as well as the rollout of comprehensive capacity building programmes across the organisation. This particular intervention, in partnership with CLEAR AA, has therefore been devised by the City to address these lacunae.
CLEAR’s intervention logic was to treat this initiative as an organisational change, rather than an ad hoc training intervention. The COJ and CLEAR AA accepted that a complex change process that would result in changed behaviour and practices, as well as improved evidence-influenced decision-making and better policy interventions could not be achieved in a short space of time. As such, this is the first phase of the initial intervention: the research focused on a situational analysis of the current status quo with respect to M&E in the COJ, both as a source of information, but also to construct the baseline against which future progress would be measured. Emphasis was on finding agreement between key stakeholders with respect to the areas in which change is being sought, the change strategies to be followed, the programme theory to be implemented, identify prioritised interventions and the avenues for change plotted. Secondly, the intervention involved implementing an initial round of capacity building interventions for key groupings of employees who were expected to support the implementation of the M&E system across the organisation, most notably staff responsible for strategic planning, research, policy formulation, performance monitoring, reporting and evaluation. The third component of the intervention was to provide technical support adopting conceptual thinking for strengthening institutional capacity and behaviour change in relation to M&E, such as establishing a community of practice among M&E practitioners within the City as well as the foundations for implementing an M&E resource centre.
Research methods and design of the diagnostic1
Quantitative and qualitative methods of data collection were used. The methodological approach consisted of a desktop review, online survey, key informant interviews with senior managers and three workshops from which focus groups were drawn. The data were used to assess the current status of M&E in the COJ.
Desktop review
A number of key policy and programme documents from the government that were critical to understanding the lack of alignment with the City’s current reporting structures were reviewed. This phase revealed challenges in the alignment of key indicators for quarterly reports used to track the performance of the four clusters of departments tasked with delivering on the GDS 2040. This review also provided insights into the causality and change dimension of the complexity framework . We describe these further later on on Page??
Online organisational survey
The first iteration of a baseline was developed using an online survey which could serve as an M&E barometer for the City to measure capacity changes over time as it tested M&E skills and training in the City and the evolving nature of reporting. The target group for the diagnostic was a population of 77 officials with direct M&E responsibilities as these were the individual champions expected to institute an organisational change. The officials were largely technical specialists and planners drawn from various departments and entities in the City. Almost every major department was represented.
The survey had 54 responses out of 77 invitees, covering 70% of the targeted population. Figure 2 provides a summary of the profiles of M&E practitioners in the COJ. These officials operate in the M&E related field and, with greater exposure, were expected to be champions and agents of change in the COJ M&E. The respondents were a relatively experienced group with over half of them (55%) aged between 40 and 50 years. Nearly two-thirds were women. A significant proportion (42%) have worked in the City for over nine years, indicating their potential of holding institutional knowledge. In addition to work experience, a fifth of the team had formal training in M&E. While many of the respondents have developed a good understanding of indicators and a system of performance monitoring in the City, this area of expertise needs to be strengthened. Though small in number, this group was representative of the larger M&E fraternity which was expected to perform complex analytical functions without the requisite training to do so. The survey results reinforced that this group of respondents was critical for the future development of M&E in the City. As per the planned intervention, this group was part of the cohort that was trained in the second part of the CLEAR ECD intervention that we have described above.
 |
FIGURE 2: Vital statistics of monitoring and evaluation officials in the City of Johannesburg (n = 54). |
|
In-depth interviews
Results from the survey were central in developing the interview instrument for key informant interviews. Eleven in-depth interviews were interspersed with workshops and these were conducted with senior officials and technical specialists. A structured interview guide was developed according to CLEAR’s six sphere framework (Crawley, 2016). The framework consists of the following elements; ideological (values and beliefs), social, political, contextual, logistical (human & financial resources required for M&E), technical (skills, technologies, capacity) and the value system.
Workshops
Three workshops were held between June and July 2016 with a total of 45 officials. The first workshop formed focus groups to identify the strengths and weaknesses of the current M&E system, drawing on simple M&E questions to probe people’s familiarity with how they used these terms in their daily work. This initial process failed as the quality of conversation remained superficial and revealed a lack of common understanding of core M&E concepts. The sense of impatience for technical learning by COJ officials forced the CLEAR team to redesign the second workshop to respond to the desire for pedagogical learning and the need to use a focus group approach that could draw out a more strategic conversation.
The second workshop began with training on core concepts around evaluation and was remarkable in creating a common reference point in how officials engaged in the focus group discussions. The second workshop used CLEAR’s six-sphere framework (Crawley 2016), outlined in greater detail in this special issue, in order to tease out strategic reflections on what the bottlenecks were in using evidence to inform decision-making.
Figure 3 shows a diagnostic model which consists of six spheres that cover all aspects of organisational development. Questions related to the COJ were posed for each sphere. Participants were asked to identify major strengths and weaknesses of the M&E system across each sphere. They were also asked to identify a ‘desired state of change’, for example ‘Identify important factors that can improve the overall M&E environment’. Data were analysed for each sphere and coded into various themes. The team from CLEAR remarked on the leap in the quality of discussions ensuing from the use of this model for the focus groups, which formed an important source of information in understanding the relational dynamics between M&E officials and the line departments they are either in or report to.
During the third workshop, a problem tree analysis was used as the primary facilitation tool drawing on the already-developed COJs M&E framework. This workshop was used to build the foundations of a theory of change for the implementation of the M&E framework.
Embeddedness and the nature of the system
The second element of the five-dimension model is defined by the legal, regulatory and institutional context of the COJ, which has profoundly shaped its M&E culture. Findings from the in-depth interviews and workshops reveal that the legal and regulatory context of the municipal government has created an onerous upward reporting architecture that absorbs considerable organisational time and energy. This was primarily a result of inter-governmental fiscal transfers between the national and provincial governments and the COJ. This system has created an upward accountability reporting structure.
A lack of coordination of the grant administration between the three spheres of government (national, provincial and local councils) has created multiple reporting and accountability levels. The City has to demonstrate the spending of conditional grants that are allocated for capital expenditure. In 2011, the Financial and Fiscal Commission undertook a comprehensive review of the local government funding framework (LGFF) and identified multiple reporting lines by metropolitan cities to national government departments for grant funding. This prompted the move by National Treasury recently to develop guidelines for an Integrated City Development Grant (National Treasury 2013).
Results from the online survey further illustrate the point. Figure 4 shows the level of effort spent by officials on reporting activities. The graph indicates that the largest proportion (41%) of officers spent three-quarters of their time on reporting. Given that the participants of the survey were middle management, these figures could suggest a system that is burdened with reporting requirements. Of the time spent on reporting, the majority of this time is spent on reporting to internal finance and audit, mayoral committees, boards of entities (water, waste and electricity) and lastly national and government departments. The link between internal and external reporting was not established during this diagnostic.
 |
FIGURE 4: Level of effort in reporting by government officers (n = 54). |
|
The City has created an additional level of reporting to line departments in an effort to align with other key planning documents for the GDS 2040. In an attempt to respond organisationally to the implementation of Joburg 2040, executive management, through the City Manager’s office, developed a cluster system in 2012. The City has 9 line departments and 12 municipal entities grouped according to the four outcomes in Joburg 2040. This includes: good governance, human and social development, sustainable services and economic growth. Each cluster is responsible for producing a cluster plan. The cluster system is not an additional formal organisational structure but cluster representatives have to meet on a regular basis to assess their plans.
The reporting burden is further slowed down by committees. Examples of such committee meetings are: strategic executive management team meetings, operational executive management team meetings, cluster meetings, mayoral committee meetings and sub-mayoral committee meetings. These committee systems originated during the last term of office 2010–2015 from a decision taken by the mayoral committee in 2010. The rationale was to divide strategic reporting with more compliance-driven reporting and to ensure that the senior administration, which includes the executive management team and mayoral committee, only deliberate on matters of strategic importance. However, the cumulative effect of the group structure, clusters and line departments, combined with separation of strategic and compliance reporting systems has created a seemingly strenuous reporting system. Participants from the workshops corroborated the above conclusions about the burdensome reporting system. One remarked that:
‘The regulatory and institutional environment has induced malicious compliance.’ (Thabo, Director, Male)
That the reporting is referred to as malicious is of importance in the sense that the value of such reporting is not perceived as positive by M&E practitioners. Another female respondent concurred with this sentiment noting:
‘Reporting drains morale and organisational energy.’ (Ayanda, Director, Female)
The legal, regulatory and institutional context of the COJ influences the reporting environment. The regulatory and administrative environment has induced rote compliance and officials are focused on narrow upward reporting.
A second key element of this dimension is the organisational culture of the COJ with regards to its current monitoring system. The diagnostic study found that there is poor internal horizontal communication between line departments and entities. Information is not shared across departments and specifically among M&E practitioners. Partnerships across departments and entities are weak and this creates information asymmetries and duplication of data collection processes. Workshop participants expressed the view that there was a lack of communication between the central M&E unit, known as Group Strategy Policy Communications and Relations (GSPCR), and other departmental M&E units of the COJ.
Furthermore, there is a lack of incentive to provide greater rigour in relation to the quality of the data provided nor are there consequences for lack of verification of data. Workshop participants noted that there is unwillingness by their colleagues to improve poor reporting. There are various reasons for this; for example, there may be an onerous reporting system, no effective punitive measures and a lack of incentives that would promote better reporting and willingness from M&E practitioners to improve the quality of reports.
Institutions and stakeholders
The key insights from this diagnostic dimension was the lack of defining roles and responsibilities between the executive and the metropolitan legislature. This was illustrated by the strained relationship between senior executive managers and political councillors. To understand this further, we first outline the governance structure of the City in terms of the separation of powers between the legislative and the executive arms of the city council to ensure accountability to its citizens.
The City’s legislature provides political oversight through various committee structures, such as Section 79 and ward committees. The intention of political oversight is to ensure accountability in driving capital spending over a five-year political term of office, which makes members of this committee responsible for monitoring the plans, programmes and projects of city’s departments. These committees, who carry out community visits, ultimately use their observations and reports, to make recommendations to the city manager’s office on service delivery progress in relation to capital spending.
Findings from this diagnostic revealed that the councillors who make up the Section 79 oversight committees lack skills required for processing the technical information given by their administrative counterparts. This sense of inadequacy is compounded by mistrust stemming from the lack of efficacy and credibility of data derived from the City’s monitoring systems. From the officials’ perspectives, they believe that politicians overpromise results to their constituencies and yet they do not understand the risks associated with developing contracts for new projects that were not agreed to in council planning processes. When political decisions need to be actualised, officials feel the pressure to deliver on these promises that were not grounded in the data that fed into planning processes; these different perspectives have created a level of mistrust that has fuelled considerable tensions between the executive and the legislature of the city council.
Another source of tension centres around the quality of data presented to the Section 79 committees. As these committees are responsible for demonstrating progress to the mayor’s office, they rely on accurate data in reports to be able to communicate this progress. How this information is presented to politicians by officials is vital in terms of the former being able to understand what can, at times, be technically complicated matters on which they need to make recommendations. For example, the COJ has an extensive electrification programme targeting approximately 180 000 households living in slum settlements. According to Statistics South Africa (2012), there are 17 new informal shacks erected every day in Johannesburg. Therefore, the ability of the City to adequately respond to this ever-growing demand for electricity depends on maintaining effective data records of the number of electricity connections. Absence of readily available data on this issue becomes a source of contention among political and administrative decision-makers.
Another contributing factor to the weak culture of rigour in the implementation of the monitoring system in the COJ was related to the lack of a dashboard, making it difficult for politicians to understand how to weigh priorities for interventions given the magnitude of detail associated with reporting from many different line departments. Without this dashboard in place, it is difficult for politicians to assess City-wide performance, which undermines their function in providing political oversight. Inability to provide an overall assessment of the City’s performance makes it difficult for these committees to assess their own performance in their oversight function.
This directly affects the M&E champions who formed part of the assessment from which this article is drawn, as they are responsible for compiling the data for producing the reports that go to Section 79 committees. Poor quality data from various line departments were widely cited. Without credible data presented to the mayor, the members of this oversight committee feel ineffective.
Causality and change
This dimension is about the nature of processes and causality and how programmes can affect change in society and how we capture and measure this change. Key to achieving this is an ability to generate knowledge that can be processed analytically in order to understand the structural flaws in the design of programmes. This is the dimension of the City that requires evaluative thinking. In practice, this dimension captures the mayoral flagship programmes that are capital intensive and medium-term to long-term in their delivery. Mitigating problems through causal analysis of endogenous and exogenous factors is fundamental to reducing risk for the investments the City has made to bring about societal change.
The COJ created two distinct interventions in its institutional structure to effect reflective thinking and support to relevant departments and entities. The first was the creation of the Central Strategy Unit (CSU) in 2002, just after a major institutional transformation of the City, in which core functions of city services were transformed into utilities, agencies and corporatised entities. The CSU was set up to help departments make sense of their strategic plans and programmes and to drive new thinking policy and research around the design of high-impact projects. In over a decade, this department has been successful in being the locus of evaluative thinking for the City. In 2011/2012, the CSU was transformed into the GSPCR, which is now responsible for strategy and policy, knowledge management and ensuring effective M&E practice in the city (COJ 2011).
Views shared through key informant interviews and focus groups suggest that the GSPCR has lost its evaluative thinking focus and instead plays the role of a high-level coordinator to the executive manager and the mayor’s office. It has neither the capacity to help departments to develop high-impact projects nor the ability to measure results from interventions. Focus group discussions and key informant interviews revealed that the current unit for M&E in the City does not have the capacity both in terms of number of staff and technical skill or capability to advise departments appropriately. Some of the respondents from key informant interviews expressed frustration that the unit they look to for support, the GSPCR, has largely been reduced to conducting performance assessments of departmental scorecards.
The GSPCR has the Integrated Development Planning Unit within it, which carries the responsibility for coordinating departmental inputs into the City’s five-year Integrated Development Plans (IDPs). While the IDP unit within the GSPCR is supposed to support departments to develop their capacity to improve their own departmental plans, the perception from M&E officers across the city is that it simply plays a role of collating these various inputs and not providing the quality assurance needed to turn the IDP inputs from various departments into a meaningful city-wide medium-term plan that can be adequately measured on an annual basis. To substantiate this view, a recent analysis of the IDP shows the need for building capacity for measuring the ‘effect’ of programmes and projects. Discussed are some of the observations from the review of the IDP for the years 2016 to 2021 (COJ 2016a).
The review showed that 60% of key performance indicators (KPIs) are not crafted properly, with some of them worded as themes or sentences. Baselines are not informed by an analysis or benchmarks and, as such, 80% of the baselines are incorrect or are ill-suited measures for the KPIs in question. It is noteworthy that the KPIs include basic service delivery indicators on access to water, sanitation and electricity. On such indicators, the COJ has consistently achieved over 90% access across the board which is in sharp contrast to the documented low levels of access to services in African townships. This demonstrates both the paucity in strategic planning but also an inability to decipher inconsistencies in operational performance areas.
The IDP for the next five years (2016–2021) presented over 50 objectives and of these approximately 60% could not be accurately measured. As such, the IDP is an indicator itself of the capacity of the officials within the organisation to understand causality and change interventions. This point was reiterated by workshop participants who collectively raised the poor specificity of the IDP’s KPIs, baselines and targets.
Part of the problem relating to the challenges of the GSPCR in supporting departments to integrate their planning through the IDP process is related to the silo structure of the COJ as an organisation. This silo structure, as was voiced by workshop participants, acts as a constraint to the GSPCR to realise its centralising and coordinating function, which limits its ability to communicate vertically to line departments that have M&E functions, as well as to foster horizontal communication among departments. As such, this has created a culture of poor sharing of data across line departments, which has fostered duplication of data collection efforts.
Evaluation
This dimension is about the interplay between an intervention and the effect it has on the problem that it seeks to address. This dimension explores the design, methods, implementation of interventions, value and ethics considered when executing evaluations. The intervention may be a policy, programme, project or activity, as well as the institutional context in which the intervention is embedded and the processes of change that occur as a result.
Findings from the desktop analysis, in-depth interviews and workshops reveal that evaluation is in its infancy in the COJ. Approximately 66% of respondents to the COJ survey conducted for the City diagnostic on M&E indicate that there is a dedicated M&E unit in their department. The extent, however, to which evaluations are conducted within these departments is limited. One in-depth interview concluded: ‘We are new to evaluation’. When asked what the main function of their unit was, 90% of respondents indicated that ‘reporting’ activities for monitoring purposes, thus demonstrating that managing and designing evaluations was not an active practice within units.
Figure 5 shows findings from the online survey. Participants were asked if there had been an evaluation conducted in their departments over the last three years. Forty-seven per cent of participants indicated ‘no’ and a further 24% indicated that they ‘don’t know’. Over a quarter (29%) of participants who are the champions of M&E in the City indicated that an evaluation has been undertaken during the last three years.
 |
FIGURE 5: Prevalence of evaluations undertaken in the last three years in the City of Johannesburg. |
|
Testing the capacity for the organisation to ‘evaluate’ presented a methodological challenge in both the phrasing of questions for evaluation in the online survey and also the manner in which the in-depth interviews asked questions on evaluation. Eventhough the in-depth interviews revealed wide and divergent views on evaluation, there was a general consensus about the importance of evaluation. This is indicated by one focus group participant:
‘Evaluation is long-term and commands resources and energy. Auditing gives legitimacy to your organisation. However, it is evaluation that allows us to course correct.’ (Emma, female, Director)
As mentioned earlier, the lack of an effective data management system and capacity would ultimately compromise the quality of how evaluations are carried out. In-depth interviews showed that the skill to determine how data is collected and organised into an indicator that allows officials to make sense of their interventions is missing. This was identified by senior officials with one official summarising his views as follows:
‘Take BRT2 for example, what are its objectives? Well it needs to be cheaper, safer and achieve a modal shift. By modal shift we mean a shift from private to public transport as a percentage share of the total amount of commuters. If we look at cheaper is this the cost of transport as a proportion of household income? Or is it in relation to other modes of transport? How do we measure cheapness? How we measure this depends on the reliability of the data. Access to the data, how we collect the data determines our ability to conduct an evaluation whether external or internal. When do we set up a baseline, and when is the first cycle of evaluation?’(Grace, female, Assistant Director)
This quote reveals the need for a focus on access to data, collection of data and input of data into carefully designed indicators. Weak data management and organisation processes result in poor inputs into the evaluation process.
A much wider understanding of evaluation in terms of the complexity framework needs to be embedded in the organisation, which must form part of a key focus area in the longer term capacity building programme. This would necessitate building the capacity of officials to measure the interplay between the intervention and the social, economic, cultural and political environment. Notwithstanding that, there are critical improvements and capacity development that are needed to improve the data, which will allow indicators to be tracked and monitored.
Summary findings of the five dimensional analyses
This article has outlined the key approaches employed to carry out a situational analysis of the state of M&E in the COJ. It has outlined the key findings based on the Bamberger et al. (2015) complexity framework, drawing on five dimensions that capture the overall governance of the city. CLEAR AA has found this analytical tool useful for understanding the push and pull of these dimensions.
The challenges outlined in the Embeddedness as well as Institutions and Stakeholders dimensions have revealed an upward accountability pull that dominates the energy of the organisation in reporting at the expense of being able to further develop the causal analysis or evaluative dimensions in the performance management of the City. Examples of this were given in the Embeddedness section, regarding the portion of time officials spent on reporting to national government, leaving insufficient time to focus on the City’s own monitoring requirements. This was also demonstrated in the Institutions and Stakeholders section, where significant reporting efforts were channelled through the Section 79 committees. The mistrust between political councillors of the Section 79 committees and senior officials, due to poor quality data provided by the latter to the former, has contributed to much of the energies of monitoring in the City being focused on accountability feeding evidence for political decision-making, but with weak sources of evidence.
The amount of time and energy taken by city officials to meet reporting deadlines for these two separate purposes (national governments and the Section 79 committee) has undermined processes of rigour in the quality of data that is derived to feed this reporting system and, as such, led to insufficient substantiation of facts to ensure the key decisions are made based on solid evidence. This has led to neglect in evaluative thinking needed to identify whether the measures being used are useful or appropriate for tracking the performance of key programme and project interventions. This has in turn led to constraints in developing the capacity to engage in the causal analysis of where problems arise and the strategic responses needed if these problems are within the City’s areas of control.
Conclusion
The diagnostic study unravelled a number of challenges that constrained the implementation of M&E tools. The three spheres of government – national, provincial and local – and the concomitant requirements on reporting are a burden for M&E officers. This could have contributed to reduced time for the officers to conduct evaluations. The committees that were designed to monitor functions of local authorities added to this burden. The M&E capacity of managers at different levels of the government was identified as weak. Data systems in the City lacked credibility. The study provided insights into how interventions could be introduced to improve M&E use in the COJ. The close engagement with a targeted group of officials, who are the M&E custodians within the COJ, was designed with a perspective of wanting to build the capacity within the COJ to grow their own programme for medium-term to long-term periods.
While CLEAR AA developed a methodology focused on a series of different interventions over a six-month period, the team deviated from its planned methodologies in order to adapt to what it observed as emergent needs in the course of the three workshops that were used for focus group discussions. For instance, the group work in the first workshop was structured with questions that were sufficiently general to entice people into engaging in the experiences from their respective departments. But they were insufficiently detailed to draw out the strategic insights the research team was looking for. Furthermore, many of the workshop participants were hungry for acquiring tools through technical training even though this had been scheduled for more structured training a few months later. The CLEAR team responded to this by providing basic training on evaluation concepts at the beginning of the second workshop, providing a common reference point for the focus groups set up later in the day. Secondly, the CLEAR team brought in a diagnostic tool that encouraged participants to dig deeper and respond more strategically in focus group discussions in terms of the cultural and political blockages to evaluative thinking in their respective work environments.
This responsiveness was also factored into the third workshop when CLEAR AA brought in speakers from other institutional organisations. In this instance, the Southern Africa World Food Programme presented how they had built up a monitoring system and a sophisticated evaluation system within a three-year period across 15 countries. Such a short turnaround time for an institution that deals with complexity to a greater degree than the COJ itself does was an inspiration to the City’s M&E officers.
This responsiveness was further reflected in taking the findings from the diagnostic and feeding it into the formal training programme planned as part of this phase. The intent was to ensure that the generic training in M&E was contextualised with local case studies drawn from the experiences of COJ officials. As a result of their participation in the earlier phases of the diagnostic study (in-depth interviews, workshops and surveys), and using a tailored, customised M&E curriculum that was relevant to the COJ, CLEAR AA trainers observed great enthusiasm among their learners as the material was deemed relevant. These participants have provided suggestions of how they want to use the training and how they would like to cascade it to other relevant departments in the COJ, thereby helping to co-develop a longer term training programme for the City. The course facilitators have suggested that the enthusiasm in learning could be credited to the process employed in terms of CLEAR AA delivering a diagnostic and preliminary training programme in a participatory and inclusive way that got relatively disempowered M&E officials excited about the change management process they were now going to help drive.
Acknowledgements
The report is a result of a partnership between the Centre for Learning on Evaluation and Results for Anglophone Africa (CLEAR AA) and the City of Johannesburg. Senior officers from the City of Johannesburg contributed valuable information at different stages of the diagnostic study. Their role is acknowledged. The City of Johannesburg commissioned CLEAR AA to carry out a city diagnostic. The content of this work has informed the substance of this article.
Competing interests
The authors declare that they have no financial or personal relationship that may have inappropriately influenced them in writing this article.
Authors’ contributions
L.N. was project leader, trainer and co-author. L.S. was project director, conceptualised the article and was co-author. S.N. was lead project consultant and co-author.
References
Bamberger, M., Vasessen, J. & Raimondo, E. (eds.), 2015, Dealing with complexity in development evaluation: A practical approach, Sage, Los Angeles, CA.
Barnes, M., Matka, E. & Sullivan, H., 2003, ‘Evidence, understanding and evaluation in non-linear Systems’, Evaluation 9(3), 265–284. https://doi.org/10.1177/13563890030093003
Basheka, B.C. & Byamugisha, A., 2015, ‘The state of monitoring and evaluation (M&E) as a discipline in Africa: From infancy to adulthood?’, African Journal of Public Affairs 8(3), 75–95.
Byrne, D., 2013, ‘Evaluating complex social interventions in a complex world’, Evaluation 19(3), 217–228. https://doi.org/10.1177/1356389013495617
Byrne, D. & Callaghan, G., 2014, Complexity theory and the social sciences: The state of the art, Routledge, New York.
City of Johannesburg, 2011, Joburg 2040: Growth and development strategy, Central Strategy Unit, City of Johannesburg, Johannesburg.
City of Johannesburg, 2012, Monitoring and evaluation framework, City of Johannesburg, Johannesburg.
City of Johannesburg, 2016a, Integrated development plan 2016–2021, City of Johannesburg, Johannesburg.
City of Johannesburg, 2016b, Developing & implementing a comprehensive capacity building programme for Monitoring and Evaluation in the City of Johannesburg, unpublished report prepared by CLEAR AA, University of the Witwatersrand, Johannesburg.
Crawley, K., 2017, The Six Sphere Framework: a practical tool for assessing M&E systems, paper to be presented at the 8th African Evaluation Association (Afrea) conference, CLEAR AA, University of the Witwatersrand, Johannesburg, 27–30 March 2017.
Goldman, I., Rabie, B. & Abrahams, M., 2015, ‘Special edition of African Evaluation Journal on the national evaluation system’, African Evaluation Journal 3(1), Art. #166, 1–4. https://doi.org/10.4102/aej.v3i1.166
Local Government Sector Education and Training Authority (LGSETA), L.G.S.E. and T.A, 2017, Sector Skills Plan, LGSETA, Johannesburg.
Marra, M., 2011, ‘Micro, meso and macro dimensions of change: A new agenda for the evaluation of structural policies’, in K. Forss, M. Marra & R. Shwartz (eds.), Evaluating the complex. Beyon attribution, contribution and beyond, pp. 171–203, Transaction Publishers, New Brunswick.
Martens, K.R., 2011, ‘M.Q. Patton. 2011. Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press. 373 pages’, The Canadian Journal of Program Evaluation 26(2), 108–110.
Morin, D. & Montuori, A., 2008, On complexity, Hampton Press, New York.
National Treasury, 2013, Annual report 2013–14, National Treasury, Republic of South Africa, Pretoria.
OECD, 2016, Making cities work for all: Data and actions for inclusive growth, OECD Publishing, Paris.
Statistics South Africa, 2012, Statistical release (Revised) Census 2011, Statistics South Africa, Pretoria.
Walton, M., 2014, ‘Applying complexity theory: A review to inform evaluation design’., Evaluation and Program Planning 45: 119–126. https://doi.org/10.1016/j.evalprogplan.2014.04.002
World Bank South Africa overview, viewed 27 October 2016, from http://www.worldbank.org/en/country/southafrica/overview
Footnotes
1. Valuable data were gathered using different strategies. However, constraints relating to data have to be acknowledged and taken into consideration. The different data gathering methods yielded different response rates. For instance, the quantitative survey instrument targeted a population of 77 M&E officers and 54 responses were received. Only half of the 25 targeted in-depth interviewees were available to be interviewed because of local government elections being underway. In analysing the data profiles for officials who failed to respond or attend interviews, there was no reason to conclude that this was a different group that could have biased findings in one particular direction. A potential source of bias could arise from the COJ practitioners. There is a possibility that respondents could have presented biased views since they had vested interests in the topics. Changes in the COJ would affect them directly. However, this was mitigated by use of facilitators and data collectors who were senior and very experienced researchers. Data from the survey was self-reported and it was quantitative in nature. Focus group discussions and in-depth interviews provided complementary data but were not necessarily designed to triangulate each other.
2. REA VAYA Bus Rapid Transit is a multi-billion rand investment by all three spheres of the South African government to improve mobility within the City of Johannesburg.
| 
Hide
Show all
Email this article (Login required)
