Abstract
Background: The higher education (HE) policy framework identifies funding inter alia as one of the steering mechanisms, for the transformation of HE in South Africa. However, the role of monitoring and evaluation (M&E) in facilitating this transformation is still emerging. While the instrumental use of evaluation findings is common, research on how process use can also play a critical role in facilitating sector change is missing.
Objectives: The study investigated the use of process evaluations in enhancing the performance of the University Capacity Development Grant in a previously disadvantaged institution in the Eastern Cape province.
Method: Purposive sampling was used as a method to select the participants. The case study design, interpretive paradigm and qualitative methods approach were adopted, and data were collected through individual semi-structured interviews, focus groups and document review. The study sampled 17 participants, including the Department of Higher Education and Training officials, the University Capacity Development Programme (UCDP) management and two staff focus groups.
Results: If process use is to be strengthened, the evaluation process must be highly adaptable to specific programmatic circumstances and organisational contexts.
Conclusion: Evaluation process use, in its practical form, must go beyond issues of participation from an individual perspective and include organisational and stakeholder management for effective implementation.
Contribution: The study’s results are pivotal as they would assist different organisations worldwide that are running similar projects as the UCDP but have not yet understood the benefits that would be obtained by key stakeholders involved in an evaluation process.
Keywords: evaluation use; process use; Walter Sisulu University; higher education; University Capacity Development Programme; monitoring and evaluation.
Introduction
The study investigates the effectiveness of the evaluation process use to improve the performance of the University Capacity Development Programme (UCDP). Implementation evaluations are commonly conducted, and outcome-oriented evaluations often lead to process-related lessons based on the conduct of the evaluation itself. However, empirical research on evaluation use still focusses disproportionately on the instrumental use of recommendations despite research around transformational evaluation emphasising the importance of lessons learned from the evaluation process itself. This study looks at the important consideration of process use in a higher education institution (HEI) with transformational aspirations.
Evaluation process use was first conceptualised when Alkin and Taut (2002) presented process and findings as two distinct categories of evaluation use. Process use focusses on changes to individuals’ thinking or behaviour and changes in organisational procedures and culture that happen as a result of the influence of those that were involved in the evaluation process before the results of the evaluation (Patton 1998). These changes happen as a result of learning throughout the process at different levels of the institution, and in this case, learning is pivotal as it would provide experience and strengthen the implementation of other similar projects in the university.
The proponents of evaluation use speak to how the evaluation process can foster use, primarily focussing on issues of participation and stakeholder engagement (Ledermann 2012; McNulty 2012). While this study confirmed that these issues are important, it has also found that there is not sufficient literature that discusses evaluation process-related considerations that engage with individual decision-making and organisational context (Cottrell et al. 2015).
This study sought to advance knowledge by investigating the effect of process use as a mechanism of performance in a UCDP Learning and Teaching project at this particular HEI. The aim is to better understand the nature of knowledge and skills acquired by the project leaders and implementers who were involved in the evaluation. The importance of this lies in the presumption that participants in this programme may have increased their competence by first applying what they learned throughout the evaluation process in the implementation of other programmes and further improving the performance of the project.
Currently, the implementation of university projects often ends at an output and does not reflect changes to beneficiaries (outcome and impact) and key stakeholders of the project, which limits the intervention’s developmental role. Higher education institutions are hubs of learning, therefore they provide a unique insight into the abilities of institutions to reflect, adapt and respond to the process of evaluations. This study has implications for how process use can be measured and understood, and how evaluations are implemented, giving a deeper understanding of how evaluation processes can influence individual and institutional outcomes.
The study seeks to address the knowledge gap that exists within the field of monitoring and evaluation (M&E), where the processes use in organisation are concerned. For example, Chiesa, Frattini and Lazzarotti (2007) and Nederhand and Klijn (2019), alluded to the fact that little or no literature seeks to address the measures that are put in place for the change of individual thinking or behaviour as a result of process use in the higher education (HE) context (Cottrell et al. 2015). As a result, there is a strong understanding of what process use is, but an insufficient understanding of the result of process use. There is no consensus about the institutional and individual changes that can come about as a result of process use, and this study aims to contribute to this gap.
This study considered the effect of process use as a mechanism of performance improvement in the context of learning and teaching projects in the South African university context. The study explored the different experiences of all the stakeholders who were involved in the process of developing M&E and further participated in the evaluation process and looked at both the individual and institutional factors that shape the results of process use. These stakeholders include the Department of Higher Education and Training (DHET), the executive management of the Walter Sisulu University (WSU) such as the Deputy Vice-Chancellor (DVC) who was the coordinator of the project, two Senior Directors Teaching and Learning and Research, project leaders, project implementers and project beneficiaries. The results of these investigations assisted in assessing whether there has been any effect on skills development and performance.
The aim and objectives that drove the direction of the paper were structured according to King and Alkin’s (2019) prescriptive theory, which provides a basis for categorising different ways in which process use can take place. Process use, as presented by King and Alkin, includes how learning changes individual thinking or behaviour, programme or organisational procedures and culture among those who have been involved in the evaluative practice. This study adapts this framework to understand the contributions of evaluation process use, by also considering programmatic performance, because this is an explicit goal of the evaluation. This framework was identified as the most appropriate in organising principles for this study due both to its resonance with other theoretical frameworks often applied to studies on evaluation use, such as Crawley’s (2017) six-sphere framework and Mapitsa and Khumalo’s (2018) framework on evaluation and capacity. All these frameworks reflect individual, organisational and cultural dimensions of evaluation. However, King and Alkin’s prescriptive theory’s specific application to process use makes it an appropriate framework for this study.
With that in mind, the aims and objectives of the study are as follows:
- To assess the process changes within the UCDP at WSU as a result of evaluations conducted.
Sub-objectives
- To examine the thinking or behavioural change of procedures developed through process use.
- To establish skills acquired by the project implementers through their involvement in the process.
- To establish the changes in project performance and organisational culture because of the involvement of certain key stakeholders in the evaluation process.
- To recommend procedures that could be put in place for project implementers in enhancing project performance through evaluation process use.
Based on the aforementioned objectives, background literature on process use is presented next, structured according to the thematic areas that Alkin identifies.
Individual thinking or behavioural change of procedures developed
A study conducted by Cottrell et al. (2015) indicates that the lack of clearly defined reporting and evaluation measures limit the ability to learn from the past and that a more effective activity is likely to benefit the project. These benefits include the positive change to the programme’s quality and impact, building relationships and educating the stakeholders who are part of the process. Ledermann (2012) concurs that indeed in a context-bound situation, in low-pressure and low-conflict situations, evaluation through an evaluator can cause change by bringing new ideas that can awaken people especially if they are of good quality. While these authors explain the situations where evaluation use can bring change in individual thinking and behaviours as well as in the environment; Patton (2020) however, believes that a major factor in evaluation use is evaluation credibility and evaluator credibility that is determined or perceived and actual evaluator competency. He indicates that this would bring more change together with a good practice project to the stakeholders involved in the process.
Patton (2020:280) further explains that ‘The Good Practice project is a lever for change, but primarily a means for gaining knowledge about the strengths and weaknesses, and the margins for improvement’.
Arnaboldi and Azzone (2010) support the latter statement by explaining that good practices were noted where there is an interaction between the key role players in the project implementation. The authors agree that effective change and good practices are likely to be noted where transparency is available and there are seamless interactions and involvement between the key stakeholders. It is further illustrated that these interactions are meant to upskill the key players who are not familiar with the evaluation process. The literature reviewed highlighted the benefits for stakeholders involved in the evaluation process including positive change in the quality of the programme and new ideas that could be brought including the importance of evaluator involvement. Furthermore, it underlined changes in major stakeholders with regard to skills and knowledge gained including performance. Therefore, this study, through interviews, sought to understand what other behavioural changes to individuals and procedures developed as a result of the evaluation process, because they are understudied in the literature reviewed.
The skill set acquired by the project leaders
The key highlight from the literature reviewed reveals that during the evaluation process, there must be skill transference from the evaluator to the stakeholders involved that will in turn encourage use (Arnaboldi & Azzone 2010).
Moreover, involvement in the process provides incidental learning of different skills by those involved (Patton 2007). In addition, some scholars believe that stakeholders’ involvement assists in sharing ideas that improve implementation and have enduring benefits for individuals and organisations Elo et al. (2014); Aapaoja, Haapasalo and Pia Söderström (2013) and Patton (1998), whereas others believe that the involvement of different stakeholders does not affect the performance but has an effect on the innovation of the project (Nederhand & Klijn 2019). Forss, Rebien and Carlsson (2002) sum the skills acquired through process use by explaining that it may change the management’s thinking about the future options and that it may become the treatment rather than just a normal evaluation based on the skills acquired during the process.
They further state that the strength of process use is that it benefits different stakeholders in terms of knowledge and skills, namely, those who carry out the evaluation – the beneficiaries and the commissioners and project managers. Fletcher and Dyson (2013) conclude by stating that incorporating evaluation process use has improved the level of accountability, within the project team, stronger inter-team relationships and increased knowledge sharing within the project.
While Forss et al. (2002) believe that there are skills that may be acquired by different stakeholders. They also believe that it is not common to identify lessons learned through the process. Therefore, this study sought to understand the actual skills that may be learned by involved stakeholders in the process that may improve the implementation and performance of the project.
Moreover, the study ascertained whether any skills were transferred by evaluators during the process as illustrated in the literature. This is particularly important given the push in the evaluation sector towards capacity building as a result of the evaluation process itself. The literature reviewed in this chapter highlighted different issues regarding the relationship between stakeholder involvement and project performance.
The changes in the performance of university capacity development grant are related to the involvement of certain key stakeholders
The literature revealed that measuring performance in the project is key as it provides support to decision-makers, stimulates learning and improves coordination and organisational results (Chiesa et al. 2007). However, Patton (1998) indicates the dual possibility that it may provide good or bad results when evaluation is used as a measurement tool. A study by Aapaoja et al. (2013) where the early involvement of key stakeholders is interrogated emphasises that to achieve effective performance of the project, different stakeholders shall be included at different stages of the project. This submission emanates from the fact that their importance differs in stages and that the number of stakeholders should be limited to a number that is manageable to avoid conflict between these members as the interests may differ.
Nederhand and Klijn (2019) provide a different angle of measuring performance in that indicators can be used to measure performance and these indicators are categorised into the effectiveness of the intervention offered, the support, the role played by the intervention, the intervention feasibility in the future and finally how cost-effective is this intervention. Mascia et al. (2014) add that when performance measurement is conducted through M&E, the assessment of a project or programme’s progress towards desired levels of specific activities, outputs and outcomes is key and followed. While different measures are provided by different scholars in measuring performance, Boston, Bradstock and Eng (2021) explain that the system cannot be a blanket approach but rather could be designed for a specific section or context in the public sector. As such the debates provided a blueprint of the basis from which the results would be measured, as the policies may be drafted in such a way that they suit the needs of the key decision-makers.
The literature about the involvement of stakeholders in improving project implementation and increased skills from implementers is still scanty. Therefore, this theme sought to understand if there was any relationship between stakeholder involvement and project performance, specifically at the level of stakeholders that may have an effect. Thus, the bases are used for measuring performance by both commissioners and funders in the projects. Given the argument provided with regard to different perspectives of process use when linked to performance measurement, it is vital to further look at other arguments from different scholars with regard to how involvement in the process may affect change in implementation and institutional culture, which is discussed in the next section.
The organisational change of culture
Different scholars have indicated that in many cases, the success of the project relies significantly on the number of senior staff involved in the process (hierarchical culture rather than organisational culture) who in turn are also the decision-makers (Tanner & Willingh 2014; McNulty 2012). Other scholars have referred to resistance to change, political dynamics and extremely traditional ways of doing things as the challenges that hinder change in implementation (Ledermann 2012; Patton 1998). McNulty (2012) argues that while knowledge management, capacity building and communication are important aspects for improving evaluation use; they may however, be relatively less important compared to the more persistent and systemic cultural and psychological dimensions that may, in turn, improve the performance of any project or organisation holistically. Therefore, while culture might have negative effects on the project implementation if it is viewed positively, it is likely to have positive spin-offs in the project depending on the evaluator’s role and how internal stakeholders gain trust in the evaluator (Fletcher & Dyson 2013).
A different angle of describing the change in culture in a study conducted by Fletcher and Dyson (2013) is, that for an effective assessment of the change it shall be viewed from the shift in assumptions, values and behaviour that underpins the existing culture of any entity to accommodate social justice. It is further explained that the shift assists in uncovering the unknown meanings that shift the mindset. While they mentioned that trust built by evaluators with the project managers is the benefit; on the other side, the challenge of boundaries has also been noted. Drawing from the argument by Tanner and Willingh (2014), this study seeks an understanding of the effect of the involvement of senior personnel in the process and the role played by the evaluator in curbing the challenges that are associated with organisational culture and other dynamics.
Research methods and design
The study employed a qualitative research approach, which enabled the researcher to generate in-depth information regarding the human experiences of the UCDP project coordinators and project implementers of the evaluation process (Chilisa 2012). This study further employed a case study design because the unit of analysis is a single institution. The case study provided an opportunity for the researcher to have an in-depth view of a more authentic understanding of the process use effect in the context of University Capacity Development Grant (UCDG) as implemented by the university under investigation and the changes that were experienced.
The study’s overall sample provided in Table 1 was 17 key participants where five were project leaders, the Senior Director of Teaching and Learning at the WSU, the Deputy Director of Teaching and Learning at the DHET and two focus groups made of five student-related project leaders and five staff-related project leaders. The study adopted a purposive sampling approach, selecting participants based on their expertise and first-hand knowledge about the area of interest of this research. The criteria for sample selection targeted the DHET participant as one key stakeholder. The DHET personnel was selected to get an understanding of the extent the objectives of the project have been met, by getting their perceptions of UCDG implementation and performance of the project in the University. Then, there were two levels in the management structure of the UCDG at WSU and the two focus groups of the project per student development-related and staff-related projects. At WSU, the UCDG management structure consists of the Deputy Vice-Chancellor, Academic Affairs and Research, Senior Director of Learning and Teaching (Project coordinator), the project manager who is also the project leader, five other project leaders and the project implementers on the ground.
| TABLE 1: List of participants in the data collection process. |
The data collection instruments used were semi-structured interviews, focus groups and document reviews. Semi-structured interviews and focus groups were selected to get a deeper understanding of different participants’ experiences. Lastly, a document review was selected to triangulate the claims made through the semi-structured interviews and focus groups. This included both qualitative and quantitative data around the project’s implementation and performance. Interviews were guided by a semi-structured instrument that probed the various aspects of project implementation in line with prescriptive theory. Both groups were included to assess the interaction between project implementers and the project leaders, and how information gets cascaded to the relevant target groups. Documents were used to gather additional data to corroborate the information provided by participants. This assisted in strengthening the results for better triangulation and the data collected were stored in a password-protected computer, and the backup was kept in a safe. The data collected were transcribed, coded, pattern searched and analysed according to themes presented in Alkins’s prescriptive theory.
Ethical considerations
An application for full ethical approval was made to Wits School of Governance Human Research Ethics Committee and ethics consent was received on 23 September 2021. The ethics approval number is WSG-2021-43. This study ensured confidentiality and protection against any harm or exposure to harm of the participants that partook in the interviews. A research protocol was developed and submitted to the Wits Ethics Committee for review. All the participants who partook in the interview were assured of confidentiality as much as possible and their identification or names will not be published to ensure anonymity. A consent form outlining the objectives and purpose of the research provided by Wits University was provided to all participants in the interviews to confirm that they agreed to participate in the study. The consent forms to individual participants and the university adopted as the case study were used to ensure that the normal process was followed despite being part of the project that was evaluated to ensure objectivity. The discretion to withdraw was given to research participants at any point in their participation.
Results
This study considered the ways in which the evaluation process shapes individual thinking and behaviour, with specific consideration given to skills building and capacity development, connections between evaluation process use and project performance, including how not only evaluators, but project managers can work to strengthen process use in evaluations. The core finding is that while multiple instances of process use can be found, the drivers of process use extend far beyond typical considerations of facilitation and participation, reflecting organisational dynamics that should be better understood to better tailor evaluation processes to facilitate use.
The first key finding is that evaluation process use may change individual thinking, but other factors also contribute to changes in attitude, such as a change in leadership. As such, it is important to understand process use in an organisational context. A second finding is that process use can be uneven across different stakeholders and should be measured in ways that are appropriate to the organisational context.
A further finding is that all participants at the project leaders’ level who were involved in the evaluation process acquired skills that would shape their mindset in implementing the project; there was even a limited degree of ‘training the trainers’, with those who were not involved in the evaluation still benefiting to a limited degree from the skills gained by those who participated.
These findings are further unpacked, presented and discussed according to Alkin’s framework next.
Individual thinking or behavioural change of procedures developed through process use
The main research question seeks first to understand whether process use was seen within the organisation as a result of the evaluation. One participant illustrated a sentiment shared by many, which is that:
‘It gave us as a university an opportunity to reflect on how we have been doing things and it allowed us to change how we have been doing things.’ (Participant 4, interviews, Executive Director Student Affairs)
Individual behavioural change can be viewed at four different levels of the project, namely at an individual, project, institutional and funder level. At the individual, project and institutional levels, participants agreed that their mindset has changed ever since the exposure to the evaluation process in such a way that they took a step back and reflected on how they have been implementing the projects in general, which means that in a way, their involvement in the evaluation process has changed their mindset.
The study further revealed that while process use has brought in skills and understanding of organisational procedures to both coordinators and project leaders that were involved in the process, there is still a notable predicament of skills transfer to implementers of the project. While we do see later that there were some skills strengthened as a result of the evaluation process, they were not themselves sufficient to address a capacity gap that goes beyond the scope of a single evaluation. This further agrees with the submission made by the DHET representative and participant four, that the project leaders that were involved in the process, while there is a notable change from them, they were doing it for themselves, not for the improvement of the project.
In considering the ways in which the evaluation process shifted individual knowledge and behaviour, the evaluation did contribute to some change in the mindset. However, other factors also contributed to the change. For example, when probed about a specific change in attitude mentioned, one participant stated that:
‘So the change in leadership is the reason for the change in the project implementation itself, so that is not caused by Centre for Research on Evaluation, Science and Technology (CREST), but it has it has caused by the change in attitude in the environment … the attitude now of Senior Management, understands the work of academic staff development.’ (Participant 7, interviews, Teaching and Learning Manager 2)
This suggests that it is important to consider the evaluation in the context of broader organisational trends and contextual factors, to understand the ways in which the evaluation process can best be adapted to encourage use.
At the funder level, the study found that there is no meaningful change in terms of thinking and the way the project is implemented. The funder feels that the UCDP implementation and the two sister grants, teaching development grant (TDG) and research development grant (RDG), that were discontinued to form the UCDG are the mirror image and have no meaningful change. Improved procedures and meeting reporting deadlines were accompanied by challenges of not meeting targets, as this was a common practice even in the previous triennium. The different viewpoints from other participants as opposed to the funder illustrate that, change cannot solely be measured in a short space of time but in the long term, as when comparing the previous and current triennium, the funder indicated that a small movement is noticeable after the introduction of the evaluators. When viewing submissions from individual and funder levels, there is a corroboration with regard to change; however, the expectation is at different stages from both ends. The DHET participant said:
‘But from where I’m sitting, I don’t see much of a meaningful change from the way the UCDG is run, or the UCDP is run at WSU from you know, the two sister grants were run.’ (Participant 6, interviews, DHET Deputy Director Teaching and Learning)
In addition to the analysis provided by the funder, the progress reports trends from 2014/15 to 2019 and assessment reports were analysed to substantiate the claims by the DHET participant that there is not much change with regard to the UCDP implementation.
When analysing the performance of the two trienniums, the TDG tenure 2014/15 – 2017 has better performed (12% unspent) as compared to the UCDG (2018–2019), which has demonstrated the deterioration in terms of expenditure patterns (15% unspent) and validates the claims by the funder. This analysis illustrates the university’s ability to use its earmarked UCDG as another measurement tool utilised by the DHET to measure performance.
The skill set acquired by the project leaders during the evaluation process
The main question in this theme focussed on whether any notable skills were acquired by the stakeholders involved in the evaluation process. The findings revealed that the skills acquired during the evaluation process can be discussed at two levels, namely at the project leaders’ level who were involved in the process and those who were not involved. This finding is in line with the theoretical framework that advocates for interactions between different stakeholders during the evaluation that will influence learning (King & Alkin 2019). While those who participated in the evaluation gained the most significant skills, those who interacted with participants but were not themselves involved also benefitted, to a lesser degree. One participant stated that:
‘So it was a nice way of learning how to manage a project, especially the M & E strategies that are used because all that it did was moving from “uh,” the baseline would just be moving as you wish.’ (Participant 4, interviews, Executive Director Student Affairs)
‘I learned, so much in terms of how to run a successful project.’ (Participant 4, interviews, Executive Director Student Affairs)
Beneficiaries identified communication as one skill that is lacking among project leaders. Communication weakness from the project leaders was seen to pose a risk to the project and the institution as the skill transfer may limit the project implementers, therefore limiting the benefits to recipients through the programme. It is evident that while these skills have been obtained, further steps are needed to apply these skills, some of which require systemic and cultural shifts within the organisation to facilitate effective communication.
Although project leaders do not believe that the skills were exclusively a result of their participation in the evaluation, evaluators commended some project leaders who were involved in the process for assisting them with communicating the evaluation’s purpose and processes. The study found that at the project implementers’ level, those who did not participate in the evaluation process often did not have sufficient communication skills to facilitate the implementation of the evaluation. This was evidenced when the two project implementers during the project life cycle changed their portfolios to be project leaders, which gave them a deeper insight after starting to have interactions with evaluators. Through this, it became apparent that skills had not sufficiently transferred to all levels to ensure the efficient and effective implementation of the project.
When looking at the components of what process use seeks to change, the lack of skill transfer from the stakeholders who were involved in project implementation might delay some of these changes at an institutional level. The implementers on the ground are key stakeholders who shall be directly involved in the process, so that they may improve their skills in terms of implementing interventions. Moreover, limited skills at a certain level of the project may further prolong the change that may be seen in the project as it has been stated that if project implementers are not involved in the evaluation, there will always be that reliance on only the project leaders who were part of the process.
Changes in the performance of university capacity development grant
The authors wanted to assess the changes in the performance of the UCDG that would be attributed to stakeholders who were part of the evaluation process. While programmatic performance is not part of Alkin’s framework, it was added in adapting the framework, given that it is the core goal of the evaluation that was undertaken.
One of the most important findings on the effect of the evaluation process on performance was that the DHET was strongly involved in the evaluation, such as through communicating with the evaluators about WSU’s organisational culture. This was negatively perceived by certain participants from the university and means that from the university’s perspective, the inclusion of many stakeholders without clearly defined roles may affect the evaluation process negatively, as it would also contribute to the exclusion of certain stakeholders. For example, in this case, the DHET participant showed more involvement and understanding as compared to the university stakeholders in illustrating the university culture. This illustrates a risk that a process that is open to participation can be co-opted by individuals with more time to be active or vocal in an evaluation process, even if they do not hold core knowledge for the evaluation.
Additionally, there seemed to be an assumption that in the case of the evaluation, many hands would boost performance, but in fact, many participants would be there for compliance purposes and not necessarily be able to make meaningful inputs. The DHET participant concurred with the finding that the involvement of many stakeholders in the evaluation process may not be relevant if there are no clarified roles. He indicated that as DHET they have witnessed many situations where projects are decided by a few individuals on what shall be in and out which demonstrates a lack of integrated systems. As a result, all other stakeholders would only be there to implement what was planned by a few individuals with no further contribution. The implications for the lack of understanding of the context by institutional stakeholders would be attributed to the staff turnover rate, specifically, the stakeholders that were involved in the process and playing the leadership role that has been discovered to be derailing progress in terms of performance as more training is required and new stakeholders come up with different ideologies and philosophies:
‘The change in leadership actually reversed, … some of the things that we have learned in that many people were brought in, … people who will come with “uh” different understandings, theories, and perceptions of what the UCDG was.’ (Participant 7, interviews, Teaching and Learning Manager 2)
This implies that while participation is certainly positive, it is more meaningful if it is intentionally curated, rather than being left entirely up to the evaluand.
Overall, this suggests that process use can play an important role in improving the performance of the project, and this is an area that would benefit from further exploration. It has been stated that the improvement noted in integrating systems is attributable to the arrival of the new DVC who holds a strategic position and is hands-on in the project and that has brought changes in terms of structural issues, the performance of the project and cultural changes in the institution. When one considers the counterargument above, there is an indication that if stakeholders who have occupied key strategic roles in the institution may be involved in the evaluation process, more positive changes would be achieved, including further changes in performance. The effect of the more important stakeholders that hold a strategic role involvement in the evaluation process needs further investigation, as ‘participation’ is often left as an open concept, without further interrogating different levels and mechanisms of participation.
Lastly, the stakeholders who were part of the evaluation process from the beginning might not be very transparent in terms of the information and even upskilling the new project leaders so that their confidence is boosted. Therefore, these challenges may delay the performance of the project and cascading of the information to the relevant places such as faculties and departments, and this is in line with the finding in the first theme.
Organisational changes of culture
The last question was to understand the change in the project and university culture that would be associated with evaluation use. The study has noted a shift in the traditional way of reporting from both the institutional and funder or DHET levels. It has been indicated that previously it would be sufficient to report the number of activities performed per project; however, after the involvement of evaluators, the reporting did improve to such an extent that progress was tracked up to the outcomes and impact of the project and further evidence to prove the achievement or non-achievement of the objectives. This is another factor that illustrated a shift of mindset in the project leaders and implementers that may not solely be attributed to process use.
The DHET participant believed that some deep-rooted structural and cultural challenges needed to be uprooted for the performance and change to be glaring. As a result, the funders claimed that from the previous project, there were no meaningful changes in terms of implementation of the UCDP from their perspective. To support the latter statement one participant indicated that the mindsets have not changed yet in some staff members within the institution. As such things are done only for compliance rather than understanding the importance of certain procedures. The submission by the funder is supported by the claims made by one participant who mentioned the issue of bureaucracy in financial procedures that delay the spending of project funds. The participant believes that:
‘I think the university red tape; and the financial procedures delay a lot of usage of the funds because of the policies and procedures that we have to follow.’ (Participant 5, interviews, Teaching and Learning Manager 1)
A further interrogation was made on the assessment report that was provided by the evaluators with regard to how the project has since changed its implementation to achieve its objectives. Table 2 depicts the results from the evaluators:
| TABLE 2: Three main dimensions scorecard. |
To unpack the findings from evaluators, the following summary is provided:
- The ‘strength’ and ‘positive features’ of the UCDG report are quite detailed, reporting on implementation and achievement of outcomes as well as lessons learned. This reflects a culture or organisational learning that the team should build on for the next or other projects. In addition, project underspend was relatively low at 15%.
- Some areas that require the most improvement refer to the understanding of the logic model and its components. The relatively average score for dimensions around conceptualisation – description of activities and outputs, and especially of outcomes and outcome indicators, point to an area into which the team needs to put more effort in the future.
This points to variations among improvements emerging from the evaluation, and in particular, the different ways that changes made are experienced by different stakeholders. The institutional and evaluator components reinforce each other, but the funders are still not satisfied with the implementation. These differences are associated with the fact that when the DHET measures the change in implementation they align it more with financial trends while the institution and evaluators, on the other hand, consider the financial trends but also consider the progress in actual implementation and reporting as literature explained when measuring performance (DAC 2002; Mascia et al. 2014). In support of the DHET viewpoint, project implementers claim that culturally the institution is still lacking support at certain levels, as some interventions may not be implemented based on the fact that there is a lack of buy-in from the faculties.
The role played by the evaluators should be key in breaking the barriers of culture through the calibre of key stakeholders and understanding the context (Ledermann 2012). The evaluators have to ensure that the stakeholders involved are at a level where they can be able to tabulate cultural issues during the engagements so that guidance should try and align with those cultural issues. Should the evaluators not play this role, it may be difficult to transition from the old culture to the new developments (Tanner & Willingh 2014). Lastly, the assumption would be that key or senior management who hold strategic positions were not part of the engagements with evaluators that made it difficult to instil the importance of evaluation and how can it be infused in the institutional context and further steer the direction as advised by evaluators.
The final finding revealed that there is not much change in the project as the interventions that are implemented are a continuation of the old interventions and were not based on any research or evidence. It was further explained that culturally, a bottom-up approach shall be adopted where interventions shall be problematised, from the people who need and experienced challenges that require these interventions, so that when they are designed, they respond to the actual needs of many rather than a few individuals. This may change the claims that were made by the DHET that the institution has returned a significant amount of money from which on their side they associate it with inefficiencies within the institution.
Discussion of themes emerged
This section provides a discussion of key findings as related to the literature on different themes from Alkin’s framework that guided the study.
Individual thinking or behavioural change of procedures developed through process use
The researcher broadly interrogated the noticeable changes from the way the project was implemented and further the procedures that were developed through individual changes or behavioural changes that may improve the effectiveness of interventions.
At an individual level, the evidence demonstrated that there was a change to individual thinking as the mindset of most project leaders and project coordinators that were involved in the evaluation process has shifted in the way they looked at project implementation from which this change is associated with their involvement in the evaluation process. Some project leaders do not attribute these changes exclusively to the evaluation process.
Other factors such as structural, cultural, change of leadership and even the people who are involved in the project may contribute to or hinder the process. Even evaluators have a role in influencing the stakeholder selection that may influence organisational change that may come as a result of involvement in the evaluation process.
Similarly, at the institutional level, specifically, the recipients explained that there is a shift in the mindset as positive competition has been noted based on the influence of the interventions at the faculty and departmental level. This is in line with the study by Cottrell et al. (2015), which highlighted that more effective activity is likely to provide benefits to the project and includes a positive change to the quality and impact of the programme since the implementation of the evaluation. Additionally, it is building relationships and educating the stakeholders who are part of the evaluation process, many of whom reported positive changes in capacity.
At the funders’ level, procedural changes are still in nascent stages; however, there are still gaps that have been noted specifically with regard to the targets. These gaps are in line with the literature that advocates for mechanisms that shall be adopted for complex situations, these gaps may be a result of other factors that were alluded to by the DHET representatives such as cultural, structural and so on (Ledermann 2012).
The first key finding that prevail in this theme is that while evaluation process use may change the individual thinking and procedures, other factors also contribute to the change of attitude. As such, it is important to understand individual decision-making processes in organisational and institutional contexts, to strengthen the possibility of process use.
The skill set acquired by the project leaders through their involvement in the process
The level of knowledge and exposure as alluded to by some key stakeholders, the experience of managing the project to its entirety, the confidence obtained, the interaction with different stakeholders at different levels, theoretical framework understanding, inter alia are the skills that have been acquired by the project leaders that were involved in the evaluation process. This is particularly noteworthy because some project leaders who did not participate in the evaluation process still managed to gain certain evaluation-related skills through organisational cascading. However, there is still room for improvement in terms of ensuring a flow of information to the project implementers who seemed to lack the necessary skills in the implementation of the project as the ultimate goal is to improve the institutional procedures and culture for the effective implementation of different projects. The study by McNulty (2012) agrees with this finding that knowledge management, capacity building and communications are important tactical issues for improving evaluation use, while Patton (1998) believes that involvement or engaging in the evaluation process has enduring benefits for both individual and organisational levels. These benefits may in turn improve the performance of any project or organisation holistically as these skills have been noted to improve the project implementation to some extent.
In addition, this study further aligns with Patton (2007) who explains that incidental learning may be found as a result of stakeholder engagement as some project leaders learned even beyond mentioning how they have to understand the theoretical framework in the context of M&E. However, the study has demonstrated the weaknesses in communication skills that may lead to challenges such as delay in the change in implementation of a project that does not conform to characteristics of process use. This weakness implies that the interventions may not reach the targeted audience if there are no proper channels of communication (McNulty 2012).
The findings in this study illustrate that skills acquired or learning achieved during the evaluation process may only be confined to those who participated in the process if there is no proper plan of transferring the skills to other members of the project who were not involved in the process including the project implementers. Therefore, to notice change in the implementation of the project, there needs to be a proper plan in place on how the skills shall be transferred to different key stakeholders who were not involved in the process.
Changes in the performance of university capacity development grant related to the involvement of certain key stakeholders in process use
The submissions made by different participants indicated that the change cannot be brought by one factor, but a multitude of interconnected factors related to an evaluation will bring change. Therefore, the involvement of key stakeholders in the evaluation process and the involvement of the senior management level can be an added advantage to the institution to bring positive spin-offs in terms of implementation and attitude. However, one thing that became clear in the findings is that there is no glaring relationship between the number of stakeholders involved in the process and the performance of the project as long as those stakeholders do not influence the organisation at senior positions. This submission is in line with Tanner and Von Willingh (2014) when they address the hierarchal structure, meaning the seniority of staff members and the role they play in the cultural encapsulation. They further explain that if senior staff are not involved, decision-makers might delay the project changes and further limit the scope of work specifically to project leaders and implementers.
This connects more to the calibre of stakeholders involved in the evaluation process based on the influence they have in the strategic levels of the university. Even though in many circumstances it also depends on the agenda of the key stakeholder that is involved; therefore, the involvement of senior personnel in the process can either be positive or negative depending on the particular agenda.
In addition, the stakeholders should be limited to a number that is manageable to avoid conflict between these members as the interests may differ. This brings in the role that should be played by the evaluator as guided by the skin in the game principle as alluded to in the blue marble principles to mitigate conflict from different stakeholders (Patton 2020). Therefore, currently, no evidence can solely demonstrate a link between stakeholder involvement in the process and the performance of the UCDG project.
Furthermore, the study found that the DHET, evaluators and institutions have different approaches to framing performance; while DHET places greater emphasis on financial performance, evaluators and institutions prioritise project implementation. The evaluators found gaps in implementation, which the institution experienced as a performance gap, but DHET’s primary concern was with the return of unutilised funds. All stakeholders agreed that M&E should reflect results across the entirety of the intervention’s results chain but recognised that their systems emphasise specific levels.
The funders’ determination of UCDP performance, the commissioners and evaluators were mainly based on the progress of the project through the result chain and whether it has met its targets and objectives. Overall, there is convergence among all participants that there is a slight change in performance in the project, but their views differ in terms of the extent of the noticeable change. Some believe that there is a slight change even though it is far from maturity, whereas others believe that there is no change at all. This study has contributed to the theory by looking at the relationship between stakeholder involvement and performance, which has demonstrated that only senior management involvement might influence the performance if they are involved in the process.
The organisational changes of culture as a result of evaluation process use
At the project level, the implementation of the UCDG has shifted positively, which has changed the old cultural way of doing things. Previously, UCDG was centralised in the Teaching and Learning Centre and after the involvement of evaluators, activities were stretched to cover the broad spectrum in the university. However, a more integrated way of working was adopted, which has served to break the silo mentalities that were previously there. This agrees with the study by Aapaoja et al. (2013) that the interaction between different stakeholders has resulted in creative solutions and the sharing of ideas and Tanner and Von Willingh (2014) that the more hierarchal culture changes the more changes in the project implementation will prevail. One participant stated that:
‘I think it’s not the same thing the fact that it was repackaged as a UCDG has introduced other elements that were off the radar “uh” before the UCDG was there.’ (Participant 1, interviews, Senior Director Learning and Teaching)
However, at the institutional level, the findings revealed that for process use to make a noticeable change in organisational culture, there is still a long way to go as the project implementers complained that there is still a culture of separating the project and academic enterprise, and as such there is a lack of buy-in from faculties that inhibit a wider uptake of the evaluation results. This finding illustrates a disconnect in terms of communication between project leaders and project implementers as the project coordinator indicated that the faculties have now designed their faculty plans that incorporate the interventions. The DHET explained that factors such as culture may hinder progress and McNulty (2012) agrees with the position when he stated that more persistent and systemic cultural and psychological dimensions may either improve or derail the performance of any project or organisation holistically.
Holistically, there is no convergence in terms of change in the implementation that has a direct influence on the cultural changes; different stakeholders reported divergent perspectives on which cultural changes could be observed and how these related to the evaluation process. However, there is convergence from all evidence that conceptually, the project still needs focus even following the evaluation, as the interventions are carrying over from the previous project and they are not tailored to address real needs of constituencies.
Summary and areas of further research
The evaluation literature assumes that process use is a positive evaluation outcome, that is most likely to be achieved if an evaluation succeeds in ensuring participatory engagement, and targets building evaluation capacity through a learning-by-doing approach by all evaluation stakeholders. However, this study has uncovered a range of complexities in process use that would hinder its success. It suggests that for process used to be meaningful, there needs to be a full understanding and acknowledgement of the unique institutional complexities, in addition to diverse stakeholder dynamics. Adapting to this level of contextual understanding may be a challenge for many evaluations, which happen in the context of real constraints of time, budget, engagement and capacity.
Some interesting areas that merit further research include:
- For process used to be strengthened, it must include a strong understanding of organisational and institutional dynamics that can shape the process.
- Participation in an evaluation process is critical for process use, but for it to optimally support use, participation should be delineated, with roles and responsibilities outlined.
- When conducting further research on the enablers and constraints to evaluation process use, consideration should be given not only to the evaluation practitioners but also to managers and their organisational context.
There is a need for further development of individual, institutional and cultural results of process use
Limitations of the study
The study was limited to only one university in South Africa in the Eastern Cape and the study particularly focusses on the UCDP project as its unit of analysis. Moreover, in the focus group, five participants were expected but only four participated in each group. However, to overcome the limitations different instruments were used to collect data, through four layers of semi-structured interviews conducted including external stakeholders and beneficiaries.
Conclusion
A key finding of this study is that different stakeholders understand both the project performance and the evaluation’s implementation process differently. Institutions, funders and evaluators all have different incentives to use the evaluation, as well as different institutional contexts. What could seem participatory and inclusive in one context could seem like failed leadership or preparation in another. Therefore, going forward all measurement tools should be integrated to form a holistic view that would be acceptable to all three key stakeholders. Furthermore, process use should be contextualised based on other institutional context factors that influence implementation processes and evaluation use.
These findings will assist the institution not only in focussing on the results of the project but also on the skills acquired by the people involved in the evaluation process and how they can make use of the capacity building from evaluators to build learning in the process.
While the research focussed on the changes that are brought about by evaluation process use, this research has indicated that other intertwined factors need to be considered in the discussion that affect the change. Understanding these factors and taking them into account in evaluation design and implementation have the potential to significantly strengthen process use. These factors include institutional culture, structural factors, leadership change and systemic factors. This questions blanket assumptions in the evaluation use literature that participation and capacity building inherently strengthen use. It also suggests considerably more research needs to be done to better understand institutional drivers of process use. Lastly, there is a need to develop guidance for both evaluators and managers around how institutional context should be taken into account in evaluation commissioning and designs.
Acknowledgements
The authors would like to thank Dr Luyanda Marhaya for his contribution on the success of this article.
This article is partially based on the author’s thesis entitled ‘Evaluation process use in the University Capacity Development Programme at Walter Sisulu University’ towards the degree of Master of Management in the Wits School of Governance, University of the Witwatersrand, South Africa, 2020, with supervisor Dr Caitlin Blaser Mapitsa. It is available here: https://wiredspace.wits.ac.za/server/api/core/bitstreams/f17376d3-3daa-4884-8e7f-e4cc56057657/content.
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Authors’ contributions
M.M. is the main author who has brought together the contents of this article. C.B.M. provided guidance on writing and editing the article. C.B.M. also reviewed and approved the contents of this article.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
The data used in this manuscript received ethical approval from the University of the Witwatersrand, receiving clearance number WSG-2021-43. According to the ethical approval received, data cannot be made publicly available or shared. However, the authors are available for further discussion around the methodology or data management processes.
Disclaimer
The views and opinions expressed in this article are those of the authors and are the product of professional research. The article does not necessarily reflect the official policy or position of any affiliated institution, funder, agency or that of the publisher. The authors are responsible for this article’s results, findings and content.
References
Aapaoja, A., Haapasalo, H. & Söderström, P., 2013, ‘Early stakeholder involvement in the project definition phase: Case renovation’, ISRN Industrial Engineering 2013, 1–14. https://doi.org/10.1155/2013/953915
Alkin, M.C. & Taut, S.M., 2002, ‘Unbundling evaluation use’, Studies in Educational Evaluation 29(1), 1–12. https://doi.org/10.1016/S0191-491X(03)90001-0
Arnaboldi, M. & Azzone, G., 2010, ‘Constructing performance measurement in the public sector’, Critical Perspectives on Accounting 21(4), 266–282. https://doi.org/10.1016/j.cpa.2010.01.016
Boston, J., Bradstock, A. & Eng, D., 2021, ‘Why ethics matters’, in J. Boston, A. Bradstock & D. Eng (eds.), Public Policy, pp. 1–19, ANU Press, UTC.
Chiesa, V., Frattini, F., Lazzarotti, V., Manzini, R., 2007, ‘Measuring Performance in New Product Development Projects Designing a PMS for NPD Projects: A Literature Synthesis’, Project Management Journal 10(2), 45–59. https://doi.org/10.1002/pmj
Chilisa, B., 2012, Indigenous research methodologies, SAGE, Los Angeles, CA.
Cottrell, E., Whitlock, E., Kato, E., Uhl, S., Belinson, S., Chang, C. et al., 2015, ‘Defining the benefits and challenges of stakeholder engagement in systematic reviews’, Comparative Effectiveness Research 2015, 13–19. https://doi.org/10.2147/cer.s69605
Crawley, K.D., 2017, ‘The six-sphere framework: A practical tool for assessing monitoring and evaluation systems’, African Evaluation Journal 5(1), a193. https://doi.org/10.4102/aej.v5i1.193
DAC, 2002, Glossary of key terms in evaluation and results based management, Organisation for Economic Co-operation and Development, Paris.
Elo, S., Kääriäinen, M., Kanste, O., Pölkki, T., Utriainen, K. & Kyngäs, H., 2014, ‘Qualitative content analysis’, SAGE Open 4(1), 215824401452263. https://doi.org/10.1177/2158244014522633
Fletcher, G. & Dyson, S., 2013, ‘Evaluation as work in progress: Stories of shared learning and development’, Evaluation 19(4), 419–430. https://doi.org/10.1177/1356389013505176
Forss, K., Rebien, C.C. & Carlsson, J., 2002, ‘Process use of evaluations types of use that precede lessons learned and feedback’, Evaluation 8(1), 29–45. https://doi.org/10.1177/1358902002008001515
King, J.A. & Alkin, M.C., 2019, ‘The centrality of use: Theories of evaluation use and influence and thoughts on the first 50 years of use research’, American Journal of Evaluation, 431–458. https://doi.org/10.1177/1098214018796328
Ledermann, S., 2012, ‘Exploring the necessary conditions for evaluation use in program change’, American Journal of Evaluation 33(2), 159–178. https://doi.org/10.1177/1098214011411573
Mapitsa, C.B. & Khumalo, L., 2018, ‘Diagnosing monitoring and evaluation capacity in Africa’, African Evaluation Journal 6(1), 1–10. https://doi.org/10.4102/aej.v6i1.255
Mascia, M.B., Pailler S., Thieme M.L., Rowe A., Bottrill M.C., Danielsen F., 2014, ‘Commonalities and complementarities among approaches to conservation monitoring and evaluation’, Biological Conservation, 258–267. https://doi.org/10.1016/j.biocon.2013.11.017
McNulty, J., 2012, ‘Symbolic uses of evaluation in the international aid sector: Arguments for critical reflection’, Evidence and Policy 8(4), 495–509. https://doi.org/10.1332/174426412X660115
Nederhand, J. & Klijn, E.H., 2019, ‘Stakeholder involvement in public-private partnerships: Its influence on the innovative character of projects and project performance’, Administration and Society 51(8), 1200–1226. https://doi.org/10.1177/0095399716684887
Patton, M.Q., 2020, Blue marble evaluation premises and principles, in M.Q. Patton, K. McKegg & N. Wehipeihana (eds.), 1st edn., The Guilford Press, London.
Patton, M.Q., 1998, ‘Discovery of process use’, Evaluation 4(2), 225–233. https://doi.org/10.1177/13563899822208437
Patton, M.Q., 2007, ‘Process use as a usefulism’, New Directions for Evaluation 2007(116), 99–112. https://doi.org/10.1002/ev.246
Tanner, M. & Willingh, U.V., 2014, Factors leading to the success and failure of agile projects implemented in traditional waterfall environments, Slovenia, viewed 06 December 2021, from https://www.researchgate.net/publication/312187721.2021.12.006.
| 
Hide
Show all
Email this article (Login required)
