The world is experiencing an increasing demand for effective utilisation of evaluation results (Porter & Goldman 2013), and public universities in different contexts cannot be an exception. In fact, evaluation findings in public universities would go a long way in improving their performance levels and help in promoting administrative accountability as reported by Matsiliza (2012). Hardlife and Zhou (2013) suggest that evaluation results utilisation signifies a gradual shift from the traditional implementation-based approach to the contemporary results-based approach, which in no small measure has penetrated the university list of expectations. The contemporary development process acknowledges the importance of evidence in decision-making and data as an indispensable element of the sustainable development agenda (United Nations 2015). Unfortunately, there remains limited utilisation of evaluation results, which seems to be the consequence of the setting of the organisation not matching the purpose of the evaluation as demonstrated by Widmer and Neuenschwander (2004).

Kusek and Rist (2004) and OECD (2002) opine that evaluation is the systematic and objective process of assessment of an ongoing or completed intervention to determine the relevance, efficiency, effectiveness, impact and sustainability. They argue that an evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of the commissioners of evaluations. In agreement with Scriven (1996), the authors remind us that while evaluation is still a very young field of study and often claimed by several other disciplines such as philosophy, political science, psychology, sociology, anthropology, education, economics, communication, public administration, information technology, statistics and measurement, it is an old practice. An attempt to trace its roots as a profession brings about the ‘chicken-egg dilemma’ because there is hardly any science of what comes first (Basheka & Byamugisha 2015).

Ledermann (2012) equally reminds his audience that the questions of whether or not evaluations are used are as old as the evaluation business itself, and this serves to confirm that it is indeed an old practice but a new discipline as more scholarship on the same is relatively new. Patton (2002) suggests that the history of evaluations and their utilisation were holy processes in the management cycle of organisations, arguing that from the bible, the first evaluation of the 10 days dietary project that got Daniel, Hannaniah, Mishael and Azariah fed on vegetables and water healthier and better fed saw its findings put to practical use (Dan 1:1–20). Højlund (2014) defends the utilisation of evaluations from a rationalism perspective and notes that evaluation itself was born in a time of belief in a better world through rational interventions and social engineering. His argument is that evaluation is inherently rationalist, causal and evolutionary in nature. This same view is shared by Cousins, Goh and Clark (2004) whose definition of evaluation regarded it as a systematic inquiry leading to judgements about a project, programme, policy or organisation’s merit, worth and significance, and support for programme (or organisational) decision-making has important implications for evaluation.

Sandison, Vaux and Mitchell (2006) aver that the utilisation of evaluation has been a topic of lively debate in the development of the public sector since the 1970s. They demonstrate that since the 1970s, the neat and linear connection between evaluation findings and policy or programme improvement has been increasingly challenged. The rest of the article covers the theoretical and conceptual landscape, the problem statement and research questions, methodology, findings from the study, conclusions and policy implications. The theoretical and conceptual framework is addressed first.

This study was underpinned by the utilisation-focused theoretical framework proposed by Patton (2008), who argued that utilisation-focused evaluation (UFE) begins with the premise that evaluations should be judged by their utility and actual use. UFEs should advocate for close collaboration between evaluators and intended evaluation users. His arguments stemmed from the fact that close collaboration enhances understanding and buy-in by the intended users of evaluations, which results in increased commitment to use evaluation findings. It further allows stakeholders to improve the quality of their evaluation processes through joint planning, implementation, monitoring and self-evaluation (Widmer & Neuenschwander 2004). King and Stevahn (2013) assert that interactions with stakeholders make or break any evaluation process. Their argument is that utilisation of evaluation hinges on the process for conducting an evaluation. They add that effective interactions amongst the stakeholders of the evaluation result in a more effective evaluation process.

Smits and Champagne (2008) argue for practical participatory evaluation as a supplementary approach to evidence-based decision-making and accountability in policymaking. This in turn contributes to learning amongst the stakeholders with the intention of reinforcing understanding, the ownership of results and a sense of obligation to follow through on the results. Their argument is built on Turnbull (1999) who concludes that participative processes warrant stakeholders to be involved in evaluation decision-making early enough and to share joint responsibility for the evaluation report with the evaluator. In light of the current study, the UFE theory requires that the institutional design in terms of procedural rules, the evaluation processes as well as the evaluation capacity is adequate to support the utilisation of evaluation results as in the conceptual framework.

Utilisation of evaluation is the use of the findings of an evaluation as well as the implementation of the recommendations of the evaluation. Johnson, Greenseid and Toal (2009) explain that evaluation use is ‘any application of evaluation processes, products, or findings to produce an effect’. Evaluation utilisation demonstrates the consequence of evaluation studies. It answers the question, ‘So what after presenting the findings of an evaluation?’. It therefore underscores the linkage between evaluation and policy. This is because the aim of evaluation is to assist people and public organisations to improve their plans, policies and practices on behalf of citizens (Weiss 1999). Utilisation of evaluation results also ensures sustainability (Schaumburg-Müller 1996). In this study, utilisation is assessed in terms of its five strands of instrumental, conceptual, process-related, symbolic and general utilisation (Balthasar 2008).

Instrumental utilisation of an evaluation is the implementation of the recommendations. This is the intended, targeted and direct use of evaluation by the decision-makers in the intervention. According to Rich (1991), instrumental utilisation refers to ‘utilization that can be documented’; however, Mayne (1994) regards instrumental utilisation of evaluations as the implementation of evaluation results and recommendations. Vedung (1997) describes it as utilising evaluations as means in goal-directed problem-solving processes. However, conceptual utilisation is the change in opinions, attitudes or ideas regarding certain aspects of the evaluated programme as the consequence of an evaluation (Balthasar 2009). Vedung (1997) shows that conceptual utilisation occurs when cognitive, affective and normative insights are gained through evaluations. In the same way, Weiss (1977) observes it as an ongoing sedimentation of perceptions, theories, concepts, ways of looking at the world and enlightenment. Conceptual utilisation as presented by Rossi, Lipsey and Freemen (2004) is the utilisation of evaluation findings to enhance knowledge about the type of intervention under study with an intention of influencing the thinking about issues in a general way.

Process-related utilisation as described by Patton (1997) is one that results in the sharing of the problem under investigation and develops strong networks for the commissioners of the evaluations. This same route is taken by Henry and Mark (2003) who explain it as the action or learning that takes place as a result of evaluation findings or as a result of participation in evaluation procedures. Symbolic utilisation occurs when decision-makers use evaluations to confirm their perspective and to obtain legitimation for themselves (Henry & Rog 1998). Henry and Mark (2003) conclude that it is the use of evaluation to claim a rational basis for action, or inaction, or to justify pre-existing positions. Moleko (2011) identifies the symbolic utilisation of evaluation results when evaluation becomes an instrument of political manoeuvring to the Pork-Barrel approach. From this perspective, evaluations are used as a justification for what decision-makers are interested in doing. Relatedly, Patton (2008) regards symbolic utilisation as the token utilisation made of an evaluation result to fulfil a requirement to do evaluation or to show support for an intervention area. A combination of all these four types of evaluation utilisation therefore gives the general utilisation; in this study, general utilisation was used for the general benefit of utilisation.

Evaluation is strongly dependent on its social and organisational context (Dahler-Larsen 2012). This shows that the extent to which evaluation results are utilised is linked to the institutional context. In this regard, the researchers’ choice of institutional design is supported by the empirical studies of Balthasar (2006, 2008) and Højlund (2014) that suggest the use of institutional design to explain the utilisation of evaluation findings. Their empirical contribution in this regard motivated the researchers to study the institutional explanation for the utilisation of evaluation results. Other studies in the field of evaluation utilisation have dwelt on environment and process-related factors (Cousins & Leitherwood 1986). For example, Lester and Wilds (1990) talk of contextual variables such as the nature of the political environment where policy analysis occurs, the nature of the problem, issue salience and bureaucratic variables, user characteristics, clear definition of objectives by decision-maker, decision-maker interest, decision-maker style and decision-maker participation, whereas Bayley (2008) presents the characteristics of the evaluation as factors that influence utilisation of evaluation results.

Mackay (2006) uses institutionalisation to describe the creation of a monitoring and evaluation (M&E) system that produces monitoring information and evaluation findings which are judged valuable by key stakeholders, which are used in the pursuit of good governance and where there is sufficient demand for the M&E function to ensure its funding and its sustainability for the foreseeable future. Mackay (2006) and Kusek and Rist (2004) demonstrate that the utilisation of results should be embedded within the operation framework of public sector organisations. Kusek and Rist advise that the success of any results-based M&E system depends on how lessons learned are incorporated into the decision-making process of the institution. This requires sustaining the M&E system within the organisation that involves: demand for accountability, clear roles and responsibilities, trustworthy and credible information, accountability, capacity and incentives. Mackay (2006) buttresses that while in African countries public organisation collect a range of performance information, the same is hardly utilised because its quality of data is often poor. Therefore, to enhance the utilisation of evaluation results, Mackay advises that public sector organisations should build reliable data systems that support the M&E function. Moreover, Dhakal (2014) concludes that institutionalisation of evidence-based policymaking, planning and decision-making practices is the panacea for timely demand and use of evaluations in the government sector.

The strength of an evaluation is measured by the extent to which the findings and recommendations are utilised (Patton 1997). Utilisation of evaluations has been appreciated by numerous scholars in the field of evaluations (Patton 1997; Rebora & Turri 2011; Widmer & Neuenschwander 2004). The extent to which these evaluations are utilised has been associated with the design of the institutions for which and in which evaluations are carried out. On this subject, Balthasar (2006, 2007 in Ledermann 2012, 2009), has systematically presented the effect of the institutional design on the utilisation of evaluations. In addition, for the case of Kyambogo University (KYU), notwithstanding the numerous evaluations that have been carried out, available evidence indicates that the utilisation level for the evaluation results is still weak. This has resulted in perpetual low performance levels as indicated by the copious strikes from both students and staff fraternity (GOU 2015a). This low utilisation level has been blamed on the institutional design. Therefore, through this study, the researchers intended to build on the work of Balthasar (2006, 2007 in Ledermann 2012, 2009) to examine the effect of the institution’s procedural rules, processes and capacities on the utilisation of evaluations at KYU.

The following explicit research questions were explored:

This study was conducted through a cross-sectional survey design. Cross-sectional survey research design is a present-oriented methodology that is used to investigate populations by selecting samples to analyse and discover occurrences (Oso & Onen 2009). It was used to study a group of people just one time, in a single session, focusing on the institutional design and utilisation of evaluations at KYU. Surveys are designed to provide a picture of how things are at a specific time. Cross-sectional survey design was adopted because it helps the researcher gather data from a sample of a wider population at a particular time (Amin 2005) and use such data to make inferences about the wider population.

A sample size of 118 respondents, who were selected using simple random sampling and purposive sampling, was considered. Simple random sampling selects a sample without bias from the target or accessible population (Oso & Onen 2009). The sampling frame used was a list of academic staff that was sought from the Directorate of Human Resources to help in determining the respondents. The study also used purposive sampling to select administrative staff from the Directorate of Planning and Development (DPD) that houses the M&E function because they are specific and know the subject matter. Purposive sampling enables the researcher to decide whom to include in the sample based on their typicality (Oso & Onen 2009).

The study used both a questionnaire survey for quantitative data as well as key informant interviews and document analysis for qualitative data. The questionnaire survey instrument was self-administered to respondents in relation to the dimensions and indicators of institutional design as well as those of utilisation of evaluation results. Closed-ended questions in a Likert scale following a five-category response continuum were used: strongly agree, agree, not sure, disagree and strongly disagree. Questionnaires are often a one-time data-gathering device on the variables of interest to the researcher (Amin 2005). Key informant interviews were used to explore which elements of and how institutional design affect the utilisation of evaluations at KYU. These were given to the selected staff of the DPD. Interview method was used because it provides an excellent opportunity for the study to take note of issues that cannot be directly observed or difficult to put down in writing, thus capturing the meanings beyond the words (Oso & Onen 2009).

In processing the analysis of data relating to all research questions of the study, the researchers used the Statistical Package for Social Scientists (SPSS) where data were entered, edited, cleaned and sorted. This programme was used to do univariate analysis. Through this, the study described the demographic attributes of respondents as well as the attributes of utilisation of evaluation results, institutional evaluation procedural rules, processes and capacity. Univariate analysis of these variables was used to obtain descriptive data in the form of means, frequencies and percentages of the respondents. In establishing the relationships amongst variables, bi-variate analysis was done using Spearman’s rank order correlation to establish relationships amongst the study variables (Amin 2005). The correlation coefficient (r) takes a value between -1 and 1, with 1 indicating perfect positive linear correlation and -1 indicating a perfect negative linear correlation. A positive correlation shows a positive association between the variables (increasing values in one variable correspond to increasing values in the other variable), whereas a negative correlation shows a negative association between the variables (increasing values in one variable correspond to decreasing values in the other variable). A relationship value close to 0 shows no association between the variables (Amin 2005). The quantitative data were collated with qualitative data, and various responses from respondents were categorised into common responses that related to the objectives of the study as advised by Amin (2005). These were qualitative data that were obtained from interviews and various documents and analysed by content analysis (Kothari 2004). This helped to corroborate the data obtained through the questionnaires. Therefore, each piece of work was read through thoroughly to identify where the themes belonged and hence presented, interpreted and analysed.

The background characteristics of the respondents included sex, age, faculty or school and academic qualification. The total number of respondents to the study was 118 sampled across the seven faculties or schools that make the university. Out of these, 66.95% were male and 33.05% were female. A total of 30.5% of the respondents were from Faculty of Arts and Social Sciences, 18.6% from the School of Management and Entrepreneurship, the Faculties of Education and Special Needs each had 13.6%, and other faculties such as Vocational Studies, Sciences and Engineering each had 9.3%, 7.6% and 6.8% of the respondents to the study, respectively. A total of 65.3% had a Master’s Degree, 14.4% had a Bachelor’s Degree, 11.0% had a Postgraduate Diploma and 9.3% had a PhD. In terms of age distribution, 44.1% were aged between 18 and 35 years, 28.8% were aged between 36 and 45 years, 24.6% were aged between 46 and 59 years and 2.5% were aged 60 years and above. These characteristics are a true reflection of KYU because, for instance, Faculty of Arts and Social Sciences is the biggest faculty at the university. This implies that participation and inclusiveness of the staff who constitute the majority are critical for buy-in and ownership and hence utilisation of the evaluation results.

This shows that the university staff in terms of academic qualifications are largely young academics holding lower staff positions of assistant lecturers. This is because the minimum requirement for appointment as a lecturer is possession of a Master’s Degree and evidence of enrolment for a PhD at KYU (GOU 2015a). It is noted that this is a good minimum requirement but does not cut across all Uganda’s public universities. It is one of the highest requirements in the country because, in some other public universities in Uganda, people with only a Master’s Degree are recruited at lecturer level. This signifies a need for harmonisation of minimum requirements for academic positions. Low academic qualifications were earlier noted by Kyambogo University (2013) that showed the proportion of PhD holders to the total number of academic staff at the university to be below NCHE standards and blamed the situation on the ban on recruitment from 2011 to 2012. In addition, Baryamureeba (2015) contextually observed that in Uganda, PhD training is still undeveloped.

The results show that procedural rules have a positive average effect on utilisation of evaluation results but are less likely to influence the utilisation of evaluation. The positive relation between procedural rules and utilisation of evaluation results is traced from the availability of laws and frameworks in terms of policies and manuals to guide the utilisation processes. The relation is average because legal framework, for instance the UTIA, is ineffective (Baryamureeba 2015). GOU (2015a) observed that respondents interviewed by the IGG investigation team were unanimously of the view that the current composition of the University Council is unmanageable. The 27 (minimum) University Council members are diverse, including the following: (1) representatives of a sector relevant to the university depending on its objectives and mission, appointed by the relevant body of that sector, (2) representative of the Ministry of Education, (3) three appointees of the Minister of Education from the public, (4) three representatives of the administrative staff associations, (5) two representatives of the students’ association, (6) three members appointed by the University Council from the public, (7) representative of the Ministry of Finance, (8) representative from Ministry in charge of higher education and (9) representative of the District Council in whose jurisdiction the university is found; for the case of KYU, this is Kampala Capital City Authority (KCCA). Such composition has always resulted in University Councils being dominated by members of staff of the university with at least 16 of the total number. This makes it very difficult to take firm policy decisions towards streamlining the administration of the university.

The results are in agreement with Firme et al. (2009) who guide that a set of guidelines establishes rules and procedures to properly conduct planning, implementation and effective utilisation of evaluation results, in all levels of possible implementation.

Regarding the triggering of evaluations, it is evident that who triggers the evaluation matters a lot. Mayne, Divorski and Lemaire (1998) argue that once evaluations are triggered by those responsible for implementation of the measures, difficulties are faced in asking questions of effect and relevance of the measures and programmes. This is because diverse forms of institutionalisation disagree in their ability to deal with the varying information requirements of the target groups. In this case, relatedly, Balthasar (2008) posits that triggering of the evaluation by the unit responsible for the measures or implementation of the examination within the office promotes process-related utilisation. Williams, De Laat and Stern (2002), on the other hand, aver that the independent evaluations need to be carried out by people who are not involved in the implementation of a measure, in contrary to Conley-Tyler (2005) who argues that internal and external evaluators can be independent depending on the evaluation role they choose. In this regard, Schaumburg-Müller (1996) shows that establishment of a unit responsible for the evaluation function in an institution is an important indicator of demand for evaluation and its utilisation. He cites Colombia where evaluations are based on legislation or constitution. Interestingly, Højlund (2014) notes that an organisation with a culture of evaluation and measurement is likely to have a culture that supports its desire to use knowledge instrumentally.

The results show that institutional evaluation processes have a positive average effect on the utilisation of evaluation results and that this effect is statistically significant. The averageness could be because of the bureaucratic tendencies of public administration with vertical administrative structure. This significant effect shows that the process through which an evaluation is carried out is very important in explaining whether the results will be implemented. So critical is the issue of participation of stakeholders which informs ownership of results. The process of doing the evaluations needs to be participatory and consultative so that an input of stakeholders is sourced and where possible is considered (Kyambogo University 2013). Short of that, results are referred to as the evaluators’, which increases the distance between the evaluators and the evaluees (Balthasar 2008).

The results show that the institutional evaluation capacity has a positive and a statistically significant effect on utilisation of evaluation results. Evaluation capacity enhances the ability of the organisation to carry out good evaluations and hence utilise results. This is line with Conley-Tyler’s (2005) finding that building staff capacity may be a strong factor in some cases, but may make no sense for an organisation that is only going to conduct one evaluation once in a very long time, say a decade. In the same line, Léautier (2012) educates us that evaluation capacity to conduct evaluations as well as capacity to use evaluations is very critical.

The current institutional procedural rules are significant and averagely positive in explaining the utilisation of evaluation results in KYU. This is because the existing legal frameworks, notably the law that establishes and guides the governance of public universities, are marred by issues that complicate the functioning of these universities. The positive sign would nevertheless signify that once rules are improved on, then they guide the planning, the costs incurred in the evaluations as well as the implementation of the recommendations from the evaluations. Amongst the procedural rules, those ranked highly by respondents are the rules that pertain to the assumption of costs and it is followed by the participation and involvement of stakeholders. Therefore, the cost implications need be clear and the stakeholders need to actively participate in the evaluations so that they own up to the results and support the utilisation of the findings in improving the performance of the public universities in Uganda.

The current institutional evaluation process has a positive average and significant relation with the utilisation of evaluation results at KYU. This implies that when the evaluation is carried out through a good process, then the results will be good and acceptable and therefore utilisable, hence utilisation. Amongst the processes for evaluation, it is critical that the evaluators are selected on merit through a competitive process. This is deemed to increase confidence in the evaluation results. At Kyambogo, it was evident that many evaluations are commissioned on an ad hoc basis, but what is clear is that the membership is on merit and the evaluators try as much as possible to consult widely prior to the generation of the final report to the commissioners of the evaluation. The null hypothesis relating to institutional evaluation process in Chapter 1 is accepted.

The issue of institutional evaluation capacity was found to be highly related to the utilisation of evaluation results at KYU. Amongst the indicators of institutional evaluation capacity were the competences of the individuals to manage evaluation and the unit responsible for evaluations. These were ranked over and above all others as far as evaluations are concerned. The individual respondents to the study questionnaire themselves believed in their competences to manage evaluations, which is critical in self-esteem. Otherwise, it would be bad if they doubted their skills and abilities to manage evaluation. This unit responsible for evaluations helps to coordinate and harmonise evaluation issues in the university, especially in the field of academic evaluation so that quality assurance is enhanced in the higher education sub-sector. Therefore, it is paramount to strengthen the institutional evaluation capacity such that good evaluations are commissioned, overseen and the results are utilised. This is fundamental such that the utilisation of evaluation results is not left to the goodwill of individuals at the university.

The present study provides a useful departure point for high-education technocrats and public universities’ managers in Uganda to examine the policies and practices so that results of evaluations are utilised: