Evaluation findings are increasingly becoming valuable for policy makers in Kenya. The Directorate of Monitoring and Evaluation is responsible for providing reliable data findings for decision-makers. They are in turn expected to access the data and information through the National Integrated Monitoring and Evaluation System (NIMES). Unfortunately, the directorate hardly receives timely data as required, thus is unable to make timely decision within the ministry of agriculture, livestock and irrigation in Kisumu County.
The aim of this study was to assess the extent of operationalisation of NIMES through utilisation of the electronic project management information system (e-ProMIS) within the three agricultural departments.
Through single-point face-to-face interviews using semi-structured questionnaires, mixed methods approach and Likert scale were applied to assess the level of operationalisation of, staff competences in, and satisfaction with NIMES. Both random and purposive sampling was used. Using mixed methods approach, primary and secondary data were collected from 10 key indicators and fitted in a binary logistic regression model to assess the level of operationalisation of NIMES.
This article shows that operationalisation of NIMES is unsatisfactory, and data collected are incorrectly formatted. None of the departmental personnel charged with uploading relevant data in e-ProMIS neither accessed nor utilised the platform. There were no champions supporting NIMES, thus no reports generated from the system.
Factors hindering operationalisation of NIMES were: dysfunctional monitoring and evaluation (M&E) systems, limited human capacity on M&E, lack of NIMES champions, limited availability of data, unclear information flow to decision makers and inadequate integration of NIMES in planning and budgeting.
Globally, monitoring and evaluation (M&E) has increasingly come to the fore over the past three decades in being regarded as the key in providing evidence of programme and organisational performance (Christie
Colombia, for example, developed its National Results-Based Management and Evaluation System (SINERGIA) to aid in enhancing the country’s reform towards performance-based management, particularly at the central administration through promoting joint planning and budgeting using system-generated data. The system achieved a high level of development and customisation and is held up as an example of best practices by multilateral organisations, donor agencies and other governments (Manuel
In the United Kingdom, the government’s performance targets contained in the public sector agreements between the Treasury and each of the 18 main departments ensured that the M&E system contained the state department’s overall goal, priority objectives and key performance targets that are also annually reported on (Mackay
Despite the pivotal role of M&E in providing credible information for continental development and decision making (Bhattacherjee
Like in many African countries, the introduction of National Integrated Monitoring and Evaluation System (NIMES) in Kenya in 2004 coupled with the policy on e-government created room for M&E to be an integral part of policy formulation and implementation process at the national level (GOK
The study is undertaken because the Kisumu County government, its Ministry of Agriculture and Rural Development as well as the three departments of agriculture, fisheries and livestock face challenges in operationalising NIMES. These challenges range from limited capacity of the departmental staff to undertake M&E functions, including regular data collection, analysis and consolidation of periodic reports and other relevant information, to the utilisation of M&E findings to inform departmental decision-making and budgeting. The factors determining the operationalisation of NIMES in the targeted departments are undocumented, while the extent to which available data have been uploaded onto the electronic project management information system (e-ProMIS) platform, and eventually fed into NIMES are not articulated. It is further unclear how these three departments have complied with the NIMES standard protocols and procedures in reporting on key indicators of performance within the three components.
Based on these gaps, the main purpose of the study is to analyse the extent to which NIMES has been operationalised in the three departments. It is anticipated that unless NIMES is operationalised using the required e-ProMIS platform, the MED will not be able to fulfil its mandate of adequately reporting on progress in the implementation of public policies, programmes and projects, including the county’s contribution to the County Integrated Development Plan (CIDP
This article shows the M&E capacity gaps that exist at the individual and departmental level in their efforts towards operationalisation of NIMES through utilisation of the MED-recommended e-ProMIS platform. It provides the heads of these departments and their staff with relevant information and approaches on how to fast track their programme and project performance in the implementation of CIDP, MTP, Vision 2030, CAADP and SDGs. It generates vital lessons for the departmental staff to address existing gaps and initiate operationalisation of NIMES, not only in the county, but also in other counties and government departments. It also identifies effective ways of developing M&E capacity within and across the departments, besides indicating how to integrate NIMES in the annual departmental planning and budgeting.
An effective NIMES is focused on enhancing the policy environment for M&E usage, strengthening M&E skills and developing the physical infrastructure necessary to support the demands for M&E data and information management. The Kisumu County’s CIMES is designed to ensure regular reporting on implementation progress of the country’s priority policies, projects and programmes outlined in key policy documents such as the MTP, the CIDP, devolved funds programmes, the National Accountability Management Framework and the Performance Contracts and the Performance Appraisal System. The national NIMES that incorporates the CIMES is designed to report on the government’s commitments to other international frameworks such as the SDGs, the New Partnership for African Development’s (NEPAD) and the African Peer Review Mechanism (APRM). Both NIMES and CIMES aim at strengthening governance by improving transparency, strengthening accountability relationships and building a performance culture within the national and county governments to support better policy making, budget decision-making and management.
An effective NIMES incorporates evaluation, hereby defined as the systematic process of analysing information obtained during regular inspection of project and programme activities undertaken by the departments. The M&E officers should upload the evaluation findings and data on the e-ProMIS platform for use by the departmental personnel and institutional administration. An effective NIMES should also have a functional CIMES, hereby referred to as an integrated and observation system for data management at the county level. Kenya has a total of 47 counties, headed by county governors, and managed by county executive committees and senior management staff, including the Head of M&E. Through CIMES, the county governments should verify whether the activities of each county’s priority projects and programmes are happening according to schedules, and whether resources are correctly spent. A major reference document to help the county government, especially the Ministry of Agriculture and its departments of agriculture, fisheries and livestock to assess its performance, is the 10-year CIDP (
Both NIMES and CIMES are supported by the e-ProMIS. This is a Web-based data collection, tracking, analysis and planning tool that ensures coordination of development efforts, data dissemination, alignment and harmonisation of projects with the county strategy. Thus, effective operationalisation of NIMES refers to the systematic integration of NIMES into the decision-making process and creation of a permanent demand for its use within the three departments. The departments are regarded to have operationalised NIMES when the M&E personnel and departmental heads ensure that they regularly follow the guidelines provided in NIMES standard protocols and procedures, and that their decision-making is guided by the information available in NIMES/CIMES.
The Kenya Vision 2030 refers to the national level long-term objectives of the country that aims at making the country achieve middle-income status by 2030. It has a series of 5-year medium-term plans that translate the long-term objective into medium-term priorities, objectives and programmes. The Draft M&E Policy of 2012 that articulates the government’s commitment to manage for development results at all levels supports this Vision 2030. This policy provides a clear framework for strengthening the coverage, quality and utility of the assessment of public policies, programmes and projects by proposing that finances for M&E are clearly allocated within the national budget. It focuses on providing credible and greater evidence to the executive government, the legislature and other actors to make informed policy and programmatic decisions, and to hold the public sector accountable for utilisation of allocated resources. The policy further sets the basis for a transparent process for the citizenry and development stakeholders to mutually appraise results by outlining the principles for a strong M&E system to track achievement of the Vision 2030. All the public policies, strategies, programmes and projects managed by ministries, departments and agencies, county governments, parastatals and executing agencies of public programmes are obligated to apply this policy.
Generally, an institution or department is regarded to have an operational integrated M&E system when the following elements are in place: effective leadership and a champion to coordinate and spearhead alignment of M&E tools and frameworks (GAO
In this article, these elements are clustered into four components across individual, institutional and systemic levels. These components comprise: (1) M&E planning and reporting, (2) M&E structures and human resources, (3) M&E processes and procedures and (4) data and information management.
Components of a fully operationalised National Integrated Monitoring and Evaluation System.
In this article, the operationalisation of NIMES (here taken as the dependent variable) is assumed to be heavily influenced by three independent variables, namely: the extent to which each of the three departments has been capacitated to operationalise NIMES; the level to which each of the departments access and manage data based on the requirements by the Directorate of M&E; and the degree to which NIMES receives support from senior departmental officials and staff. The framework further posits that an operationalised NIMES within the county and its constituent departments is likely to contribute to institutional growth, development and performance against the strategic objectives and targets. On the other hand, two intermediate or extraneous variables are assumed to influence the independent and dependent variables. For example, adherence to M&E policy guidelines (①) is anticipated to influence operationalisation of NIMES, such that the departments that access and implement the policy guidelines as indicated by the MED are expected to be better positioned to operationalise NIMES than those that do not. Similarly, the presence of an enabling institutional environment (②) is assumed to influence operationalisation of NIMES.
Conceptual framework for operationalisation of National Integrated Monitoring and Evaluation System (NIMS).
As indicated above, the level of operationalisation of NIMES as well as the utilisation of the e-ProMIS platform by each of the departments is fitted as the dependent variable, where a binary response dummy variable of 1 is assigned if NIMES is operationalised by the department, and 0 if otherwise. Operationalising NIMES (❶) in these departments hinges on studies conducted by Boudreau (
In this article, the dependent variable (❶) is assumed to heavily depend on three independent variables (see
‘Capacitated departmental staff on e-ProMIS’ such that the more the number of trained personnel within these departments, the higher the likelihood of operationalising NIMES by ensuring that it is maintained and utilised in generating valuable information for policy makers for timely decision-making, and vice versa.
‘Frequency of data uploads onto the e-ProMIS platform’ in that the departments with more frequent data and information upload onto the platform are more organised in their routine M&E activities.
‘Total budget allocated for M&E functions’ in that the departments that allocate significant amounts of budgets (at least 2.5% of the total annual budgets) to support M&E activities are more likely to operationalise NIMES than those with lower budgets.
‘Capacitated staff capable of uploading data on e-ProMIS’, thus updating the NIMES such that the extent of capacity strengthening of each department and the number of departmental staff or officers able to upload data on e-ProMIS according to protocols and procedures, especially those that have benefited from training on management of e-ProMIS platform are positively correlated.
‘Access to relevant M&E tools and approaches’ in that the departments with quick access to relevant M&E tools, protocols, instruments, approaches and methodologies are more likely to adopt NIMES and operationalise its standards and procedures than those without.
‘Staff engaged in M&E activities, such as periodic progress reviews, data capture, and work planning and budgeting’ such that enhanced participation of departmental staff in M&E activities according to NIMES standards and procedures is likely to positively influence transparency, accountability and NIMES sustainability within these departments.
‘Personnel satisfaction with NIMES’ such that the more the departmental staff are satisfied with the benefits derived from the functional and user-friendly NIMES, the higher the probability that they would ensure its sustainable implementation.
‘Availability of NIMES champions’ and enhanced ‘departmental priority in NIMES maintenance’ in that the departments that prioritise the maintenance of NIMES and ensure that they have champions for the operationalisation of NIMES are better able to institute the system.
‘Senior staff buy-in of NIMES’ such that those departments with ownership and buy-in by the senior staff members, including heads of department, deputy directors and directors are better positioned to operationalise NIMES than those without such ownership and buy-in by management.
‘Obligatory use of NIMES in reporting and planning’ in that departments that make it obligatory to only produce biannual and annual reports using data generated by the e-ProMIS platform and NIMES are able to more quickly operationalise NIMES than those that allow the reports to be generated via other approaches other than through NIMES.
In this article, the case study design was adopted to answer questions like ‘how?’ or ‘why?’ with regard to operationalisation of NIMES in the county. Out of the 286 staff within the three departments, 43 staff (26 men; 17 women) are full-time personnel who are responsible for M&E, knowledge management and ICT. To ensure data sufficiency and minimise data saturation, the countywide sampling frame and unit of analysis comprised 31 (19 men; 12 women) out of the 43 responsible personnel (
Sample size distribution.
Ministry or department | Respondent | Population | Sample |
---|---|---|---|
Entire ministry | Director – ICT/M&E | 1 | 1 |
Agriculture | Head of ICT & M&E | 1 | 1 |
Deputy – ICT | 1 | 1 | |
ICT Officers | 5 | 3 | |
Deputy – M&E | 1 | 1 | |
M&E officers | 8 | 5 | |
Livestock | Head of ICT & M&E | 1 | 1 |
Deputy – ICT | 1 | 1 | |
ICT officers | 4 | 2 | |
Deputy – M&E | 1 | 1 | |
M&E officers | 7 | 5 | |
Fisheries | Head of ICT & M&E | 1 | 1 |
Deputy – ICT | 1 | 1 | |
ICT officers | 3 | 2 | |
Deputy – M&E | 1 | 1 | |
M&E officers | 6 | 4 | |
M&E, monitoring and evaluation; ICT, information and communication technology.
Primary and secondary data were collected using mixed methods approach. Primary data were collected from the 31 departmental staff using a semi-structured questionnaire. A pre-tested semi-structured Likert-type questionnaire that assessed the extent of respondents’ satisfaction with NIMES was also administered. The five-point Likert scale was adopted as it yields a distribution resembling a normal distribution curve (Likert
Statistical data were analysed using Statistical Package for Social Sciences (SPSS) programme, besides fitting a binary logistic regression model to estimate the probability of selected independent variables in influencing operationalisation of NIMES in each department. The probability of each department utilising NIMES standards and procedures was estimated using the following model:
where
where
This article followed all ethical standards for a research without direct contact with human or animal subjects.
Under this independent variable, data sets are generated on five key indicators, namely: (1) number of full-time staff engaged in M&E activities, (2) number of personnel trained on e-ProMIS, (3) total amount of budget allocated to M&E activities, (4) frequency of data uploads on e-ProMIS and (5) number of staff skilled to upload data on e-ProMIS.
The effectiveness of M&E systems and operationalisation of NIMES depends on the proportion of staff who have accessed M&E training, and the proportion of annual budget dedicated to M&E activities. Results show that the three departments have different staff categories – technical, support and administration. Out of the 286 staff, 43 (26 men; 17 women) are full-time personnel who are responsible for M&E, knowledge management and ICT (
Distribution of sampled respondents.
Regarding budgetary allocation for M&E activities, the results show that each of the departments has dedicated an average of 3.6% of the annual budgets of US$ 1 865 249 to M&E functions. However, very little progress has been made in ensuring that appropriate M&E tools and protocols are developed and used in data collection. There is also no evidence of utilisation of NIMES through e-ProMIS platform, because no consistently archived data or time series data are available.
Results show an intermittent pattern in the frequency of data uploads onto the e-ProMIS platform. In as much as 12 personnel (7 men; 5 women) have gone through training on e-ProMIS (four personnel per department), only seven of them (4 men; 3 women) can partially upload data on e-ProMIS (Agriculture = 4; Livestock = 2; Fisheries = 1). This shows some capacity gaps in fisheries and livestock departments because only half of the trained personnel could partially upload data. In as much as the personnel have not uploaded data on e-ProMIS platform, they are prepared to start using the platform. The rest of the personnel have neither accessed the ‘
With regard to data quality, the results show a significant relationship (
Under this independent variable, data sets are generated from two indicators, namely: (1) number of personnel accessing relevant M&E tools and approaches, as well as the ease of access to these tools; and (2) number of staff engaged in M&E activities, especially periodic progress reviews, data capture and work planning and budgeting.
The study shows that all the 23 personnel (14 men; 9 women) who have benefited from training on M&E have also accessed relevant M&E tools and approaches (Agriculture = 11; Livestock = 7; Fisheries = 5). The commonest tools they have accessed include: logical framework template, direct observation checklists, structured questionnaires, focus group discussion guide, in-depth interview guide and theory of change guide. However, they have hardly been involved in the development of new M&E tools. Results also show that in as much as only 12 personnel (7 men; 5 women) periodically collect, but infrequently upload, the data, the regulations stipulated in the NIMES that require the M&E personnel to co-develop the data collection tools with inputs from ICT unit are not followed. This explains why data are not frequently uploaded, because they are collected or stored in different formats and exclude standard indicators required in specific fields within the e-ProMIS. Similarly, variations exist among the departments with respect to the kind of M&E documents and standard operating procedures that guide their M&E functions and processes. The department of fisheries uses the performance measurement plan, while the departments of agriculture and livestock use the M&E plan. None of the departments uses the M&E framework, the M&E strategy, the Handbook of Indicators from MED or the M&E policy, all of which are meant to enhance conformity with NIMES. Results further show that all the personnel are engaged in M&E activities in one way or an other, especially in annual progress reviews, work planning and budgeting. However, it is evident that some of the work planning and budgeting sessions hardly use M&E data to inform decision-making and future programming.
Under this independent variable, data sets are generated from four indicators, namely: (1) level of personnel satisfaction with NIMES, (2) extent of availability of NIMES champions and enhanced departmental priority in NIMES maintenance, (3) level of senior staff buy-in of NIMES and (4) extent of utilisation of NIMES in reporting and planning.
Results show varied levels of personnel satisfaction with NIMES across the departments. Out of the 31 personnel sampled in this study, 16 (10 men; 6 women) are satisfied with NIMES (Agriculture = 8; Livestock = 5; Fisheries = 3). The satisfied personnel comprise those that have been trained on e-ProMIS (12) and are currently attempting some data uploads (7). Those showing dissatisfaction cite lack of clarity in guidelines provided and availability of limited access to the required data. With regard to availability of champions to support operationalisation of NIMES, the departments indicate that there is very minimal patronage from high-level policy makers and ministry officials. In as much as only the department of agriculture indicates that they have a champion, it is unclear how this champion has supported the operationalisation of NIMES. This confirms the limited departmental priority in NIMES maintenance.
Notwithstanding the limited engagement of a champion on NIMES, there is a general observation that the senior staff have demonstrated their buy-in of NIMES (Agriculture = 10; Livestock = 7; Fisheries = 5). This buy-in is exhibited in their support for budgetary allocation of up to 3.6% annually on M&E. They have also supported the staff in undertaking targeted M&E studies. Regarding utilisation of NIMES in reporting and planning, all the departments indicate they indirectly use generated data from running projects. However, the extent of actual utilisation of NIMES in this process is very limited as indicated by 12 respondents from the departments (Agriculture = 6; Livestock = 4; Fisheries = 2). In general, it is observable that the departments record very minimal NIMES patronage culture. This should alert the policy makers to review the entire process of operationalising NIMES.
Based on the five-point Likert scale that assessed the level of stakeholder satisfaction with NIMES, an average score of 3.12 (indicating indifference) is generated, with average responses from the departments varying from 2.3 (agreeing) to 3.8 (disagreeing) with posited statements (
Likert scale ratings on attitudinal assessment.
The data generated from the indicators are used to fit a model to determine the extent of operationalisation of NIMES in the three departments. The logit model (Maddala
Thus, the likelihood that a department operationalises NIMES (
where ε is the error term with a logistic distribution. Based on
where the dependent variable,
Based on
where
where
Using
The model with all predictor variables is statistically significant (
Logistic regression predicting likelihood of operationalising National Integrated Monitoring and Evaluation System (NIMS).
Variable | Standard error | Wald | Odds ratio | ||
---|---|---|---|---|---|
Number of staff capacitated on e-ProMIS (ST) | −0.288 | 0.22700 | 16.400 | 0.02 |
1.810 |
Frequency of data uploads (FU) | −0.250 | 0.01300 | 25.590 | 0.02 |
1.200 |
Total budget allocated to M&E activities (BG) | 0.069 | 0.00003 | 3.840 | 0.05 |
8.240 |
Number of staff uploading data (OT) | 1.040 | 0.04000 | 3.760 | 0.05 |
3.190 |
Personnel access to M&E tools (ACC) | −0.204 | 0.10230 | 2.105 | 0.06ns | 0.909 |
Number of M&E staff (NS) | 0.050 | 0.02000 | 2.700 | 0.04 |
5.130 |
Personnel satisfaction with NIMES (SAT) | −0.045 | 0.09535 | 3.407 | 0.08ns | 1.103 |
Availability of NIMES champions (CHA) | 0.007 | 0.04410 | 2.740 | 0.09 | 1.072 |
Senior staff buy-in of NIMES (APP) | 0.268 | 1.55100 | 3.183 | 0.02 |
4.830 |
Departments update and use NIMES to generate reports (UGR) | 0.330 | 0.12200 | 4.710 | 0.04 |
1.610 |
Constant | 0.027 | 0.37410 | 29.740 | 0.00 | 9.670 |
, highly significant.
ns, non significant at
From the study findings, it is evident that the factors influencing operationalisation of NIMES in the ministry of agriculture can be summarised in
Factors influencing operationalisation of National Integrated Monitoring and Evaluation System (NIMS).
Based on the study findings, it is evident that the three departments can still operationalise NIMES by ensuring that the personnel responsible for M&E, knowledge management and ICT undertake the tasks illustrated in
Requisite stages for operationalisation of National Integrated Monitoring and Evaluation System (NIMS).
In as much as the three departments have been capacitated through trainings and budgetary allocations for M&E functions to operationalise NIMES via the utilisation of e-ProMIS platform, these departments have not made any significant steps towards realising this objective. Only 7 out of the 23 departmental personnel trained and charged with uploading relevant data onto NIMES have partially accessed and used the e-ProMIS platform. This poor performance of the departments in utilising the skills and resources availed to them compromises the credibility of reports submitted to the MED to enable them to prepare relevant policy documents for the Parliamentary Budget Committee and Parliament for approval.
Given the non-adherence to the NIMES through the MED M&E-approved e-ProMIS, there is unsatisfactory performance of the departments towards achieving their M&E objectives based on parameters for NIMES. None of the requirements of NIMES is adhered to, including the requisite stages of ensuring credible M&E processes such as data collection, data analysis, development of M&E tools and adherence to NIMES protocols and standards.
Among the key factors that influence operationalisation of NIMES are dysfunctional M&E systems and units, sub-optimal departmental and human capacity on M&E, as well as limited enabling environment for joint planning, budgeting, peer review of performance and mutual accountability. Other factors include: limited departmental buy-in and ownership of the process; limited NIMES’ champions within the departments and the entire county government; pronounced individual and institutional M&E capacity gaps; limited availability of data and information as well as unclear information flow to decision– makers on the performance and implementation of strategic plans; partial production of vital lessons for further operationalisation of NIMES; limited integration of NIMES as part of public management practice and culture; and inadequate integration of NIMES in planning and budgeting.
A major hypothesis postulated in this article is that unless NIMES is operationalised using the required e-ProMIS platform, the Directorate of M&E will not be able to adequately report on the progress made in implementation of public policies, programmes and projects, and thus the county’s contribution to the CIDP, the MTP, the Vision 2030, the CAADP and SDGs will be unclear. This hypothesis is confirmed, thereby further exacerbating challenges for MED to provide credible data for decision-making.
This article recommends that: (1) the heads of departments should facilitate capacity strengthening of departmental staff to enable them to meet their M&E objectives and operationalise NIMES; (2) the policy makers should articulate roles and responsibilities for M&E personnel, including data uploads on NIMES; (3) the departments should appoint champions who ensure that the personnel operationalise NIMES; (4) the heads of departments should ensure inter-departmental meetings and exchange of information and lessons learned and (5) policy makers should enforce integration of NIMES in annual planning and budgeting.
The author declares that he has no financial or personal relationships that may have inappropriately influenced him in writing this article.
I declare that I am the sole author of this research article.
The author declares that he did not receive any support in form of grants, equipment, drugs or other support from any source to undertake this task, or to facilitate the conduct of the work he has described in this article or the writing of the article itself.
The data sets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
The author declares that the views expressed in the submitted article are his own and not an official position of the institution or funder.