Abstract
Background: A care facility for people with disabilities struggles to obtain financial support for its Parent Education and Support Programme. The programme’s design includes two implementers, an occupational therapist and a community-based worker, increasing its core costs. To enhance the likelihood of donor support, the Facility considered choosing the best-suited implementer for the programme. To help inform this decision, a formal methodological approach to high-level decision-making called multi-criteria decision analysis (MCDA) was utilised.
Objective: Through a case study, this paper demonstrates how the MCDA methodology, using the analytical hierarchical process (AHP), was applied in a programme evaluation context.
Method: Decision models were constructed using the AHP MCDA method and elicited rater judgments. Raters were drawn from four stakeholder groups: Programme beneficiaries, management, donors, and experts in disability and rehabilitation. This was followed by assigning criteria weights, establishing local priorities for each alternative, and aggregating the judgments. The model was then synthesised, and a sensitivity analysis was conducted.
Results: The findings revealed that specific outcomes were attributed to each implementer, and thus, deciding to employ only one implementer would have had serious consequences for the programme’s quality and the achievement of intended outcomes.
Conclusion: The results confirmed the usefulness of AHP MCDA for programme design decisions.
Contribution: This article contributes by enhancing the understanding of the AHP MCDA methodology. Secondly, it demonstrates the suitability of this methodology for programme designers, evaluators, or non-profit organisations (NPOs) who need to make informed decisions about the design and implementation of interventions.
Keywords: multi-criteria decision analysis; analytical hierarchical process; disability care; programme implementers; design decisions.
Introduction
Non-profit organisations (NPOs) worldwide rely on subsidies to implement social programmes. However, in developing countries like South Africa, funding is limited, and numerous NPOs compete to secure funding from local and international donors. The Cape Town based non-profit care facility for people with disabilities under study has, for years, struggled to source funding for its Parent Education and Support Programme (hereafter referred to as the Programme).
Feedback on its funding applications suggests that one of the reasons is the high cost of its intervention compared to those of similar facilities in the disability care sector. This funding dilemma is not unique to this NPO. Perroni et al. (2019) noted the unwillingness of some donors to fund NPOs with high core costs. The high costs stem from the Facility’s decision to employ and incorporate the expertise of an occupational therapist and a community-based worker in the Programme.
The NPO considered adjusting its dual-implementer design to lower programme costs and, in so doing, be able to secure donor funding. However, they wanted to make an informed decision about which implementer would be best suited to implement the Programme. A multi-criteria decision analysis (MCDA) was conducted using the analytic hierarchy process (AHP) to explore who the preferred implementer would be.
The article begins by introducing the care facility, their intervention and the problems faced. This is followed by an overview of MCDA and the AHP, and why this unique methodology was applied in a programme evaluation context. The method section outlines the steps one would follow to conduct an AHP MCDA. Each step is illustrated through the case study example. Finally, a summary and brief discussion of the AHP MCDA results are presented along with our concluding thoughts about the usefulness of this methodology for programme design decisions.
The care facility
The care facility under study, based in Cape Town, South Africa, provides educational and rehabilitation services to children, adolescents and adults with disabilities, as well as support for and education of their family members regarding care. Persons with disabilities are deemed a vulnerable population group requiring specific and unique care and rehabilitation interventions (Chappell & Lorenzo 2012; Department of Social Development 2016; World Health Organization [WHO] 2002). These services are usually offered in hospitals and clinics.
In developing countries, however, clinics and hospitals are located far from many patients with disabilities. The lack of public transport further compounds the problem of accessibility of such services by the underprivileged. These individuals are also not able to afford rehabilitative care (Chappell & Lorenzo 2012; Sherry 2014). This has led to NPOs implementing community-based rehabilitation interventions. Community-based rehabilitation is both a philosophy and a strategy for making rehabilitative care more need-specific and accessible.
Overview of the Parent Education and Support Programme
Through their work, the care facility under study became aware that in South Africa, parents of children with disabilities are sometimes unsure of their role in preparing their children for school or lack the confidence and knowledge to engage in child development activities at home. This is why the facility implemented the Parent Education and Support Programme in two impoverished and under-resourced communities in Cape Town.
The Programme aims to empower parents to support their children’s school readiness. The Programme is run by an occupational therapist and a community-based worker, who present workshops to the parents of these children. These workshops educate parents on what is expected of their child developmentally and the milestones necessary for school readiness. Parents are taught how to use various aids and activities to support their children in developing and performing multiple life skills. The workshops also include information sessions on the following: clear speech, the nutritional requirements of children, health, sleep, emotional security, homework routines and the importance of play.
Rationale for employing occupational therapists
Occupational therapists are professional healthcare workers with extensive tertiary training in disabilities and appropriate rehabilitation practices. They provide specialist rehabilitative care to enable people with disabilities to become more independent. Community-based workers, while playing a vital role in care for people with disabilities, are not necessarily trained in rehabilitative care. Community-based workers’ primary role is to identify people with disabilities in the local community and to improve their access to care.
The WHO’s (2020) guidelines on community-based rehabilitation support NPOs in employing both occupational therapists and community-based workers for such interventions. This staffing arrangement has the added benefit of transferring skills from professionals to community-based workers, thereby continuously upskilling community-based workers (Orkin et al. 2021). This holds significant long-term benefits for the local community. Moreover, the care sector’s burden is redistributed to communities, alleviating the pressure on already overburdened clinics and other institutions.
The dilemma of high costs
Donors are inclined to favour output-driven programmes that yield measurable short-term results and have the potential to be scaled (Banks, Hulme & Edwards 2015; Perroni et al. 2019). Non-profit organisations, however, seek to address pressing community needs (Elbers & Arts 2011), which often require long-term strategies focused on transformation (Andrews 2014; Elbers & Arts 2011). The situation is exacerbated by donors earmarking funding for individual activities and often being unwilling to fund long-term operating costs (Castaneda, Garen & Thornton 2008; Elbers & Arts 2011; Perroni et al. 2019).
In an experimental study, Perroni et al. (2019) found that when donors were presented with various initiatives, they were less likely to contribute to those with comparatively higher fixed costs, even when these interventions followed more effective strategies. Such is the case with the care facility under study, which follows implementation strategies that have yielded high-quality, efficient and effective assistance to beneficiaries.
In these instances, NPOs face a predicament: they either realign their programme strategies to meet donor requirements or risk losing donor support (Banks et al. 2015). Researchers have documented that many NPOs concede to the pressure and opt to adjust programme design and activities to satisfy donors (Andrews 2014; Banks et al. 2015; Elbers & Arts 2011). However, one should question whether these realignments have detrimental effects on programme quality and effectiveness. Non-profit organisations are arguably effective because of grassroots orientation and strategies, and the ability to innovate and learn through first-hand experience of the issues they seek to address (Mansuri & Vijayendra 2013). Sadly, these are the very strategies that NPOs are often forced to abandon to meet donor requirements (Banks et al. 2014).
The potential donors of the care facility under study have argued that the costs of employing an occupational therapist to implement its intervention are too high and that the costs could be reduced using one implementer. The care facility needed more information before hastily adjusting its programme design. To help inform this important decision, we utilised the MCDA methodology.
Research method and design
The MCDA methodology deconstructs the Programme design and implementation decision into three constituent elements: decision goal, alternatives and criteria. Programme stakeholders (including donors) provided the data by making various judgements at different stages in the MCDA process. Once analysed and aggregated, these judgements identify the preferred course of action.
What is multi-criteria decision analysis?
Organisational decisions are based on gathering information, evaluating and assessing trade-offs (Hummel, Bridges & Ijzerman 2014; Marsh et al. 2016; Mu & Pereyra-Rojas 2017). Such decision-making becomes complex when: (1) there are multiple or conflicting alternatives to consider; (2) the information used to make decisions is incomplete or imperfect; and/or (3) multiple decision-makers are involved, and different perspectives and motives thus need to be considered (Dodgson et al. 2009; Dolan, 2000; Duffy 2021). Multi-criteria decision analysis is an umbrella term for frameworks for analysis that simplify such decision-making (Dodgson et al. 2009). It provides a formal methodological approach to high-level decision-making, ensuring that the decision, its options and the consequences are all adequately considered by multiple stakeholders (Marsh et al. 2016).
Multi-criteria decision analysis techniques deconstruct the decision into composite elements. For each modelled decision, alternatives (possible actions) are identified and criteria (standards) are set, based upon which decision-makers judge the alternative courses of action. These alternatives and criteria are judged relative to other elements and in terms of their importance in achieving the overall goal related to the decision (Mu & Pereyra-Rojas 2017). This enables decision-makers to identify an ideal course of action in response to the problem, evaluate the strengths and weaknesses of all possible courses and discuss the trade-offs of different options.
Multi-criteria decision analysis aids decision-making and explicitly justifies the decision – why it was chosen over alternatives (Dodgson et al. 2009; Marsh et al. 2016). Additionally, using MCDA frameworks helps to reduce bias and increases the probability of making appropriate decisions (Mu & Pereyra-Rojas 2017).
A brief history of multi-criteria decision analysis
With its roots in behavioural decision theory, MCDA was already used before the 1960s by consumer experts to assess consumer data and judgements (Fishburn & Lavalle 1999). In the years that followed, MCDA was used for preference modelling. Simulations of large-scale problems, decisions, decision-maker preferences, and evaluative procedures were analysed (Fishburn & Lavalle 1999). For example, in the 1970s, MCDA was used to make decisions about location, transportation and investment problems (Roy & Vanderpooten 1996).
Because of the structure of the problem being a crucial consideration in MCDA, research in the field in the late 1990s focused on how different analysts approached problem structuring and how this step influenced the subsequent analysis (Marttunen, Lienert & Belton 2017). Today, MCDA is widely used in technology decisions and risk–benefit analyses in healthcare, transport, combatting climate change and energy provision (Marsh et al. 2016). The method also facilitates decision-making regarding budget and resource allocations, and the choice to pursue one intervention over another (Marsh et al. 2016).
Justification for using multi-criteria decision analysis in the context of this study
There has been a noticeable move from the traditional cost-benefit analysis towards using MCDA in healthcare. Where cost-benefit analysis aims to express all costs and benefits of different interventions or policies in a single – usually monetary – unit, MCDA makes the trade-offs of each option explicit (Saarikoski et al. 2016). This method provides greater transparency than cost-benefit analyses with regard to which options were considered in making a decision, as well as the advantages and disadvantages of each. Perhaps more important is that MCDA makes the criteria used to make important decisions visible.
Stirling (2006) calls the lack of transparency of cost-benefit analysis its ‘black box’ and discusses why this is problematic. Although cost-benefit analyses could arguably include intangible elements (such as moral, ethical, or social considerations or benefits) in the analysis, the method was not designed to account for these factors, nor does it consider the various viewpoints of stakeholders regarding these intangible elements. Instead, the method’s output typically consists of one figure indicating which option is most cost-beneficial.
In this study a method to weigh the pros and cons of the alternatives (the occupational therapist and community-based worker) while accommodating multiple – and possibly conflicting – opinions about the best-suited alternative was needed. As such, the MCDA methodology was considered appropriate for addressing the current study’s problem.
The analytical hierarchical process multi-criteria decision analysis technique
Over the years, several schools of thought have emerged on multiple-criteria analysis techniques. These techniques vary substantially from field to field and also depend on the context of the decision. How the decision is deconstructed, how the elements are presented to stakeholders for judgement and the algorithm used to determine the elements’ importance depend on the method used. For this research, the evaluators used the AHP technique.
The AHP, developed by Saaty in the 1980s (Mu & Pereyra-Rojas 2017), is a mathematical decision-making theory. This method is highly suitable when a decision has two or more alternatives and multiple stakeholders weigh in on the decision-making process, in other words, when group decision-making is required (Mu & Pereyra-Rojas 2017). The AHP captures and quantifies each stakeholder’s comparison judgements at different stages of application of the process. The result is an assigned performance score for each stage, which, when aggregated, indicates which alternative is the overall preferred choice. Another advantage of using the AHP technique is that tangible criteria (e.g., costs) and intangible criteria (e.g., the abilities or knowledge of an implementer) can be incorporated into the methodology. This allows stakeholders to use various criteria to judge alternatives (Mu & Pereyra-Rojas 2017).
Data gathering and analysis
The following eight steps for AHP MCDA, modified from the work of Mu and Pereyra-Rojas (2017), were applied in the present study (Duffy 2021):
- Step 1: Identifying decision-makers
- Step 2: Constructing decision models
- Step 3: Eliciting decision-makers’ judgements
- Step 4: Establishing criteria weights
- Step 5: Establishing local priorities for alternatives
- Step 6: Aggregating judgements
- Step 7: Model synthesis
- Step 8: Sensitivity analysis
The following sections use the case study to demonstrate the technical procedure followed in each step to gather and analyse the data.
Step 1: Identifying decision-makers
The first step in AHP MCDA is identifying decision-makers who serve as data providers. There are no definitive guidelines for selecting decision-makers, but the literature suggests they must be affected by the decision (Marsh et al. 2016). In a programme evaluation context, these decision-makers could be individuals affected by programme design and implementation decisions, including primary or direct and secondary stakeholders. The authors decided on the following eligibility criteria to identify decision-makers: (1) persons with a solid association with or involvement in the Programme, (2) persons with intimate knowledge of the Programme, (3) persons with expert knowledge of similar programmes and (4) persons with considerable interest in the success of the Programme (Duffy 2021). The authors identified four stakeholder groups that met one or more eligibility criteria: Programme beneficiaries, Programme management, Programme donors and experts in disability and rehabilitation.
Final sample: Based on the eligibility criteria above, the sampling frame consisted of 16 current stakeholders of the Parent Education and Support Programme: 3 beneficiaries (parents), 8 management staff, 2 donors and 3 experts in disability and rehabilitation. Because of the small size of the sampling frame, invitations to participate in the research were sent via electronic mail to each of these stakeholders. The final sample comprised 6 data providers (a response rate of 38%). It is important to note that while the sample size is important in research, an advantage of the AHP is that it does not require a large sample (Dias & Ioannou 1996). The final sample of 6 was, therefore, adequate. A profile of the stakeholders is presented in Table 1.
| TABLE 1: Data providers (and response rate) for the multi-criteria decision analysis presented by stakeholder group. |
Step 2: Constructing decision models
The second step in AHP MCDA is organising the decision into three constituent elements: the decision’s goal, the alternatives and the criteria according to which these alternatives will be judged (Saaty 2001). These elements constitute a decision hierarchy or framework (Saaty 2001). The decision’s goal was: who is the preferred implementer of the Parent Education and Support Programme? The alternatives or options are the various courses of action that could be implemented to attain the decision’s goal. The decision had two alternatives: an occupational therapist and a community-based worker. Lastly, the criteria are the standards against which each alternative is judged. In other words, a decision-maker should consider the measures when comparing the two alternatives and judge which is more likely to achieve the decision goal. These criteria are a crucial part of any MCDA and are typically developed by the evaluator (Marsh et al. 2016; Mu & Pereyra-Rojas 2016).
For guidance on what to include as criteria for the Parent Education and Support Programme, MCDA and AHP literature was consulted. However no examples of MCDA using the AHP for disability and rehabilitation or parenting programmes were found. In the limited instances of its use that somewhat resembled the study, AHP was used for employee selection (Mojaheed et al. 2013; Zolfani et al. 2012). In these studies, the criteria included: knowledge of a product, education, relevant experience, ability to work as a team, ability to think strategically, computer skills, fluency in a given language and communication skills. Unfortunately, none of these criteria aligned with our decision goal and its alternatives. While this made the application of the AHP novel in our context, it also meant we needed to design a strategy to develop a list of criteria for the Programme. The authors decided on a three-pronged approach to developing the criteria.
Firstly, the Programme’s desired outcomes and objectives was used as criteria (identified in the Programme’s theory of change – see Figure 1).
Given that the implementer (either an occupational therapist or a community-based worker) would be responsible for achieving these outcomes, it made sense that implementers should be judged on their ability to contribute to attaining the programme outcomes. For example, a desired outcome for the Programme is that parents feel encouraged to support their child’s development. By using this outcome as a criterion, decision-makers needed to consider and judge which alternative (i.e., which type of implementer) they preferred for attaining this programme outcome. It is important to note that we conducted a programme theory evaluation for the care facility as part of the research and found that the outcomes and objectives of the intervention were aligned with similar disability programmes and were plausible.
Secondly, programme practicalities were reviewed and included in the analysis. Given that the decision was prompted by donor unwillingness to support high-cost programmes, the assumed cost of employing the implementer was incorporated as a criterion in the AHP.
Lastly, an expert on disability and rehabilitation at a tertiary education institution was consulted.1 They reviewed the proposed criteria and deemed them comprehensive.
The AHP components (the decision, criteria and alternatives) were then organised into a decision hierarchy. The hierarchy or framework depicts the relationships between the components. Figure 2 presents the AHP decision framework for the Parent Education and Support Programme.
 |
FIGURE 2: The decision framework for the programme. |
|
As shown in Figure 2, our decision goal was to identify the preferred implementer of the Parent Education and Support Programme. The two alternatives were a community worker and an occupational therapist (Duffy 2021). The criteria against which these two alternatives were judged were cost, designing programme activities, educating parents, preparing parents, supporting parents (programme outcomes) and an implementer’s qualification. This framework can be developed manually (on paper) or using software. Saaty’s Super Decisions software was used for this research to create the decision hierarchy and analyse the MCDA data. This decision was based on Super Decisions being one of the few free advanced software packages. Another consideration was that comprehensive manuals and user guides on conducting AHP using Super Decisions are available online (Dodgson et al. 2009; Duffy 2021; Mu & Pereyra-Rojas 2016; Saaty 2001).
Step 3: Eliciting judgements
Once the decision hierarchy had been created, the next step was to elicit judgements from the decision-makers. The AHP utilises pairwise comparisons in two phases. The first phase establishes which criteria are most important, and these should be weighted as such (Mu & Pereyra-Rojas 2016). Decision-makers are presented with two criteria at a time for this phase (Duffy 2021). They must choose which criterion they deem more important and the extent to which it is more important.
Surveys are used to elicit these judgements from decision-makers. Here, again, the researcher can choose to use paper-based or online questionnaires. A paid version of Qualtrics was used to create and disseminate our MCDA questionnaire to the stakeholders (Duffy 2021). Qualtrics was selected because the researchers were familiar with the software. A copy of our questionnaire can be found in our publicly accessible data repository.2
An important consideration is the response scale used in MCDA questionnaires, as it forms the basis for the AHP calculations. While other scales for pairwise comparisons have been proposed, Saaty’s (2012) ordinal scale is the most commonly used. The scale is designed to indicate the degree to which a criterion is more, less, or equally important than another. The scale consists of verbal statements and numeric values (Duffy 2021). These values start at nine (on the extreme left), representing a criterion, go down to one (middle-scale point) and then back up to nine (on the extreme right), representing the opposing criterion. The value of one means the criteria are equally important, whereas nine (on either end of the spectrum) indicates that the criterion is more important than the other. Table 2 portrays Saaty’s (2012) scale.
Figure 3 demonstrates an example of a pairwise comparison of criteria. In this example, the decision-maker was presented with two criteria: (1) an implementer’s ability to prepare parents to support children’s school readiness and (2) an implementer’s ability to design programme activities (Duffy 2021). The decision-maker had to decide whether these criteria were equally important or whether one was more important than the other. They then indicated their decision and to what extent on Saaty’s (2012) scale.
 |
FIGURE 3: An example of a pairwise comparison for rating criteria importance. |
|
Decision-makers are presented with several iterations of pairwise criteria, until each criterion has been rated (according to importance) against the other. The same sequencing was used for all stakeholders. This can be viewed in our questionnaire.
Once all the criteria had been paired, the second phase of pairwise comparison was performed to determine the preferred alternative (Mu & Pereyra-Rojas 2016). The decision-makers were presented with a pairwise comparison of alternatives and one criterion. They were asked to indicate which alternative they preferred concerning the single criterion and to what extent they preferred the alternative. An example of this pairwise comparison is presented in Figure 4. In this example, a decision-maker is shown the criterion: ‘Educate parents about children’s development’ and asked to indicate their preferred implementer.
 |
FIGURE 4: An example of a pairwise comparison for alternative preference. |
|
In this example, the decision-maker chooses which alternative best suits the criterion. Several pairwise comparisons of alternatives are presented until each decision-maker has indicated the preferred alternative for each criterion. It is important to note that the decision-makers may choose a neutral response; in other words, they may deem both implementers equally qualified and skilled to satisfy the criterion.
In some MCDA techniques, criteria and alternatives are measured on different scales. In contrast, the AHP technique standardises the judgements of the decision-makers by using the same scale for all comparisons of criteria and alternatives (Saaty 2001).
Once the judgements for the pairwise comparisons of the criteria and the alternatives had been elicited from each decision-maker, the data were collated in decision-making software with the functionality to conduct MCDA using the AHP, namely the Super Decisions program (Duffy 2021). All of the decision-makers had completed the entire questionnaire, and thus, there were no missing data. A copy of the data can be found in our publicly accessible data repository.3 Super Decisions captures each decision-maker’s MCDA judgements in a comparison matrix. An example of a comparison matrix from our case is provided in Table 3.
| TABLE 3: An example of a comparison matrix. |
The white cells illustrate the responses of the decision-makers on Saatys (2012) scale, that is, their judgements. The Super Decisions software automatically generates the values shown in the grey cells, indicating the mathematical inverse of the chosen value, thereby creating comparison matrices to analyse the judgements of each decision-maker.
Step 4: Establishing criteria weights
The Super Decisions software uses the comparison matrix to compute a decision-maker’s rated importance from each pairwise criteria comparison, producing a criterion’s overall weighting. The criterion the decision-maker ranked as most important will have the highest computed weighting.
Two indexes are provided when analysing the criteria weights: the standardised weighting and the ideal weighting. The standardised weighting statistic reflects the importance of a criterion in relation to all the other criteria (Duffy 2021; Mu & Pereyra-Rojas 2016). As such, the standardised weighting can be read as a percentage indicating the extent to which a criterion is essential. These weightings add up to a value of 1 (Duffy 2021; Hummel et al. 2014).
The idealised weighting statistic reflects which criterion is deemed the most important, regardless of how the other criterion performed (Saaty 2001). Thus, an ideal statistic of 1.0000 reflects that the decision-maker has identified a criterion as the most important (Duffy 2021). For any AHP MCDA result to be considered valid, the consistency ratio (CR) must be below 0.10 (Mu & Pereyra-Rojas 2016; Saaty 2001).
Step 5: Establishing local priorities for alternatives
The overall aim of Step 5 is to calculate the local priorities for the alternatives of each decision-maker. In other words, the software computes which alternative each decision-maker prefers. As in Step 4, the judgements of a decision-maker are calculated to obtain their individualised local priority output (Hummel et al. 2014; Saaty 2001).
Two indexes are computed for the local priorities. The standardised priority statistic (which can be read as a percentage) shows the extent to which the alternative is preferred when considering the other alternative(s); these indexes add up to a value of 1. An idealised priority statistic of 1.0000 indicates the preferred alternative regardless of how the other alternative performed (Saaty 2001).
Step 6: Aggregating judgements
At this stage in the process, the criteria weights and local priorities of the alternatives for each decision-maker have been established. Naturally, there would be differences in each decision-maker’s weightings and preferences if they held different opinions. However, we were interested in an overall conclusion regarding the preferred implementer. As such, the six decision-makers’ criteria weightings and alternative preferences needed to be aggregated. To account for possible outliers, the general rule is to take all the individual decision-makers’ results and calculate the geometric mean instead of the arithmetic mean (Mu & Pereyra-Rojas 2016).
Aggregating the individual judgements in the AHP using the geometric mean is necessary because the geometric mean preserves the reciprocal properties of the pairwise comparisons (Liu et al. 2017). Table 4 displays the aggregated weighted criteria, and Table 5 shows the aggregated local priorities from our AHP MCDA. Under the next few steps, we present the output tables from our MCDA results as illustrative of the process. Thus, for this article’s purposes, the results are somewhat intertwined with the results to aid understanding of the MCDA methodology.
| TABLE 4: Aggregated weighted criteria for the parent education and support programme. |
| TABLE 5: Aggregated local priorities for the parent education and support programme. |
Overall, the aggregated results in Table 4 showed that an implementer’s ability to provide developmental education (‘Educating parents on the expected development of pre-schoolers’) was considered the most important criterion, with a standardised weight of 0.2523 and an idealised weight of 1.0000 (Duffy 2021). The criterion weighted as least important was cost, with a standardised weight of 0.0554 and an idealised weighting of 0.2197 (Duffy 2021).
Step 7: Model synthesis
Step 7 is to synthesise the MCDA results by considering the weighting of each criterion and the preferred alternative linked to each. This results in an overall priority score for each alternative, which is the critical output of the MCDA.
In our case (refer to Table 6), community workers received a standardised overall priority of 0.5055 (50.6%), whereas occupational therapists received a standardised overall priority of 0.4945 (49.5%) (Duffy 2021). Idealised overall priorities for community workers and occupational therapists were 1.0000 and 0.9782, respectively (Duffy 2021).
| TABLE 6: Synthesised results (overall priority) for the parent education and support programme. |
Step 8: Sensitivity analysis
Although the MCDA decision was answered in Step 7, Step 8 is a sensitivity analysis. Mu and Pereyra-Rojas (2017:20) describe a sensitivity analysis as a ‘what-if analysis’. This analysis determines if the output from Step 7 would change if the criteria were weighted differently (Duffy 2021). This step is important because it determines the extent to which the weightings of the criteria influence the final overall priority of the MCDA (Duffy 2021). Thus, the sensitivity analysis answers the question: Would Step 7’s results hold if the criteria were weighted differently? (Duffy 2021; Saaty 2001). With the sensitivity analysis, MCDA users can establish how strongly (and to what degree) the chosen alternative is preferred.
One technique for performing a sensitivity analysis is to assign equal weightings to all criteria and then re-run the analyses to see how the results would differ (Mu & Pereyra-Rojas 2016). In these circumstances, each criterion’s local priority is adjusted by dividing it by the total number of criteria in a model (Mu & Pereyra-Rojas 2016). This simulates a scenario where all decision-makers assign the exact weighting for the criteria and have the same preferences regarding alternatives.
As shown in Table 7, our results changed when we performed a sensitivity analysis on our MCDA output. An occupational therapist (standard overall priority of 0.5184 [51.8%] and an ideal overall priority of 1.000) was preferred over a community worker (standard overall priority of 0.4816 [48.2%] and an ideal overall priority of 0.9289) (Duffy 2021).
| TABLE 7: Synthesised results (overall priority) for the parent education and support programme after sensitivity analysis. |
With the sensitivity analysis results, one can reflect on how different the results would be if all criteria had the same weighting. This also enhances confidence in the overall decision. In our case, the standard overall priorities for a community worker (50.6%) and an occupational therapist (49.5%) were comparable when we synthesised the results. Even though the sensitivity analysis depicted a change in the preferred implementer, the standard priorities did not change the overall picture. Both implementers held nearly 50% of the preference score in the synthesised and sensitivity analysis results. Having conducted the sensitivity analysis, we were confident that both implementers were equally important.
Ethical considerations
The authors’ institutional ethics committee granted ethical approval to conduct the study (Approval Reference: REC 2017/07/017). There were no risks associated with the research, and participation was voluntary. Consent to participate was obtained from each respondent, and anonymity was ensured.
Results
The Programme’s outcome criteria were ranked using MCDA. Linked to each criterion, the MCDA results established whether an occupational therapist or community-based worker was deemed best suited and the preferred implementer to achieve an outcome. Overall, the AHP MCDA enabled us to determine whether stakeholders prioritised the costs of employing an implementer or whether other considerations were more important. Each stakeholder’s preferences were assessed through multiple decision statements to conclude their preferences.
As shown in Table 5, overall, the decision-makers preferred a community worker for three programme criteria and an occupational therapist for the other three criteria. All stakeholders (including the donor) indicated cost as the least important consideration.
Overall, while a community-based worker was identified as the preferred implementer of the Parent Education and Support Programme (with an overall preference score of 50.5%), the results suggested that both a community-based worker and an occupational therapist are necessary. The preference for a community-based worker was linked to two of the four programme outcome criteria: ‘Parental development education’ and ‘Parental support’.
The community-based worker achieved a slightly higher overall preference score because of the two outcome criteria linked to the community-based worker being ranked as more important. An occupational therapist was preferred for the other outcome criteria: ‘Designing programme activities’ and ‘Preparing parents for their child’s school readiness’.
Discussion
Intuitively, these results make sense. If one assesses the outcome criteria for which an occupational therapist was preferred, these can be categorised as technical skills. Occupational therapists undergo extensive specialist training in disability care and rehabilitation and are equipped with technical competencies. For example, occupational therapists are trained to facilitate social integration by coaching individuals on how to act in different social settings. They are, therefore, better positioned than community-based workers to implement activities such as designing programme activities and determining content. The preferences for the community-based worker were based on psychosocial considerations. Psychosocial components included, but were not limited to, supporting parents and communicating essential health and educational information to parents.
Social science literature asserts that a communication barrier often exists between health professionals, such as occupational therapists, and communities in South Africa (Scheffler, Visagie & Schneider 2015). This is because most occupational therapists in South Africa speak English, whereas, for most South Africans, English is their second or third language (Scheffler et al. 2015). Community-based workers also tend to be members of their communities and are, therefore, more likely to share a home language with the beneficiaries. In this way, they can translate essential health and educational information to community members, making it more accessible. Being proficient in beneficiaries’ home language and having basic skills and knowledge of rehabilitation uniquely situates community-based workers to break communication barriers and share essential information. As community members, community-based workers are also likely to have pre-existing relationships with beneficiaries and their families and carers. Given their background, they are better positioned to deal with the psychosocial aspects of a programme because community members and beneficiaries will see them as relatable and approachable.
The almost 50/50 preference split between the two implementers suggests a justification for having both implementers and their areas of expertise in the care facility’s programme. The results also identify which outcomes would be affected if only one implementer were employed. The results of the AHP MCDA supported the care facility’s initial dual-implementer model, and they decided not to adjust the intervention’s design and implementation strategy.
Study limitations
The authors acknowledge that the number of programme staff and beneficiaries outnumbered the single donor who agreed to participate in our study. When these groups are unequally represented, aggregating the responses as part of the AHP could skew the results to represent the priorities and views of a particular stakeholder group. However, all six data providers’ criteria weightings and local priorities were closely aligned in our case. Thus, the results showed a view shared by all the stakeholder groups.
A survey was used to collect the preference judgements from all the stakeholders. Feedback from our pilot study indicated that respondents found the surveys time-consuming. More importantly, however, respondents also reported feeling pressured to select one alternative over another, even though the option of a neutral response was offered. Therefore, future researchers and practitioners who use Saaty’s (2012) scale must ensure that their respondents are repeatedly assured that they can indicate an equal preference for the two alternatives presented.
Finally, developing decision criteria is crucial to conducting MCDA (Marsh et al. 2016). It has been argued that criteria are the make-or-break component of any MCDA because they are the standards used to judge decision alternatives (Mu & Pereyra-Rojas 2016). Therefore, the criteria must be all-encompassing and include all areas of the programme. If there are significant gaps in the criteria, the findings drawn from the analysis may be inaccurate. Best-practice guidelines suggest developing criteria by adapting existing criteria used in previous MCDA studies (Marsh et al. 2016). Ideally, we would have developed criteria for our MCDA using previous programme evaluation research. However, given the absence of prior studies in this domain, we had to build an entirely new MCDA decision model.
Considering that the key question guiding our MCDA was the best-suited and preferred implementer, developing criteria based on programme outcomes and whether a particular implementer could help achieve these made sense. To ensure our criteria were appropriate, we did a quality-check phase that involved inviting disability experts to scrutinise and critique our decision models before we conducted the MCDA.
Conclusion
The authors undertook this evaluation because the care facility under study had difficulty securing funding, based partly on their decision to employ an occupational therapist and a community-based worker in its Parent Education and Support Programme. Feedback from potential donors suggested that the costs associated with having both implementers resulted in unjustifiable, high implementation costs. This feedback made the care facility consider adjusting its dual-implementer design of the Programme.
The authors customised the AHP MCDA methodology and applied it uniquely to this programme evaluation context to enable the care facility to decide which implementer would be preferred. The methodology involved multiple stakeholders, making judgements about their preferred implementer at various stages in the process.
The final model of the AHP MCDA suggested that both implementers were preferred for different programme components and supported the involvement of an occupational therapist and a community-based worker in the Programme. The results uncovered how different Programme outcomes were attributed to each implementer. Thus, if the care facility decided to forgo one of the implementers, the Programme would not achieve its intended results, and the quality of the intervention would be affected. This crucial finding led the care facility to decide not to adjust the design of the Programme.
Applying the AHP MCDA highlighted these implications and demonstrated how practical this methodology can be for programme design and implementation decisions. Overall, MCDA using the AHP showed promise in this joint evaluation and disability research, and the results were undoubtedly valuable to the care facility.
This article contributes new knowledge in several ways. Firstly, MCDA is not common in evaluation research. To our knowledge, this is the first time MCDA has been used to aid decision-making about programme design and implementation in the non-profit sector. Secondly, the use of MCDA in the disability care sector is also unique. Finally, by demonstrating the use of MCDA in this case study, this article aims to enhance the understanding of the innovative methodology among programme designers, evaluators, researchers and practitioners and highlight its potential to assist in programme decision-making.
Acknowledgements
Competing interests
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
Authors’ contributions
C.D. conceptualised the idea and supervised the data collection, analysis and interpretation, which was initially carried out by L.M. The lead writing, manuscript drafting, critical revisions and manuscript finalising were performed by C.D.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
The data sets are available online without restriction from: Duffy (2021). The application of multi-criteria decision analysis in a program evaluation context – MCDA Questionnaire (Version 1) [Data set]. University of Cape Town. https://doi.org/10.25375/uct.15073104.v1 and Duffy (2021). The application of multi-criteria decision analysis in a program evaluation context - Raw data from the collated questionnaire responses (Version 1) [Data set]. University of Cape Town. https://doi.org/10.25375/uct.15073122.v1.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors and the publisher.
References
Andrews, A., 2014, ‘Downward accountability in unequal alliances: Explaining NGO responses to Zapatista demands’, World Development 54, 99–113. https://doi.org/10.1016/j.worlddev.2013.07.009
Banks, N., Hulme, D. & Edwards, M., 2015, ‘NGOs, states, and donors revisited: Still too close for comfort?’, World Development 66, 707–718. https://doi.org/10.1016/j.worlddev.2014.09.028
Castaneda, M.A., Garen, J. & Thornton, J., 2008, ‘Competition, contractibility, and the market for donors to non-profits’, The Journal of Law, Economics, & Organization 24(1), 215–246. https://doi.org/10.1093/jleo/ewm036
Chappell, P. & Lorenzo, T., 2012, ‘Exploring capacity for disability-inclusive development’, in T. Lorenzo (ed.), Marrying community development and rehabilitation: Reality or aspiration for disabled people, Disability Innovations Africa, Cape Town.
Department of Social Development, 2016, White paper on the rights of persons with disabilities, Government Gazette No. 39792, pp. 4–197, viewed 07 June 2020, from http://www.gov.za/sites/www.gov.za/files/39792_gon230.pdf.
Dias Jr, A. & Ioannou, P.G., 1996, ‘Company and project evaluation model for privately promoted infrastructure projects’, Journal of Construction Engineering and Management 122(1), 71–82. https://doi.org/10.1061/(ASCE)0733-9364(1996)122:1(71)
Dodgson, J.S., Spackman, M., Pearman, A. & Phillips, L.D., 2009, Multi-criteria analysis: A manual, Department for Communities and Local Government, London.
Duffy, C.G., 2021, The application of multi-criteria decision analysis in a program evaluation context - MCDA Questionnaire, Version 1, Data set, University of Cape Town, viewed n.d., from https://doi.org/10.25375/uct.15073104.v1.
Elbers, W. & Arts, B., 2011, ‘Keeping body and soul together: Southern NGOs’ strategic responses to donor constraints’, International Review of Administrative Sciences 77(4), 713–732. https://doi.org/10.1177/0020852311419388
Fishburn, P.C. & Lavalle, I.H., 1999, ‘MCDA: Theory, practice and the future’, Journal of Multi-criteria Decision Analysis 8(1), 1–2. https://doi.org/10.1002/(SICI)1099-1360(199901)8:1<1::AID-MCDA205>3.0.CO;2-E
Hummel, J.M., Bridges, J.F. & Ijzerman, M.J., 2014, ‘Group decision making with the analytic hierarchy process in benefit-risk assessment: A tutorial’, The Patient – Patient-Centered Outcomes Research 7(2), 129–140. https://doi.org/10.1007/s40271-014-0050-7
Liu, F., Peng, Y., Zhang, W. & Pedrycz, W., 2017, ‘On consistency in AHP and fuzzy AHP’, Journal of Systems Science and Information 5(2), 128–147. https://doi.org/10.21078/JSSI-2017-128-20
Mansuri, G. & Vijayendra, R., 2013, Localizing development: Does participation work?, World Bank, Washington, DC.
Marsh, K., IJzerman, M., Thokala, P., Baltussen, R., Boysen, M., Kaló, Z. et al., 2016, ‘Multiple criteria decision analysis for health care decision making emerging good practices: Report 2 of the ISPOR MCDA emerging good practices task force’, Value Health 19(2), 125–137. https://doi.org/10.1016/j.jval.2015.12.016
Marttunen, M., Lienert, J. & Belton, V., 2017, ‘Structuring problems for multi-criteria decision analysis in practice: A literature review of method combinations’, European Journal of Operational Research 263(1), 1–17. https://doi.org/10.1016/j.ejor.2017.04.041
Minne, L., 2018, ‘A theory evaluation and programme implementer decision analysis for two therapy-driven programmes operating in the disability and rehabilitation sector’, Master’s thesis, University of Cape Town, viewed 07 June 2022, from https://open.uct.ac.za/handle/11427/29419.
Mojaheed, M., Marjani, M.E., Afshari, A.R. & Marjani, S., 2013, ‘Using ELECTRE-AHP as a mixed method for personnel selection’, in Proceedings of the 12th International Symposium on the analytic hierarchy process, Kuala Lumpur, Malaysia, Creative Decision Foundation, Pittsburgh, July 01, 2013, pp. 23–26.
Mu, E. & Pereyra-Rojas, M., 2016, Practical decision making: An introduction to the analytic hierarchy process (AHP) using super decisions, vol. 2, Springer, New York, NY.
Orkin, A.M., Rao, S., Venugopal, J., Kithulegoda, N., Wegier, P., Ritchie, S.D. et al., 2021, ‘Conceptual framework for task shifting and task sharing: An international Delphi study’, Human Resources For Health 19, 61. https://doi.org/10.1186/s12960-021-00605-z
Perroni, C., Pogrebna, G., Sandford, S. & Scharf, K., 2019, ‘Are donors afraid of core costs? Economies of scale and contestability in charity markets’, The Economic Journal 129(622), 2608–2636. https://doi.org/10.1093/ej/uez006
Roy, B. & Vanderpooten, D., 1996, ‘The European school of MCDA: Emergence, basic features and current works’, Journal of Multi-criteria Decision Analysis 5(1), 22–38. https://doi.org/10.1002/(SICI)1099-1360(199603)5:1<22::AID-MCDA93>3.0.CO;2-F
Saarikoski, H., Mustajoki, J., Barton, D.N., Geneletti, D., Langemeyer, J., Gomez-Baggethun, E. et al., 2016, ‘Multi-criteria decision analysis and cost–benefit analysis: Comparing alternative frameworks for integrated valuation of ecosystem services’, Ecosystem Services 22, 238–249. https://doi.org/10.1016/j.ecoser.2016.10.014
Saaty, T.L., 2001, Decision making with dependence and feedback: The analytic network process: The organization and prioritization of complexity, RWS Publications, Pittsburgh, PA.
Saaty, T.L., 2012, Decision making for leaders: The analytic hierarchy process for decisions in a complex world, 3rd rev. edn., RWS Publications, Pittsburgh, PA.
Scheffler, E., Visagie, S. & Schneider, M., 2015, ‘The impact of health service variables on healthcare access in a low resourced urban setting in the Western Cape, South Africa’, African Journal of Primary Health Care & Family Medicine 7(1), 1–11. https://doi.org/10.4102/phcfm.v7i1.820
Sherry, K., 2014, ‘Disability and rehabilitation: Essential considerations for equitable, accessible and poverty-reducing health care in South Africa’, South African Health Review 2014(1), 89–99, viewed 13 June 2022, from http://www.health-e.org.za/wp-content/uploads/2015/10/HST-SAHR-2014-15-Complete.pdf.
Stirling, A., 2006, ‘Analysis, participation and power: Justification and closure in participatory multi-criteria analysis’, Land Use Policy 23(1), 95–107. https://doi.org/10.1016/j.landusepol.2004.08.010
World Health Organization (WHO), 2002, Towards a common language for functioning, disability, and health: ICF, World Health Organization, Geneva, viewed 08 June 2020, from https://www.who.int/classifications/icf/icfbeginnersguide.pdf?ua=1.
Zolfani, S.H., Rezaeiniya, N., Aghdaie, M.H. & Zavadskas, E.K., 2012, ‘Quality control manager selection based on AHP-COPRAS-G methods: A case in Iran’, Economic Research-Ekonomska Istraživanja 25(1), 72–86. https://doi.org/10.1080/1331677X.2012.1151749
Footnotes
1. University of Cape Town.
2. Available after the review because of the double-blind process.
3. Available after the review, because of the double-blind process.
| 
Hide
Show all
Email this article (Login required)
