An evaluation of a transport subsidisation programme in the Western Cape, South Africa, was undertaken to capture the outcomes and benefits of the service from the perspective of the bus user.
The objective of this article was to compare the findings from the adopted parallel mixed-methods design that included a perception survey and the Most Significant Change (MSC) technique. The article presented the advantages of each approach and reflected on the benefits and challenges in applying the MSC technique.
Data on the intended outcomes of the Provincial Transport Operations Grant programme were collected from 458 commuters on four bus routes through a structured close-ended questionnaire. The MSC technique was applied to collect 69 stories of change that captured changes regarded as most significant by beneficiaries of the public transport subsidy.
It was found that the survey better captured the intended and predetermined objectives of the programme, while the large response group allowed for comparisons and cross-tabulations. The MSC interviews better captured the real-life experience of participants and identified the benefits most valued by commuters, including outcomes not specifically anticipated by the programme. It also served to clarify contradicting responses or ratings on the closed-ended questionnaire and informed further bivariate analysis of the structured questionnaire data.
To maximise benefits from the MSC technique, sufficient time is needed to solicit value responses from respondents, while decision-makers should allow time for multiple iterations and discussions at different levels of the hierarchy.
Pre-1994 spatial development policies that separated racial groups into geographically segregated neighbourhoods left South Africa with sprawling cities that increase the cost of public service delivery. Low-density areas in cities make transport networks extremely inefficient, resulting in high transport costs and long commuting hours, according to the Government Technical Advisory Centre (Republic of South Africa
The Western Cape Department of Transport and Public Works (DTPW) ‘is responsible for the management and administration of road-based subsidised public transport and the associated PTOG allocation’ (Western Cape Government
From December 2016 to March 2017, an evaluation of the PTOG programme was undertaken on selected routes to capture the outcomes and benefits of the bus service from a bus user’s perspective. The grant and change theory of the programme identify the intended outcomes of the programme as the increased reliability, accessibility, safety and affordability of public transport to increase mobility of GABS commuters.
The objectives of this article are to compare the data and findings of the two data collection instruments that formed part of the parallel mixed-methods design; to compare the data and findings offered by the two respective instruments on the benefits and outcomes for beneficiary of the Provincial Transport Operations Grant in the Western Cape, South Africa; and to offer recommendations on the application of the Most Significant Change (MSC) Technique in evaluation outcome designs.
The MSC technique is a form of participatory monitoring and evaluation (M&E) that relies on collecting qualitative change stories directly from programme beneficiaries and the assessment of the impact of reported change in terms of the future focus of the programme (Dart
As the primary purpose of MSC is programme improvement, it aligns well with the framework of the purpose of evaluation findings (Patton 1997, as cited in Dart & Davies
Wilson (
Most Significant Change searches for significant outcomes through an inductive process and tends to generate mainly positive information (Dart
There is an increasing body of research that presents the lessons learnt from the use of MSC (Choy & Lidstone
According to Wilder and Walpole (
data lack the contextual information that helps clarify the causal link between an observed change and the project activities, overlook unanticipated changes, and fail to unearth any flaws in a conceptual model or logical framework. (Whitehouse, as cited in Wilder & Walpole
While indicator-based monitoring tends to reduce complex organisational, social and economic developments to single numbers (Davies & Dart
The study adopted a parallel mixed-methods design to capture the perceptions and viewpoints of commuters benefiting from the travel operations grant through a commuter perception survey using a structured questionnaire and the MSC technique using an interview guide. Questions in the survey capture responses that reflected purpose and the intended outcomes of the grant, while the MSC interviews solicited the respondents’ viewpoint on the changes most valued, including intended and non-intentional changes caused by the programme. The study targeted four purposively selected bus routes: Cape Town to Durbanville, Bellville to Malmesbury, Bellville to Chatsworth and Cape Town to Khayelitsha. Respondents were selected to reflect various age groups and occupations. On average, the Durbanville route captured viewpoints from a higher income market segment, while the Khayelitsha route captured a lower income segment. The Malmesbury and Chatsworth lines were recently introduced by GABS and their inclusion provided an opportunity to capture bus users’ perceptions on their recent transport mode change.
The selection of respondents was based on a non-probability sampling, using a combination of accidental and quota sampling methods. Sampling was accidental based on bus users at a particular platform at the time of data collection that varied in times and dates. A quota of 500 interviews were targeted and achieved, but later data analysis excluded the responses from commuters who do not travel the full bus route to the end destination as this would skew the analysis on, for example, travel times, financial affordability and adherence to bus schedules.
Data were collected through two research instruments. The first was a structured questionnaire, administered via face-to-face interviews conducted in English, Afrikaans or isiXhosa that captured demographic details of respondents, as well as the perception of commuters on Likert scale closed-ended questions that related to the change areas of direct interest to the programme. A total of 529 commuters completed the questionnaire, but only responses from the 458 travelling on the targeted bus routes were considered in the analysis. Responses from commuters who do not travel the full route (from the selected start to the selected end destination) were omitted from the analysis.
The second instrument was an MSC interview guide, administered via individual interviews conducted in English that explored the impacts and benefits from the commuter perspective through four open-ended questions that captured any changes they observed in their own lives, or those of fellow passengers, as a result of the bus route, and why they regarded those changes as significant. The intended programme outcomes were not referred to in these questions. Stories were collected from commuters while waiting at the bus stop or travelling with the bus en route to the destination. The selection of commuters was based on availability (thus, accidental sampling of those who happened to be at the stop or on the bus). This was done when bus occupancy levels allowed it. In total, 69 stories of change (MSC interviews) were collected across the four targeted bus routes. Each respondent’s story was individually captured with verbatim quotes from the audio recordings.
IBM SPSS Statistics was used to analyse the data collected through the structured questionnaire to present descriptive statistics and bivariate correlational statistics. The process to analyse the MSC interview data started with the identification of an initial set of change domains as identified by the PTOG manager and the M&E unit. The analysis of the stories of change collected from commuters revealed a wide variety of social and economic changes regarded as significant by commuters and not catered for in the purpose of the grant. An analysis of the 69 responses rendered nine change domains, presented in the list below, with an indication of the number of responses that related to each domain
The MSC technique advocates for multiple levels of story selection accompanied by an explicit value clarification process explaining why a particular change story is regarded as significant and important. Shah (
[
However, in some cases selection only occurs once.
The DTPW M&E unit formed the first selection level in the selection process. Monitoring and evaluation unit members conducted an in-depth analysis of each story followed by a discussion to identify the change domains reflected. Stories were regarded as most significant when they represented a diverse list of domains of change, representing the more comprehensive socio-economic change that the department was pursuing through the PTOG programme. Preference was given to those stories that reflected changes aligned to the outcome statement as in the grant framework that specifically focused on affordability and improving the affordability of the bus service for the user. To capture different socio-economic conditions of commuters, care was taken to include change stories from each route. The team filtered the stories per route and identified the MSC stories for each route (Chatsworth, Durbanville, Khayelitsha and Malmesbury) with more emphasis on stories from the recently introduced Chatsworth and Malmesbury routes and stories that reflected on a modal shift from taxis, cars or trains to the bus. A final round of assessment revisited the initial set of stories and selected the story that best represented each change domain to be added to the list of selected stories. This process rendered a final list of 20 change stories.
The shortlisted 20 change stories were sent to the second selection level, namely the programme staff responsible for the administration of the provisional grants. The second level was asked to select the MSC story for each of the nine change domains and to provide a short motivation for the choice made. The final nine selected stories were to be presented to the senior management team of the DTPW to provide strategic direction for subsequent designs of the programme. Unfortunately, the last two steps were not executed in full and the MSC hierarchical value clarification process did not achieve full benefit.
Permission for the completion of the research ‘Benefits of transport subsidisation: Comparing findings from a customer perception survey and Most Significant Change Technique interviews’ was obtained from the REC: Humanities, Stellenbosch University. Project number: SPLPAD-2019-10420. Project Title: Benefits of transport subsidisation: Comparing data from different data collection methods.
In comparing the data obtained through the structured questionnaire versus the MSC interviews, the structured questionnaire was better suited to explore the intended benefits of the programme against predefined programme objectives, as well as comparisons between routes. The structured questionnaire ensured that data for change domains identified as strategic by the programme were collected for quarterly and annual reporting to the department. However, the structured questionnaire failed to provide insight into unintended positive (or negative) consequences of the programme. The MSC interviews allowed for the identification of new change domains that were valued by the participants.
For example, in terms of travel time, findings from the structured questionnaire present the average time spent on the bus as follows:
A total of 32% of respondents indicated that they spend one hour on the bus, 19% indicated that they spend one hour and thirty minutes travelling on the bus, with 14% indicating that they spend two hours on the bus. In comparing the routes in terms of travel time spent on the bus, most respondents indicated that they spend an hour on the bus, with the Malmesbury route showing the longest range in travelling time. (Western Cape Government
While the structured questionnaire did not capture the implications of reported data from a commuter perspective, the stories of change were useful in explaining the changes in the lives of the beneficiary, as reflected by the following responses of MSC interviews:
‘I come home almost two hours earlier at night, and I can leave home an hour later in the mornings’. [
‘I can be at work and home on time. Before I never used to be at work on time.’ (Respondent 16, female, Malmesbury route)
‘[
The structured Likert style also allowed for on-average comparisons across lines and cross-tabulation comparisons between variables. For example, it was possible to cross-tabulate the respondents’ perceived affordability of the service to their self-reported household income:
When analysed per route, the routes that are regarded as sometimes not affordable and very unaffordable are Chatsworth (53%), Khayelitsha (45%) and Malmesbury (41%). An overwhelming majority of the Durbanville route respondents found the bus either very affordable or mostly affordable (91%). (Western Cape Government
A comparison of data affordability and gross household income across the different bus routes reveals … that lower income bracket groups on average tend to regard the bus service as less affordable. With the exception of the Durbanville route, the average household income across the routes is between R2000 to R9000. (Western Cape Government
It is interesting that on the Chatsworth and Malmesbury routes, the lowest household income bracket group indicated that the bus service is mostly affordable. (Western Cape Government
In the latter case, a limitation of the structured questionnaire was that it did not allow for further comments that may have enhanced understanding of conflicting responses or ratings received. The stories of change were valuable in filling this gap. In trying to understand why Chatsworth respondents found the bus service affordable, despite low reported household income, the stories offered the following:
‘…the bus…is cheaper than taking a taxi…’ (Respondent 32, female, Chatsworth route)
‘… the bus is more affordable than making use of his own car and the related fuel cost…’ (Respondent 7, male, Chatsworth route)
This explains why commuters perceived the bus as very affordable, though the percentage of household income spent on transport was still well above the national policy target.
The stories of change were also useful for understanding the interconnectedness between changes experienced from the end-user perspective, even when these did not offer significant correlations in the sub-variate analysis. The following change story serves well to indicate the interdependence between change variables:
The most important change for her [
‘Affordability definitely, it is very expensive for me to travel in and out with my car because it is too far. It cost me a third of the price of travelling with my own car per day.’ (Respondent 6, female, Malmesbury route)
Furthermore, she explains, ‘It is more
The initial change domains identified as important to the programme and that informed the design of the structured questionnaire were accessibility, affordability, mobility, reliability and socio-economic changes. The analysis of the collected stories of change served to confirm the realisation of some of these changes (reliability, affordability and accessibility), while highlighting the change results regarded as valuable by the commuter (safety, reduced travel time, mobility, convenience, comfort and social networks). Although the stories collected were usually only a few sentences long, they were useful in identifying the most important benefits for participants and for constructing short narratives reflecting the impact of the bus service on users. Understanding what is important to the beneficiary is useful for realigning the programme to maximise these benefits.
Both the structured questionnaire and the MSC interviews offered valuable insights on the perceptions of participants. While extensive training was required to ensure consistent administering of the structured questionnaire by the team of data collectors, less training was required to administer the MSC interviews. However, the apparent simplicity of the MSC technique may be deceiving. The capturing of data for the MSC interviews was, however, found to be very resource-intensive to collect and capture, further complicated by the fact that most interviews took place while standing in a moving bus. Respondents were quick to provide the changes they observed in their lives, but struggled to unpack the more personal reasons
While respondents find it easy to report on the changes they experience, soliciting meaningful value reflections on the significance of these changes requires more skilled interviewers. Careful structuring of the MSC interview guide and training of the data collection team are essential for the value clarification process. When conducting interviews in the respondents’ second language, the sentence structure should be simple and interviewers should have clear guidelines on soliciting in-depth, value-laden information without leading the respondents.
As discussed in the methodology, only one level of hierarchical selection of stories took place, while the MSC technique calls for multiple levels of value clarification. While the first round of analysis only focused on multi-faceted stories, the M&E unit adopted a more value-driven approach in the second round of selection. Three domains –
An opportunity to realign the programme with new change domains regarded as significant by the participants was also seemingly lost. For example, an improvement in perceived safety emerged as a valued change domain from commuters who recently changed from train to bus transport. A few change stories referred to the prevalence of criminality either on the train or at the train stops. In an emotive story, a respondent remarked:
‘The bus service has made me value life because if it had not been for the bus services I would have lost my life. There was a time we experienced train surfing which killed some of my friends but luckily I never died.’ (Respondent 49, male, Khayelitsha route)
He recalled that, on the trains, older commuters used to push younger ones to their deaths, but on buses this did not occur. The respondent attributed this to a sense of unity and ubuntu amongst bus commuters.
While this experience demonstrates the significance of improved safety offered by bus services, this story was excluded from the shortlisted set of 20 stories because it offered a negative perspective on another programme run by the same department. As the MSC technique does not offer specific guidelines on how to deal with negative change stories, powerful stories may be rejected in favour of more positive stories, thereby limiting programme decision-makers’ potential to identify and address programme limitations.
In this evaluation, participants’ insights derived through the MSC interviews complemented data collected against predefined programme objectives. Applying multiple methods offered richer understanding of the impact of the programme.
The structured questionnaire offered the following key advantages:
enabled responses to intended and predetermined objectives of the programme
enabled comparisons and cross-tabulations between questions and responses
a larger sample can be efficiently managed, enabling generalisations (if proper sampling strategy is applied).
The MSC interviews complemented these findings by offering:
short narratives on the real-life experience of individual participants, rather than the average experience of all respondents
the most important and valuable benefits for participants, not limited by the intended and predefined objectives
comments to clarify responses or ratings on the questionnaire
connections between change domains for further bivariate analysis
new change domains valued by the participants.
The MSC technique is not suitable for quick evaluations (Serrat
The authors acknowledge the support from the Department of Transport and Public Works who commissioned the evaluation study and offered access to the relevant information. We also acknowledge the data collection assistance by the following 2017 SPL HonsBCom students: L.A. Brown, C. Cedras. N. Daries, R. Muchena, N. M Mvundlela, N. Mwala, L.T. Ngoma, E. Sitler, J. Von Bardeleben, D. Vraagom.
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
This article is based on a joint research project by the listed authors. Background, instrument design, discussion and conclusions are equally attributed to both authors.
The authors acknowledge the financial support from the Department of Transport and Public Works who commissioned the evaluation study and offered access to the relevant information. The authors commend the organisation for their continued commitment to critically evaluate their programmes, and also acknowledge the logistical support from the Cape Higher Education Consortium.
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Views expressed in this article are those of the authors and not the official position of Stellenbosch University, the Cape Higher Education Consortium or the Department of Transport and Public Works.
Many stories responded to more than one change domain targeted by the transport grant. The response count is therefore higher than the number of stories collected.