Original Research

Benefits of transport subsidisation: Comparing findings from a customer perception survey and Most Significant Change Technique interviews

Babette Rabie, Adéle Burger

Received: 31 Jan. 2019; Accepted: 02 May 2019; Published: 20 June 2019

Copyright: © 2019. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: An evaluation of a transport subsidisation programme in the Western Cape, South Africa, was undertaken to capture the outcomes and benefits of the service from the perspective of the bus user.

Objectives: The objective of this article was to compare the findings from the adopted parallel mixed-methods design that included a perception survey and the Most Significant Change (MSC) technique. The article presented the advantages of each approach and reflected on the benefits and challenges in applying the MSC technique.

Method: Data on the intended outcomes of the Provincial Transport Operations Grant programme were collected from 458 commuters on four bus routes through a structured close-ended questionnaire. The MSC technique was applied to collect 69 stories of change that captured changes regarded as most significant by beneficiaries of the public transport subsidy.

Results: It was found that the survey better captured the intended and predetermined objectives of the programme, while the large response group allowed for comparisons and cross-tabulations. The MSC interviews better captured the real-life experience of participants and identified the benefits most valued by commuters, including outcomes not specifically anticipated by the programme. It also served to clarify contradicting responses or ratings on the closed-ended questionnaire and informed further bivariate analysis of the structured questionnaire data.

Conclusion: To maximise benefits from the MSC technique, sufficient time is needed to solicit value responses from respondents, while decision-makers should allow time for multiple iterations and discussions at different levels of the hierarchy.

Keywords: mixed-methods design; outcome evaluation; value clarification; methodology reflections; public transport programme.

Background

Pre-1994 spatial development policies that separated racial groups into geographically segregated neighbourhoods left South Africa with sprawling cities that increase the cost of public service delivery. Low-density areas in cities make transport networks extremely inefficient, resulting in high transport costs and long commuting hours, according to the Government Technical Advisory Centre (Republic of South Africa 2013:1), which has determined that ‘[t]he higher costs, combined with affordability constraints, mean that passenger fares tend to cover a smaller proportion of the operating costs of public transport in South African cities’. To cover operating cost, government spent nearly R140 billion on public transport subsidies in major cities between 2012/2013 and 2016/2017 (Republic of South Africa 2013). Subsidisation is mostly through the conditional Public Transport Operations Grant (PTOG) that ‘subsidise public transport services in poor communities thus making these services accessible and affordable’ (Republic of South Africa 2017:112).

The Western Cape Department of Transport and Public Works (DTPW) ‘is responsible for the management and administration of road-based subsidised public transport and the associated PTOG allocation’ (Western Cape Government 2017a:52). Department of Transport and Public Works applies the PTOG to subsidise bus services in the City of Cape Town through financial support to Golden Arrow Bus Services (GABS) (Western Cape Government 2017a). Golden Arrow Bus Services serves 220 000 passengers per weekday across 1300 routes in the area (GABS 2018). Approximately 1.3 million passenger trips were subsidised during the 2016/2017 financial year for accessing work, education and other services (Western Cape Government 2017a).

From December 2016 to March 2017, an evaluation of the PTOG programme was undertaken on selected routes to capture the outcomes and benefits of the bus service from a bus user’s perspective. The grant and change theory of the programme identify the intended outcomes of the programme as the increased reliability, accessibility, safety and affordability of public transport to increase mobility of GABS commuters.

The objectives of this article are to compare the data and findings of the two data collection instruments that formed part of the parallel mixed-methods design; to compare the data and findings offered by the two respective instruments on the benefits and outcomes for beneficiary of the Provincial Transport Operations Grant in the Western Cape, South Africa; and to offer recommendations on the application of the Most Significant Change (MSC) Technique in evaluation outcome designs.

The Most Significant Change Technique

The MSC technique is a form of participatory monitoring and evaluation (M&E) that relies on collecting qualitative change stories directly from programme beneficiaries and the assessment of the impact of reported change in terms of the future focus of the programme (Dart 2005:262; Davies & Dart 2005). According Dart and Davies (2003:138), MSC has been referred to as ‘the evolutionary approach to organisational learning’, ‘the story approach’ and ‘monitoring without indicators’.

As the primary purpose of MSC is programme improvement, it aligns well with the framework of the purpose of evaluation findings (Patton 1997, as cited in Dart & Davies 2003:139) and highlights identifying and clarifying the outcomes of programmes and why these programmes are valuable (Dart 2005:22). The method concentrates on anecdotal evidence of change that is often missed by conventional quantitative monitoring techniques (Wilder & Walpole 2008). Most Significant Change adds value to measuring qualitative change indicators that are difficult to measure (Connors et al. 2017; Kloosterman, Benning & Fyles 2012).

Wilson (2014) considered MSC well-suited for public sector programmes that are continuous in nature, where outcomes may vary significantly between beneficiaries or where there may not be prior agreement between stakeholders on which outcomes are most important to the programme. The participatory nature of the method renders it suitable to community-driven programmes (Ho et al. 2015).

Most Significant Change searches for significant outcomes through an inductive process and tends to generate mainly positive information (Dart 2005:262). The process starts by inviting stakeholders to identify a set of broad, loosely defined change areas informed, but not restricted, by the programme objectives and open to individual interpretation by the respondents (Dart 2005:262–263; Davies & Dart 2005; Wilder & Walpole 2008). Secondly, stories providing factual descriptions of observations (Wilder & Walpole 2008) are collected from the programme beneficiaries through simple, non-leading questions that allow respondents to reflect on their experience and the benefits gained (Dart 2005:263). Thirdly, respondents are asked to assign their story to one or more change dimensions (Davies & Dart 2005; Limato et al. 2018) and to state why the change was significant to them (Davies & Dart 2005). Fourthly, captured stories are discussed at various levels of authority in the organisation responsible for the programme (Dart 2005:263; Davies & Dart 2005; Wilder & Walpole 2008). At each level, stories are sifted to identify the single most significant account of change in each domain (Davies & Dart 2005). Participants are again asked to explain their reasons for selecting particular change stories. This identifies the values most important to various decision-makers, and enables the organisation to crystallise its desired results to inform the future-focused activities (Dart 2005:263; Davies & Dart 2005). This encourages both upward and downward accountability (Wilder & Walpole 2008). Finally, information is fed back to the programme implementers (Dart 2005:263; Limato et al. 2018).

There is an increasing body of research that presents the lessons learnt from the use of MSC (Choy & Lidstone 2013; Hall 2014; Kraft & Prytherch 2016; Wilder & Walpole 2008; Willetts & Crawford 2007; Wrigley 2006).

According to Wilder and Walpole (2008:529), MSC offers ‘value as a monitoring tool to… improve project adaptive management and responsiveness’, as indicator:

data lack the contextual information that helps clarify the causal link between an observed change and the project activities, overlook unanticipated changes, and fail to unearth any flaws in a conceptual model or logical framework. (Whitehouse, as cited in Wilder & Walpole 2008:529)

While indicator-based monitoring tends to reduce complex organisational, social and economic developments to single numbers (Davies & Dart 2005), MSC offers a rich picture of what is happening (Davies & Dart 2005; Willetts & Crawford 2007) and provides a deeper understanding of why the change happened (Wilder & Walpole 2008) and why it is regarded as significant. Choy and Lidstone (2013) found that the stories deliver a richer picture of the immediate impact of the programme, while Limato et al. (2018) found that it not only offered beneficiaries the opportunity to express how they experienced change, but also revealed the different experiences of beneficiaries. Unrestricted by the preconceptions of desired outcomes, the ability of the MSC technique to uncover unintended changes makes it a valuable addition to conventional M&E methods (Davies & Dart 2005; Kraft & Prytherch 2016; Willetts & Crawford 2007). As the MSC process requires participants to analyse outcomes critically, it clarifies (Davies & Dart 2005) or enlightens individual and shared value systems (Choy & Lidstone 2013), which may help managers to understand their response to the needs of beneficiaries (Wilder & Walpole 2008) and promote adaptive management (Dart & Davies cited in Wilder & Walpole 2008; Kraft & Prytherch 2016).

Methods

The study adopted a parallel mixed-methods design to capture the perceptions and viewpoints of commuters benefiting from the travel operations grant through a commuter perception survey using a structured questionnaire and the MSC technique using an interview guide. Questions in the survey capture responses that reflected purpose and the intended outcomes of the grant, while the MSC interviews solicited the respondents’ viewpoint on the changes most valued, including intended and non-intentional changes caused by the programme. The study targeted four purposively selected bus routes: Cape Town to Durbanville, Bellville to Malmesbury, Bellville to Chatsworth and Cape Town to Khayelitsha. Respondents were selected to reflect various age groups and occupations. On average, the Durbanville route captured viewpoints from a higher income market segment, while the Khayelitsha route captured a lower income segment. The Malmesbury and Chatsworth lines were recently introduced by GABS and their inclusion provided an opportunity to capture bus users’ perceptions on their recent transport mode change.

The selection of respondents was based on a non-probability sampling, using a combination of accidental and quota sampling methods. Sampling was accidental based on bus users at a particular platform at the time of data collection that varied in times and dates. A quota of 500 interviews were targeted and achieved, but later data analysis excluded the responses from commuters who do not travel the full bus route to the end destination as this would skew the analysis on, for example, travel times, financial affordability and adherence to bus schedules.

Data collection instruments

Data were collected through two research instruments. The first was a structured questionnaire, administered via face-to-face interviews conducted in English, Afrikaans or isiXhosa that captured demographic details of respondents, as well as the perception of commuters on Likert scale closed-ended questions that related to the change areas of direct interest to the programme. A total of 529 commuters completed the questionnaire, but only responses from the 458 travelling on the targeted bus routes were considered in the analysis. Responses from commuters who do not travel the full route (from the selected start to the selected end destination) were omitted from the analysis.

The second instrument was an MSC interview guide, administered via individual interviews conducted in English that explored the impacts and benefits from the commuter perspective through four open-ended questions that captured any changes they observed in their own lives, or those of fellow passengers, as a result of the bus route, and why they regarded those changes as significant. The intended programme outcomes were not referred to in these questions. Stories were collected from commuters while waiting at the bus stop or travelling with the bus en route to the destination. The selection of commuters was based on availability (thus, accidental sampling of those who happened to be at the stop or on the bus). This was done when bus occupancy levels allowed it. In total, 69 stories of change (MSC interviews) were collected across the four targeted bus routes. Each respondent’s story was individually captured with verbatim quotes from the audio recordings.

Analysis

IBM SPSS Statistics was used to analyse the data collected through the structured questionnaire to present descriptive statistics and bivariate correlational statistics. The process to analyse the MSC interview data started with the identification of an initial set of change domains as identified by the PTOG manager and the M&E unit. The analysis of the stories of change collected from commuters revealed a wide variety of social and economic changes regarded as significant by commuters and not catered for in the purpose of the grant. An analysis of the 69 responses rendered nine change domains, presented in the list below, with an indication of the number of responses that related to each domain¹:

• Reliability (37 responses), reflecting commuters’ perception of the bus service departing and arriving as scheduled.
• Affordability (32 responses), reflecting the relative affordability of the bus to similar transportation modes such as a mini-bus taxi, own transport or multi-modal service.
• Safety (21 responses), reflecting commuters’ perception of personal safety while on board the bus and at bus stops, in comparison to other transport modes.
• Reduced travel time (17 responses), thereby freeing up time for other work and personal responsibilities.
• Accessibility (13 responses), measured in terms of the proximity of the departure and destination bus stop to the residence and destination of the commuter.
• Mobility (12 responses) because of the availability of alternative modes of transport.
• Convenience (11 responses), reflecting commuters’ perception of the degree of travel stress experienced in comparison to other transport options, for example, own transport, multi-modal transport and connections.
• Comfort (10 responses), measured in terms of the availability of seating on the bus and ease of the journey.
• Social networks (6 responses), focused on the social engagement with other commuters during the journey or while waiting at the bus stop.

Story selection

The MSC technique advocates for multiple levels of story selection accompanied by an explicit value clarification process explaining why a particular change story is regarded as significant and important. Shah (2014) argues that:

[T]he systemic process of selection and retention of stories that occurs through these [multiple levels] helps to increase validity by ensuring that the narrations hold relevance and importance not only for the storytellers, but also for a broader group of stakeholders…. (p. 265)

However, in some cases selection only occurs once.

The DTPW M&E unit formed the first selection level in the selection process. Monitoring and evaluation unit members conducted an in-depth analysis of each story followed by a discussion to identify the change domains reflected. Stories were regarded as most significant when they represented a diverse list of domains of change, representing the more comprehensive socio-economic change that the department was pursuing through the PTOG programme. Preference was given to those stories that reflected changes aligned to the outcome statement as in the grant framework that specifically focused on affordability and improving the affordability of the bus service for the user. To capture different socio-economic conditions of commuters, care was taken to include change stories from each route. The team filtered the stories per route and identified the MSC stories for each route (Chatsworth, Durbanville, Khayelitsha and Malmesbury) with more emphasis on stories from the recently introduced Chatsworth and Malmesbury routes and stories that reflected on a modal shift from taxis, cars or trains to the bus. A final round of assessment revisited the initial set of stories and selected the story that best represented each change domain to be added to the list of selected stories. This process rendered a final list of 20 change stories.

The shortlisted 20 change stories were sent to the second selection level, namely the programme staff responsible for the administration of the provisional grants. The second level was asked to select the MSC story for each of the nine change domains and to provide a short motivation for the choice made. The final nine selected stories were to be presented to the senior management team of the DTPW to provide strategic direction for subsequent designs of the programme. Unfortunately, the last two steps were not executed in full and the MSC hierarchical value clarification process did not achieve full benefit.

Ethical considerations

Permission for the completion of the research ‘Benefits of transport subsidisation: Comparing findings from a customer perception survey and Most Significant Change Technique interviews’ was obtained from the REC: Humanities, Stellenbosch University. Project number: SPLPAD-2019-10420. Project Title: Benefits of transport subsidisation: Comparing data from different data collection methods.

Results

In comparing the data obtained through the structured questionnaire versus the MSC interviews, the structured questionnaire was better suited to explore the intended benefits of the programme against predefined programme objectives, as well as comparisons between routes. The structured questionnaire ensured that data for change domains identified as strategic by the programme were collected for quarterly and annual reporting to the department. However, the structured questionnaire failed to provide insight into unintended positive (or negative) consequences of the programme. The MSC interviews allowed for the identification of new change domains that were valued by the participants.

For example, in terms of travel time, findings from the structured questionnaire present the average time spent on the bus as follows:

A total of 32% of respondents indicated that they spend one hour on the bus, 19% indicated that they spend one hour and thirty minutes travelling on the bus, with 14% indicating that they spend two hours on the bus. In comparing the routes in terms of travel time spent on the bus, most respondents indicated that they spend an hour on the bus, with the Malmesbury route showing the longest range in travelling time. (Western Cape Government 2017b:16)

While the structured questionnaire did not capture the implications of reported data from a commuter perspective, the stories of change were useful in explaining the changes in the lives of the beneficiary, as reflected by the following responses of MSC interviews:

‘I come home almost two hours earlier at night, and I can leave home an hour later in the mornings’. [I] “can do a lot of things at home”, like spend time with family and catch up on chores in the house….’ (Respondent 9, female, Malmesbury route)

‘I can be at work and home on time. Before I never used to be at work on time.’ (Respondent 16, female, Malmesbury route)

‘[I] now have time to prepare food for [my] child and to play with her.’ (Respondent 16, female, Malmesbury route)

The structured Likert style also allowed for on-average comparisons across lines and cross-tabulation comparisons between variables. For example, it was possible to cross-tabulate the respondents’ perceived affordability of the service to their self-reported household income:

When analysed per route, the routes that are regarded as sometimes not affordable and very unaffordable are Chatsworth (53%), Khayelitsha (45%) and Malmesbury (41%). An overwhelming majority of the Durbanville route respondents found the bus either very affordable or mostly affordable (91%). (Western Cape Government 2017b:19)

A comparison of data affordability and gross household income across the different bus routes reveals … that lower income bracket groups on average tend to regard the bus service as less affordable. With the exception of the Durbanville route, the average household income across the routes is between R2000 to R9000. (Western Cape Government 2017b:21)

It is interesting that on the Chatsworth and Malmesbury routes, the lowest household income bracket group indicated that the bus service is mostly affordable. (Western Cape Government 2017b:22)

In the latter case, a limitation of the structured questionnaire was that it did not allow for further comments that may have enhanced understanding of conflicting responses or ratings received. The stories of change were valuable in filling this gap. In trying to understand why Chatsworth respondents found the bus service affordable, despite low reported household income, the stories offered the following:

‘…the bus…is cheaper than taking a taxi…’ (Respondent 32, female, Chatsworth route)

‘… the bus is more affordable than making use of his own car and the related fuel cost…’ (Respondent 7, male, Chatsworth route)

This explains why commuters perceived the bus as very affordable, though the percentage of household income spent on transport was still well above the national policy target.

The stories of change were also useful for understanding the interconnectedness between changes experienced from the end-user perspective, even when these did not offer significant correlations in the sub-variate analysis. The following change story serves well to indicate the interdependence between change variables:

The most important change for her [since changing her transport mode from the train and own car to the bus] was that the bus service is much more affordable. She states):

‘Affordability definitely, it is very expensive for me to travel in and out with my car because it is too far. It cost me a third of the price of travelling with my own car per day.’ (Respondent 6, female, Malmesbury route)

Furthermore, she explains, ‘It is more convenient for me as I do not have to sit and concentrate on the road’ (Respondent 6, female, Malmesbury route)). Using the bus allows her to relax and not worry about the traffic. She loves to use the free time to catch up on her reading. She adds, ‘I do not have to worry about parking’ (Respondent 6, female, Malmesbury route)). She said that the train works out cheaper, but it is a lot less reliable, and there is only one train. When using the train she was late at times. On one occasion she got home at 23:45 on a Friday night because the train lines had problems. At that time the streets are unsafe. Using the new bus route makes her more punctual and enables her to be at work and at home on time.

The initial change domains identified as important to the programme and that informed the design of the structured questionnaire were accessibility, affordability, mobility, reliability and socio-economic changes. The analysis of the collected stories of change served to confirm the realisation of some of these changes (reliability, affordability and accessibility), while highlighting the change results regarded as valuable by the commuter (safety, reduced travel time, mobility, convenience, comfort and social networks). Although the stories collected were usually only a few sentences long, they were useful in identifying the most important benefits for participants and for constructing short narratives reflecting the impact of the bus service on users. Understanding what is important to the beneficiary is useful for realigning the programme to maximise these benefits.

Discussion

Both the structured questionnaire and the MSC interviews offered valuable insights on the perceptions of participants. While extensive training was required to ensure consistent administering of the structured questionnaire by the team of data collectors, less training was required to administer the MSC interviews. However, the apparent simplicity of the MSC technique may be deceiving. The capturing of data for the MSC interviews was, however, found to be very resource-intensive to collect and capture, further complicated by the fact that most interviews took place while standing in a moving bus. Respondents were quick to provide the changes they observed in their lives, but struggled to unpack the more personal reasons why the reported change was significant to them. The public setting of the interviews and the fact that most respondents were not responding in their first language but in English may explain the difficulty in conveying complex values and perceptions. This concurs with the experience of Wilder and Walpole (2008:535) who observed that questions requiring analytical thought, such as the question: ‘Why is this story significant?’, were rarely answered well.

While respondents find it easy to report on the changes they experience, soliciting meaningful value reflections on the significance of these changes requires more skilled interviewers. Careful structuring of the MSC interview guide and training of the data collection team are essential for the value clarification process. When conducting interviews in the respondents’ second language, the sentence structure should be simple and interviewers should have clear guidelines on soliciting in-depth, value-laden information without leading the respondents.

As discussed in the methodology, only one level of hierarchical selection of stories took place, while the MSC technique calls for multiple levels of value clarification. While the first round of analysis only focused on multi-faceted stories, the M&E unit adopted a more value-driven approach in the second round of selection. Three domains – affordability, reliability and convenience – that the team deemed most significant were used to identify further stories. Unfortunately, the next level of value clarification and story selected was impeded by time constraints that limited engagement by programme staff. Some value of the technique was lost because the organisation did not have the opportunity to clarify values at different decision-making levels that may inform future direction of the programme.

An opportunity to realign the programme with new change domains regarded as significant by the participants was also seemingly lost. For example, an improvement in perceived safety emerged as a valued change domain from commuters who recently changed from train to bus transport. A few change stories referred to the prevalence of criminality either on the train or at the train stops. In an emotive story, a respondent remarked:

‘The bus service has made me value life because if it had not been for the bus services I would have lost my life. There was a time we experienced train surfing which killed some of my friends but luckily I never died.’ (Respondent 49, male, Khayelitsha route)

He recalled that, on the trains, older commuters used to push younger ones to their deaths, but on buses this did not occur. The respondent attributed this to a sense of unity and ubuntu amongst bus commuters.

While this experience demonstrates the significance of improved safety offered by bus services, this story was excluded from the shortlisted set of 20 stories because it offered a negative perspective on another programme run by the same department. As the MSC technique does not offer specific guidelines on how to deal with negative change stories, powerful stories may be rejected in favour of more positive stories, thereby limiting programme decision-makers’ potential to identify and address programme limitations.

Conclusion

In this evaluation, participants’ insights derived through the MSC interviews complemented data collected against predefined programme objectives. Applying multiple methods offered richer understanding of the impact of the programme.

The structured questionnaire offered the following key advantages:

• enabled responses to intended and predetermined objectives of the programme
• enabled comparisons and cross-tabulations between questions and responses
• a larger sample can be efficiently managed, enabling generalisations (if proper sampling strategy is applied).

The MSC interviews complemented these findings by offering:

• short narratives on the real-life experience of individual participants, rather than the average experience of all respondents
• the most important and valuable benefits for participants, not limited by the intended and predefined objectives
• comments to clarify responses or ratings on the questionnaire
• connections between change domains for further bivariate analysis
• new change domains valued by the participants.

The MSC technique is not suitable for quick evaluations (Serrat 2010). To maximise benefits from the MSC technique, data collectors need to be able to solicit value responses from respondents, while decision-makers should allow time for multiple iterations and discussions at different levels of the hierarchy. Ideally, the selection levels would commence with programme managers who operate closest to the programme and the beneficiaries. Guidelines should be offered in how to deal with negative stories or unintended consequences, to ensure that these filter through to the next level. The next level of value clarification can be accommodated in a strategic support unit (like an M&E unit), who would not only select stories closely aligned to the values and principles of the organisation but also identify new emerging change domains. At the top of the hierarchy, the strategic management committee would select the stories that best represent the core future values that the organisation wishes to pursue. Finally, the process should provide for efficient feedback loops, where strategic-level decision-making and prioritisation filter down to the programme implementation level.

Acknowledgements

The authors acknowledge the support from the Department of Transport and Public Works who commissioned the evaluation study and offered access to the relevant information. We also acknowledge the data collection assistance by the following 2017 SPL HonsBCom students: L.A. Brown, C. Cedras. N. Daries, R. Muchena, N. M Mvundlela, N. Mwala, L.T. Ngoma, E. Sitler, J. Von Bardeleben, D. Vraagom.

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

This article is based on a joint research project by the listed authors. Background, instrument design, discussion and conclusions are equally attributed to both authors.

The authors acknowledge the financial support from the Department of Transport and Public Works who commissioned the evaluation study and offered access to the relevant information. The authors commend the organisation for their continued commitment to critically evaluate their programmes, and also acknowledge the logistical support from the Cape Higher Education Consortium.

Data sharing is not applicable to this article as no new data were created or analysed in this study.

Views expressed in this article are those of the authors and not the official position of Stellenbosch University, the Cape Higher Education Consortium or the Department of Transport and Public Works.

References

Footnote

1. Many stories responded to more than one change domain targeted by the transport grant. The response count is therefore higher than the number of stories collected.