The paper describes the implementation evaluation of the business process services (BPS) incentive programme undertaken by the Department of Trade and Industry (the dti) and the Department of Performance Monitoring and Evaluation (DPME) as part of the 2012/2013 National Evaluation Plan. The evaluation started on 31 October 2012 and the final report was approved on 17 May 2013. The evaluation covers the period from the inception of the programme in January 2011 to December 2012. The BPS incentive programme was implemented to stimulate the business process sector which contributes to economic growth largely through employment creation. The main objectives of the programme are to attract investment and create employment opportunities through offshoring activities. Twenty-six indicators across the five Development Assistance Community (DAC) evaluation criteria were developed. A multi-method approach was undertaken to collect data for each of the indicators. The key findings relate to the operation of the programme and a number of suggestions were made as to how to strengthen it. Overall 3807 jobs have been created through the BPS programme during the period under review. Estimated total investment provided by firms is approximately R2.7 billion. Amongst others, the study recommended that the design of the programme be reviewed and extended, potentially to a five-year period in order to maintain the competitiveness of South Africa as a business process off-shoring destination. It is essential to address the skills shortage to ensure the growth and sustainability of the South African BPS industry and finally the uptake of the incentive programme.

The South African National Evaluation Policy Framework, approved by Cabinet on 23 November 2011, seeks to ensure that credible and objective evidence from evaluation is used to improve performance and is incorporated in processes of planning, budgeting, organisational improvement, policy review, as well as on-going programme and project management. The main focus of the policy framework is on priority evaluations specified in national and provincial evaluation plans to be approved by Cabinet and provincial executive councils. As detailed in the policy framework, evaluations within the National Evaluation Plan (NEP) are undertaken by the relevant departments in partnership with the Department of Performance Monitoring and Evaluation (DPME) (DPME 2015:1).

The evaluation of business process services (BPS) was undertaken as part of the first NEP approved by Cabinet on 13 June 2012, covering eight evaluations. Although there was a programme review in 2010, the Department of Trade and Industry (the dti) in collaboration with DPME recognised a need to evaluate the implementation mechanisms of the revised BPS incentive programme from its inception in January 2011 until July 2012, as a way of improving on the uptake of the programme, which would in turn lead to faster job creation by the benefiting firms. The purpose of this evaluation was to investigate whether the BPS incentive programme was achieving its objectives.

A steering committee was established, chaired by the dti and secretariat services provided by the DPME. The role of the steering committee was to oversee the implementation of the evaluation, including approval of the evaluation products. The DPME commissioned the evaluation using its national panel of evaluators. Genesis Analytics (Pty) Ltd was selected to conduct the evaluation. The evaluation started on 31 October 2012 and the final report was approved by the steering committee as technically sound and factually correct on 17 May 2013.

A strong element of capacity building was incorporated into the process, as two interns from the DPME were part of the evaluation team and served on the steering committee. Their responsibilities related to the desktop review, fieldwork and report writing. The interns had previously worked for research companies as assistant researchers. Both the service provider and the interns found the process to be valuable and rewarding.

As early as 2005, BPS and off-shoring had been identified as an emerging sector with the potential for sustained job creation and contribution to economic growth by most governments and the private sector globally. The Accelerated Shared Growth Initiative (ASGI-SA), formally launched in February 2006, identified the business process outsourcing and off-shoring (BPO& O) sector as one of the country’s top three priority sectors to stimulate growth. Following from this, Cabinet approved a substantial Government Assistance and Support (GAS) programme aimed at improving processes for expanding existing BPO& O operations, deepening the skills pool, improving administrative processes and introducing investment incentives. BPO& O incentive had no explicit theory of change and log frame. The intervention was the result of a government decision based on the identified priority to create jobs. The GAS programme provided approximately R1 billion in investment incentives to cover part of the capital expenditure costs incurred by investors during setup.

The South African government introduced a BPO& O incentive programme in July 2007. During the period July 2007 to March 2010, the incentive resulted in the creation of at least 6000 new jobs and attracted R303 million in direct investments.¹ A comprehensive review of the BPO& O incentive programme resulted in a revised BPS incentive programme, which became effective in January 2011. Aligned with government’s overall strategy, the BPS incentive programme aims to attract investment and create employment in South Africa through off-shoring activities.² It was envisaged that the programme would result in the creation of a total number of 15 149 jobs over 3 years and support 22 firms in the 2011 and 2012 financial year.

The BPS incentive comprises two components, namely a base incentive and a graduated bonus incentive. The base incentive comprises a three-year operational expenses grant which decreases in line with narrowing the cost gap between South Africa and other destinations. This incentive is disbursed on the basis of actual jobs created and sustained and is paid quarterly over a period of three years. The bonus incentive is offered if the applicant exceeds annual offshore job creation targets. This is paid once in the year the bonus level is first achieved.

The incentive is only available to applicants who are servicing the offshore market; to be eligible, a project must create at least 50 new offshore jobs by the end of the three years. A minimum of ten offshore jobs qualifies a project for the first disbursement. Furthermore, a project cannot displace existing jobs in South Africa and projects may not receive concurrent incentives under the BPO& O incentive scheme.

Data made available by the dti identify 32 projects participating in the BPS incentive scheme as of March 2013. The 32 projects are run by 31 companies comprising service providers and captive operators. For the service providers, an approved project may involve services to more than one client but grouped as one project for the purposes of the incentive application. Of the service providers, 45% are international companies (mainly from the UK) who have set up operations in South Africa to offer outsourcing services to other international companies. This reflects the increasing attractiveness of South Africa as an outsourcing destination.

The projects under the incentive scheme are distributed across service types and geographical locations. The majority (66%) of the projects are in front-office operations, split almost evenly between inbound (57%) and outbound operations (43%) (Figure 1). Back-office services account for 22% of the total operations and are split almost evenly between operations that focus on human resources (HR) and pay roll, information technology (IT), knowledge process outsourcing (KPO), legal process outsourcing (LPO) and finance and accounting outsourcing (FAO), with KPO and LPO services constituting a slightly larger proportion of operations. Thirteen per cent of projects are shared services operations, which are primarily a combination of HR, FAO and customer query services for the captive operation.

FIGURE 1: Split of projects by service type – March 2013.

The 32 projects are distributed across KwaZulu-Natal, Gauteng and the Western Cape (Table 1). The projects in KwaZulu-Natal are mostly outbound contact centres whilst those in the Western Cape are mainly inbound contact centres, with a significant number of back-office operations. There are four shared services operations taking part in the incentive scheme; three of these are located in Gauteng and one in the Western Cape.

Although there was a programme review in 2010, the dti in collaboration with the DPME decided to conduct an implementation evaluation of the revised BPS incentive programme.

The purpose of the evaluation was to assess:

• whether the BPS incentive scheme was achieving its policy goals
• whether it was implemented as planned
• how its performance could be improved.

The key evaluation questions that guided the evaluation and informed the development of the methodology and analysis framework were the following:

• Are the objectives of the BPS incentive programme being achieved?
• What are the key factors influencing the success of the BPS sector in South Africa?
• Is the design of the incentive programme supporting the achievement of programme objectives?
• What is the current rate of job creation through the BPS incentive scheme?
• How cost-effective and competitive is South Africa’s BPS incentive programme relative to those of competing countries?
• How can the programme be up-scaled for greater impact and what are the barriers to growing the BPS sector in South Africa?
• How can the BPS sector be sustained post-incentive?

A number of the evaluation questions related specifically to an assessment of the BPS incentive scheme, whilst others had a broader scope, assessing the overall industry. It is important to note that there were a number of elements of the BPS value proposition that were explicitly excluded from the evaluation, including:

• Monyetla Work Readiness Programme™
• marketing
• industry mobilisation
• standards
• talent development initiative
• quality of sector operators
• industry organisation.

During the research process, it was impossible to totally exclude certain aspects of the above elements, especially where there were direct links to the evaluation of the BPS incentive scheme. As a result, the findings and recommendations highlight issues relevant to these themes, even though they are not central to the evaluation.

This study made use of the internationally accepted Development Assistance Community (DAC) evaluation criteria. The DAC criteria provide five measures against which each programme or project should be assessed, namely relevance, effectiveness, efficiency, impact and sustainability. For the evaluation of the BPS incentive scheme, the DAC criteria were adapted to encompass the following five categories, which adequately reflect the objectives of the implementation evaluation:

• Efficiency measures the extent to which resources allocated to the programme were used efficiently to deliver a quality programme (i.e. the efficiency of the project approval, selection, disbursement, deployment and management cycle). In this evaluation, efficiency assessed the extent to which the incentive scheme is administered and marketed well and the extent to which internal processes are effective.
• Relevance assesses the extent to which the incentive scheme is consistent with national and local priorities and also with the needs of the beneficiaries. This involves analysing the extent to which the incentive scheme enables investment and job creation, the quality of the jobs created and to what extent the design of the scheme maximises these.
• Effectiveness assesses the achievement of the incentive scheme’s objectives and the extent to which the incentive scheme is on track to meet its targets and objectives of job creation and attracting foreign direct investment (FDI). Impact has been incorporated into the achievement criteria, as impact is measured through the achievement of the impact level indicators to date.
• Sustainability investigates whether or not the benefits reaped as a result of the incentive scheme are likely to continue after the incentive scheme ends. In particular, this measure assesses if the incentive scheme is catalysing an uncompetitive environment to become competitive, or creating an unsustainable distortionary environment.
• Additionality focuses on the extent to which the incentive has created sustainable employment that would not have occurred otherwise.

Twenty-six indicators comprising quantitative and qualitative questions across the five evaluation criteria described above were developed. In order to maintain an objective and standardised approach to the assessment of the indicators, qualitative responses were coded into a quantitative scale wherever possible to ensure that the indicators could be aggregated in order to provide comparable findings.

The evaluation included the collection and analysis of primary and secondary research information on the BPS incentive scheme since its introduction in 2011. A multi-method approach was undertaken to collect data for each of the indicators. This involved a document review of the available project documentation (Table 2), analysis of existing programme data on projected and actual jobs created and projected FDI, a cost-competitiveness analysis of South Africa’s BPS incentive scheme compared with key competitors, and 34 stakeholder interviews with service providers and investors active in the industry, relevant government department personnel, and industry body representatives (Table 3).

Interviews were conducted in Johannesburg, KwaZulu-Natal and Cape Town with representatives from 24 of the 28 participating companies, one rejected or cancelled applicant, BPS industry representatives, nine key government officials involved in the BPS incentive scheme and one industry body.

Quantitative data were collected from the application approval reports of benefiting companies and the BPS company data provided by the dti. The data analysis included all 32 projects which were on the incentive scheme at the time of the evaluation. The quantitative analysis focused on the following elements:

• the achievement of each project’s job creation against its targets
• the total number of jobs created
• the projected investment to be generated from the incentive scheme
• the distribution of the projects by province and service type
• the type, size and ownership of companies on the incentive scheme.

This analysis was crucial to accurately map the range of BPS incentive participants – identify their position in the industry, the duration of their participation and the type of service they provided. This quantitative review further uncovered gaps and additional pertinent questions to be included in the interview process.

A core component of the evaluation of the BPS scheme involved an assessment of the cost-effectiveness and competitiveness of South Africa’s BPS programme compared with competing countries.³ The work focused on two main areas, namely benchmarking South African BPS cost-competiveness as well as the projected cost gap between South Africa and other offshore locations (India and Poland) over the next 3 to 5 years.

The historical (2010) and current benchmarking view of South African BPS cost-competiveness focused on benchmarking South Africa’s fully loaded cost per FTE for contact centre services (compared to Egypt, Kenya, Poland, Philippines and India) and for financial back-office services (compared to India and Poland).

The methodology for this portion of the study included market intelligence and secondary research, proprietary tools, databases and experiences, and selective primary interactions with market participants.⁴

Although the sample size is small it is reflective of the total number of firms participating in the incentive scheme. The findings therefore reflect only the experience of the participants in the scheme, and not potential investors or firms not involved in the scheme.

At the beginning of the fieldwork, there were 28 companies taking part in the incentive scheme. The aim was to engage each participant in the scheme; however, four were unable to participate in the research within the necessary timeframe. Three additional companies approved at the end of January 2013 were not included in the qualitative research process owing to the timing of the evaluation but were included in the quantitative analysis. Despite attempts to interview companies who had their applications rejected or cancelled, only one such interview was completed. Thus the respondent sample has a bias towards participating companies.

Ideally, each provincial industry body as well as the national body should have been interviewed. However, only one person representing both the Business Process enabling South Africa (BPeSA) Western Cape and BPeSA National was interviewed and there was no representation from industry bodies from Gauteng or KwaZulu-Natal.

Across the various programme data sources, there were a few inconsistencies relating to the figures for project performance; one programme data source would state a marginally different figure for the number of jobs created by a project compared to another data source. As the average number of jobs created is calculated by an electronic calculator developed by an external firm, inconsistencies in the data and the underlying calculations were difficult to clarify with the dti.

The investment values used in this paper are based on reported investment according to submitted claim sheets. The investment figures are not audited and are therefore reported as estimates. These values include operational expenditure, which in the majority of cases includes salaries. As there was no set template to guide how these figures are reported, these figures were taken as additional in some cases, and cumulative in others, as recommended by the dti.

Given the number of interviews successfully conducted and the amount of information yielded, it is highly unlikely that the limitations identified above have significantly affected the analysis and recommendations.

A wide range of documentation was reviewed to inform the evaluation, including academic literature, industry-oriented and government-produced publications. A synthesis of the literature reviewed allowed the evaluation team to categorise the information into three relevant themes, namely the global BPO context, the South African BPS context and the theory of incentive schemes. A brief outline of these findings is provided below. The literature review enabled the team to understand the research context in greater detail and formed the basis of the analysis framework. Key outcomes of the literature review process included:

• a better understanding of the history and context of South Africa’s BPS industry
• knowledge of the global BPO industry
• understanding of the theory of incentives, with particular focus on incentives for outsourcing industries
• identification of gaps in the documents and data for the BPS incentives scheme evaluation.

The development of the main themes and the identification of the gaps in the data formed the basis of the analysis framework and the respective indicators.

There is a growing trend of companies looking to outsourcing⁵ as a means of not only increasing efficiency but also improving industry expertise and customer service. This trend is evidenced in the significant growth rate of the industry which, over the years, has resulted in a substantially sized industry. In 2012, global BPO revenues were estimated to be in excess of $175 billion, global IT outsourcing revenues were estimated to be $290 billion, and offshore outsourcing was expected to represent approximately $85 billion of these two revenue streams (London School of Economics 2012). Figures for the industry’s growth rate vary but the general estimate is 8% – 12% per annum for the period 2012–2019 (London School of Economics 2012).

Typically it is the developed markets, such as the United Kingdom, the United States and Australia, which outsource to lower-cost destinations. These outsourcing destinations are categorised as tier I or tier II countries, where tier I refers to countries with a mature and experienced BPO market, such as India and the Philippines, and tier II refers to countries with less mature BPO markets, such as Mexico, South Africa and Brazil. Despite India’s overwhelming dominance as a BPO location, there is evidence of a recent trend of large companies moving their outsourcing functions to other outsourcing destinations in search of higher customer satisfaction. As a way of attracting this investment, the majority of BPO destinations offer competitive incentives which lower their cost of operations relative to other destinations and thus attract potential investors. These incentives are structured in a variety of ways, including tax holidays, training grants, investment grants and combinations of these.

The South African government identified the BPO sector as a priority sector for job creation and attracting investment. In order to encourage investment in this sector, the government identified the key constraints to the growth of the sector, which included high costs of doing business, lack of required talent, lack of investor support, lack of marketing and lack of experience. Following this, the dti, the Business Trust and BPeSA partnered together to implement seven work streams to overcome the identified constraints.

Since July 2007, the dti has offered an incentive programme to promote BPS as a means of bridging the cost gap of South Africa compared to other offshore locations, thus making South Africa more competitive.

Although the incentives add to South Africa’s competitiveness, cost is not South Africa’s primary strategic advantage over other destinations. The country’s strategic advantage is its value proposition,⁶ which includes a skilled English-speaking talent pool, deep domain skills, first-world experience, a robust enabling environment and significant cost savings.

There is contentious debate regarding implementing incentives and whether or not one should interfere with the market economy. Opponents of incentive schemes suggest that markets can function sufficiently without government intervention and that if left alone, an efficient competitive equilibrium will result, maximising social welfare. On this premise, if the BPS industry is in a perfectly competitive equilibrium then the incentive reduces price and increases consumption beyond the competitive equilibrium, resulting in a departure from competitive equilibrium. The World Bank and other proponents of the Washington Consensus discourage the use of incentive schemes in developing countries, other than if the motive behind the scheme is improving national welfare in the form of an increased technology level, job creation, improved work opportunities and economic growth. From an South African economic perspective, the BPS incentive scheme has the characteristics of an incentive which promotes job creation for previously unemployed South Africans.

Therefore there is not a simple, one-size-fits-all solution to introducing and maintaining incentives. When assessing whether or not to introduce or continue with the kind of market intervention that incentive schemes present, there needs to be a full analysis of the costs and benefits that the incentive has on both the industry as a whole and the national economy, and the consequential effects on both the industry and the broader economy once the incentive ends. If positive spill-over effects are to materialise, firms need to invest in technology, improve learning capacity and maintain a competitive local business environment to ensure that the externalities can be absorbed for long-term sustainability in the industry.

The findings of the qualitative and quantitative research processes are presented according to the five evaluation criteria, and may be summarised as follows.

Efficiency: In general, the administration of the incentive scheme was assessed as being efficient. An overwhelming majority (88%) of respondents described the communication and explanation of the details of the scheme during the application process as being efficient, and the information received as sufficient for promoting timely and informed decision-making. It was noted that the paper-based application and claim process can be improved, and that in some instances the administrative team is not able to respond to technical queries. A quarter (25%) of participants reported that they had been exposed to promotion of the BPS incentive scheme within South Africa.

Relevance: Of the 25 firms, 13 were of the opinion that the incentive scheme should be designed such that the amount of incentive received is graded by the type of service provided or the agent skill level, whilst 13 of the 25 firms noted that the BPS incentive has a positive influence on their investment decisions. Mainly captive operators⁷ operating in South Africa or service providers who are based in South Africa indicated that the incentive scheme does not directly influence their investment decisions.

Effectiveness: The achievement of the scheme is related to its success in meeting its FDI and job creation targets, as well as to industry participant perceptions of the incentive scheme and of the competitiveness of the South African BPS industry as a whole. However, no targets are set for job creation or FDI. The annual Medium-term Expenditure Framework (MTEF)-linked budget allocations (by the National Treasury) are used as a guide for job creation. The annual allocation can be revised upwards, through a formal virement process, but the upward revision has not been necessary. Since the start of the incentive scheme, 3807 jobs have been created or supported by the BPS incentive scheme. The actual jobs created are substantially less than the 11 597 projected by companies when submitting their incentive applications because projected jobs on approval of applications were not realistic as they were based on potential contracts; however, the actual number of jobs were based on signed contracts.

FDI is not clearly defined, nor is FDI data consistently captured or audited. Instead, data on total investment are provided by firms as part of their claim documentation. Reported investment to date is approximately R2.7 billion, which refers primarily to operational expenditure. This reported value meets, and in fact slightly exceeds, the investment value projected by the scheme for the same period. In the absence of targets for these indicators, it is difficult to design and operate an effective performance management system. Sixteen of the 25 firms stated that the strength of the incentive scheme related to its impact on reducing the cost of operations in South Africa. Skills were noted as one of the main areas of concern in the industry – 9 of the 25 firms noted that the incentive scheme does not adequately deal with this issue.

Sustainability: There is significant market uncertainty regarding the future of the incentive scheme, which is scheduled to end in the 2014 and 2015 financial year. The survey revealed that 100% of respondents had received no communication from the government regarding the future of the scheme. The majority of respondents also noted that, unless the level of skills increases commensurately with industry growth, the current skills pool will not be sufficient to meet the industry’s demands or to enable sustained sector investment and growth.

Additionality: This criterion assesses the role of the incentive in directly catalysing investment and jobs, and the extent to which the jobs created and investment made in the BPS industry would (not) have gone ahead without the incentive scheme. Half (14 out of 28) of the firms stated that their investment in the industry was strongly influenced by the presence of incentives, which implies that the incentive scheme has indeed catalysed significant additional activity and investment and has resulted in job creation and investment that would not have occurred otherwise.

The implementation review provided the evaluation team with sufficient information to make recommendations to improve both the operation of the programme and consequently, its prospects for long-term impact. In addition to an assessment of the five standard evaluation criteria discussed above, the analysis and recommendations respond to the specific evaluation questions highlighted in the section ‘Purpose and focus of the evaluation’.

There is no forward-looking target set for job creation or FDI. Assessing the performance of the BPS incentive scheme in relation to job creation is limited to a comparison of jobs created in relation to the maximum budget allocation for the scheme in alignment with the MTEF. Currently, budget allocation is based on the scheme’s past performance. This means of operational planning is not conducive to proactive, opportunity-led and incentive-driven management of the scheme.

Similarly, it is not stipulated as a part of the scheme’s objectives exactly how much FDI the scheme is expected to attract, nor is actual FDI systematically reported. In certain instances it is reported and recorded as a cumulative value over the life of a particular project, and in others it measures annual figures relevant to a particular claim period. Therefore no assessment can be made as to whether the value of FDI is meeting expectations at the time the scheme was initiated.

A policy target needs to be set for both jobs created and FDI attracted. Although the incentive scheme is already run over half its term, it is essential that appropriately researched targets are set for the remainder of the scheme’s duration.

In the current context of no policy target, the dti should aim to achieve a higher jobs created or sustained figure than implied by the MTEF budget. The necessary arrangements should be made with the Treasury to ensure support of these targets and to ensure that future budget allocations are not based purely on past performance. Firms should be encouraged to project more accurately how many jobs would be created; an additional financial incentive might be provided to firms that create 90% or more of the projected jobs.

Any relevant investment sustained must be accurately and consistently reported by firms – either cumulatively or additionally over the life of a project.

Monitoring of the scheme’s performance needs to be improved, particularly in relation to collection of data on actual FDI and jobs created. If discrepancies arise in recorded data, they need to be resolved in each case and standardised to ensure that accurate figures are reported and logged.

Promotion of the scheme, particularly to domestic stakeholders who have significant international client bases, needs to be improved so as to increase the number of participants in the scheme, which will in turn result in increased job creation and FDI.

One positive aspect of the incentive scheme is that it is straightforward to understand and operate. However, most participants felt that the duration of the scheme should have been five years, making it directly comparable with incentive schemes in other countries. The incentive duration also had implications for South Africa’s cost-competitiveness, as potential investors are comparing it against other countries’ offerings over five years. The extent to which the duration of our incentive scheme may have discouraged investment in South Africa to date is not clear.

The BPS industry is split into two broad categories, namely front-office and back-office operators. These differ not only in their service offerings, but in their employment requirements, required employee skill levels, average size, and cost of operations. Back-office operations require highly skilled employees and in most cases are smaller operations. As such, a number of existing and potential back-office operations are unable to take advantage of the scheme because they do not meet the minimum requirement of 50 employees. In addition, given their smaller size, higher unit costs and higher labour cost, the incentive amount contributes proportionately less to reducing their cost of operations than is the case for lower-cost front-office operations. The incentive scheme is not designed to take into account these differences in service types.

The administration of the scheme is perceived to be efficient; no major issues were noted regarding the application or claims processes. However, paper-based application and claims processes as well as administrators’ inability to respond to technical queries relating to these processes and criteria in certain instances were identified as areas for improvement.

A graded scheme by type of service should be introduced to provide higher incentives for firms offering a higher value niche service, particularly in the case of back-office firms. The requirements for back-office operators should also reflect their smaller size, and the minimum requirement should be reduced to 20 jobs for such firms.

The application and claims processes should be translated from the current paper-based platform to an electronic, web-based platform, which will increase the efficiency of the process and will enhance the scope for effective, timely monitoring and information processing. The dti is in the process of setting up such a system.

Any adjustments to the design of the scheme must not complicate the application and claims processes.

The responsibility of communication of information specific to the policy guidelines and operational and strategic issues of the incentive scheme should be allocated to a single contact person, who should be well informed and equipped to respond to relevant technical queries.

In the 2012 and 2013 year, the number of projects on the scheme grew by 60% to 32, whilst the number of jobs created or supported grew by 48% to 3807. Whilst this is a significant increase over the two years, when assessed in relation to the maximum allocation as per the MTEF budget and the potential for growth in the industry, there is definite scope for improvement.

Increasing the number of firms on the scheme and consequently the rate of jobs created should be made a priority. This is linked both to improving the promotion strategy of the incentive scheme to attract more investors and to the introduction of graded incentives so as to capture a greater portion of the entire BPS value chain.

Again, companies should be encouraged to make more realistic projections.

The incentive scheme has contributed to making South Africa more competitive than it was in 2010 as fully loaded operating cost per job for contact centre work has declined by approximately 5% per annum. South Africa’s cost gap has reduced against all five benchmark countries, as compared to prior to the BPS incentive scheme. However, when a full assessment is made over the life of the scheme and compared to other countries, South Africa’s cost disadvantage is set to increase rather than decrease. This is primarily as a result of two factors: the reducing nature of the incentive scheme, and the three-year duration of the scheme as compared to schemes in competitor countries which typically run for five years.

The uncertainty regarding the future of the incentive scheme was noted by every participant as a major source of concern. This uncertainty is beginning to affect adversely South Africa’s competitiveness as an investment destination. There is no specific strategy to promote the incentive scheme. It is essential that the incentive is adequately promoted to investors as a distinct feature of South Africa’s BPS offering, to ensure that South Africa’s competitiveness is accurately understood by potential investors.

The incentive scheme is playing a role in reducing the cost gap between South Africa and its competitors and should be extended in duration to maximise this benefit.

Communication from the dti regarding the future of the scheme must be prioritised; communication of the fact that the extension of the scheme is currently being explored will alleviate current anxiety.

There needs to be a more concerted effort to promote the incentive scheme prominently within South Africa’s value proposition to domestic service providers and captives. Any promotion of the incentive scheme must be communicated to all industry participants so as to ensure consistency in the message that is delivered to potential investors.

Better coordination is needed between the entity responsible for the promotion of the incentive scheme and industry bodies and companies, to develop a coherent strategy for communicating and promoting the scheme to investors.

Upon completion of the evaluation, the dti submitted a management response, followed by an improvement plan which was developed by the key stakeholders. The improvement plan comprised a list of recommendations endorsed by the dti and an action plan against each recommendation. The improvement plan includes, amongst others, strategies for a review of the existing incentive programme and how the programme could be continued beyond its current three-year duration (2011–2014), so as to drive long-term competitiveness and growth.

An important development following the evaluation was the review of the incentive, which has been finalised. The revised BPS incentive which was launched by the dti in London on 14 October 2014 has fundamentally addressed all the study recommendations. The key distinguishing factors of the revised BPS incentive are that the programme will run for five years and that the base incentive will be split into a two-tier incentive for complex and non-complex jobs. The purpose is to attract more complex jobs such as LPO and shared services subsectors which attract a higher skilled workforce. The graduated bonus incentive has been maintained; however, it will be provided to approved applicants that create and sustain more than 400 offshore jobs. The programme is operational from 01 October 2014 until 31 March 2019.

The poor quality of programme monitoring data tends to prolong evaluations. Evaluators often have to mine the data, which makes it difficult to ensure credible, verifiable findings. In some cases this has resulted in some redesign challenges mid-way through the evaluation process in order to achieve the required evaluation outcome (Goldman et al. 2015:6). In the case of the BPS evaluation, data were captured in multiple Microsoft Excel spreadsheets and the evaluators repeatedly had to double check if it was the correct and up-to-date version with the correct data. In the end, it was not possible for the evaluators to validate or verify the actual numbers in the spreadsheet provided by the programme team.

The BPS incentive had no explicit theory of change and log frame prior to the evaluation. One of the deliverables from the evaluators was to produce the theory of change for BPS, test it and propose a suitable theory of change. This has become the first phase for all evaluations in the NEP. The other issue with respect to an inherent lack of results-based planning was that targets were not set for the priority indicators as a programme – in this case the number of jobs created. Therefore, when the evaluators were expected to assess achievement of results, it was not possible to determine whether the results achieved were in line with expectations or not. This was a priority recommendation emanating from the evaluation.

The BPS evaluation was one of the first evaluations completed by an external service provider for the DPME. It was a learning curve for all parties concerned, but on the whole was a positive change from the traditional evaluation role when working directly with a line department. The DPME’s role is to play a facilitative role in procuring the service provider as well as the secretariat for the consultative steering committee. In comparison to working with line departments directly, working through the national evaluation system offered additional structure and neutrality and encouraged greater independence for the evaluator. In the case of the BPS evaluation, the DPME project manager played this role effectively throughout the process.

The challenge of evaluation, particularly in the public sector, remains the fact that evaluation may be perceived as a threat that can expose certain weaknesses within the organisation. This often results in defensive actions and attempts to refute the evaluation findings, rather than to focus on the lessons learned. Whilst the BPS evaluation identified a minimal number of shortcomings, stakeholder response demonstrates that a culture of learning is not yet instilled in government and that broader support is required to ensure engagement with evaluation findings.

Whilst this evaluation was an implementation evaluation, structured to be a holistic and comprehensive evaluation using the DAC criteria to go beyond the basic understanding of the achievement of results, unfortunately; the stakeholders were unduly focused on impact – and the achievement of what the evaluators felt were arbitrary pre-determined targets. Unfortunately, this undue emphasis on the quantitative results may have distracted the stakeholders from the other findings, thus losing some of the value that could have been achieved through the evaluation process. Whilst the preference for quantitative results can be understood given the quantitative focus of the dti’s work, broader support is required throughout government for both quantitative and qualitative evaluation findings to inform optimal changes to policy, programme design, and the measurement of results.

Following the challenges with evaluability of the BPS and other programmes, the DPME has resolved to carry out evaluability assessments of programmes proposed for evaluation. A robust evaluability assessment tool would be developed and applied to determine the evaluability of selected government programmes for the following year.

The problems identified include poorly articulated programme outcomes, unclear target populations, missing or poorly defined theories of change, the lack of appropriately defined indicators, and poor quality or missing data on these programmes. These problems particularly affect the potential to undertake impact and economic evaluations, with the result that the evaluations become formative in nature. Usually, at the concept note and terms of reference development stage, there seems to be clarity on what information and data are available, but once the study is commissioned, the service providers discover that the proposed evaluation type is difficult or impossible because the conditions for successful evaluation have not been met at the design, development and implementation stages of the programmes (Backhouse & Anambao 2014:4).

The envisaged DPME evaluability assessment tool development will focus on determining what needs to be in place to ensure that programmes can be assessed using the desired type of assessment and methods, or to suggest alternative types and methods of assessment that are feasible and likely to provide useful information on programme performance and achievements as well as identification of unintended results and risks (Backhouse & Anambao 2014).

The evaluation was a useful exercise that highlighted the positive attributes of the BPS incentive scheme and identified the areas requiring attention to improve the scheme going forward. Importantly, the evaluation emphasised that the BPS incentive scheme does not operate in a vacuum and that the scheme’s potential for success is intricately linked to the potential of the BPS industry in South Africa.

South Africa’s success as a BPO destination will always be linked to its cost-competitiveness. Although not aiming to become the lowest cost operator, the incentive scheme was introduced to decrease substantially the cost of operating in South Africa.

On the other hand, the quality of service is a key factor influencing South Africa’s success; the country is often the selected destination because of the higher quality of service provided compared to other destinations. A drop in the existing comparatively high standard of service will have adverse consequences for the success of South Africa’s BPS sector. Given that for the most part the BPS incentive scheme is successful, well run and is received positively in the industry, it is important that the findings that go beyond the scheme itself are considered to ensure that the South African BPS value proposition remains relevant.

The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.

N.M. (Department of Trade and Industry) was the lead author and contributed to methodology, background, context and use of the evaluation. A.W. (Genesis Analytics) contributed to the literature review, evaluation methodology, results, recommendations and conclusions as well as perspectives from the service provider. J.M. (Department of Planning, Monitoring and Evaluation) contributed to the abstract, the significance of the evaluation, the background to the evaluation as well as lessons from the evaluation. R.S. (Department of Trade and Industry) contributed to the section on the use of evaluation results.

1. ‘Jobs’ refers to actual incentivised jobs and direct investment is based on projected capital expenditure for 3 years.

2. ‘Off-shoring activities’ refer to when a company relocates IT-enabled business processes that it used to perform in-house to a foreign location.

3. Genesis Analytics outsourced the cost-competitiveness analysis to the Everest Group, a leading global services advisory firm, with extensive experience of business process services, particularly in South Africa.

4. Egypt: Information Technology Industry Development Agency; India: National Association of Software and Services Companies (NASSCOM); Kenya: Kenya BPO and Contact Centre Society; Philippines: Business Processing Association of the Philippines; Poland: Polish Information and Foreign Investment Agency.

5. Outsourcing occurs when a company uses third-party providers to perform activities/services.

6. Value proposition is the unique selling point or offering for the country, specific to that sector.

7. Captive operators are companies that use a wholly owned subsidiary for own services instead of a third-party owned vendor, that is they do not outsource.