Given the low levels of financial literacy in South Africa, financial education projects have a significant role to play in reducing some of the demand side barriers to financial inclusion. Measuring the impact of a financial education project is important to assess whether the project achieves its ultimate objectives, for justifying scaling up and for policy design. However, impact evaluations alone are not sufficient in describing the success or failure of a project.
This study aims to show that, particularly in a South African context, where investment in financial education interventions is mandated by the Financial Sector Codes, impact should not be the only criterion assessed when evaluating financial education projects.
This study was informed by a literature review, a synthesis of team experience on a range of financial education projects in South Africa and the development of case studies.
Describing the success or failure of a project needs to go beyond impact and explore factors such as project relevance, design and quality. In order to verify these other factors, different types of evaluations are necessary at the various stages of the project’s life-cycle.
Expanding the learning objective beyond the exclusive identification of whether financial behaviour was achieved is particularly important where financial education projects, and the monitoring and evaluation thereof, is mandated. In the African context, where resources are scarce, money for monitoring and evaluation should be selectively channelled into determining project relevance, effectiveness, efficiency and then only impact.
African countries are characterised by low levels of financial literacy and high barriers to financial inclusion (Messy & Monticone
There are a number of components to financial inclusion that should be considered when measuring the effectiveness of a financial education project (Atkinson & Messy
Trajectory of individual competencies contributing towards financial behaviour.
Currently, impetus is on measuring the impact of financial education projects by measuring changes in beneficiaries’ financial behaviour (Bayer, Bernheim & Scholz
What learnings can be generated from the evaluation of financial education projects?
To what extent can evaluators expect positive impact from financial education projects if the projects’ relevance, design and effective implementation have not been ascertained?
How can different types of evaluations contribute to the ultimate achievement of impact?
This study is structured as follows:
Section 2 discusses the research methodology used to inform this article.
Section 3 focuses on why measuring impact in financial education projects matters, but why it should not be the only criterion that is assessed. A key point in this section is that a primary objective of evaluations should be to provide feedback into the design and operation of the project to reinforce its ability to achieve its objectives.
Section 4 explores the concept of the ‘life-cycle’ of an financial education project and introduces the notion that different types of evaluations, and hence different evaluation designs, are appropriate at different points in a project’s life-cycle. These various evaluation methodologies are then presented with a description of when each should be used.
Section 5 highlights the importance of process evaluations, which allow practitioners to track the progress of a project’s development and performance across its life-cycle and serve as a critical complement to impact evaluation particularly in the African context.
The methodology employed to develop this study consisted of three steps: A review of key literature on financial education and the evaluation thereof, focusing on the African, and specifically South African, context. The selection of literature for this review was based on the following four predetermined inclusion criteria:
Intervention type: Only financial education–focused studies were considered.
Geography: Studies with African-based financial education projects were prioritised.
Date of intervention/publication: Studies/interventions published/implemented from January 2000 to April 2015 where prioritised.
Publication language: Only studies published in English were considered.
The following three exclusion criteria were also used for the selection of literature:
Studies that do not link directly to the financial education context with behaviour change outcomes.
Studies that are not explicitly evaluation oriented.
Studies in a non-African context.
A team synthesis session to reflect on and integrate learnings around implementing and evaluating financial education projects in South Africa was attended by four specialists in financial education in South Africa. Participation in this session was restricted to individuals with practical experience in implementing, monitoring and evaluating financial education projects across South Africa. The session followed a structured format; firstly, each participant presented their experiences evaluating financial education projects in South Africa, and secondly, a plenary discussion was facilitated maximising learnings from the evaluation of financial education projects.
Primary research case studies of financial education projects rolled out in South Africa were included to get practical insights into evaluating financial education projects.
The main limitation to undertaking this research was a lack of published work on financial education in South Africa and Africa more broadly. This is a relatively immature field in the African context, and thus, published articles, in particular academic articles, are not easily accessible.
The future of good financial education in South Africa depends on learning from doing - evaluations must unpack more than the achievement of behaviour change.
Impact evaluations of financial education projects are widely undertaken to assess the changes accrued as a result of the project. This identifies whether the project is having any effect, which can be used to inform the following:
Ensuring value: Evidence that a project has a proven impact is critical in determining the value of investment, public funds or funds from private sector financial institutions as is the case in South Africa. If the intervention is not shown to have a positive impact, it is most likely not addressing a real need and not implemented appropriately, thus indicating that either the project should be modified or the money should be spent elsewhere.
Impact is important for policy design: Beyond the realm of the funders, implementers and beneficiaries, a project’s ability to have an impact has important implications for policy design. If a particular strategy or intervention can be shown to positively affect a particular population group it may result in a policy change, such as the government encouraging or binding a specific sector to invest in similar initiatives.
Justifying scaling up: If a project can be shown to have a positive impact, it allows implementers to make an argument for increasing the size and reach of the project. Naturally, other factors also need to be considered, such as the scalability of the model and budget, but if the project is not having an impact, there is no case to be made for the project budget and reach to be extended.
Having identified why establishing impact of a project is important, experience from the research shows that impact is likely only achieved if a project was well-designed, identified relevant messaging and delivery channels for the target audience and was implemented effectively and to a particular quality standard. Therefore, in this nascent field of financial education, evaluation should seek to confirm the theory, relevance and quality of interventions to ascertain that impact is even possible to expect in a given project context. Learning from past projects is critical to ensuring that future projects are designed appropriately for the intended audience, social and economic environment and the behavioural nuances that each context presents. Focusing resources on proving the impact of financial education does not contribute significantly to the financial education knowledge base in Africa. Rather, this knowledge base is more effectively and efficiently expanded through evaluations that generate learnings around projects’ relevance, effectiveness, efficiency and sustainability and then only impact (OECD
Measuring financial education in South Africa.
In South Africa, financial service providers are required by the Financial Sector Codes to spend a minimum of 0.4% net profit after tax on financial education, a portion of which must go towards the monitoring and evaluation of these initiatives. The Financial Sector Codes specify that this must be done by an external service provider; however, the manner in which monitoring and evaluation must be conducted is not defined.
Evaluations can allow implementers to learn about what works and what does not in their context and to establish best practices that can improve the implementation of existing projects and inform the design of new ones. Impact evaluations do not provide a holistic understanding of projects’ implementation, nor do they provide detailed information into the relevance, quality and appropriateness of the project for a given context.
Understanding contextual diversities has been identified as a financial education best practice as social and cultural idiosyncrasies play a large role in project success (Finmark Trust
Finally, financial education is intended to improve beneficiaries’ financial capability, and ultimately financial inclusion. While there are no standardised definitions of financial capability and financial inclusion, financial capability is generally understood to refer to an individual’s financial knowledge and skills to make good financial decisions and financial inclusion is understood to consist of two elements: good financial decision-making (the ‘demand side’) and access to suitable products and services (the ‘supply side’) (Milton
Different types of evaluations serve different purposes throughout the life of a project. In order to elicit the desired information from an evaluation, not only does the right type of evaluation need to be implemented, it must be applied at the right point in the ‘life-cycle’ of a project.
Many evaluations yield poor results or fail to detect the positive contribution of a project as a result of being conducted at the wrong stage. Given the discourse, project owners too often now expect projects to be evaluated using rigorous impact methodologies at too early a stage in their development, for example pilot interventions that are used to test material and delivery channel for a particular target audience.
Project life-cycle and evaluation type.
A
Based on the diagnostic evaluation, the theory of change should be drawn up if it has not already been drawn up, or should be reviewed if there is an existing one in place. A
Once the project is underway, an
The above evaluations inform project conception and implementation planning with guidance on the most suitable design of a particular financial education intervention and how best to implement it. Once this has been established, an
It is not necessary to undertake each of these evaluations at each point in your project’s life-cycle. The selection of the appropriate evaluation type will depend on the future of the project and the intended use of the evaluation. One needs to give due consideration to the purpose of the evaluation and selection of the appropriate evaluation type to achieve the learning objective. Projects are typically given separate funding windows for each stage in their life-cycle. As such, a portion of this should always be set aside for monitoring and evaluation such that projects do not progress to the next stage in the life-cycle before the relevant evaluation has been undertaken to ensure that the project is designed and implemented appropriately to achieve its intended impact. Finally, impact evaluation should be considered when stakeholders are sufficiently confident that a project’s relevance, theory and quality will lead to the expected impact – thus lending to expanded scope and scaling.
At earlier stages of a project’s life-cycle, evaluations should primarily focus on the suitability of the project’s design and the theoretical ability to impact its beneficiaries positively, in addition to whether it is doing that.
An advantage of conducting implementation evaluations during earlier stages of a project’s life-cycle is that they allow for a holistic investigation into the project’s performance. Data required for implementation evaluations should be both qualitative (e.g. diaries, interviews, focus group discussions) and quantitative (e.g. indicators measuring knowledge about financial concepts, comfort knowing where to go for recourse, savings, and other MIS data). Evaluations can focus on indicators that assess the overall functionality of the project itself. The OECD’s Development Assistance Committee (DAC) Criteria for Evaluating Development Assistance (OECD
Implementation evaluations can be seen as an investigative tool that provides real-time insight into the effective functioning of the project, if effective at all. They can serve as an early warning system, allowing implementers to address operational issues as they arise and provide information for the constant improvement of the project’s design and are ultimately essential in order to maximise chances of seeing results.
Implementation evaluations are particularly valuable when piloting new financial education projects. As noted above, the social and cultural idiosyncrasies of a context should be considered when designing financial education projects. As such, South Africa’s diversity and cultural nuances present many challenges to replicating projects from both international and domestic contexts. As such, a common practice in the South African context is to run a pilot project before rolling the project out in full scale, thus enabling evaluators to identify necessary changes to the project design to ensure its effectiveness. There are a number of examples of this practice, including:
The Imali Matters Pilot Project run by the Department of Trade and Industry, Finmark Trust, African Bank and the Credit Information Ombudsman, which delivered face-to-face consumer counselling at walk-in centres across the three main cities in South Africa (Eighty 20 Consulting
Association for Savings & Investment SA (ASISA) Foundation’s pilot project, SaverWayaWaya, implemented in Hammanskraal, Gauteng, is discussed in
Learning from a pilot.
The ASISA Foundation, established to deliver effective Consumer Financial Education projects nationally on behalf of its members, insisted on incorporating rigorous M&E alongside the 2014 pilot implementation project that would inform the Foundation and its members about what works and what does not. The 2014 evaluation assessed the five DAC criteria (described above) in an attempt to uncover success factors for financial education projects. The particular project specifically sought to investigate the use of a ‘life stages’ approach as well as delivery methods comparing workshops versus workshops incorporating industrial theatre.
The ASISA Foundation staff and the industry steering committee were very interested in the findings of the 2014 evaluation. The findings from the evaluation of the pilot were used to design the 2015 project and have informed a more relevant and targeted project.
The ASISA Foundation, established to deliver effective Consumer Financial Education projects nationally on behalf of its members, insisted on incorporating rigorous M&E alongside the 2014 pilot implementation project that would inform the Foundation and its members about what works and what does not. The 2014 evaluation assessed the five DAC criteria (described above) in an attempt to uncover success factors for financial education projects. The particular project specifically sought to investigate the use of a ‘life stages’ approach as well as delivery methods comparing workshops versus workshops incorporating industrial theatre.
The ASISA Foundation staff and the industry steering committee were very interested in the findings of the 2014 evaluation. The findings from the evaluation of the pilot were used to design the 2015 project and have informed a more relevant and targeted project.
Ultimately, impact evaluations should only be conducted when there is confidence that a project is nearly perfect in its implementation (e.g. quality of material, appropriate messaging, accessible and relevant delivery channels). The most rigorous type of impact evaluation, randomised control trials, are thus ultimately most useful once a project has reached a certain level of maturity (Chambers
There has been an increased focus on embedded financial education among formal and informal financial service providers who hope to improve their customers’ financial capability and ability to make informed financial decisions. These initiatives aim to pass on financial messages to beneficiaries at points of contact in such a way that there is a seamless integration between an individual’s interactions with the financial service provider and the financial education they receive. When evaluating these initiatives, this embeddedness makes it considerably more difficult to attribute particular changes in beneficiaries’ knowledge, skills, attitudes and particularly behaviour, to either the financial education or the responsible provision of financial services. We should rather take a holistic view of projects like this, focusing solely on whether the desired impact is achieved, regardless of the relative contribution of the financial education or financial service. What is therefore critical is that the potential for harmony and reinforcement between the two is maximised.
Generating lessons around project relevance, design and quality is critical given the current importance placed on financial education in South Africa. Given the heterogeneous nature of the South African context, forming a knowledge base that includes learnings around contextual nuances, and how to best address these, is fundamental to ensuring that projects are implemented in a manner that will likely lead to the greatest impact. In order to elicit this information and to ensure that the objectives of the project are met, different types of evaluations must be considered and conducted throughout the project life-cycle. Furthermore, not only does the right type of evaluation need to be implemented but it must also be initiated at the right point in the ‘life-cycle’ of the project and take into account the additional factors that contribute to an individual’s financial well-being. Many evaluations yield poor results or fail to detect the positive contribution of a project as a result of being implemented at the wrong stage. Most often, projects are evaluated with rigorous methodologies at too early a stage in their development or have focused on inappropriate evaluation questions given an intervention’s implementation stage, maturity, design and context.
The ability to demonstrate a project’s impact is important to ensure the continuation of project funding, for justifying scaling up, to ensure value for money and for policy design. However, we should only take hard and fast decisions on impact once there is confidence in the design and implementation.
The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.
E.M. was the primary drafter and researcher. She was responsible for reviewing past literature on the topic and synthesising the team’s experiences of financial education in South Africa. A.W. was responsible for providing strategic insight to the document and guided the team’s review process given her international expertise in financial education. C.S. was responsible for drafting the case studies and for providing strategic guidance based on her practical experience working with financial education in South Africa.