About the Author(s)


Emily Massey Email
Genesis Analytics, South Africa

Alyna Wyatt
Genesis Analytics, South Africa

Caitlin Smit
Genesis Analytics, South Africa

Citation


Massey, E., Wyatt, A. & Smit, C., 2016, ‘Evaluating financial education initiatives in South Africa: the importance of multiple evaluation approaches’, African Evaluation Journal 4(1), a125. http://dx.doi.org/10.4102/aej.v4i1.125

Original Research

Evaluating financial education initiatives in South Africa: The importance of multiple evaluation approaches

Emily Massey, Alyna Wyatt, Caitlin Smit

Received: 06 Mar. 2015; Accepted: 25 May 2015; Published: 10 June 2016

Copyright: © 2016. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: Given the low levels of financial literacy in South Africa, financial education projects have a significant role to play in reducing some of the demand side barriers to financial inclusion. Measuring the impact of a financial education project is important to assess whether the project achieves its ultimate objectives, for justifying scaling up and for policy design. However, impact evaluations alone are not sufficient in describing the success or failure of a project.

Objectives: This study aims to show that, particularly in a South African context, where investment in financial education interventions is mandated by the Financial Sector Codes, impact should not be the only criterion assessed when evaluating financial education projects.

Research method and design: This study was informed by a literature review, a synthesis of team experience on a range of financial education projects in South Africa and the development of case studies.

Results: Describing the success or failure of a project needs to go beyond impact and explore factors such as project relevance, design and quality. In order to verify these other factors, different types of evaluations are necessary at the various stages of the project’s life-cycle.

Conclusion: Expanding the learning objective beyond the exclusive identification of whether financial behaviour was achieved is particularly important where financial education projects, and the monitoring and evaluation thereof, is mandated. In the African context, where resources are scarce, money for monitoring and evaluation should be selectively channelled into determining project relevance, effectiveness, efficiency and then only impact.

Introduction

African countries are characterised by low levels of financial literacy and high barriers to financial inclusion (Messy & Monticone 2012). A range of factors contribute to low levels of financial inclusion in Africa. These include supply side factors, such as restrictive terms and conditions on products and services, high fees and physical access; demand side factors, such as low income and low financial literacy; psychological factors and cultural barriers (Messy & Monticone 2012). While the extent to which financial education directly contributes to financial inclusion remains unproven, it is widely considered that appropriate financial education projects help to reduce demand side barriers to financial inclusion (Atkinson & Messy 2013).

There are a number of components to financial inclusion that should be considered when measuring the effectiveness ofa financial education project (Atkinson & Messy 2011). As illustrated in Figure 1, financial education involves a sequence of changes. At the outset, financial education is believed to lead to improvements in an individual’s knowledge and skills, and this in turn progresses to changes in the individual’s attitude and behaviour.

FIGURE 1: Trajectory of individual competencies contributing towards financial behaviour.

Currently, impetus is on measuring the impact of financial education projects by measuring changes in beneficiaries’ financial behaviour (Bayer, Bernheim & Scholz 2008; Bruhn, Ibarra & McKenzie 2014; Jamison, Karlan & Zinman 2014), and evaluating financial education projects is typically focused on determining the impact in order to justify the current funding or to obtain future funding (Lyons et al. 2006). This study argues that measuring only the behaviour change associated with a financial education project will not be sufficient in ensuring long-term financial inclusion and financial well-being. There is significant value in generating lessons around relevant project design, effective implementation and the achievement of immediate and intermediate financial education outcomes including gains in knowledge, skills and attitudes. Specifically, this study seeks to answer the following research questions:

  1. What learnings can be generated from the evaluation of financial education projects?

  2. To what extent can evaluators expect positive impact from financial education projects if the projects’ relevance, design and effective implementation have not been ascertained?

  3. How can different types of evaluations contribute to the ultimate achievement of impact?

This study is structured as follows:

  • Section 2 discusses the research methodology used to inform this article.
  • Section 3 focuses on why measuring impact in financial education projects matters, but why it should not be the only criterion that is assessed. A key point in this section is that a primary objective of evaluations should be to provide feedback into the design and operation of the project to reinforce its ability to achieve its objectives.
  • Section 4 explores the concept of the ‘life-cycle’ of an financial education project and introduces the notion that different types of evaluations, and hence different evaluation designs, are appropriate at different points in a project’s life-cycle. These various evaluation methodologies are then presented with a description of when each should be used.
  • Section 5 highlights the importance of process evaluations, which allow practitioners to track the progress of a project’s development and performance across its life-cycle and serve as a critical complement to impact evaluation particularly in the African context.

Research method and design

The methodology employed to develop this study consisted of three steps: A review of key literature on financial education and the evaluation thereof, focusing on the African, and specifically South African, context. The selection of literature for this review was based on the following four predetermined inclusion criteria:

  1. Intervention type: Only financial education–focused studies were considered.

  2. Geography: Studies with African-based financial education projects were prioritised.

  3. Date of intervention/publication: Studies/interventions published/implemented from January 2000 to April 2015 where prioritised.

  4. Publication language: Only studies published in English were considered.

The following three exclusion criteria were also used for the selection of literature:

  1. Studies that do not link directly to the financial education context with behaviour change outcomes.

  2. Studies that are not explicitly evaluation oriented.

  3. Studies in a non-African context.

A team synthesis session to reflect on and integrate learnings around implementing and evaluating financial educationprojects in South Africa was attended by four specialists in financial education in South Africa. Participation in this session was restricted to individuals with practical experience in implementing, monitoring and evaluating financial education projects across South Africa. The session followed a structured format; firstly, each participant presented theirexperiences evaluating financial education projects in South Africa, and secondly, a plenary discussion was facilitated maximising learnings from the evaluation of financial education projects.

Primary research case studies of financial education projects rolled out in South Africa were included to get practical insights into evaluating financial education projects.

The main limitation to undertaking this research was a lack of published work on financial education in South Africa and Africa more broadly. This is a relatively immature field in the African context, and thus, published articles, in particular academic articles, are not easily accessible.

Results

The future of good financial education in South Africa depends on learning from doing - evaluations must unpack more than the achievement of behaviour change.

Impact evaluations of financial education projects are widely undertaken to assess the changes accrued as a result of the project. This identifies whether the project is having any effect, which can be used to inform the following:

  • Ensuring value: Evidence that a project has a proven impact is critical in determining the value of investment, public funds or funds from private sector financial institutions as is the case in South Africa. If the intervention is not shown to have a positive impact, it is most likely not addressing a real need and not implemented appropriately, thus indicating that either the project should be modified or the money should be spent elsewhere.
  • Impact is important for policy design: Beyond the realm of the funders, implementers and beneficiaries, a project’s ability to have an impact has important implications for policy design. If a particular strategy or intervention can be shown to positively affect a particular population group it may result in a policy change, such as the government encouraging or binding a specific sector to invest in similar initiatives.
  • Justifying scaling up: If a project can be shown to have a positive impact, it allows implementers to makean argument for increasing the size and reach of the project. Naturally, other factors also need to be considered, such as the scalability of the model and budget, but if the project is not having an impact, there is no case to be made for the project budget and reach to be extended.

Having identified why establishing impact of a project is important, experience from the research shows that impact is likely only achieved if a project was well-designed, identified relevant messaging and delivery channels for the target audience and was implemented effectively and to a particular quality standard. Therefore, in this nascent field of financial education, evaluation should seek to confirm the theory, relevance and quality of interventions to ascertain that impact is even possible to expect in a given project context. Learning from past projects is critical to ensuring that future projects are designed appropriately for the intended audience, social and economic environment and the behavioural nuances that each context presents. Focusing resources on proving the impact of financial education does not contribute significantly to the financial education knowledge base in Africa. Rather, this knowledge base is more effectively and efficiently expanded through evaluations that generate learnings around projects’ relevance, effectiveness, efficiency and sustainability and then only impact (OECD 1991; 2002). This is particularly critical in the South African context where all financial institutions are mandated to implement financial education projects by the regulator and would benefit more substantially from lessons learnt about practical application issues including potential best practices and challenges encountered (Box 1).

BOX 1: Measuring financial education in South Africa.

Evaluations can allow implementers to learn about what works and what does not in their context and to establish best practices that can improve the implementation of existing projects and inform the design of new ones. Impact evaluations do not provide a holistic understanding of projects’ implementation, nor do they provide detailed information into the relevance, quality and appropriateness of the project for a given context.

Understanding contextual diversities has been identified as a financial education best practice as social and cultural idiosyncrasies play a large role in project success (Finmark Trust 2013). Evaluations provide insight into the contextual nuances of a project and whether a project that has been proven to be effective in a given context is likely to be transferrable to another context. Given the contextual nuances across South Africa, the stratified population, the sophistication of the financial sector and the immaturity of the South African financial education landscape, copying a project from a given context to another context is not always appropriate. Evaluations should help to identify how and where projects can be copied into a different context.

Finally, financial education is intended to improve beneficiaries’ financial capability, and ultimately financial inclusion. While there are no standardised definitions of financial capability and financial inclusion, financial capability is generally understood to refer to an individual’s financial knowledge and skills to make good financial decisions and financial inclusion is understood to consist of two elements: good financial decision-making (the ‘demand side’) and access to suitable products and services (the ‘supply side’) (Milton 2008). Financial inclusion is a multi-faceted state, influenced by a number of pillars including employment, sustainable income and control over personal finances. As such, financial education alone cannot drive significant changes in beneficiaries’ financial behaviour or financial status. However, many projects mistakenly evaluate financial inclusion indicators when determining the financial education intervention’s impact, despite the fact that the intervention likely only addressed knowledge, skills and attitudes regarding financial literacy and planning issues. This ultimately results in the misconception that financial education is ineffective. The impetus for measuring changes in beneficiaries’ knowledge rather than behaviour is illustrated by the South African Financial Services Board’s Consumer Education Department’s emphasis on measuring knowledge- and skills-oriented indicators within their own projects, rather than behaviour change.

Evaluation methodologies, project cycle and maturity

Different types of evaluations serve different purposes throughout the life of a project. In order to elicit the desired information from an evaluation, not only does the right type of evaluation need to be implemented, it must be applied at the right point in the ‘life-cycle’ of a project.

Many evaluations yield poor results or fail to detect the positive contribution of a project as a result of being conducted at the wrong stage. Given the discourse, project owners too often now expect projects to be evaluated using rigorous impact methodologies at too early a stage in their development, for example pilot interventions that are used to test material and delivery channel for a particular target audience.

Figure 2 illustrates the various types of evaluation, as outlined by the South African National Evaluation Policy Framework (DPME 2011), that are relevant at each stage of a financial education intervention life-cycle. While the spectrum of potential evaluation types is broader than what has been defined below, the evaluations defined in Figure 2 are those that have been identified by the South African government and more specifically are relevant for financial education projects in South Africa. These are elaborated on in the text that follows.

FIGURE 2: Project life-cycle and evaluation type.

A diagnostic evaluation is the preparatory research that informs about the status quo prior to a financial education project; this should be used to inform the financial education project design and as such should be carried out prior to the design or planning of the financial education project. This evaluation identifies the problem that the project is intended to address, the causes of the problem and the consequences thereof; furthermore, this evaluation identifies the likely effectiveness of the project (DPME 2011). This is a crucial step in the evaluation continuum as it highlights issues, specifically context-specific issues, which could arise during the implementation of the project which can then be mitigated in the design phase.

Based on the diagnostic evaluation, the theory of change should be drawn up if it has not already been drawn up, or should be reviewed if there is an existing one in place. A design evaluation can be used to analyse this theory of change, through assessing the logic and consistency of the project. This type of evaluation can be carried out before the project starts or in early implementation stages. It is a relatively quick evaluation to carry out and will further inform the quality of the indicators used to measure the project and the assumptions on which the project is based (DPME 2011). Although this is a quick process to carry out, it is an important step as it further identifies potential issues which otherwise would not be identified. This enables the project designer to mitigate these issues as they are identified at an early stage in the project.

Once the project is underway, an implementation evaluation should be carried out to determine whether the project’s ‘operational mechanisms support achievement of the objectives’ (DPME 2011:9). This evaluation explores the inputs, the activities and the subsequent outputs and outcomes to analyse the operational processes and how the efficiency and efficacy of these can be improved. This evaluation enables the project implementers to correct operational inefficiencies before these have a detrimental effect on the outcome of the project. This further informs the extent to which the project is likely to achieve its objectives and allows for adjustments to the project such that these can be achieved.

The above evaluations inform project conception and implementation planning with guidance on the most suitable design of a particular financial education intervention and how best to implement it. Once this has been established, an impact evaluation can then be considered, establishing the extent to which changes in the target group’s financial behaviours are attributable to the financial education project. The above evaluations are necessary to ensure that the impact and attribution of the project can be accurately measured; otherwise, adverse project design and implementation effects may influence the measurement of attribution and impact. Although an impact evaluation is only conducted at key milestones, or the completion of an intervention, it needs to be planned early in the project’s design such that a baseline can be established.

It is not necessary to undertake each of these evaluations at each point in your project’s life-cycle. The selection of the appropriate evaluation type will depend on the future of the project and the intended use of the evaluation. One needs to give due consideration to the purpose of the evaluation and selection of the appropriate evaluation type to achieve the learning objective. Projects are typically given separate funding windows for each stage in their life-cycle. As such, a portion of this should always be set aside for monitoring and evaluation such that projects do not progress to the next stage in the life-cycle before the relevant evaluation has been undertaken to ensure that the project is designed and implemented appropriately to achieve its intended impact. Finally, impact evaluation should be considered when stakeholders are sufficiently confident that a project’s relevance, theory and quality will lead to the expected impact – thus lending to expanded scope and scaling.

Discussion

Implementation evaluations for informing improved programming

At earlier stages of a project’s life-cycle, evaluations should primarily focus on the suitability of the project’s design and the theoretical ability to impact its beneficiaries positively, in addition to whether it is doing that.

An advantage of conducting implementation evaluations during earlier stages of a project’s life-cycle is that they allow for a holistic investigation into the project’s performance. Data required for implementation evaluations should be both qualitative (e.g. diaries, interviews, focus group discussions) and quantitative (e.g. indicators measuring knowledge about financial concepts, comfort knowing where to go for recourse, savings, and other MIS data). Evaluations can focus on indicators that assess the overall functionality of the project itself. The OECD’s Development Assistance Committee (DAC) Criteria for Evaluating Development Assistance (OECD 1991), for example, provide a useful framework for this type of holistic approach, encouraging investigation into the project’s Relevance, Effectiveness, Efficiency and Sustainability, in addition to Impact.

Implementation evaluations can be seen as an investigative tool that provides real-time insight into the effective functioning of the project, if effective at all. They can serve as an early warning system, allowing implementers to address operational issues as they arise and provide information for the constant improvement of the project’s design and are ultimately essential in order to maximise chances of seeing results.

Implementation evaluations are particularly valuable when piloting new financial education projects. As noted above, the social and cultural idiosyncrasies of a context should be considered when designing financial education projects. As such, South Africa’s diversity and cultural nuances present many challenges to replicating projects from both international and domestic contexts. As such, a common practice in the South African context is to run a pilot project before rolling the project out in full scale, thus enabling evaluators to identify necessary changes to the project design to ensure its effectiveness. There are a number of examples of this practice, including:

  • The Imali Matters Pilot Project run by the Department of Trade and Industry, Finmark Trust, African Bank and the Credit Information Ombudsman, which delivered face-to-face consumer counselling at walk-in centres across the three main cities in South Africa (Eighty 20 Consulting 2011).
  • Association for Savings & Investment SA (ASISA) Foundation’s pilot project, SaverWayaWaya, implemented in Hammanskraal, Gauteng, is discussed in Box 2.
BOX 2: Learning from a pilot.

The ASISA Foundation, established to deliver effective Consumer Financial Education projects nationally on behalf of its members, insisted on incorporating rigorous M&E alongside the 2014 pilot implementation project that would inform the Foundation and its members about what works and what does not. The 2014 evaluation assessed the five DAC criteria (described above) in an attempt to uncover success factors for financial education projects. The particular project specifically sought to investigate the use of a ‘life stages’ approach as well as delivery methods comparing workshops versus workshops incorporating industrial theatre.

The ASISA Foundation staff and the industry steering committee were very interested in the findings of the 2014 evaluation. The findings from the evaluation of the pilot were used to design the 2015 project and have informed a more relevant and targeted project.

Ultimately, impact evaluations should only be conducted when there is confidence that a project is nearly perfect in its implementation (e.g. quality of material, appropriate messaging, accessible and relevant delivery channels). The most rigorous type of impact evaluation, randomised control trials, are thus ultimately most useful once a project has reached a certain level of maturity (Chambers et al. 2009; Donaldson & Christie 2004). There should be sufficient indication which suggests that the project does have a positive impact, and the randomised control trial should serve not only to test this on a more rigorous level but also to provide evidence in support of specific actions, such as scaling up, informing policy or replication.

Evaluating embedded financial education projects

There has been an increased focus on embedded financial education among formal and informal financial service providers who hope to improve their customers’ financial capability and ability to make informed financial decisions. These initiatives aim to pass on financial messages to beneficiaries at points of contact in such a way that there is a seamless integration between an individual’s interactions with the financial service provider and the financial education they receive. When evaluating these initiatives, this embeddedness makes it considerably more difficult to attribute particular changes in beneficiaries’ knowledge, skills, attitudes and particularly behaviour, to either the financial education or the responsible provision of financial services. We should rather take a holistic view of projects like this, focusing solely on whether the desired impact is achieved, regardless of the relative contribution of the financial education or financial service. What is therefore critical is that the potential for harmony and reinforcement between the two is maximised.

Conclusion

Generating lessons around project relevance, design and quality is critical given the current importance placed on financial education in South Africa. Given the heterogeneous nature of the South African context, forming a knowledge base that includes learnings around contextual nuances, and how to best address these, is fundamental to ensuring that projects are implemented in a manner that will likely lead to the greatest impact. In order to elicit this information and to ensure that the objectives of the project are met, different types of evaluations must be considered and conducted throughout the project life-cycle. Furthermore, not only does the right type of evaluation need to be implemented but it must also be initiated at the right point in the ‘life-cycle’ of the project and take into account the additional factors that contribute to an individual’s financial well-being. Many evaluations yield poor results or fail to detect the positive contribution of a project as a result of being implemented at the wrong stage. Most often, projects are evaluated with rigorous methodologies at too early a stage in their development or have focused on inappropriate evaluation questions given an intervention’s implementation stage, maturity, design and context.

The ability to demonstrate a project’s impact is important to ensure the continuation of project funding, for justifying scaling up, to ensure value for money and for policy design. However, we should only take hard and fast decisions on impact once there is confidence in the design and implementation.

Acknowledgements

Competing interests

The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.

Authors’ contributions

E.M. was the primary drafter and researcher. She was responsible for reviewing past literature on the topic and synthesising the team’s experiences of financial education in South Africa. A.W. was responsible for providing strategic insight to the document and guided the team’s review process given her international expertise in financial education. C.S. was responsible for drafting the case studies and for providing strategic guidance based on her practical experience working with financial education in South Africa.

References

ASISA Foundation, 2014, Evaluation of the ASISA Foundation pilot consumer education project in Hammanskraal, ASISA Foundation.

Atkinson, A. & Messy, F., 2011, ‘Assessing Financial Literacy in 12 Countries: An OECD/INFE International Pilot Exercise’, Journal of Pension Economics and Finance 10, 657–665. http://dx.doi.org/10.1017/S1474747211000539

Atkinson, A. & Messy, F., 2013, Promoting financial inclusion through financial education, OECD/INFE evidence, policies and practice, OECD Working Papers on Finance, Insurance and Private Pensions, No. 34, OECD Publishing, Paris, viewed 15 May 2015, from http://dx.doi.org/10.1787/5k3xz6m88smp-en

Bayer, P.J., Bernheim, B.D. & Scholz, J.K., 2008, ‘The effects of financial education in the workplace: Evidence from a survey of employers’, Economic Inquiry 47, 605–624.

Broad-Based Black Economic Empowerment Act, No. 53 of 2003, 2003, Government Gazzette, 569(35914), 26 November, Government notice no. 997, Government Printer, Pretoria.

Bruhn, M., Ibarra, G.L. & McKenzie, D., 2014, ‘The minimal impact of a large-scale financial education program in Mexico City’, Journal of Development Economics, 108, 184–189.

Chambers, R., Karlan, D., Ravallion, M. & Rogers, P., 2009, Designing impact evaluations: Different perspectives, International Initiative for Impact Evaluation Working Paper 4, International Initiative for Impact Evaluation (3ie), New Delhi.

Department of Performance Monitoring and Evaluation, 2011, National Evaluation Policy Framework, DPME Publications, Pretoria.

Donaldson, S. & Christie, C., 2004, ‘Determining causality in program evaluation and applied research: Should experimental evidence be the gold standard?’, Journal of MultiDisciplinary Evaluation 3, 60–77.

Eighty 20 Consulting, 2011, Imali matters – Final report. Unpublished.

Financial Education Fund, 2010, ‘Monitoring and evaluation toolkit for financial education initiatives’, viewed 24 February 2015, from http://carefinedportal.weebly.com/uploads/2/0/5/5/20550716/8._fef_me_toolkit__dec_31_2010.pdf

Finmark Trust, 2013, ‘Financial education Toolkit’, viewed 24 February 2015, from http://www.finmark.org.za/wp-content/uploads/pubs/FEtoolkit_unabridged_Developing-effectiveFEstrat_SaveActEvaluation_FNL_June2013.pdf

Jamison, J.C., Karlan, D. & Zinman, J., 2014, ‘Financial education and access to savings accounts: Complements or substitutes? Evidence from Ugandan youth clubs’, NBER Working Paper 20135.

Lyons, A., Chang, Y. & Scherpf, E. 2006, ‘Translating Financial Education into Behavior Change for Low-Income Populations’, Financial Counseling and Planning 17, 27–45.

Messy, F. & Monticone, C., 2012, The status of financial education in Africa, Organisation for Economic Co-Operation and Development (OECD) Working Papers on Finance, Insurance and Private Pensions, No. 25, OECD Publishing, Paris.

Milton, L., 2008, ‘Financial inclusion in the UK: Review of policy and practice’, Joseph Rowntree Foundation, New York, viewed 5 May 2015, from http://www.jrf.org.uk/sites/files/jrf/2234.pdf

OECD, 1991, DAC principles for evaluation of development assistance, OCED Publishing, Paris.

OECD, 2002, Glossary of key terms in evaluation and results based management, OCED Publishing, Paris.

OECD, 2010, Detailed guide to evaluating financial education programs, OECD Publishing, Paris.


 

Crossref Citations

1. Financial education for HIV‐vulnerable youth, orphans, and vulnerable children: A systematic review of outcome evidence
Nanci Lee, Sabrina Beeler Stücklin, Patricia Lopez Rodriguez, Meryem El Alaoui Faris, Ida Mukaka
Campbell Systematic Reviews  vol: 16  issue: 1  year: 2020  
doi: 10.1002/cl2.1071