Evaluating a Financial Literacy Program: the Case of the Australian MoneyMinded Program - PowerPoint PPT Presentation


PPT – Evaluating a Financial Literacy Program: the Case of the Australian MoneyMinded Program PowerPoint presentation | free to download - id: 1e2edc-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Evaluating a Financial Literacy Program: the Case of the Australian MoneyMinded Program


Everyday Banking and Financial Products. Rights and Responsibilities ... ANZ plans to partner with around 100 community organisations during the next 5 ... – PowerPoint PPT presentation

Number of Views:46
Avg rating:3.0/5.0
Slides: 38
Provided by: Home1720
Learn more at: http://mams.rmit.edu.au


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Evaluating a Financial Literacy Program: the Case of the Australian MoneyMinded Program

Evaluating a Financial Literacy Program the Case
of the Australian MoneyMinded Program
  • Rob Brooks
  • Monash University

Presentation Structure
  • Background / Context
  • Objectives
  • Overview of evaluation framework
  • Evaluation of the MoneyMinded program in the
    context of the framework
  • Conclusion

Background / Context
  • The promotion of greater levels of financial
    inclusion is now seen to be a major public policy
    and corporate initiative across a variety of
  • Core elements of programs to promote greater
    financial inclusion include matched savings
    programs and financial literacy education

Background / Context
  • United States American Dream Demonstration
  • United Kingdom Saving Gateway
  • Australia Saver Plus
  • Need for effective evaluation of programs in
    their role in improving financial inclusion
  • Fox, Bartholomae and Lee (2005)
  • To date, no single framework to guide the
    evaluation process

Background / Context
  • Fox, Bartholomae and Lee extend the framework of
    Jacobs (1988) to develop a 5 tiered evaluation
    framework against which evaluations of financial
    literacy education programs can be conducted

  • To explore this evaluation framework in the
    context of a financial literacy program designed
    and developed in a different country,
    specifically the MoneyMinded financial literacy
    program, offered in Australia
  • Extension of work of Fox, Bartholomae and Lee
  • Provides information on the generalisability of
  • Results will be of interest to a range of
    countries given comparable initiatives relating
    to financial inclusion across various countries

The Evaluation Framework
  • 5 tiers, based on the work of Jacobs (1988)
  • Pre-implementation
  • Accountability
  • Program clarification
  • Progress towards objectives
  • Program Impact

  • Involves a needs assessment on the financial
    literacy training required by participants
  • Needs identified from more general financial
    literacy testing
  • Specific needs assessment around program

  • Involves the collection of data on the education
    and services provided as part of the program, the
    costs involved and demographic information on

Program Clarification
  • Involves ongoing assessment of the program's
    strengths and weaknesses, and plans to review the
    program in the light of such assessments

Progress Towards Objectives
  • Involves the collection of data on the impacts of
    the program on participants

Program Impact
  • Builds on the previous stage in terms of long and
    short term impacts, typically involving
    comparison of participants and non-participants

The MoneyMinded Program
  • General objective To assist people to make
    informed financial decisions and take control of
    their finances for their future
  • 6 topics separated into 17 workshops
  • Planning and Saving
  • Easy Payments
  • Understanding Paperwork
  • Living with Debt
  • Everyday Banking and Financial Products
  • Rights and Responsibilities
  • (original version which has now been revised)

The MoneyMinded Program
  • Allows individual participants to choose which
    workshops they wish to complete
  • Developed in part as a response to the results
    from the first-ever major survey of consumer
    financial literacy in Australia (ANZ, 2003)
  • Results of the survey identified the following
  • The groups within the community that could
    benefit from improved financial literacy
  • The issues that could be addressed in such an
    education program
  • Involvement of community groups in the delivery
    of program

The MoneyMinded Program
  • ANZ plans to partner with around 100 community
    organisations during the next 5 years to deliver
    the program to 100,000 people nationally
  • Evaluation is critical for a program as
    comprehensive and extensive as MoneyMinded
  • Importance of evaluation being collected and
    analysed in a systematic manner

Evaluating the Money Minded Program
  • Pre-implementation needs assessment
  • Not a formal needs analysis at the individual
    participant level
  • Based on the results of the financial literacy
    survey reported in ANZ (2003)
  • Telephone survey customised to particular needs
    and circumstances

Pre-implementation Stage (Contd)
  • ANZ survey identified the following groups were
    strongly over-represented in the lowest financial
    literacy quintile
  • Lowest education levels
  • Not working or in unskilled work
  • Lower income levels
  • Lower savings levels
  • Single
  • Extreme ends of the age profile
  • Females

ANZ financial literacy surveys in 2003 and 2005
Groups over-represented in the lowest financial
literacy quintiles
Group 2003 Survey () 2005 Survey ()
Females 24 25
Education Less than Year 10 42 43
Looking for work 32 31
Semi-skilled 28 34
Unskilled 40 38
Single Living Alone 26 26
Aged 18 24 31 33
Aged 70 31 37
Renting 29 27
Pre-implementation Stage (Contd)
  • The involvement of community groups in
    facilitating delivery of the MoneyMinded program
    has assisted in targeting the over-represented
  • Community group involvement has also proven to be
    successful in the recruitment of participants for
    the Saver Plus matched savings program
  • Needs assessment through administration of a
    pre-training questionnaire, which captured
    demographic information and data on the areas
    that participants identified where they most
    needed financial literacy education in terms of
    the modules offered as part of the MoneyMinded

MoneyMinded modules identified by participants as
being areas where they had an interest in
acquiring greater financial knowledge
Topic Area Number of Interested Participants of Total Participants
Planning and saving 101 71.1
Living with and managing your debt 79 55.6
Your consumer rights and responsibilities 68 47.9
Understanding financial paperwork 68 47.9
Your everyday banking and financial products 61 43.0
Your payment options 51 35.9
Pre-implementation Stage (Contd)
  • Together with the fact that the MoneyMinded
    workshop is composed of individual workshops the
    participation in which is chosen at the
    individual level suggests that the design
    performs well on the criteria relevant to the
    pre-implementation stage

Accountability Stage
  • Requires that details be collected on a range of
    matters including the education and services
    provided and base participant information
  • Basic participant information
  • Pre-training questionnaire collected data on
    demographics, mathematical and computer literacy,
    Internet usage, current savings and spending
    behaviour, savings intentions and current usage
    of a range of financial products
  • Pre-training questionnaire also captured data on
    the financial literacy of participants

Socio-demographic Characteristics of MoneyMinded
Number of Participants of Total
Male 19 13.4
Female 123 86.6
15 34 34 24.3
35 44 63 45.0
45 54 29 20.7
55 14 10.0
Primary / Some Secondary 30 22.8
All Secondary 24 18.2
TAFE / Workplace Training 32 24.2
University 46 32.3
Employment Status
Full time 50 35.4
Part time 45 31.9
Casual 10 7.1
Unemployed / Not looking 10 7.1
Student 1 07
Home duties 15 10.6
Retired 10 7.1
Accountability Stage (Contd)
  • The collection of such data provides the
    potential for a rich analysis to inform the
    program clarification, progress towards
    objectives and program outcome stages of any
    evaluation by allowing an exploration of the
    influences of these demographic and financial

Program Clarification Stage
  • Requires an assessment of the program strengths
    and weaknesses, and the use of such information
    in any re-assessment of program goals and
  • This element is present in the MoneyMinded
  • Initial roll-out to a small number of community
    organisations and participants allows for the
    type of assessment and re-thinking that is at the
    core of the program clarification stage

Program Clarification Stage (Contd)
  • The MoneyMinded evaluation also involved the
    administration of post-training questionnaires to
  • Required participants to provide an assessment of
    each workshop completed in terms of how useful
    they found the workshop to be, how satisfied they
    were with the topics and material covered in the
    workshop and an assessment of overall program
  • Post-training evaluation data when coupled with
    the pre-training data provides scope for a rich
    analysis at the program clarification stage
  • High overall ratings for both satisfaction (90
    reported being very satisfied or satisfied with
    the overall program) and usefulness (87 found
    the workshops to be extremely useful or useful)

Program Clarification Stage (Contd)
  • A richer analysis for the program clarification
    is provided by considering whether there are
    differences in the satisfaction and usefulness
    ratings across categories in the demographic and
    financial data collected in the pre-training
  • There were statistically significant differences
    in terms of usefulness and satisfaction ratings
    across participant categories
  • Higher ratings were provided by those in
    categories that captured lower incomes, education
    levels, employment status, mathematical and
    computer literacy, savings levels and intentions
    and base levels of financial literacy

Group ?2 d.f. ?2 value p-value
Program Satisfaction Program Satisfaction Program Satisfaction Program Satisfaction
Income Source 2 5.282 0.071
Employment Status 4 11.692 0.020
Income Level 6 12.823 0.046
Savings Intentions 6 12.634 0.049
Savings Level 6 4.756 0.576
Spending Behaviour 6 7.699 0.261
Understanding Bills 2 1.641 0.440
Credit card usage 2 5.700 0.058
Workshop usefulness Workshop usefulness Workshop usefulness Workshop usefulness
Income Source 2 14.345 0.001
Employment Status 4 29.169 0.000
Income Level 12 51.544 0.000
Savings Intentions 6 30.733 0.000
Savings Level 6 29.656 0.000
Spending Behaviour 6 74.844 0.000
Understanding Bills 6 48.953 0.000
Credit card usage 6 23.757 0.001
Workshop Satisfaction Workshop Satisfaction Workshop Satisfaction Workshop Satisfaction
Income Source 2 6.210 0.045
Employment Status 4 26.590 0.000
Income Level 12 22.480 0.032
Savings Intentions 6 30.453 0.000
Savings Level 6 29.337 0.000
Spending Behaviour 6 37.084 0.000
Understanding Bills 6 44.261 0.000
Credit card usage 6 32.603 0.000
Selection of the results on whether the ratings
of satisfaction and usefulness provided by
participants vary across the socio-demographic,
financial product usage and financial literacy
Program Clarification Stage (Contd)
  • Appropriate statistical analysis can be
    beneficial at the program clarification stage of
    a structured evaluation process
  • Bulk of data collected in the MoneyMinded
    evaluation was categorical in nature
  • Statistical testing was conducted using the ?2
    testing framework appropriate to such data
  • Obvious issue in any formal statistical testing
    is whether the data collected satisfies the
    assumptions required of the statistical procedure
    being used
  • Expected cell count of 5 for the asymptotic ?2
    approximation to work
  • Requirement not satisfied in the initial data
  • 2 approaches available 1)aggregate cells
    together and then conduct the tests 2) conduct a
    simulation analysis of the exact distribution of
    the test statistic typically involving some Monte
    Carlo technique

Program Clarification Stage (Contd)
  • Post-training questionnaire also provided useful
    data on some dimensions of program delivery
  • Website of resources made available to
  • 92 of participants did not access the website
  • Strong result which suggests use of technology in
    delivery should be assessed, and not just assumed
    in the design features
  • Also captured data on the number of individual
    workshops that participants attended majority
    attended only a single workshop, with a small
    number attending 12 workshops

Program Clarification Stage (Contd)
  • Results show that participants who attended
    multiple workshops had higher usefulness and
    satisfaction ratings than those who attended a
    single workshop differences in the ratings are
    statistically significant
  • Suggests consideration be given in the program
    design to encouraging attendance at multiple
    workshops via greater bundling, while recognising
    that workshops are already bundled by broad topic
  • Useful and constructive information to obtain at
    the program clarification stage of an evaluation

Program Clarification Stage (Contd)
  • Assessment of MoneyMinded also involved
    completion of questionnaires by facilitators who
    delivered the program, in terms of their expected
    usefulness of the workshops prior to delivery,
    and their perceptions of participant usefulness
  • Ratings were high and were broadly consistent
    with the ratings provided by participants
  • Useful data at the program clarification stage in
    informing how the program goes forward

Program Clarification Stage (Contd)
  • Final data collected of relevance to the program
    clarification stage was questionnaire data from
    both participants and facilitators in regard to
    improvements that could be made to the program in
    terms of expanding the topic coverage
  • Need for workshops on investment, superannuation
    and retirement planning
  • To be expected ties in with government policy
    in recent years to move funding of retirement
    incomes to private funding via long-term
    superannuation investments

Progress Towards Objectives Stage
  • Objective of a financial literacy education
    program is in terms of improved financial
    literacy, and as a consequence better financial
    decision-making by participants
  • In the context of an initial evaluation of the
    roll-out of what is to become a wider program,
    such a stage is difficult to effectively action,
    except in terms of interpretation of ratings of
    satisfaction and usefulness that have been
    previously utilised at a program evaluation stage

Program Impact Stage
  • Short and long-term
  • Looking for wider demonstrable evidence in terms
    of improved financial literacy and better
    financial decision making by those who have
    participated in financial literacy programs
  • Fox, Bartholomae and Lee (2005) recommend that
    this measurement of impact be done relative to a
    control group who have not participated in such
  • This stage is not relevant in the current
    assessment of MoneyMinded, given the program is
    only at an initial roll-out stage

Program Impact Stage (Contd)
  • How such an evaluation might be built for use in
    the future
  • Future broad surveys of financial literacy to
    include questions for the respondents on whether
    they have previously participated in a financial
    literacy education program
  • Allows us to compare measures of financial
    literacy across groups that have/have not
    undertaken financial literacy education, after
    making appropriate adjustments for other
    socio-economic and demographic factors known to
    influence financial literacy
  • Longer-term analysis of the savings behaviour of
    participants in a matched savings program would
    provide evidence of the joint impacts on
    financial decision-making of participation in
    both a matched savings program and financial
    literacy education

  • Exploration of the Fox, Bartholomae and Lee
    (2005) framework in the context of a specific
    program, the Australian MoneyMinded program
  • Framework developed in the context of US programs
    appears to generalise well to the Australian
    context, thus providing support to the general
    applicability of the framework
  • Formal statistical evaluation might be used at
    certain stages
  • Measurements of longer-term impacts i.e.
    behaviour change
About PowerShow.com