Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois - PowerPoint PPT Presentation

1 / 33
About This Presentation
Title:

Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois

Description:

Reluctance to divulge personal information (surveys 'too personal'; lack of trust) ... Train-the-Trainer. Testing Knowledge. Building Skills. Taking Charge. Manual ... – PowerPoint PPT presentation

Number of Views:88
Avg rating:3.0/5.0
Slides: 34
Provided by: jlh8
Category:

less

Transcript and Presenter's Notes

Title: Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois


1
Some Practical Tips for Measuring Financial
Success Dr. Angela LyonsUniversity of Illinois
Evaluating Financial Education in Troubled
Times Dr. Lance Palmer University of
Georgia Dr. Angela Lyons University of Illinois
Urbana Champaign AFCPE Extension
Pre-Conference November 2008
2
MotivationRecent Financial Challenges
  • Difficulty paying debts
  • Low saving rates
  • Rising home foreclosures
  • Mounting bankruptcies
  • Higher unemployment
  • An increased awareness and impact of natural
    disasters

Consumers are in financial trouble!!!
3
The Result.
  • A number of financial education efforts have been
    developed to address consumers financial needs
    in these troubled times.
  • 3 key programming areas
  • Bankruptcy
  • Homeownership issues
  • Natural Disasters

4
  • At the end of the day, are these
    financial education efforts making a
    difference?

5
On the front lines with the evaluation experts.
  • What do we mean by evaluation?
  • What are we really trying to measure?
  • How do we define program success?
  • How do we know if participants are improving?
  • What financial outcomes and indicators should
    we be using?
  • What constitutes a successful, or even
    acceptable, evaluation?

6
The Big Picture 5 Biggest Evaluation
Challenges in General
  • 1. Identifying the ideal approach to
    evaluation.
  • Evaluation methods and measures vary widely
    across programs and academic disciplines.
  • Wide variation in financial outcomes across
    programs.
  • Significant differences in financial needs
    across
  • individuals and families.
  • Some participants unable to
  • implement certain financial behaviors.
  • Shift towards more one-on-one individualized
    education.

7
  • 2. Defining program success.
  • Setting realistic expectations for program
    participants.
  • Choosing appropriate outcomes and indicators
    based on participants financial situation or
    other external constraints.
  • Identifying participants individual financial
    desires and applying appropriate educational
    interventions and evaluation.
  • Finding the teachable moment or motivated
    learner.
  • Recognizing that one size does not fit all.

8
  • 3. Collecting impact data from
    program participants.
  • Little incentive to complete evaluations (like
    pulling teeth).
  • Reluctance to divulge personal information
    (surveys too personal lack of trust).
  • High drop out rates, low response rates, and
    difficult to track.
  • Literacy levels (i.e., ESL, reading level).
  • Collecting sensitive data and information.
  • Tradeoff between participation and evaluation
    rigor.

9
  • 4. Designing and implementing effective program
    evaluations.
  • Evaluation process is cumbersome.
  • Lack of time, staff, and financial resources.
  • The PUSH for increased rigor and the rush to
    the finish line.
  • Rigor vs. Reality (e.g., measurement issues)
  • The limitations of one-shot evaluations.
  • (pre- and post-tests intended vs. actual
    behavior change)
  • The reality of conducting longitudinal studies
    with control groups.
  • (follow-ups and tracking of program
    participants)

10
  • 5. Data analysis and measurement issues
  • Measurement error and validity of indicators.
  • Self-reports are subject to bias.
  • Social desirability
  • Norms and rules of thumb
  • Misperceptions and over-optimism
  • Memory distortion and recall bias
  • Samples may not be representative.
  • Non-response bias
  • Program attrition
  • Self-selection
  • Low response rates (e.g., follow-ups)

11
Other measurement issues
  • Environmental factors may affect outcomes.
  • Unexpected life events
  • Program incentives (e.g., rewards, special
    benefits, enrollment programs)
  • Individualized financial advice or coaching
  • Psychological factors.
  • Inherent motivation
  • Ability
  • Attitudes
  • Stress

12
Bankruptcy Counseling and Education
  • Educational impacts are difficult to disentangle
    from the actual bankruptcy process.
  • Difficult for debtors to change certain behaviors
    when havent yet discharged debts and still under
    court supervision.
  • Conducting a follow-up is critical.
  • Dealing with a very heterogeneous population that
    is required to complete the counseling and
    education.

13
Bankruptcy Counseling and Education Contd
  • Content of the counseling and education is
    similar which can reduce the positive impact of
    the debtor education.
  • Selecting indicators that are independent of the
    debtors current financial situation and the
    bankruptcy discharge.
  • Constraints associated with curriculum content
    and timing of delivery.

14
Homeownership programs
  • Many different types of housing counseling and
    education programs
  • Transitional housing programs
  • Homeownership education
  • Post purchase counseling
  • Default and delinquency counseling
  • Reverse mortgage counseling
  • Less emphasis on group education and more focus
    on one-on-one models of financial counseling or
    financial coaching.
  • Significant grant dollars are available to
    conduct these programs, but evaluations can be
    constrained by funders requirements.

15
Post-Natural Disaster Programs
  • Using traditional pre- and post-test surveys may
    be less effective than more qualitative methods
    such as focus groups, interviews, and case
    studies.
  • Informational and financial needs may vary widely
    across disaster victims.
  • Psychological factors are likely to significantly
    affect motivation, attitudes, and perceived
    abilities which will influence evaluation
    responses.
  • Disaster victims can be very transient
    difficult to track and conduct follow-ups.
  • Very time and resource intensive.

16
Overcoming the ChallengesPlanning, planning, and
more planning.
  • Take stock of who you are What is your
    vision?
  • Be strategic What are you trying to accomplish?
  • Conduct a thorough needs assessment.
  • Spend time understanding your target audience(s).
  • Define program success for your audience. Be
    realistic!
  • Create an evaluation action plan
  • Identify resources and overcome constraints.
  • Take Action!

17
4 Steps to Creating An Effective Evaluation
Action Plan
  • Step 1 Define objectives of the program.
  • At the end of the day, what do you want to show?
  • Who will be the target audience?
  • What will be the primary delivery method?
  • Who will use the evaluation and how?
  • Step 2 Select appropriate methods for data
    collection.
  • What is the most appropriate evaluation format?
  • What types of questions will the evaluation seek
    to answer?
  • What types of indicators will be used to show
    impact?
  • Are these indicators appropriate for the target
    audience?

18
  • Step 3 Identify and overcome evaluation
    challenges.
  • What are your biggest implementation challenges?
  • What steps can you take to overcome the
    challenges?
  • What financial and non-financial resources are
    available?
  • Are there others who can help?
  • What is the program timeline?
  • Given constraints, what can you realistically
    do?
  • Step 4 Analyze and disseminate findings.
  • How will data be analyzed?
  • What do you hope to learn from findings?
  • What are the potential impacts?
  • How will the results be used and disseminated?

19
Where do we go from here?Online resources at
your fingertips
20
NEFE Financial Education Evaluation Toolkit
http//www2.nefe.org/eval/intro.html
21
NEFE Financial Education Evaluation Toolkit
  • Database
  • Post evaluation only with option for follow-up
  • Pre and post evaluation with option for follow-up
  • Stages to Change Evaluation
  • Train-the-Trainer
  • Testing Knowledge
  • Building Skills
  • Taking Charge
  • Manual
  • How-to-guide for grass-roots level organizations
  • Examples (survey instruments, executive summary,
    reports)
  • Guidance on how to organize and present impact
    data

22
Manual http//www2.nefe.org/eval/manual.html Par
t I Financial Education Overview Part II
Understanding Program Evaluation Part III The
Evaluation Planning Process Part IV Using the
Evaluation Database Part V Reporting Program
Impact
23
Appendix Sample Evaluation Instruments
24
Database http//www2.nefe.org/eval/index.php
25
Knowledge Indicators
26
Behavior Indicators
27
Qualitative Info
28
Demographics
29
University of Wisconsin-Extension http//www.uwex
.edu/ces/pdande/evaluation/index.html
30
Cornell University Extension http//staff.cce.cor
nell.edu/administration/program/evaluation/evalref
s.htm
31
Penn State Extension http//www.extension.psu.edu
/evaluation/
32
Contact Information
  • Dr. Lance Palmer
  • Assistant Professor
  • Department of Housing and Consumer Economics
  • University of Georgia
  • Phone (706) 542-4916
  • E-mail lpalmer_at_uga.edu

33
Questions
Write a Comment
User Comments (0)
About PowerShow.com