Title: Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois
1Some Practical Tips for Measuring Financial
Success Dr. Angela LyonsUniversity of Illinois
Evaluating Financial Education in Troubled
Times Dr. Lance Palmer University of
Georgia Dr. Angela Lyons University of Illinois
Urbana Champaign AFCPE Extension
Pre-Conference November 2008
2MotivationRecent Financial Challenges
- Difficulty paying debts
- Low saving rates
- Rising home foreclosures
- Mounting bankruptcies
- Higher unemployment
- An increased awareness and impact of natural
disasters
Consumers are in financial trouble!!!
3The Result.
- A number of financial education efforts have been
developed to address consumers financial needs
in these troubled times. - 3 key programming areas
- Bankruptcy
- Homeownership issues
- Natural Disasters
4- At the end of the day, are these
financial education efforts making a
difference?
5On the front lines with the evaluation experts.
- What do we mean by evaluation?
- What are we really trying to measure?
-
- How do we define program success?
- How do we know if participants are improving?
- What financial outcomes and indicators should
we be using? - What constitutes a successful, or even
acceptable, evaluation?
6The Big Picture 5 Biggest Evaluation
Challenges in General
- 1. Identifying the ideal approach to
evaluation. -
- Evaluation methods and measures vary widely
across programs and academic disciplines. - Wide variation in financial outcomes across
programs. - Significant differences in financial needs
across - individuals and families.
- Some participants unable to
- implement certain financial behaviors.
- Shift towards more one-on-one individualized
education.
7- 2. Defining program success.
-
- Setting realistic expectations for program
participants. - Choosing appropriate outcomes and indicators
based on participants financial situation or
other external constraints. - Identifying participants individual financial
desires and applying appropriate educational
interventions and evaluation. - Finding the teachable moment or motivated
learner. -
- Recognizing that one size does not fit all.
-
8- 3. Collecting impact data from
program participants. -
- Little incentive to complete evaluations (like
pulling teeth). - Reluctance to divulge personal information
(surveys too personal lack of trust). - High drop out rates, low response rates, and
difficult to track. - Literacy levels (i.e., ESL, reading level).
- Collecting sensitive data and information.
- Tradeoff between participation and evaluation
rigor.
9- 4. Designing and implementing effective program
evaluations. -
- Evaluation process is cumbersome.
- Lack of time, staff, and financial resources.
- The PUSH for increased rigor and the rush to
the finish line. - Rigor vs. Reality (e.g., measurement issues)
- The limitations of one-shot evaluations.
- (pre- and post-tests intended vs. actual
behavior change) - The reality of conducting longitudinal studies
with control groups. - (follow-ups and tracking of program
participants)
10- 5. Data analysis and measurement issues
- Measurement error and validity of indicators.
- Self-reports are subject to bias.
- Social desirability
- Norms and rules of thumb
- Misperceptions and over-optimism
- Memory distortion and recall bias
- Samples may not be representative.
- Non-response bias
- Program attrition
- Self-selection
- Low response rates (e.g., follow-ups)
11Other measurement issues
- Environmental factors may affect outcomes.
- Unexpected life events
- Program incentives (e.g., rewards, special
benefits, enrollment programs) - Individualized financial advice or coaching
- Psychological factors.
- Inherent motivation
- Ability
- Attitudes
- Stress
12Bankruptcy Counseling and Education
- Educational impacts are difficult to disentangle
from the actual bankruptcy process. - Difficult for debtors to change certain behaviors
when havent yet discharged debts and still under
court supervision. - Conducting a follow-up is critical.
- Dealing with a very heterogeneous population that
is required to complete the counseling and
education.
13Bankruptcy Counseling and Education Contd
- Content of the counseling and education is
similar which can reduce the positive impact of
the debtor education. - Selecting indicators that are independent of the
debtors current financial situation and the
bankruptcy discharge. - Constraints associated with curriculum content
and timing of delivery.
14Homeownership programs
- Many different types of housing counseling and
education programs - Transitional housing programs
- Homeownership education
- Post purchase counseling
- Default and delinquency counseling
- Reverse mortgage counseling
- Less emphasis on group education and more focus
on one-on-one models of financial counseling or
financial coaching. - Significant grant dollars are available to
conduct these programs, but evaluations can be
constrained by funders requirements.
15Post-Natural Disaster Programs
- Using traditional pre- and post-test surveys may
be less effective than more qualitative methods
such as focus groups, interviews, and case
studies. - Informational and financial needs may vary widely
across disaster victims. - Psychological factors are likely to significantly
affect motivation, attitudes, and perceived
abilities which will influence evaluation
responses. - Disaster victims can be very transient
difficult to track and conduct follow-ups. - Very time and resource intensive.
16Overcoming the ChallengesPlanning, planning, and
more planning.
- Take stock of who you are What is your
vision? - Be strategic What are you trying to accomplish?
- Conduct a thorough needs assessment.
- Spend time understanding your target audience(s).
- Define program success for your audience. Be
realistic! - Create an evaluation action plan
- Identify resources and overcome constraints.
- Take Action!
174 Steps to Creating An Effective Evaluation
Action Plan
- Step 1 Define objectives of the program.
- At the end of the day, what do you want to show?
- Who will be the target audience?
- What will be the primary delivery method?
- Who will use the evaluation and how?
- Step 2 Select appropriate methods for data
collection. - What is the most appropriate evaluation format?
- What types of questions will the evaluation seek
to answer? - What types of indicators will be used to show
impact? - Are these indicators appropriate for the target
audience?
18- Step 3 Identify and overcome evaluation
challenges. -
- What are your biggest implementation challenges?
- What steps can you take to overcome the
challenges? - What financial and non-financial resources are
available? - Are there others who can help?
- What is the program timeline?
- Given constraints, what can you realistically
do? - Step 4 Analyze and disseminate findings.
- How will data be analyzed?
- What do you hope to learn from findings?
- What are the potential impacts?
- How will the results be used and disseminated?
-
19Where do we go from here?Online resources at
your fingertips
20NEFE Financial Education Evaluation Toolkit
http//www2.nefe.org/eval/intro.html
21NEFE Financial Education Evaluation Toolkit
- Database
- Post evaluation only with option for follow-up
- Pre and post evaluation with option for follow-up
- Stages to Change Evaluation
- Train-the-Trainer
- Testing Knowledge
- Building Skills
- Taking Charge
- Manual
- How-to-guide for grass-roots level organizations
- Examples (survey instruments, executive summary,
reports) - Guidance on how to organize and present impact
data
22 Manual http//www2.nefe.org/eval/manual.html Par
t I Financial Education Overview Part II
Understanding Program Evaluation Part III The
Evaluation Planning Process Part IV Using the
Evaluation Database Part V Reporting Program
Impact
23Appendix Sample Evaluation Instruments
24 Database http//www2.nefe.org/eval/index.php
25 Knowledge Indicators
26 Behavior Indicators
27 Qualitative Info
28 Demographics
29 University of Wisconsin-Extension http//www.uwex
.edu/ces/pdande/evaluation/index.html
30 Cornell University Extension http//staff.cce.cor
nell.edu/administration/program/evaluation/evalref
s.htm
31 Penn State Extension http//www.extension.psu.edu
/evaluation/
32Contact Information
- Dr. Lance Palmer
- Assistant Professor
- Department of Housing and Consumer Economics
- University of Georgia
- Phone (706) 542-4916
- E-mail lpalmer_at_uga.edu
33Questions