Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois

1
Some Practical Tips for Measuring Financial
Success Dr. Angela LyonsUniversity of Illinois
Program Evaluation I Setting the Stage for
Outcomes-Based Success Presented by Dr. Angela
Lyons University of Illinois October 2009
2
MotivationRecent Financial Challenges

• Rising debt
• Low savings rates
• Home foreclosures
• Bankruptcies
• Unemployment

Consumers are in financial trouble!!!
3
Reality Check! How much trouble are we in? 2009
U.S. deficit 1.4 trillion
4
The Million Dollar Question

• At the end of the day, does
  financial education make a difference?

5
What have we learned so far?

• What does current research tell us?
• Why is evaluation important?
• Whats in it for you and your organization?
• Whats in it for your clients?

6
On the front lines with the evaluation experts.

• What do we mean by evaluation?
• What are we really trying to measure?
• How do we define program success?
• How do we know if participants are improving?
• What financial outcomes and indicators should
  we be using?
• What constitutes a successful, or even
  acceptable, evaluation?

7
Getting Started Thinking like an
evaluator.(Program Planning Guide)

• Take stock of who you are What is your
  vision?
• Conduct a needs assessment.
• Collect baseline information from your target
  audience.
• Identify your signature program(s).
• Identify your program objectives. Be realistic!
• Create an evaluation action plan.
• What do you want to accomplish? At the end of
  the day, what do you want to show?

8
How will you define program success?

• Setting realistic expectations for program
  participants.
• Choosing appropriate outcomes and indicators
  based on participants financial situation or
  other external constraints.
• Identifying participants individual financial
  needs and applying appropriate educational
  interventions.
• Finding the teachable moment.

9
What is an outcome-based evaluation?

• Outcomes are benefits to clients from
  participating in the program.
• What do you want your participants to know or be
  able to do when they have finished the program?
• Outcomes are usually in terms of enhanced
  learning and improved behaviors.
• Outcomes are often confused with program outputs
  or units of service (e.g., number of clients who
  went through the program).

10
The Logic Model

• A picture of the program.
• Simple representation of the program theory or
  action which explains the program and what it
  is to accomplish.
• Shows relationships between inputs, outputs, and
  outcomes.

11
The Logic Model (conti.)

• INPUTS

OUTPUTS
OUTCOMES
Resources used to develop the program are called
inputs. Time and money are the most common
inputs needed to implement educational programs.
If inputs are invested into the financial
education program, then learning opportunities
will be created for the target audience. The
created educational materials, services, and
opportunities are called the program outputs.
Changes in participants perceptions, knowledge,
and behavior that represent real impact in their
lives. The benefits derived by the participants
from the program are called outcomes.
12
University of Wisconsin - Extensionhttp//www.uwe
x.edu/ces/pdande/evaluation/evallogicmodel.html
13
Impact Hierarchy of Outcomes
14
Another useful framework.Transtheoretical Model
of Behavior Change (TTM)

• TTM integrates major psychological theories into
  a theory of behavior change.
• Used to identify the state at which individuals
  are ready and able to change their financial
  behaviors.
• Appropriate educational interventions are then
  tailored to meet individuals specific needs at
  that particular stage.

15
5 Stages of Change

• Precontemplation
• Individual not ready to take action and change
  behavior in the immediate future.
• Rarely seeks help and rarely uses information.
• Contemplation
• Individual is getting ready to take action and
  intends to change behavior in next
  6 months.
• Open to education.
• Preparation
• Individual is ready to take action and intends to
  change behavior in next 30 days.
• Practices behavior by taking small steps towards
  the goal.
• Seeks information and support, but often
  concerned that changing behavior may be too
  difficult and they may not succeed.

16
5 Stages of Change (conti.)

• Action
• Individual changes behavior and maintains
  behavior for at least 6 months.
• Believes they can change.
• Can control triggers that cause them to relapse
  into old behaviors.
• Has a support system to get them through
  challenging times.
• Maintenance
• Individual has changed behavior and it has lasted
  for more than 6 months.
• May relapse into old behaviors, but can overcome
  temptations so that behavior becomes permanent.
• Can assess the conditions under which relapse
  might occur.
• Can establish successful coping strategies.

17
Example
Financial Practice Do not plan to do Plan to do in the next month Plan to do in the next 2-6 months Have been doing for 1-6 months Have been doing for more than 6 months
Set short and long-term financial goals. ? ? ? ? ?
Save money regularly. ? ? ? ? ?
Use a spending plan to track income and expenses. ? ? ? ? ?
Maintain sufficient balances in bank account(s). ? ? ? ? ?
Pay bills on time each month. ? ? ? ? ?
Review bills each month for accuracy. ? ? ? ? ?
Comparison shop before making purchases. ? ? ? ? ?
Pay off credit card balances in full each month. ? ? ? ? ?
18
Identifying Program Objectives

• Objectives should be
• Specific
• Measurable
• Achievable and observable
• Reasonable
• Time specific
• S.M.A.R.T. objective statements should clearly
  define what you want to achieve with your
  program.
• They should list the end outcomes the program
  intends to affect or change.

19
Writing objective statements

• First-time home buyer education program
• The objectives of this program are to
• Develop first-time home buyers ability to shop
  for the lowest mortgage interest rate.
• Teach first-time home buyers how to save money
  for closing costs.
• Teach first-time home buyers how to assess
  affordable housing.
• Debt reduction education program
• The objectives of this program are to
• Develop participants ability to identify needs
  and wants separately.
• Develop participants ability to control wants
  to reduce expenditures.
• Develop participants ability to avoid impulse
  and emotional spending.

20
Achieving your objectivesSelecting appropriate
indicators

• General Indicators (objective and subjective)
• Number of programs, participants, etc.
• Knowledge gains
• Changes in attitudes and satisfaction
• Changes in skills and confidence
• Changes in intended and actual behaviors
• Specific Indicators (objective)
• Actual dollar changes (reduce debt, increase
  savings)
• Development of financial plans
• Changes in spending habits
• Building or rebuilding credit reports and credit
  scores

21
Putting it all together.

• Creating Your
• Evaluation Road Map

22
(No Transcript)
23
A few words of caution when selecting
indicators.

• Measurement error and validity of indicators.
• Financial knowledge
• Confidence level
• Financial behavior
• EXAMPLE Which of the following are valid
  indicators of behavior change?
• Participant opened a bank account.
• Participant increased savings.
• Participant avoided bankruptcy.
• Participant did not default on mortgage payments.

24
One non-profit administrator commented.

• What is driving this financial education
  movement? Why is it so important? What are we
  ultimately trying to address? Is it reducing the
  poverty gap in this country? Between those that
  have and those that dont have. And its
  widening. And those at the bottom end of the
  spectrum.what were asking them is to build
  wealth. And at the same time, what were asking
  people in this country who make 20,000 or less
  is Absent us raising your wages in this
  country, were asking you to build wealth, to
  participate in IDA programs. Were asking you to
  save with the little amount of money youre
  making. Were asking you to reduce your debt
  burden, learn how to manage your money, and clean
  up your credit history with the little amount of
  money youre working with. And we want you to
  get from point A to point B with all those
  constraints.

Source Lyons, A. C., Palmer, L., Jayaratne,
K.S.U., and Scherpf, E.  (2006). "Are We Making
the Grade? A National Overview of Financial
Education and Program Evaluation. The Journal of
Consumer Affairs, 40(2), 208-235.
25
A few additional words of wisdom

• Avoid indicators that may be subject to biases.
• Social desirability
• Norms and rules of thumb
• Misperceptions and over-optimism
• Memory distortion and recall bias
• Avoid indicators that may ask sensitive or
  personal info.
• Non-response bias
• Program attrition
• Self-selection
• Low response rates (e.g., follow-ups)

26

• Environmental factors can affect outcomes.
• Unexpected life events
• Program incentives (e.g., rewards, special
  benefits, enrollment programs)
• Individualized financial advice or coaching
• Psychological factors.
• Inherent motivation
• Ability
• Attitudes

27
There is no magic set of outcomes or
indicators.The list is endless.

• Increases in savings.
• Decreases in debt.
• Maintaining a regular budget.
• Comparison shopping.
• Increases in new accounts opened.
• Improved credit scores.
• Improved communication with spouse/partner/parents
  about finances.
• Other common indicators?

28

• How do these indicators change for various target
  populations?
• Youth?
• Underserved?
• Adults?
• Members of your organization?

29
ACTIVITY Your Evaluation Action Plan Part A

• What is your signature program (e.g., course,
  workshop, educational materials, initiative,
  campaign)?
• How are you going to define program success? At
  the end of the day, what do you want to show?
• Who is your target audience(s)?
• What is your delivery method(s) (e.g., in-person,
  telephone, Internet)?
• What financial and non-financial resources are
  available to you for evaluation?
• Who will use the evaluation and how will it be
  used?

30
Summary 4 Steps to Creating a Successful
Evaluation Action Plan

• Step 1 Define objectives of the program.
• At the end of the day, what do you want to show?
• Who will be the target audience?
• Who will use the evaluation and how?
• Step 2 Select appropriate outcomes, indicators,
  and
• methods.
• What is the most appropriate evaluation format?
• What types of questions will the evaluation seek
  to answer?
• What types of indicators will be used to show
• impact?

31

• Step 3 Identify resources and constraints.
• What financial and non-financial resources are
  available?
• Are there others who can help?
• What is the program timeline?
• Given constraints, what can you realistically
  do?
• Step 4 Analyze and disseminate findings.
• How will data be analyzed?
• What do you hope to learn from findings?
• What are the potential impacts?
• How will the results be used and disseminated?

32
Where do we go from here?Evaluation resources at
your fingertips
33
U of I Center for Economic and Financial
Education http//www.cefe.illinois.edu/
34
Educational Tools for Evaluation http//www.cefe.
illinois.edu/tools/evaluation.html
35
Checklist of evaluation resources

• Program Planning Guide
• Evaluation Action Plan
• Evaluation Road Map
• Evaluation Reading List
• Sample Evaluations
• NEFE Financial Education Evaluation Toolkit
• http//www.cefe.illinois.edu/tools/evaluation.html
  

36
University of Wisconsin-Extension http//www.uwex
.edu/ces/pdande/evaluation/index.html
37
Cornell University Extension http//staff.cce.cor
nell.edu/administration/program/evaluation/evalref
s.htm
38
Penn State Extension http//www.extension.psu.edu
/evaluation/
39

• Program Evaluation II
• Creating Your Evaluation Toolkit
• More to come soon.

40
Contact Information

• Dr. Angela Lyons
• Associate Professor
• Director, U of I Center for Economic and
  Financial Education
• University of Illinois
• Phone [phone]
• E-mail anglyons_at_illinois.edu

41
Questions