DEMONSTRATING IMPACT: AN INTRODUCTION TO OUTCOMES-BASED EVALUATION (OBE)

1
DEMONSTRATING IMPACT AN INTRODUCTION TO
OUTCOMES-BASED EVALUATION (OBE)

• Jennifer Peters, Grants Administrator, Library
  Development

2
Goals for This Session

• What is OBE
• Why evaluate?
• What is an outcome?
• How We Use OBE
• Logic Model
• Why We Use OBE
• Accountability to funder
• Value to YOU
• Resources for Further Research

3
Why Evaluate?

• Libraries Change Lives do they?
• Answer questions
• Are we achieving what we hope to achieve?
• Are we being efficient?
• Are we being effective?
• Does this program work? Would another program
  work better?
• Is this program a model for other libraries?

4
What is Outcomes-Based Evaluation?

• Tool for program design
• Tool for measuring your programs effectiveness
• Tool for planning future programs
• Based on models developed in social service
  agencies and United Ways
• Extension of the data gathering you already do
• Focused not on what goes into your program but
  what comes out of it
• Provides deeper understanding of your programs
  value to the community

5
What is an Outcome?

• Benefits or changes for individuals or
  populations during or after participating in
  program activities. (IMLS)
• Beneficial changes for program participants that
  include changes in their skills, knowledge,
  behavior, attitude, status or life condition.
  (Texas State Library and Archives Comm.)
• A target audiences changed or improved skills,
  attitude, knowledge, behaviors, status, or life
  condition brought about your project.
  (Washington State Library)
• Outcomes occur as a direct result of an
  actionsuch as training, services, or teaching.
  (Boulmetis and Dutwin)
• Changes in adaptive behavior and role status
  that are logical consequences of the services.
  (Schalock)

6
OUTCOME IMPACT

• How did your program impact the lives, skills,
  abilities, knowledge, behaviors of the person or
  group who attended your program? How was that
  person or group changed?
• How does your library impact the community?

7
Outcomes

• Not how many worms the mother bird fed her baby,
  but how well the fledgling flew
• Not how many people attended the
  smoking-cessation class, but how many people
  actually quit smoking

8
How to Implement Outcomes-Based Evaluation

• Think big picture about why you do programming
• Think big picture about your program
• Why are we doing this program?
• What community need will this program meet?
• Who do we want this program to reach?
• When do we want to do this program?
• How do we want this program to impact our
  audience?

9
Logic Model

• Describes a library program
• A planning tool to pull all the pieces of your
  program together
• Helpful because it gives you the opportunity to
  think the program through from start to finish,
  with a focus on the impact of your program, the
  identified need it will meet, and its value to
  the community (AKA outcomes)

10
Wedding Logic Model
Need Two people are in love and want to spend the rest of their life together
Inputs Money, bride, groom, friends, relatives, love, mothers-in-law, clergy, caterer, band, space, registry, dress, ring, food
Activities Ceremony, reception
Outputs 200 people at ceremony, 200 people fed and watered during reception, 100 people dancing
Outcomes Short-Term The couple is pronounced husband and wife. Intermediate Couple and guests are happy and celebrating. Husband and wife decide to have children. Long-Term Lifelong partnership. Happy family produces children who grow up to be productive citizens.
Borrowed from a presentation by Sue Ellen
Schumacher, American Association of Grant
Professionals, Nov. 2009.
11
Program Logic Model
Community Need What community need have you identified that this program will address? What general group are you targeting? Who is in need of this program?
Inputs What and who do you need in place to be able to run this program?
Activities What will need to be in place to run this program?
Outputs What are the quantitative results of program activities? What are the measures that can be objectively counted?
Outcomes What are your goals? What do you want to achieve? What changes do you want to see in program participants?
12
Community Need
Community Need What community need have you identified that this program will address? What general group are you targeting? Who is in need of this program?
Inputs
Activities
Outputs
Outcomes
13
Community Need
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs
Activities
Outputs
Outcomes
14
Inputs
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs What and who do you need in place to be able to run this program?
Activities
Outputs
Outcomes
15
Inputs
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs ESL trainer, Pathways to English curriculum, classroom space, Spanish-language newspapers ads and flyers advertising training, partnership with employer providing financial support, publicity plan to promote program in community.
Activities
Outputs
Outcomes
16
Activities
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs ESL trainer, Pathways to English curriculum, classroom space, Spanish-language newspapers ads and flyers advertising training, partnership with employer providing financial support, publicity plan to promote program in community.
Activities How are you going to run this program?
Outputs
Outcomes
17
Activities
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs ESL trainer, Pathways to English curriculum, classroom space, Spanish-language newspapers ads and flyers advertising training, partnership with employer providing financial support, publicity plan to promote program in community.
Activities Library designs promotional material, hires ESL trainer and purchases curriculum. Weekly classes are scheduled to begin September 1, 2010 and run through December 31, 2010. Testing will occur 3X/semester. Attendance will be tracked in Excel spreadsheet and testing scores entered in Pathways to English website. Library Director will invite features reporter to attend a class for publicity.
Outputs
Outcomes
18
Outputs
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs ESL trainer, Pathways to English curriculum, classroom space, Spanish-language newspapers ads and flyers advertising training, partnership with employer providing financial support, publicity plan to promote program in community.
Activities Library designs promotional material, hires ESL trainer and purchases curriculum. Weekly classes are scheduled to begin September 1, 2010 and run through December 31, 2010. Testing will occur 3X/semester. Attendance will be tracked in Excel spreadsheet and testing scores entered in Pathways to English website.
Outputs What are the quantitative results of program activities? What are the measures that can be objectively counted?
Outcomes
19
Outputs
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs ESL trainer, Pathways to English curriculum, classroom space, Spanish-language newspapers ads and flyers advertising training, partnership with employer providing financial support.
Activities Library designs promotional material, hires ESL trainer and purchases curriculum. Weekly classes are scheduled to begin September 1, 2010 and run through December 31, 2010. Testing will occur 3X/semester. Attendance will be tracked in Excel spreadsheet and testing scores entered in Pathways to English website.
Outputs Number of classes offered. Number of people who sign up for classes. Attendance figures. Number who complete class. Percentage of dropouts. Test scores of attendees.
Outcomes
20
Examples of Outputs

• Reference questions answered
• Persons attending ESL classes
• Books cataloged
• Web pages accessed
•  Items requested through inter-library loan
• Summer reading participants
• Newspapers digitized
• Computers purchased
• Library cards issued
• Classes taught

21
Outcomes
Community Need The percentage of Central Texas adults who speak English less than well has risen from 20 to 25 in the last decade, per the U.S. census. Local employers report that non-native English speaking employees have difficulty following written directions, leading to work slowdowns and high turnover. Jones Branch Library is located in a neighborhood with a 65 Hispanic population, of whom 80 are non-native English speakers. Internal surveys indicate that 50 of library patrons are interested in ESL classes. Jones Library will target adult ESOL learners needing assistance in following written instructions and directions.
Inputs ESL trainer, Pathways to English curriculum, classroom space, Spanish-language newspapers ads and flyers advertising training, partnership with employer providing financial support.
Activities Library designs promotional material, hires ESL trainer and purchases curriculum. Weekly classes are scheduled to begin September 1, 2010 and run through December 31, 2010. Testing will occur 3X/semester. Attendance will be tracked in Excel spreadsheet and testing scores entered in Pathways to English website.
Outputs Number of classes offered. Number of people who sign up for classes. Attendance figures. Number who complete class. Percentage of dropouts. Test scores of attendees.
Outcomes What is the impact that your program will have? What changes do you want to see in program participants?
22
Creating Your Outcomes

• Based on your program plan and community need,
  you should be able to easily identify 1-3
  outcomes for your target audience
• What is the need?
• How do you want people to change?
• What do you want the impact to be?
• Think about short-term outcomes/intermediate/long-
  term outcomes
• You develop them yourselfbased on what you want
  the program to achieve, and what you want to
  communicate about the program

23
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator
Data Source
Audience
Timing
Goal
Outcome 1
Indicator
Data Source
Audience
Timing
Goal
24
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator
Data Source
Audience
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator
Data Source
Audience
Timing
Goal
25
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source
Audience
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator
Data Source
Audience
Timing
Goal
26
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source
Audience
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source
Audience
Timing
Goal
27
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source
Audience
Timing
Goal
28
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience
Timing
Goal
29
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience Who will be targeted specifically? Whole audience or subgroup?
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience
Timing
Goal
30
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience Who will be targeted specifically? Whole audience or subgroup?
Timing
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience ESOL learners with fair-to-moderate English-language skills.
Timing
Goal
31
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience Who will be targeted specifically? Whole audience or subgroup?
Timing When/how often will data be collected?
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience ESOL learners with fair-to-moderate English-language skills.
Timing
Goal
32
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience Who will be targeted specifically? Whole audience or subgroup?
Timing When/how often will data be collected?
Goal
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience ESOL learners with fair-to-moderate English-language skills.
Timing At end of each 6-week session. (3 sessions within Program period)
Goal
33
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience Who will be targeted specifically? Whole audience or subgroup?
Timing When/how often will data be collected?
Goal What is the number and/or percentage of the audience that you intend to impact?
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience ESOL learners with fair-to-moderate English-language skills.
Timing At end of each 6-week session. (3 sessions within Program period)
Goal
34
Integrating Outcomes into Your Logic Model
Outcome What is the impact that your program will have? What has changed about the target audience?
Indicator What will indicate that your program has made an impact? What will you measure?
Data Source How will you measure this?
Audience Who will be targeted specifically? Whole audience or subgroup?
Timing When/how often will data be collected?
Goal What is the number and/or percentage of the audience that you intend to impact?
Outcome 1 ESOL learners gain ability to read directions and instructions in English.
Indicator ESOL learners will successfully complete Pathways to English curriculum, vol. 1.
Data Source Self-tests at end of each chapter of Pathways to English, vol. 1.
Audience ESOL learners with fair-to-moderate English-language skills.
Timing At end of each 6-week session. (3 sessions within Program period)
Goal 70 will be retained and pass all self-tests in curriculum.
35
Examples of Program Outcomes

• Students gain an understanding of basic
  navigation and how to locate a web page.
• Library patrons become aware of the resources
  available on online databases.
• Students show increased critical thinking skills.
• ESOL learners gain ability to read directions and
  instructions in English.
• Students demonstrate an appreciation of regional
  authors.
•  Participants understand the value of voting in
  local, state and national elections.
• Program attendees recognize and are sensitive to
  cultural differences in the clinical setting.
• Pre-school children recognize that the value of
  books for learning and for fun.

36
Short, Intermediate, Long-Term Outcomes
Short-Term ESOL learners gain ability to read directions and instructions in English. ESOL learners gain confidence in their ability to communicate in English.
Intermediate Employers report increased interest in hiring non-native English speakers. ESOL learners become fully fluent in English.
Long-Term Non-English speakers are fully integrated into the community. The education rate for the children of non-native English speakers increases. The unemployment rate among non-English speakers drops.
37
Why Use Outcomes-Based Evaluation?

• Most funding agencies, including foundations,
  United Ways, and federal agencies, now require an
  outcomes-based evaluation model
• For funders
• Shows a Return on Investment
• Provides measures that they can, in turn, report
  on and aggregate to show the value of their
  investment in library systems
• Provides evidence of best practices that can be
  shared with other libraries

38
Why Use Outcomes-Based Evaluation?

• For YOU
• Provides an opportunity to think critically about
  why you make the programming choices that you do
• Provides deeper understanding of your programming
  for management, staff, volunteers
• A well-planned logic model means a strong,
  cohesive program
• Gives you the opportunity to evaluate your
  program internally and plan for the future
• What worked?
• What can we do better next time?
• Gives you the opportunity to tell your story.
  Dont forget to collect a success story.
• Media
• Local officials
• Donors

39
Other Resources

• Boulmetis, John and Phyllis Dutwin. The ABCS of
  Evaluation Timeless Techniques for Program and
  Project Managers. Jossey-Bass, 2005.
• Measuring Program Outcomes A Practical Approach.
  United Way of America, 1996.
• Outcomes Framework Browser. http//portal.whatwork
  s.org/programs.aspx
• Outcomes Libraries Change Lives Oh yeah?
  Prove it. IMLS presentation to Public Libraries
  Association, 2002. www.imls.gov/ppt/PLA-02-2OBE.pp
  s
• Perspectives on Outcome Based Evaluation for
  Libraries and Museums, IMLS. http//www.imls.gov/
  pdf/pubobe.pdf
• Rubin, Rhea Joyce. Demonstrating Results Using
  Outcome Measurement in Your Library. ALA, 2006.
• Shaping Outcomes. Online course offered by
  Indiana University. http//www.shapingoutcomes.org
  /
• Texas State Library and Archives Commission.
  Outcome Measures. http//www.tsl.state.tx.us/outc
  omes/

40
Questions?

• Contact Jennifer Peters, Grants Administrator,
  Library Development Division, Texas State Library
  and Archives Commission
• jpeters_at_tsl.state.tx.us
• 512-463-5527