Title: Program Evaluation Screencast
1Program Evaluation Screencast
https//ess.echo360.vcu.edu8443/ess/echo/presenta
tion/fff3175a-9600-4bda-91fa-a21a1fe32758
- Prepared by Mary Secret
- Based on materials from the following sources
- Babbie,, E. 2014 The Practice of Social
Research, (14th edition). Boston, MAThomson
Wadsworth - Corcoran, J. Secret, M. (2013). Social Work
Research Skills Workbook. New York Oxford - Engel, R.J., Schutt, R.K., (2013). The practice
of research in social work (3rd Ed). Thousand
Oaks, CA Sage.
ECHO 360 links https//ess.echo360.vcu.edu8443/e
ss/echo/presentation/fff3175a-9600-4bda-91fa-a21a1
fe32758?ectrue https//ess.echo360.vcu.edu8443/
ess/echo/presentation/fff3175a-9600-4bda-91fa-a21a
1fe32758
2What is the purpose of program evaluation
- To investigate social programs
- To assess effectiveness of social policies and
programs.
3Program Evaluation Prologue
- is not a specific activity or method that you
can point to or associate with any particular
step of the research process - encompasses all aspects of research processes
and methods
4- Major comprehensive program evaluation can
- Include experimental and non-experimental
research designs, - Use both qualitative and quantitative approaches,
- Collect data from secondary data sources,
interview participants, - Use standardized or non-standardized measurement
instruments, - Include both probability and nonprobability
samples, - Must adhere to the standard research ethics
5- Program Evaluation
- is distinguished from other types of social
science research not by the design, method, or
the approach - but the underlying intent, the purposes that
guide the evaluation process
6What is the purpose of program evaluation
- To investigate social programs
- To assess effectiveness of social policies and
programs.
7Question FIRST!!!
The specific methods depends on the evaluation
question of interest about a specific program,
policy or intervention
- Questions to be answered
- Is the program needed?
- Do a needs assessment
- How does the program operate?
- Do a formative or process evaluation
- What is the programs impact?
- Do a summative or outcome evaluation
- How efficient is the program?
- Do a cost benefit or a cost effectiveness
analysis
8The language of evaluation Fill in the Blank
- the impact of the program the intended result
the response variable the dependent variable - Outcomes
- the services delivered or new products produced
by the program process - Outputs
- resources, raw materials, clients, and staff that
go into a program - In puts
- Population for whom the program is designed.
- Target Population
- individuals and groups who have some basis of
concern with the program, often setting the
research agenda and controlling research findings - Stakeholders
- information about service delivery system
outputs, outcomes, or operations that is
available to any program stakeholders - Feedback
9What is a Needs Assessment
- Systematically researching questions about the
needs of a target population for program planning
purposes that obtains information from - Key Informants
- expert opinions from individuals who have special
knowledge about the needs and about the existing
services - Rates under treatment
- secondary analysis of existing statistics to
estimate need for services based on number and
characteristics of clients who are already being
served. - Social Indicators
- existing statistics that reflect conditions of an
entire population.. i.e. census data, Kids Count
data.
Rubin and Babbe, (2007) Essential Research
Methods for Social Work. Brooks ColeCA
10What is a Process or Formative Evaluation
- How do you know whether or not the service was
delivered in the manner intended i.e. according
to protocol or evidence based practice model - Must measure (collect data) the Independent
Variable the intervention.. What services were
actually delivered, i.e. Number of counseling
sessions, hours of training, number of meetings,
etc etc
11What is an Outcome Evaluation Also known as
impact evaluation and summative evaluation
- Evaluation research that examines the
effectiveness of the treatment or other service - Program is independent variable (treatment)
- Outcomes are dependent variables
- Experimental design is preferred method for
maximizing internal validity because of - Random assignment into an experimental group and
a control/comparison group - Manipulation of the independent variable
12A closer look at Experimental Designs
- Research Design Notations
-
- RRandom assignment
- OObservation, data collection
- XIntervention or treatment
-
13Classic Experimental Design
A local drug treatment program wanted to assess the effectiveness of adding an 8-week yoga class to their current counseling and medication treatment. A counselor has 30 clients who had been in treatment for less than 2 weeks. She randomly assigned 15 clients to participate in yoga and other 15 participated in the regular treatment activities. She has all 30 clients complete an assessment packet that examined their substance abuse history and their social and emotional functioning. She provides the yoga meditation to the After 8 weeks, she evaluated the participants substance use and their social and emotional functioning. R Experimental Group O X1 O Pretest 8-week Posttest Yoga class medication and counseling Comparison Group O X2 O Pretest Medication Posttest and counseling
Controls for selection bias and history and
maturation and statistical regression threat to
internal validity
14Quasi-experimental design
A statewide evaluation of the ENABL program was conducted to assess its ability to increase adolescents knowledge and beliefs about pregnancy prevention. ENABL is a 4-week program aimed at preventing teenage pregnancy through abstinence. Middle school students in one school district comprised a treatment group (n 974), and students in another school district comprised a control group who received no intervention. Subjects completed a pretest and posttest reflecting knowledge and beliefs about teenage pregnancy before and after the ENABL program. NO RANDOM ASSIGNMENT Experimental Group O X O Pretest 4-week Posttest ENABL program Control Group O O Pretest Posttest
Less control for threat to internal validity ..
Possibility of selection bias
15Pretest/Posttest design (pre-experimental)
This study assessed the impact of a dating violence sensitivity group intervention. The participants were 190 high school students, ages 13 to 19 years. Questionnaire data about the knowledge and values and skills needed to avoid dating violence were collected from the participants before the 12-week group training and at the end of the training sessions. NO RANDOM ASSINGMENT NO CONTROL/COMPARISON GROUP Experimental group O X O Pretest 12-week Posttest training program
Least control for threat to internal validity ..
History, maturation,
16What about Measurement
- Use of many different types of measurement tools,
- dependent on the intent and type of
evaluation research
17USING MULTIPLE MEASURES and SEVERAL DATA
COLLECTION STRATEGIES TO EVAULATE THE FACT PROGRAM
Independent varialbe
Measures for outcome and causal mechanisms Does
the program cause change? How does change happen
Measures for input And program efficiency
Measures for Process/Implementation
Evaluation What services are being delivered, by
who, how?
18LOGIC MODEL
- many pieces of information that must be organized
and then interpreted. - need a way in which this information can be
organized.
19What is the Logic Model?
- A schematic representation of the various
components that make up a social service program. - Logic models may describe
- theory and its link to change (theory approach
model) where attention is on the how and why a
program works - outcomes (outcome approach model) where the focus
of the logic model is to connect resources and
activities to expected changes - activities (activities approach model) or
describing what the program actually does
20Logic model- outcomes example
Short term outcomes.Measured by the research
Program inputs
Program processes
Long term outcomes/ difficult to measure
Identifying the causal mechanism
21Whats an Evaluabilty Assessment
Newly emergent programs that are not fully
operational are not ready for, and indeed can
be tarnished by a summative evaluation geared to
assessing program outcomes. HOW SO??
- a systematic process that helps identify whether
program evaluation is justified, feasible, and
likely to provide useful information. - determines whether a program is ready for
evaluationeither a process or outcome
evaluation, or both. - Is the program able to produce the information
required for a process evaluation,.. AT WHAT
STAGE OF IMPLEMTATION IS THE PROGRAM? - Can a program meet the other criteria for
beginning an outcome evaluation. - determines whether a program has the basic
foundation for an evaluation to take place
Evaluability Assessment Examining the
Readiness of a Program for Evaluation. Juvenile
Justice Evaluation Center Justice Research and
Statistics Association. Program Evaluation
Briefing Series 6. May, 2003, p.
6 http//www.jrsa.org/pubs/juv-justice/evaluabilit
y-assessment.pdf
22Evaluability of a program based on
- ESTABLISHED PROGRAM
- measurable outcomes
- defined service components
- an established recruiting, enrollment, and
participation process - good understanding of the characteristics of
the target population, program participants and
program environment - ability to collect and maintain information
- adequate program size
- RESEARCH SAVVY SERVICE DELIVERY STAFF
- problem solving values and skills
- prior experience with evaluation confidence in
program - commitment to new knowledge
- openness to change