Quantitative Methods for Developing Evidence for Social Work Practice

1
Quantitative Methods for Developing Evidence for
Social Work Practice
Diane DePanfilis ddepanfilis_at_ssw.umaryland.edu

• University of Maryland School of Social Work
• Research Brown Bag Seminar
• February 8, 2007

2
Using quantitative research methods

• To evolve from
• Evidence based practice to . . .
• Evidence supported practices

3
Public Health Approach Applies to Developing
Evidence for Social Work

• This approach has four steps
• define the problem,
• identify risk and protective factors,
• develop and test prevention or intervention
  strategies, and
• evolve to widespread adoption of prevention or
  intervention principles and strategies.

4
First Questions

• Epidemiology of social problems
• Develop understanding of nature, extent, effect
  of conditions that are the focus of social work
  practice
• Develop knowledge about risk and protective
  factors for these conditions

Method Implementing descriptive strategies,
e.g., survival analysis to understand patterns
and predictors
5
Use methods to systematically review the evidence

• about what works best to address the conditions
  described
• (Subject of the last presentation)

6
Document a program model based on the best
available evidence

• A well defined set of prescribed interventions
  and procedures.
• More complex to define these procedures when the
  intervention is home based and depends on skills
  of the practitioner and when multiple methods are
  employed.

Bond, et. al. (2000). Psychiatric rehabilitation
fidelity toolkit. Cambridge, MA The Evaluation
Center.
7
Design Implement Test of Efficacy

• First Steps
• Clearly define target population with inclusion
  exclusion criteria
• Establish a logic model that clearly articulates
  the program model and how it will influence the
  targeted outcomes

8
Simple Logic Model
Inputs
Activities
Long-Term Outcomes
Short-Term Outcomes
Resources Services Intermediate
Outcomes Benefits
OCAN Annie E. Casey DHR Title IVE
Emergency Services Family Assessment Service
Planning Advocacy Multi-Family groups
Child Safety Child Well-Being Stability
Permanency
Enhance Protective Factors Decrease Risk
Factors
9
Design Test of Efficacy

• Next steps
• Establish fidelity criteria and methods for
  tracking flexibility within fidelity to the
  model
• Fidelity is defined as the extent to which
  delivery of an intervention adheres to the
  protocol or program model originally developed.
• Fidelity criteria are necessary to ensure that
  the services being studied are being implemented
  as intended or that significant differences are
  documented.

Mowbray, C.T., Holter, M.C., Teague, G.B.,
Bybee, D. (2003). Fidelity criteria
Development, measurement, and validation.
American Journal of Evaluation, 24, 315-340.
10
DANGER
TYPE III ERROR
Failing to adequately implement with fidelity
(John Lutzker)
11
Why assess program fidelity?

• To ensure internal validity
• If evaluation results are positive, we need to
  understand what intervention specifically
  influenced those results.
• If evaluation results are negative or ambiguous,
  it is important to know whether the program
  adhered to the program model or whether the
  negative or ambiguous results are because the
  program was not implemented as intended.

12
Why assess program fidelity?

• Fidelity is important in single program
  evaluations and in outcome research.
• It is important to ensure model adherence.
• It is important to confirm that the manipulation
  of the independent variable (the program)
  occurred as planned.
• For randomized clinical trials, it is also
  important to assure that the experimental
  treatment is really absent in the control
  condition.

Moncher, F. J., Prinz, R. J. (1991). Treatment
fidelity in outcome Studies. Clinical Psychology
Review, 11. 247-266.
13
Design Implement Test of Efficacy

• Next steps
• Randomly assign participants to receive the
  intervention or not receive the intervention.
• Ethics of no treatment or treatment as usual
• OR
• Use a quasi-experimental design with a comparison
  group

14
Design Implement Test of Efficacy

• Next steps
• Ensure adequate measurement of intermediate and
  final outcomes (ideally using valid and reliable
  standardized measures)
• Recruit sufficient number of participants
• Track retention and attrition
• Use Intent to Treat analyses to measure changes
  in outcomes between groups over time - -
  preferably with a follow-up of 12 months
  following the end of intervention.

15
Take Home Points

• Moving from evidence based practice to evidence
  supported practice is a long term proposition.
• Quantitative research methods can support
• descriptions of the social problem and risk and
  protective factors
• assessment of fidelity
• understanding whether the intervention actually
  contributed to the outcomes
• eventually moving from efficacy to effectiveness
• ultimately developing evidence for social work
  practice.