The Planning and Evaluation Partnership: DataBased DecisionMaking and Accountability in Substance Ab

1
The Planning and Evaluation Partnership
Data-Based Decision-Making and Accountability
in Substance Abuse Prevention

• Presented to the Department of Public Health
• August 3, 2004
• Bob Illback Margaret Pennington
• REACH of Louisville

2
Partnership Overview

• Three-year project, competitively bid, currently
  entering second year
• Consolidates and integrates a number of planning
  and evaluation activities (previously contracted
  with multiple contractors) under one umbrella
• Three major components
• KIP Survey of youth in Grades 6, 8, 10, 12
• Analysis reporting of Synar survey results
• Comprehensive evaluation system design
  implementation

3
Major Activities to be Summarized in this
Presentation

• Planning and evaluation framework
• KIP survey administration and utilization
• Synar survey analysis and reporting
• PRI and EIP evaluations
• PPG Pilot Project strategic planning and
  accountability

4
Conceptual Framework for Substance Abuse
Prevention Planning and Evaluation

• An Empirically-Grounded Strategy for Organizing
  and Communicating Evaluative Information

5
Creating a Planning and Evaluation System That

• Is grounded in a conceptual map (framework) to
  consider how and when to utilize various types of
  data
• Can be readily comprehended by users
• Has practical utility and is used
• Leads to informed decision-making at all levels
• Results in improved programs, practices, and
  outcomes
• Is marketable and competitive with commercial
  programs, but at a far lower cost

6
(No Transcript)
7
KIP Survey of School-Age Youth A Core Building
Block of the Planning and Evaluation System
8
The Survey

• Administered bi-annually
• Voluntary school district and student
  participation
• Protections for anonymity
• Passive consent model
• About 40 minutes to administer
• Sent to service bureau for scanning and data
  verification
• REACH conducts statistical analysis and various
  district- and state-level reports

9
Scale Components

• Measures ATOD use, risk/resilience, and school
  safety variables for students in grades 6, 8, 10,
  12
• Demographics Age, gender, race, zip,
  achievement, grades, SES, school attendance,
  family constellation, urban/suburban/rural
• ATOD Use (personal peer) 30 day, past year,
  lifetime, onset
• ATOD-related problems/antisocial behavior
• ATOD accessibility
• Values (personal, parental)
• School safety
• Family harmony/conflict

10
Comparison to YRBS

• The YRBS (developed by CDC in 1990) has less
  items related to ATOD, especially risk
  resilience
• Surveys 60 high schools in KY, bi-annually
• YRBS also asks about issues of diet, physical
  activity, sexual behavior, and injury- or
  violence-related behavior
• Probably equally effective in obtaining
  state-level estimates of prevalence/incidence,
  enabling cross-state comparisons
• YRBS not designed to support development of
  preventive interventions at the community level,
  or monitoring of change

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
The Essential Role of KIP

• The KIP survey is the only comprehensive measure
  of youth substance use/abuse and related factors
  in Kentucky
• Results are primary tool for evaluation and
  program planning in prevention centers and school
  districts across the state
• Growing number of school-communities are
  participating (w/ exception of Jefferson County)

25
Synar Program EvaluationComplying with the
Federal Mandate to Reduce Underage Access to
Tobacco
26
Legal and Regulatory Basis

• SAMHSA regulation requires States to
• Have laws that prohibit manufacturer, retailer or
  distributor to sell to persons under age 18
• Enforce these laws effectively, reducing such
  sales
• Conduct annual, random, unannounced inspections
  to insure compliance
• Develop strategy to achieve inspection failure
  rate of less than 20 of outlets (higher rates
  face stiff penalties, possibly 8 million)
• Submit annual report
• KRS 438.300

27
Components of the Synar Program

• Annual Buying Survey
• Annual Synar report to the Federal government of
  compliance rates
• Public education utilization of findings
• Enforcement
• Regional Prevention Centers
• Tobacco Prevention Enhancement Site
• Press releases
• Vendor education
• Planning, coordination, training

28
Who Uses the Synar Data?

• Division of Regulation and Inspection (Dept of
  Agriculture)
• DSA Community Prevention Coordinator DSA Data
  Manager
• Tobacco Environmental Strategies Prevention
  Enhancement Site
• KY-ACTION Advocacy and Education Manager
• SA Expert Panel Coordinator
• Epidemiology Workgroup (UK)
• Tobacco Use Prevention and Cessation Program
• Kentucky Tobacco Policy Research Program (UK)
• Teen Tobacco Program Coordinator (ABC)
• Long-Term Policy Research Center, KY State Center
  for Health Statistics
• Regional Prevention Centers
• Many other public and private initiatives and
  activities

29
(No Transcript)
30
What does REACH of Louisville do?

• Designing the data collection process forms
• Sampling design Dun Bradstreet SIC Codes,
  geographic distribution
• Drawing the sample, insuring accuracy and
  representativeness
• Supporting the data collection process
  providing technical assistance
• Statistical analysis of the data
• Submitting results, coordinating with CSAP
• Facilitating use of the findings for SA
  prevention

31
2004 Synar Survey Current Status

• Sample of 1,400 from about 5,000 outlets
• Verified each location with phone call, narrowed
  to about 1,100
• Inspections during June and July, follow-ups in
  August
• About 90 complete, at least 50 cases per region
• Report due to CSAP in September
• Still hovering at around 7

32
Measuring Organizational Performance in Regional
Substance Abuse Prevention ProgramsA
Strategic Planning and Accountability Project
Piloted in Four Regions
33
Why Measure Performance?

• Link overall prevention strategy to key
  activities
• Promote activities that are logically related to
  goals and intended outcomes
• Resource allocation and focus
• Management tool
• Accountability to various publics
• Federal requirements (GPRA, PPGs)
• Usefulness in selling the viability and
  relevance of prevention in the overall system of
  care

34
What areas should be measured by an
organizational performance system?

• Capacity ability of regional prevention
  activities to address priority needs
• Effectiveness extent to which goals are
  attained (intended outcomes are reached)
• Accountability appropriate, ethical, and
  cost-efficient use of resources fidelity of
  program implementation

35
Capacity Goals

• Overall capacity to deliver prevention services
  (scope)
• Capacity for evidence-based practice
• Capacity to impact on priority needs (market
  penetration)
• Capacity for collaboration and coalitions
• Capacity to facilitate environmental change

36
Effectiveness Goals

• Use, risk perception, risk resilience factors,
  perceived attitudes, availability, perceived
  norms, etc.
• Evidence of change or goal attainment associated
  with prevention program participation
• Evidence of change or goal
  attainment in regional social
  indicators

37
Accountability Goals

• Staffing and work patterns
• Service delivery/program implementation
• Responsiveness to community needs
• Cost-efficiency

38
A Composite of Statewide Prevention System
Capacity, Effectiveness, and Accountability
Measures
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
EIP PRI Evaluation Activities
44
EIP Program Evaluation

• In the last FY, approximately 3,159 youth were
  referred, about ¾ males
• 36 from traffic court, 30 come from other
  courts (CDWs, juvenile court, drug court), and
  22 from schools.
• About ¾ of these enter the educational program
  (others are instead referred for mental health
  services after initial evaluation)
• Of these, 85.8 completed the educational
  intervention
• (PRIME for Life)

45
Current Status

• Initial efforts focused on comprehensive
  software re-design and data cleaning
• Generate monthly reports to Centers
  and to Substance Abuse Prevention office
• Maintain database of demographic and participant
  variables
• Pre- and post-tests scanned and matched with
  database
• Currently conducting implementation evaluation
  analyzing post-training follow-ups (despite low
  rate of return)
• Tentative agreement on a comprehensive overhaul
  of evaluation to streamline it, achieve higher
  return rates on follow-up, and assess impact
  relative to non-completers and other populations

46
PRI Evaluation

• Analysis of PRIME for Life classes
  conducted in Regional Prevention Centers
• Class lists received and attendance verified
• Scanning of pre- and post-tests
• Statistical analysis currently underway
• Conducting implementation evaluation of the
  program in support of achieving NREP status
• Will complete a comprehensive re-design of the
  evaluation strategy to streamline, and utilize
  Scantron forms for pre- and post-tests

47
Other Activities

• Support of Expert Panel
• Grant-writing
• Participation in Epidemiology Workgroup
• Planning for data warehouse design
• Collaboration with KDOE, Center for School
  Safety, RPCs, ABC, etc.
• Coordination with CSAP (e.g., PPGs)
• Research newsletters
• Special data requests
• Training and dissemination activities
• Advanced statistical analysis GIS

48
(No Transcript)
49
Overarching Goals of Evaluation Partnership

• To improve safety and health of Kentuckians by
  providing more accurate, comprehensive, and
  accessible information about use/abuse of ATOD to
  inform the actions of federal and state
  government, community groups, local school
  districts, law enforcement, parents, and other
  concerned citizens

50
Overarching Goals (contd)

• To enable Kentucky to fully comply with federal
  requirements related to Performance Partnership
  Grants (PPGs)
• To retain the number of federal dollars coming to
  Kentucky by maintaining compliance with federal
  Synar requirements
• To support state program staff in their efforts
  to assure accountability for the expenditure of
  public funds
• To increase the number of federal and foundation
  dollars available to Kentuckians for the
  prevention of ATOD abuse through assistance in
  grant writing and availability of credible data