Healthy Schools Leadership Program HSLP Evaluation Plan - PowerPoint PPT Presentation

Loading...

PPT – Healthy Schools Leadership Program HSLP Evaluation Plan PowerPoint presentation | free to download - id: 3d50e-MzhmO



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Healthy Schools Leadership Program HSLP Evaluation Plan

Description:

Consistent with National YRBS & YTS. Development led by DOH. YRBS ... New junk food module in 2006 asks about soda pop and junk/fried food consumption ... – PowerPoint PPT presentation

Number of Views:70
Avg rating:3.0/5.0
Slides: 70
Provided by: JuliaD9
Learn more at: http://www.k12.wa.us
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Healthy Schools Leadership Program HSLP Evaluation Plan


1
Healthy Schools Leadership Program
(HSLP)Evaluation Plan
  • February 2006

2
What Im going to talk about
  • Present get your feedback on Evaluation Plan
    for HSLP 06-08 cohort
  • Generate ideas for your own assessment/evaluation
    work
  • Ideas for using the Healthy Youth Survey
  • ? Please consider this a discussion!

3
George tried to be a good monkey
4
The Primary Evaluation Question
  • Is the Healthy Schools Leadership Program (HSLP)
    effective for making schools healthier?
  • Note that this is a state-level evaluation
    question
  • I will present an ambitious scenario for
    evaluation
  • We will attempt to be as rigorous as possible
  • Im not a schools expert you are the schools
    experts

5
Caveats to Evaluation Model
  • Small numbers of schools (n15)
  • Change takes time especially for long-term
    outcomes
  • Our comparison group is the rest of the state
  • If findings are positive, some will argue that
    this is the results of your existing
    capacity/readiness
  • The plan is ambitious looking for additional
    resources to support evaluation

6
Id like to say something stronger
  • What we can say right now with certainty is
    that if you spend resources implementing
    coordinated school health it doesnt appear to
    make your academic achievement any worse.

7
Describe the Program
8
Coordinated School Health Goals
  • Health Education
  • Physical Education
  • Counseling/Support Services
  • Food/Nutrition Services
  • Healthy Environment
  • Health Services
  • Family/Community Involvement
  • Staff Wellness

9
Simplified Logic Model
B. Activities Asst Plan Evaluate
A. Inputs Teams HSLP training
C. Outputs Asst results Action Plan
E. Intermediate Outcomes Policy, procedure,
environment changes Sustainability
D. Short-term Outcomes Results from action plan
activity
F. Long-term Outcomes Staff/studnt behavior
change
G. ImpactsHealth improvemt Academic achievemt
Catalysts
Barriers
Non-HSLP External Factors
10
Getting the Right Perspective Evaluation
Questions
11
Specific Aim
  • 1. Describe the extent to which the inputs
    and activities are sufficient to generate outputs
    assessment results, action plans (A/B?C,
    Descriptive)
  • Were teams able to assemble wellness committees?
    How did they function? Were they able to complete
    assessment and planning? What priorities were
    identified? What were the barriers and motivating
    factors related to output completion?
  • Are there differences among HSLP teams in
    satisfaction with outputs? What are the factors
    (HSLP or other) associated with high/low
    satisfaction? Was HSLP a critical factor in
    achievements of high-quality outputs?
  • Data Sources Team interviews, action plans,
    Healthy Schools Report Card

12
Hypothetical Findings
  • All schools assembled teams X included
    community members
  • 12 of 15 teams created formal action plans
  • All schools identified at least one nutrition
    priority
  • School teams that were very satisfied with
    their plans were more likely to have involved
    community members and had a clear process for
    translating assessment results into an action plan

13
Specific Aim
  • 2. Describe the extent to which participation
    in the HSLP is associated with short-term
    outcomes early results of action plans (A/B?D,
    Descriptive)
  • Were school wellness teams able to use plans to
    initiate change? How did implementation activity
    begin? Which priorities did schools take action
    on first? How were responsibilities identified?
    What factors were associated with transition from
    planning to action?
  • Are there differences among HSLP schools in their
    satisfaction with/quality of early
    implementation? Were some kinds of activities
    more easily translated to quality action? What
    are the factors (HSLP or other) associated with
    high/low satisfaction? Was HSLP a critical factor
    in initiating high-quality work?
  • Data Sources Team interviews, action plan
    reports, Healthy Schools Report Card

14
Hypothetical Findings
  • 10 of 15 schools increased the overall number of
    maintenance items on their Healthy Schools
    Report Card
  • Teams reported highest satisfaction with their
    work on XXX priorities
  • XX Teams (X) said they strongly agreed that
    change would not have occurred without the HSLP
    training and technical assistance

15
Specific Aim
  • 3. Describe the extent to which schools that
    participated in HSLP change their environments
    (A/B?E, Pre/post comparison to non-HSLP schools)
  • Did HSLP school environments change, and were
    changes different than those of non-participating
    schools? Which changes were achieved?
  • Are there differences among HSLP schools that
    achieved policy/procedure/environment change vs.
    those that did not?
  • How has sustainability been addressed? What
    additional resources have been attracted for
    school health? Has institutionalization of school
    health work been achieved?
  • Data Sources SHEP, Healthy Schools Report Card,
    HYS (perceived school environments), Staff
    wellness survey (if perceived environment
    indicators included)

16
Sample Findings
Source SHEP 2004
17
Specific Aim
  • 4. Describe the extent to which schools that
    participated in HSLP change their student/staff
    behaviors (A/B ? F, Pre/post comparison to
    non-HSLP schools for students, self-comparison
    for staff)
  • Did HSLP schools student behaviors change, and
    were changes different than those of
    non-participating schools?
  • Did HSLP schools staff behaviors change?
  • Are there differences among HSLP schools that
    achieved behavior change vs. those that did not?
  • Data Sources HYS, Staff Wellness Survey

18
Sample Findings
Source Healthy Youth Survey 2002-2004
19
Specific Aim
  • 5. Describe the extent to which schools that
    participated in HSLP have had health impact (A/B
    ? G, Pre/post comparison to non-HSLP schools for
    WASL, self-comparison for absenteeism)
  • Did HSLP schools student academic achievement
    change, and were changes different than those of
    non-participating schools?
  • Did HSLP schools student and/or staff
    absenteeism rates change?
  • Are there differences among HSLP schools that
    achieved impact vs. those that did not?
  • Data Sources WASL, Student absenteeism measures,
    Staff absenteeism measures

20
Sample Findings
21
Data Sources Gathering Credible Evidence
Not everything that counts can be counted and
not everything that can be counted counts.
Albert Einstein
22
Data Sources
  • Proposing to use data you already collect, or new
    data collection related to your HSLP
    implementation
  • Exception interviews of team/administration
  • For anything where we want to look at everyone
    together, methods have to be standard
  • For some things it would be fine to have sentinel
    schools
  • See school-specific data suggestions later

23
Team Interviews
  • During Spring 06 develop interview protocols for
    assessing function and satisfaction among HSLP
    participants (see evaluation questions)
  • Repeat in Spring 08
  • Augment with interviews with other wellness
    committee members (or lead administrators?)
  • Your role participate in interviews

24
Healthy Schools Report Card
  • Conduct assessment to identify priorities for
    action planning
  • ASCD (Association for Supervision and Curriculum
    Development) has a clear process/instrument
  • Your role use good assessment to develop plans,
    provide a copy of whatever baseline (spring 06)
    and follow-up (spring 08) assessment results

25
HSRC Component Areas
  • School health program policy strategic planning
  • Coordination of school health programs
  • Social and emotional climate
  • Family and community involvement
  • School facilities and transportation
  • Health education
  • Physical education and physical activity
  • Food and nutrition services
  • School health services
  • Counseling, psychological and social work
    services
  • School-site health promotion for staff

26
HSRC Indicator Priority Rankings
  • Multiple specific topics within components
  • Rate with your team for each current status,
    perceived benefit, perceived effort
  • Color-coded result report for combined scores
  • Green short-term high priority
  • Red short-term lower priority
  • Lime long-term, highest priority
  • Blue long-term, high priority
  • Orange long-term, lower priority
  • No color Maintenance

27
HSRC Sample Component-Topic
  • School health program policy strategic planning
  • All staff members are provided with
    time/resources to comply with health program
    policies
  • Coordination of school health programs
  • School health program members regularly inform
    the principal, district superintendent, or school
    board of current developments in the school
    health program.

28
Annual Action Plans Reports
  • Design your action plan based on assessment
    results
  • Your role provide action plan annually, report
    of progress annually (format?)

29
School Health Education Profile (SHEP)
  • Survey of Secondary School Principals and Lead
    Health Teachers
  • Sponsored by CDC
  • Will be sent in Spring 06 and Spring 08
  • Your role encourage completion of the survey

30
SHEP/Profiles Data ExamplesSchool Health
Advisory Groups
  • Have a School Health Advisory Group
  • 59 of high schools
  • 56 of middle schools
  • 46 of combined secondary schools
  • 55 overall
  • Similar to 52 total in 2002

31
SHEP/Profiles Data ExamplesTobacco
  • A designated individual has primary
    responsibility for seeing that the tobacco
    prevention policy is enforced
  • 64 high school
  • 68 middle school
  • 59 secondary combined
  • 65 overall
  • Similar to 68 overall in 2002
  • No association with school advisory presence

32
SHEP/Profiles Data ExamplesNutrition
  • Students can purchase fresh fruits/vegetables
    from vending machines or at the school store,
    canteen, or snack bar
  • 49 high school
  • 45 middle school
  • 39 secondary combined
  • 46 overall
  • Slight improvement from 39 overall in 2002
  • No association with school advisory presence

33
SHEP/Profiles Data ExamplesAsthma
  • Obtain and use an Asthma Action Plan (or
    Individualized Health Plan) for all students with
    asthma
  • 77 high school
  • 77 middle school
  • 74 secondary combined
  • 76 overall
  • Improvement from 62 overall in 2002
  • Schools with advisories are twice as likely to
    obtain asthma plans as schools without advisories

34
Staff Wellness Survey
  • Designed by CDC available in March
  • Assuming we can modify (shorten) if needed
  • Assuming we (DOH/OSPI) can provide as online
    survey and give data back to school teams
  • Are there other ways to measure staff absences?
  • Your role engage staff to participate
  • Caution poor participation will not yield useful
    data!

35
Healthy Youth Survey (HYS)
  • Administered Fall of even years
  • Grades 6,8,10,12
  • How many HSLP schools participated in 2002, 2004?
  • Most powerful comparison will be 02-04-06
  • Your role register to participate in 2006 (soon!)

36
Student Absenteeism
  • NCLB requires measurement of excused/unexcused
    absences
  • Assuming no change in non-illness excused
    absences, we can monitor changes in this rate
    over time
  • Your role provide annual absence rates

37
Academic Achievement/WASL
  • Annual, building-level
  • Grades 4,7,10
  • Math, Reading, Writing
  • Your role none needed

38
Other School Factors
  • Free reduced lunch enrollment this may be an
    outcome, consider different measure of SES
  • Percent minority enrollment
  • School/district size/enrollment
  • School level (elementary, middle, high,
    combination)
  • Urban/rural location
  • Baseline levels of outcomes
  • Other (baseline) funding sources for health
    improvement
  • Your role none we can obtain

39
Too much information?
40
Timeline
41
School-level Evaluation
  • We wont evaluate you as individual teams
  • Evaluating yourself on a few points would be
    helpful in telling your story later
  • Suggest designating a lead for local evaluation
    (just like any implementation piece)

42
Sample School-level Evaluation Data
  • vending revenue data
  • key informant interviews with students or staff
  • tracking of policy enforcement activities
  • other data gathering from communities
  • health room utilization data
  • student absences use more advanced tracking
    system (Public Health Tracking System grant
    available from DOH environmental health)
  • lots of (digital) pictures of your environment
    now and later

43
Still Awake??
44
Healthy Youth Survey
45
Why do I need HYS now?
  • Contribute to assessment, planning
  • Identification of your priorities
  • Describing affected students
  • Gain attention for student health as a priority

46
HYS 2004 Collaborative Effort of OSPI, DOH,
DSHS, CTED, FPC, and RMC Research
  • State-level simple random sample of schools
  • County-level samples drawn as appropriate
  • Non-sampled schools also invited to participate
  • Survey consistently administered in the Fall of
    even years
  • Survey given to 6th, 8th, 10th, and 12th graders
  • Survey booklets have one-page tear-off answer
    sheet
  • Survey for 8th, 10th, and 12th graders uses 2
    form interleaved administration

47
HYS 2004 Participation
  • The Healthy Youth Survey 2004 was completed

By 185,095 students
In 1,013 schools
In 235 school districts
In all 39 counties.
48
Risk Protective Factor DataConsistent with CTC
MTF SurveysDevelopment led by DSHS/DASA OSPI
  • WSSAHB Questionnaire
  • Demographics
  • Alcohol, Tobacco, other drugs
  • School risk/protective factors
  • Community risk/protective factors
  • Peer-Individual-Family risk/ protective factors

49
Youth Health Behavior DataConsistent with
National YRBS YTSDevelopment led by DOH
  • YRBS Questionnaire
  • Demographics
  • Alcohol, Tobacco, Other drugs
  • Nutrition
  • Physical Activity
  • Safety Behaviors (Helmets, Seatbelts)
  • Mental Health/Depression/Suicide
  • Additional Tobacco Indicators

50
Getting the Best of Both
Form A WSSAHB-like
Form B YRBS-like
Survey Core
51
6th Grade Survey
One Form
Survey Core Mutually Agreed Items
52
Core Survey Items
  • Demographics
  • Alcohol, Tobacco and Other Drug Use Items
  • Key Violence-Related Items
  • School-Specific Asset Items
  • Depression

53
Administration procedures
  • Safe and confidential environment
  • Standardized administration procedures (e.g.,
    coordinator training, teacher training, written
    instructions, teacher stays in room but at desk,
    single class period to avoid discussion, absent
    students do not make up)
  • Students informed of importance of survey
  • No student name or other identifying information
  • Students place own answer sheet in envelope
  • Students provided resource list

54
Data Cleaning Procedures
  • Inconsistent answers
  • Evidence of faking high level of substance use
  • Dishonesty
  • Wrong grade

55
What is a Confidence Interval?
  • The reported value is probably a little different
    than the true value for all your students
  • We are 95 confident that the true value is
    within the /- range, called the confidence
    interval

56
Interpreting CIs Graphically
57
Local Report Contents
  • Introduction and Overview
  • Highlights of Local Results
  • Selected Results by Gender
  • Responses to Items
  • Risk and Protective Factors
  • List of Core Items Item index

58
For your continued exploration
  • http//www.rmccorp.com/HYS06/index.htm

http//www3.doh.wa.gov/hys/
http//www3.doh.wa.gov/hys/
59
  • Students with severe persistent asthma symptoms
    had
  • increased odds for academic risk

Source 2004 Healthy Youth Survey, state sample
only. Graph uses unadjusted estimates odds
ratios are for estimates adjusted for grade
(8-10-12) and maternal education (3-level).
60
  • Students with diabetes had increased odds for
    academic risk

Source 2004 Healthy Youth Survey, state sample
only. Graph uses unadjusted estimates odds
ratios are for estimates adjusted for grade
(8-10-12) and maternal education (3-level).
61
  • Students who smoked any cigarettes during the
    past month
  • had increased odds for academic risk

Source 2004 Healthy Youth Survey, state sample
only. Graph uses unadjusted estimates odds
ratios are for estimates adjusted for grade
(8-10-12) and maternal education (3-level).
62
Healthy Youth Survey Website
  • Some quick crosstabs examples you could run
  • Kids who smoke are less likely to report having
    access to an ATOD counselor (61 among smokers v.
    71 among non-smokers, 10th grade state sample)
  • Kids who report being bullied are also more
    likely to report depression (46 depression among
    bullied v. 29 among non-bullied, 10th grade)
  • Kids who drank more than 2 soda pops per day were
    more likely to report getting Ds and Fs (7 vs.
    15 among heavy pop-drinkers, 10th grade)

63
Healthy Youth Survey 2006 HSLP related measures
  • Physical Education
  • Number of days of PE per week
  • How long doing actual activity in PE class
  • Counseling/Student Support Services
  • Is there a school counselor to talk about ATOD
  • Is there someone you could go to for help
  • Food/Nutrition Services
  • New junk food module in 2006 asks about soda pop
    and junk/fried food consumption at school, also
    breakfast
  • Fruit vegetable 5-a-day consumption
  • Soda pops per day (previous day)

64
HYS 06 HSLP-related measures
  • Healthy Environment
  • Feeling safe at school
  • Enjoy being in school/hate being in school
  • Carry weapon on school property
  • Bullying
  • Teachers care about me
  • Using tobacco on school property
  • Been drunk or high at school
  • Tobacco rules are usually enforced
  • Health Education
  • Number of times received tobacco education
  • Number of times practices ways to say no
  • How good in school at educating about HIV

65
HYS 06 HSLP-Related Measures
  • Family/Community Involvement
  • Dinner with family at home
  • Parent/guardian discussed tobacco at home
  • How easy to get tobacco/alcohol if you wanted
    some
  • Student Support Services
  • Ever had an asthma action plan

66
Your Questions
  • What if I cant find my school or district
    report?
  • What if I just need some general help getting
    started with HYS data or the website?
  • What if I want more analysis done with our data?
  • What if I have specific questions about HYS data
    and our Health Department staff cant help me?

67
You made it!
68
What we talked about
  • Present get your feedback on Evaluation Plan
    for HSLP 06-08 cohort
  • Generate ideas for your own assessment/evaluation
    work
  • Ideas for using the Healthy Youth Survey

69
Here to help!
  • General Questions
  • Lori, Martin, Pam
  • HSLP Evaluation Detail
  • Julia Dilley 360-402-7877
  • julia.dilley_at_state.or.us
  • Healthy Youth Survey
  • Diane Pilkey 360-236-3526
  • diane.pilkey_at_doh.wa.gov
About PowerShow.com