PD 360 Impact Assessment: Initial Findings regarding the Impact of PD 360 on Student Proficiency Rates

1
PD 360 Impact AssessmentInitial Findings
regarding the Impact of PD 360 on Student
Proficiency Rates

• Prepared by Steven H. Shaha, PhD, DBA

2
Overarching Research Question

• Does teacher engagement
• in PD 360 and Observation 360,
• tools within the Educator Effectiveness System,
• significantly affect
• student success?

3
Methods

• Design Quasi-experimental, retrospective,
  pre-post, normalized treatment-control /
  participation vs. non-participation (2009-10,
  2010-11)
• Goal Multi-State, large n with comparable
  student populations (matched, controlled)
• Student Change Metric was percent students
  classified as Proficient or Advanced in
  respective States.

4
Sample

• Participation
• Systematic sample of 187 schools in 8 States
• N determined by a priori Power analysis
• Schools eligible for inclusion in the sample as
  participating Schools met the following criteria
• More than 10 teachers total
• 80 or more of teachers viewed materials
• Minimum average of 90.0 minutes of viewing per
  teacher for the school
• Districts included were only those for which
  eligible schools were included
• Normalizing for difference in socio-economic and
  demographic factors between participating Schools
  and their Districts cumulatively as the
  statistical comparison group
• Data
• Participation data were extracted from the
  Internet-based professional development
  application as surveilled
• Student performance data were captured from
  publically available, Internet-accessed sources
  (school as unit of measure, percent Proficient or
  Advanced as metric)

5
Improvements in Proficiency Rates Correlated with
PD 360
Sample Demographics

• Eight States
• 187 Schools

6
Metrics for Differentiating Advantages for
Higher Engagement Organizations

• Focus Objectives Set Up
• Observations Performed
• Percent Registered Users
• Percent of Users in Communities
• Minutes Viewed
• Forums Viewed
• Programs Viewed
• Segments Viewed
• Links Viewed
• Follow-up Questions Answered
• Reflection Questions Answered
• Focus Objectives Set Up
• Forums Posted
• Downloaded Files
• Uploaded Files
• Participation in Communities

Leadership, Implementation and Accountability
Educator Participation
Educator Engagement
These are the 15 metrics for which higher
engagement schools were significantly higher than
their lower engagement counterparts
7
PD 360 Impact AssessmentExecutive Summary
Initial Findings

• Statistically significant and resounding
  advantages were identified favoring schools with
  PD 360.
• The higher the quantified utilization, the
  greater the statistical advantage.
• High utilization led to performance advantages
  ranging from twice to 15 times the gains in
  proficiency rates versus district averages (plt.01
  and plt.001, respectively).
• Low utilization led to gains in proficiency rates
  from 10 to twice the district averages (plt.01
  and plt.001, respectively).

8
Improvements in Proficiency Rates Correlated with
PD 360
High Utilization results in more the TWICE the
gains in Math proficiency rates than does Low
Utilization (plt.001)
High Utilizers Outperformed all others
But even low utilizers outperformed no PD 360
High Utilization results in nearly TWICE the
gains in Reading proficiency rates than does Low
Utilization (plt.001)
9
Improvements in Proficiency Rates Correlated with
PD 360
High Utilization results in more the FIFTEEN
times the gains in Math proficiency rates than
the Districts average (plt.001)
High Utilizers outperformed no PD 360
High Utilization results in nearly TWICE the
gains in Reading proficiency rates than the
Districts average (plt.001)
15.8 times greater gains in Math vs. District
90.6 greater gains in Reading vs. District
10
Improvements in Proficiency Rates Correlated with
PD 360
Low Utilization results in more the SEVEN times
the gains in Math proficiency rates than the
Districts average (plt.001)
But even Low Utilizers outperformed no PD 360
Low Utilization results in nearly 10 greater
gains in Reading proficiency rates than the
Districts average (plt.01)
9.3 greater gains in Reading vs. District
7.4 times greater gains in Math vs. District
11
Improvements in Proficiency Rates Correlated with
PD 360
High Minutes per User result in more the TWELVE
times the gains in Math proficiency rates than
Districts avg. (plt.001)
More Minutes per User resulted in Higher
Performance
High Minutes per User result in nearly TWICE the
gains in Reading proficiency rates than the
Districts average (plt.001)
71.5 greater gains in Reading vs. District
12.0 times greater gains in Math vs. District
12
Improvements in Proficiency Rates Correlated with
PD 360
Low Minutes per User result in more the TEN times
the gains in Math proficiency rates than the
Districts average (plt.001)
But even Low Minutes per User outperformed no PD
360
Low Minutes per User result in more than 30
greater gains in Reading proficiency rates than
the Districts average (plt.001)
32.8 greater gains in Reading vs. District
10.4 times greater gains in Math vs. District
13
Improvements in Proficiency Rates Correlated with
PD 360
High Percentage of Users that Viewed resulted in
more the THIRTEEN times the gains in Math
proficiency rates than the Districts average
(plt.001)
Higher percentage of Viewers resulted in Higher
Performance
High Pct of Users that Viewed resulted in TWICE
the gains in Reading proficiency rates than the
Districts average (plt.001)
95.8 greater gains in Reading vs. District
13.5 times greater gains in Math vs. District
14
Improvements in Proficiency Rates Correlated with
PD 360
Low Percent of Users that Viewed resulted in more
the NINE times the gains in Math proficiency
rates than the Districts average (plt.001)
But even Low percentage of Viewers outperformed
no PD 360
Low Percent of Users that Viewed resulted in more
than 20 greater gains in Reading proficiency
rates than the Districts average (plt.001)
23.6 greater gains in Reading vs. District
9.4 times greater gains in Math vs. District
15
Improvements in Proficiency Rates Correlated with
PD 360
High Number of Users that Viewed resulted in more
the THIRTEEN times the gains in Math proficiency
rates than the Districts average (plt.001)
More Viewers resulted in Higher Performance
High Number of Users that Viewed resulted in
nearly TWICE the gains in Reading proficiency
rates than the Districts average (plt.001)
92.5 greater gains in Reading vs. District
13.4 times greater gains in Math vs. District
16
Improvements in Proficiency Rates Correlated with
PD 360
Low Number of Users that Viewed resulted in
nearly TEN times the gains in Math proficiency
rates than the Districts average (plt.001)
But even fewer viewers outperformed no PD 360
Low Number of Users that Viewed resulted in more
than 30 greater gains in Reading proficiency
rates than the Districts average (plt.001)
31.6 greater gains in Reading vs. District
9.7 times greater gains in Math vs. District