Evaluation of the DC Opportunity Scholarship Program: Achievement Impacts After Three Years of Treatment Exposure

1
Evaluation of the DC Opportunity Scholarship
ProgramAchievement Impacts After Three Years of
Treatment Exposure

• Conducted by
• Westat, University of Arkansas, Chesapeake
  Research Associates, Georgetown University
• Presented by
• Patrick Wolf, Principal Investigator
• Reversion to the Mean or Does Dosage Matter?
• IES Annual Research Conference
• June 8, 2009
• Washington, DC

2
Presentation Overview

• The Program
• The Study
• Achievement Impacts in Year 3
• Achievement Impacts Over Time
• Dosage in the Context of School Choice

3
The Opportunity Scholarship Program

• DC School Choice Incentive Act signed in early
  2004
• 14 million/yr for 1,700 student vouchers worth
  up to 7,500
• Eligible students must be
• DC residents
• entering grades K-12
• income lt 185 percent of poverty
• Priority to applicants from needs improvement
  (SINI) schools
• 68 participating private schools in D.C.

4
The Impact Evaluation

• Randomized Control Trial (RCT) Design baseline
  eligible applicants assigned by lottery to be
  offered (treatment group) or not offered
  (control group) a scholarship
• Impact Sample
• First two cohorts in 2004 (Cohort 1) and 2005
  (Cohort 2)
• 1,387 students assigned to treatment, 921 to
  control
• Cohort 1 outcomes lagged one year and combined
  with Cohort 2
• Outcomes tracked annually SAT-9 in reading and
  math, parent surveys, student surveys, principal
  surveys (public and private)
• Analysis Regression-adjusted intent-to-treat
  (ITT) impact, Bloom-adjusted impact-on-treated
  (IOT), Instrumental Variable Analysis of effect
  of private schooling

5
DC Choice Year 3 Achievement Impacts

• Overall
• Reading impact of 4.5 scale score points (.13 SD
  or 3.1 months of learning)
• No impact in math
• Subgroups by student baseline conditions
• Reading impacts for (1) non-SINI, (2) higher
  performing, (3) female, (4) K-8, (5) Cohort 1
• No impacts for (1) SINI, (2) lower performing,
  (3) male, (4) 9-12, (5) Cohort 2
• No impacts for any subgroups in math

6
Regression Adjusted Impact and Confidence
Interval Year 3 Reading
Scale score points
7
Regression Adjusted Impact and Confidence
Interval Year 3 Math
8
DC Choice Achievement Impacts Over Time

• Overall effects across 3 outcome years
• Apparent trend of cumulating impacts in reading
• No apparent trend in math
• Subgroup reading effects across 3 outcome years
• Of 5 with significant impacts in year 3, all but
  Cohort 1 suggest cumulating impacts
• Of 5 without impacts in year 3, X suggest
  cumulating impacts
• Data not pooled across years so significance of
  trends unknown

9
Impacts Over Three Year for the Full Sample
Scale score points
10
Impacts Over Three Years SINI at Baseline
Subgroups
Scale score points
11
Impacts Over Three Years Baseline Performance
Subgroups
Scale score points
12
Impacts Over Three Years Gender Subgroups
Scale score points
13
Impacts Over Three Years Grade Subgroups
Scale score points
14
Impacts Over Three Years Cohort Subgroups
Scale score points
15
The DC Choice Evaluation and Treatment Dosage

• Impact patterns differ by domain
• Math may have suffered reversion to the mean
• Reading impacts appear to cumulate
• No obvious reason for the difference
• School choice interventions may require long
  treatment exposure to demonstrate results
• Starts with adjustment to a school switch
• May then place student on steeper growth curve
• Bring on Year 4!

16
Contact Info

• Patrick Wolf, Ph.D.
• Professor and 21st Century Endowed Chair in
  School Choice
• Department of Education Reform
• College of Education and Health Professions
• 201 Graduate Education Building
• University of Arkansas
• Fayetteville, AR  72701
• Phone [phone]
• FAX [phone]
• pwolf_at_uark.edu