Use of Growth Models to Measure Adequate Yearly Progress (AYP) Under the No Child Left Behind (NCLB) Act

1
Use of Growth Models to Measure Adequate Yearly
Progress (AYP) Under the No Child Left Behind
(NCLB) Act

• Lou Jacobson Presentation at the
• IES Research Conference
• June 7, 2007

2
Topics to be Covered

• Research for REL Appalachia
• Year-1 study examining the likelihood Virginia
  schools would have all students proficient in
  reading math by 2014.
• Todays focus is on reading results for all
  students together.
• The study also covers math and each of 6
  subgroups.
• The report will be on the REL website in about a
  month.
• Year-2 study examines a second key NCLB
  goalidentifying under-performing schools
• Use growth models developed in year-1.
• Examine the accountability systems in Virginia,
  Kentucky, and Tennessee using school level data.

3
Virginias NCLB Status Standard

• 100 of the students in a school must be
  proficient in math and reading by 2014
• Adequate Yearly Progress (AYP) requires meeting
  interim standards annually.

4
Starting Reading Proficiency Scores
5
As proficiency levels rise, how will growth
change?

• Strategy for answering this question
• Estimate the relationship between the 2002
  proficiency level and the change in proficiency
  2002-05.
• Forecast growth assuming that as proficiency
  rises schools starting at low levels will follow
  the same path as schools starting out at high
  levels.

6
Growth is Inversely Related to Starting Point
7
Our Growth Model

• ?P a ß P0 e
• where
• ?P (P3 P0)/3 (annualized proficiency
  change)
• Ps,t is the percent of test-takers scoring
• proficient in school s in year t
• a the intercept coefficient
• ß the slope coefficient
• e error term
• Averaging the proficiency change over 3 years
  reduces the effect of random variations, which
  accounts for as much as 75 percent of
  year-to-year change.

8
Model Estimates
School Type Intercept a Proficiency Level ß Adj. R2 N
Elementary 16.4 (0.41) -.189 (0.005) .543 1089
Middle 10.3 (0.70) -.114 (0.010) .308 296
High 19.6 (1.00) -.221 (0.012) .526 287
Standard errors in parentheses
9
What Does the Model Say about High Schools?

• Annual proficiency growth fell by 2.2 percentage
  points for each 10-point gain in proficiency.
• It becomes progressively more difficult to boost
  proficiency as higher and higher proficiency
  levels are reached.
• Schools are unlikely to raise proficiency by a
  constant number of students as the pool of
  non-proficient students shrinks.

10
How High Will Proficiency Rise?

• Is there a proficiency ceiling?
• VA schools have had major increases in
  proficiency, BUT
• Proficiency levels will plateau in about 3 years
• How do we know? Solve the growth equation for
  zero growth.
• C a / ß
• where
• C the proficiency ceiling
• a the intercept coefficient
• ß the slope coefficient
• Where are the plateaus?

11
Steady State Proficiency Levels
13 points
20 points
27 points
Upper Limit
Lower limit 1 std
dev below average
12
Implications of These Findings

• Few, if any, schools will sustain 100
  proficiency levels, if current patterns continue.
• Schools under-performing the status standard can
  avoid being labeled in need of improvement
  because there is an alternative NCLB standard.
• Safe Harbor (Growth) Standard requires that
  schools not meeting the status standard must
  reduce the percentage of non-proficient students
  by 10 percent from one year to the next.
• A school at 80 percent proficiency has to
  increase proficiency by 2 percentage points
  100-80 20 10 of 20 2
• A school at 90 percent proficiency has to
  increase proficiency by 1 percentage point.
• In contrast, the Virginia Status Standard rises
  by 4 points per year.

13
Can a growth standard identify under-performers?

• Yes, if we can determine how proficiency changes
  among schools facing similar challenges.
• To do this it is necessary to develop a
  peer-group standardthe standard experts agree
  provides the best means to identify schools
  needing improvement.

14
Developing a Peer-Group Standard

• Use our growth model to measure the difference
  between actual and predicted growth over 3-years.
• Identify under-performing schools based on
  performance being below average by a
  statistically significant amount.
• Group schools based on exogenous characteristics
  such as students entry level proficiency.

15
Distribution of High Schools Sorted by Difference
Between Actual and Predicted Performance
Under performing
Over performing
16
What the figure tells us

• The difference between actual and predicted
  growth is close to normally distributed.
• Most schools actual performance is within one
  standard deviation of predicted levels.
• A few schools under-perform by large and
  statistically significant amounts.
• A few schools over-perform by large and
  statistically significant amounts.

17
Advantages of Using the Growth Model

• The growth model provides
• A rating for each school that tells us how far
  its performance is from average for schools with
  similar baseline proficiency.
• Measures of the statistical significance of the
  ratings.
• In contrast, Virginias NCLB AYP standards
  provide
• Indiscriminant pass/fail measures.
• Ambiguous indicators of whether performance is
  truly below a standard.

18
Benefits of the Peer-Group Concept

• Improvement efforts can be focused on schools
  needing improvement that should be able to make
  substantial progress.
• Incentives to improve performance will be
  strengthened by not asking educators to do the
  impossible.
• External pressure to correct problems will be
  increased because confidence that problems are
  real and can be effectively addressed will grow.

19
Benefits of Identifying Over-Performing Schools

• Comparisons between over-performing schools and
  under-performing schools can identify factors
  that are
• not under the control of school officials that
  should be taken into account in creating
  peer-groups.
• under the control of school officials that make a
  substantial difference among schools facing
  similar challenges.

20
Can Safe Harbor Standards Identify
Under-Performers?

• Yes, if
• Growth is averaged over several years.
• Tests of the statistical significance are
  applied.
• However, regression-based standards better
  measure average performance of schools at
  different proficiency levels.

21
Some Schools are Misclassified by Current
Standards
Over performing
Under performing
Some of these schools miss AYP
Some of these schools make AYP
22
Safe Harbor Pass/Fail Based on Annual Average
Growth by Performance Quintile
23
Safe Harbor Pass/Fail Based on 3-Year Average
Growth by Performance Quintile
24
Summary Proficiency Forecasting

• The simple linear model using base-period
  proficiency fits the data well and produces
  reasonably small confidence intervals.
• Additional factors are unlikely to have much of
  an effect.
• The future may not precisely track the past, but
  is likely to come reasonably close.
• It would be useful to see how well our model
  works in projecting proficiency in other states.

25
Summary Peer-Group Modeling

• Our peer-group model comes reasonably close to
• Revealing how far above or below average is
  actual performance.
• Identifying cases where deficits are
  statistically significant.
• Averaging growth over several years greatly
  improves the statistical reliability of the
  measures.
• Our model addresses widely recognized problems
  that reduce support for the positive aspects of
  NCLB.