Summary Statements

1
SummaryStatements
2
The problem . . .

• Progress data included
• 5 progress categories
• For each of 3 outcomes
• Total of 15 numbers reported each year
• Too many interrelated targets to make sense of
• OSEP asked for a recommendation

3
Thinking through the summary statements

• ECO presented options to states and ECO work
  groups via conference calls
• Two sessions at December, 2008 EC Conference
• Posted on the ECO web site for comments
• ECO made recommendation to OSEP

4
Final Deliberation

• OSEP put the summary statements out for public
  comment
• Comments came in that were thoughtful, but not
  necessarily consistent with one another
• Advantages and disadvantages to all options

5

• Paper documenting the process on the ECO website
• Setting Targets for Child Outcomes

6
The Summary Statements

• Of those children who entered the program below
  age expectations in each Outcome, the percent who
  substantially increased their rate of growth by
  the time they exited the program.
• The percent of children who were functioning
  within age expectations in each Outcome by the
  time they exited the program.

7
Example of State Progress Data for 2008-2009
8
Summary Statement Data

• Required Summary Statement 1
• Of those children who entered the program
  below age expectations in each Outcome, the
  percent who substantially increased their rate of
  growth by the time they exited the program 75
•  
• Required Summary Statement 2
• The percent of children who were functioning
  within age expectations in each Outcome by the
  time they exited the program 54

9
Where do the s come from?

• Measurement for Summary Statement 1
• Percent of infants and toddlers reported
  in progress category (c) plus of infants and
  toddlers reported in category (d) divided by
  of infants and toddlers reported in progress
  category (a) plus of infants and toddlers
  reported in progress category (b) plus of
  infants and toddlers reported in progress
  category (c) plus of infants and toddlers
  reported in progress category (d) times 100.

10
Where do the s come from?
760 (a, b, c, and d) or 76 of the children
entered the program functioning below age
expectations
240 (e) or 24 of the children entered and exited
functioning at age expectations
11
Where do the s come from?
570 (c and d) of the 760 (a, b, c, and d) changed
their growth trajectories (made greater than
expected progress)
270 300 570 760
75
12
Where do the s come from?
Summary Statements Calculator -April 14, 2009
13
Where do the s come from?

• Measurement for Summary Statement 2 Percent
  of infants and toddlers reported in progress
  category (d) plus of infants and toddlers
  reported in progress category (e) divided by the
  total of infants and toddlers reported in
  progress categories (a) (b) (c) (d) (e)
  times 100.

14
Where do the s come from?
30 of the children reached age expectations by
exit and 24 of the children entered and exited
at age expectations
300240 540 1000
54
15
So remind me again what this means

• What can we say about the childrens progress?

16
What can we say?

• Part C Outcome 1 successful social relationships
  with peers and adults, following rules for social
  interactions
• 96 of children participating in Part C made
  progress in their social relationships while they
  were enrolled.
• The 4 of children who did not make progress
  included children with the most severe
  disabilities and/or degenerative conditions. Can
  you describe them?

17

• 24 of the children participating in Part C were
  functioning at age expectations at entry and at
  exit in this outcome area. Can you describe them?
• 54 of the children were functioning at age
  expectations in this outcome area when they
  exited the program. (summary statement 2)
• 30 started out behind and caught up
• 24 entered and exited at age expectations

18

• 75 of the children who entered the program below
  age expectations made greater than expected
  gains, made substantial increases in their rates
  of growth. i.e. changed their growth
  trajectories (summary statement 1)

19
What other data might you want to share?

• The public likes to see scores going up!
• Increase in mean scores from entry to exit
• e.g. 4.3 to 5.6 on the COSF
• Increase in raw scores
• Increase in scale scores
• What else?

20
Setting Targets
21
What well cover today

• Two strategies for examining data
• Data quality
• Potential for program improvement
• Parameters, guidance for target settings from OSEP

22
Can you trust the data?

• Begin by identifying outliers
• Examples look at the percentages reported for
  certain categories across local programs

23
Percentages reported in category a across 30
local programs
24
Remove the outliers

• State percentage for a with all data 3.9
• Revised percentage for a with outliers removed
  2.4

25
Percentages reported in category e across 30
local programs
26
Remove the outliers

• State percentage for e with all data 32.1
• Revised percentage for e with outliers removed
  27.7

27
Example of data with outliers removed

• Clean data (without the outliers) may be a
  more accurate picture of where you are starting

28
Suggested strategy

• Analyze your data with your local LEA/program
  outliers included and excluded so you can gauge
  the impact they are having on your state level
  data.

29
Note Note Note

• Consider clean data when deciding about
  reasonable targets, BUT
• Turn in the original data to OSEP in the SPP
  report!
• You can discuss the clean data in the rationale
  for your targets.

30
Which local programs can be targeted for program
improvement?

• Compare the summary statement data by local
  program to identify which programs have the most
  potential for improvement.

31
Summary Statement Percentages by Local Program
32
Considerations

• What do you know about the programs/LEAs with the
  least and the most progress in the summary
  statements? i.e. the programs w/
• the lowest and highest percentages of children at
  age expectation at exit
• the lowest and highest percentage of children
  making greater than expected gains

33
Examples of Key Questions

• Are the children similar at entry?
• Are the higher performing programs/LEAs
  participating in special projects? e.g. a state
  initiative, TACSEI or CELL?
• Are there systems issues in lower performing
  programs/LEAs that would explain differences in
  outcomes? e.g. personnel shortages

34
Bottom-line Question

• Could either system or practice focused
  improvement activities targeted toward the lowest
  performing programs/LEAs improve the child
  outcomes?

35
The Math of Target Setting

• How much would the data change if the lowest
  local programs moved toward the mean?
• Improvements in the lowest programs will result
  in improvement in your statewide data
• Experiment with your data to determine what
  targets are reasonable in your state