Update%20from%20ECO:%20Possible%20Approaches%20to%20Measuring%20Outcomes

1
Update from ECO Possible Approaches to
Measuring Outcomes
Kathleen Hebbeler SRI International
Prepared for the OSEP Early Childhood Meeting
Washington, DC February 2005
2
Todays presentation

• Update on the outcomes
• Approaches to measurement
• Next steps and timeline

3
Update on the Child and Family Outcomes

• Have been gathering input for 12 months
• Input from advisors, work groups, presentations,
  conference calls, public input solicited through
  web site

4
What should be in the outcomes?

• The outcomes should be statements of what EI and
  ECSE are trying to do for families and children.
• Statements at the most fundamental level of what
  these programs are all about.

5
Considerations and criteria

• Consistent with IDEA and legislative intent
• Apply to the entire birth to 5 age span
• Apply to all families and children in EI and ECSE
• Easily compatible with outcomes of other early
  childhood initiatives/ programs

6
Input Recieved

• 2 conference calls with gt200 participants total
• On-line survey (61 responses)
• Emails (35)
• On-line discussion forum (1)

7
Who responded?

• Fairly equally divided between Part C and
  Preschool
• About half local and half state level
• About 10 family members

8
Family Outcomes posted for comments

• Families know their rights and advocate
  effectively for their child.
• 2. Families understand their child's abilities
  and special needs
• 3. Families help their child develop and learn.
• 4. Families have the supports they want.
• 5. Families participate in desired services and
  activities that are available to all families in
  their community.

9
Feedback from the field

• The word advocate in 1 has adversarial
  connotations to some
• 2 should include strengths
• Controversy over outcomes 4 and 5
• The word want in 4
• 4 and 5 (and some thought 3) went beyond the
  requirements of Part B Section 619 Preschool
  Programs

10
Family Outcomes

• Families understand their childrens strengths,
  abilities and special needs
• Families know their rights and effectively
  communicate their childrens needs.
• Families help their children develop and learn
• Families feel they have adequate social supports
• Families are able to access services, programs,
  and activities that are available to all families
  in their communities

11
Recommendations

• All five family outcomes apply to Part C
• Family outcomes numbers 1 and 2, and maybe 3,
  apply to 619.

12
Child Outcomes as posted for comment

1. Children have positive social relationships
2. Children acquire and use knowledge and skills
3. Children take action to meet their needs

13
Feedback from the field

• The child outcomes were supported by the majority
  (gt80) of those providing comments, across Part
  C, Part B 619, and families
• Outcome 3 was the most difficult to understand
  respondents suggested adding appropriate

14
Child Outcomes

• 1. Children have positive social relationships
• 2. Children acquire and use knowledge and skills
• 3. Children take appropriate action to meet
  their needs

15
Recommendation

• All three child outcomes apply to both Part C and
  619

16
Feedback on the Preface

• Nearly all supported the concepts
• Many excellent suggestions for wording
  refinements/ minor edits

17
Measurement of Child Outcomes
18
Definition of Measurement

• Measurement - Tools and techniques to quantify
  the childs status on aspects of functioning for
  example
• Norm-based assessments
• Curriculum-based measures
• Rating scales

19
Criteria and Considerations

• Children with disabilities already have
  assessment data -- but not from the same
  assessments
• Highly probable most data will be collected by
  local providers
• Need to minimize burden try to incorporate into
  existing procedures

20
Criteria and Considerations

• Children with disabilities may be participating
  in other assessment systems
• Outcome measurement should be as valid and
  reliable as possible
• Quality of data will get better over the first
  few years

21
Pitfall Alert!

• What we want for children and families (desired
  outcomes) reflects our values
• What we want for young children may not be the
  same as what can be easily measured
• When what is valued and what can be measured
  easily or well do not line up, EITHER
• Outcomes will be determined by what can be
  measured easily or well, OR
• Some of the outcomes will be more easily measured
  or more adequately measured than others

22

• There is no perfect system
• but you get to choose the shortcomings you will
  live with

23
Standardized Assessment and Young Children

• Reliability
• Interested in behavior over multiple settings but
  many assessments reflect behavior in one setting
  (validity, generalizability)
• Appropriateness of assessment for children with
  disabilities
• Accommodations
• Qualifications of assessors

24
Measurement Options for Producing Aggregated Data
at the State Level

• All children in state assessed with same measure
• Different assessments are used and they are
  converted to the same metric

25
State
Local
Local
State
Local
OSEP
State
State
26
So What Is the Metric?

• Some kind of scale
• Two alternative scales are possible
• Age-anchored
• Continuous progress
• Each has strengths and weaknesses

27
Measurement Decision

• Givens
• Different children will have different kinds of
  assessment data
• The same child will have several different kinds
  of assessment data
• Challenge How to convert multiple pieces of
  information (e.g., scores from several
  standardized assessments, informed clinical
  opinion, parent report) to a single score?

28
Assessment 1
Conversion Process
Assessment 2
Single Score
Parent Report
Informed Clinical Opinion
29
How to get to a single score from multiple
sources of information

1. Mathematical conversion from assessment data
2. Mathematical conversion with an exception for
   clinical judgment or parent input
3. Team rating that uses all information on the
   child to derive a score

30
Team derives rating

• Structured process to derive rating
• Guidance materials would be produced for how to
  derive ratings incorporating mathematical
  conversions for common (good) assessment
  instruments
• Training materials would be developed

31
Team derives rating

• Pro
• IDEA places great value on team decision-making
  (eligibility, service decision)
• Literature suggests teams can produce reliable
  ratings data
• Could be part of IFSP or IEP process (does this
  make it more acceptable to providers?)

32
Team derives rating

• Consistent with best practices in child
  assessment
• Uses multiple sources of information
• Does not put undue weight on standardized
  assessments
• Develop systems of analyses so that test scores
  are interpreted as part of a broader assessment
  that may include observations, portfolios, or
  ratings from teachers and/or parents.
• - Natl Institute for Early Education Research

33
Team derives rating

• Con
• Requires functioning team
• Requires some knowledge of typical child
  development depending on rating scale
• Would require lots of training

34
Team derives rating Unanswered questions

• What would be required to get reliable ratings?
• Would the resulting data be seen as credible?

35
Next steps

• Feb - Fall 2005 Collect input on measurement
  approach
• Spring 2005 Develop measurement approach and
  accompanying materials. Small scale pilot work
  on measurement approach for 1-2 child outcomes
  and all family outcomes
• Summer 2005 Continued development work and
  pilot work on the other outcomes

36
Next steps (continued)

• Fall 2005 Large scale field testing of
  measurement of child and family outcomes
• Winter 2005 Revisions based on field testing
• Early Spring 2006 Release of procedures and
  materials for use by all states
• Summer 2006 and onward Revision to materials
  based on research on implementation. Ongoing
  development of training materials.

37

• All of this is very tentative
• Each next step assumes the previous step is
  completed on time and the recommendation is
  accepted.

38

• Follow developments on the web site
• Contact us with questions