Quality Assurance: Looking for Quality Data

1
Quality Assurance Looking for Quality Data
I know it is in here somewhere
Presented by The Early Childhood Outcomes Center
Revised January 2013
2
Activity

• What factors work to improve the quality of your
  data?
• What factors work to lessen the quality of your
  data?
• How to address these factors?

3
Take Home Message

• If you conclude the data are not (yet) valid,
  they cannot be used for program effectiveness,
  program improvement or anything else.
• What do you if the data are not as good as they
  should be?
• Answer Continue to improve data collection
  through ongoing quality assurance

4
Many steps to ensuring quality data
Before Good data collection/Training Good data system and data entry procedures
During Ongoing supervision of implementation Feedback to implementers Refresher training
After Review of COSF records Data analyses for validity checks
5
Many steps to ensuring quality data
Before Good data collection/Training Good data system and data entry procedures
During Ongoing supervision of implementation Feedback to implementers Refresher training
After Review of COSF records Data analyses for validity checks
6
Promoting quality data

• Training and support before and during data
  collection
• Analysis of the data after data collection
• Data system and verification after data collection

7
Promoting Quality Data

• Through training and communication related to
• Assessment
• Understanding the COS process
• Age expectations
• Data entry

8
Promoting Quality Data

• Through training materials, such as
• Video team and child examples
• Written child examples
• Quizzes for ensuring learning
• Refresher trainings
• Beware of Drift!!

9
Promoting Quality Data

• Through data systems and verification, such as
• Data system error checks
• Good data entry procedures

10
Many steps to ensuring quality data
Before Good data collection/Training Good data system and data entry procedures
During Ongoing supervision of implementation Feedback to implementers Refresher training
After Review of COSF records Data analyses for validity checks
11
Ongoing supervision

• Review of the process
• Is the process high quality?
• Are teams reaching the correct rating?
• Methods
• Observation
• Videos

12
Quality Review of COS Team Discussion

1. Do all team members participate in the
   discussion?
2. Is parent input considered in the rating? Give
   examples.
3. Is the team documenting the rating discussion?
   Give examples.
4. Does the team discuss multiple assessment
   sources? What are they?

13
Quality Review of COS Team Discussion

• Does the team describe the childs functioning,
  rather than just test scores? Give examples.
• Does the discussion include the childs full
  range of functioning, including skills and
  behaviors that are age appropriate, immediate
  foundational, and leading to immediate
  foundational? Give examples.

14
Quality review through process checks

• Provider surveys
• Self assessment of competence
• Knowledge checks
• Process descriptions (who participates?)
• Identification of barriers
• Kansas survey
• Alaskas survey

15
Questions from Alaskas Survey
3. How would you rate your own level of proficiency with the COSF process? (please select only one)
I am confident I know how to do it, and I do it well
I know how to do it, but I need some more practice and assistance
I understand it to a point, but I need more training
I do not know how to do this yet

4. Some cases are different from others, but of the choices below, which process seems to be the most typical in your experience? (please select only one)
I gather information and determine COSF ratings on my own
I gather information and consult with another provider to determine COSF ratings
I gather information and consult with the family to determine COSF ratings
I gather information, discuss it with a team and the team determines the COSF ratings
16
Ongoing Supervision

• Feedback to teams is critical
• Refresher training
• Beware of
• Auto pilot
• Drift

17
Quality Review of COS Team Discussion Activity

• Observe team video
• Evaluate quality

18
Many steps to ensuring quality data
Before Good data collection/Training Good data system and data entry procedures
During Ongoing supervision of implementation Feedback to implementers Refresher training
After Review of COSF records Data analyses for validity checks
19
Quality Review of Completed COS Forms

• Is the COSF complete?
• Is there adequate evidence for the basis for the
  rating?
• Does the evidence match the appropriate outcome
  area?
• Is the evidence based on functional behaviors?

20
Quality Review of Completed COS Forms

• Is there evidence that the childs functioning
  across settings and situations considered?
• Are the ratings consistent with the evidence?

21
Quality Review of COS Forms Activity

• Review completed COS Form with errors

22
Promoting quality data through data analysis

• Examine the data for inconsistencies
• If/when you find something strange, look for
  other data that might help explain it.
• Is the variation caused by something other than
  bad data?

23
The validity of your data is questionable if

• The overall pattern in the data looks strange
• Compared to what you expect
• Compared to other data
• Compared to similar states/regions/school
  districts

24
COS Ratings - Fall
Rating Class 1 Class 2 Class 3 Class 4
1 3 1 1 2
2 4 1 2 2
3 5 2 3 3
4 6 3 2 4
5 1 4 5 4
6 1 5 5 4
7 0 4 2 1
25
Outcome 3 Appropriate Action
Spring Spring
Fall 1 2 3 4 5 6 7 total
1 1 4 2         7
2 1 1 5 6 9 3 1 26
3   2 15 14 27 19 6 83
4   4 4 21 39 28 12 108
5   1 12 14 71 86 48 232
6   1   3 21 48 63 136
7       2 18 23 56 99
Review Total 2 13 38 60 185 207 186 691
26
OSEP Categories
OSEP Categories Class 1 () Class 2 () Class 3 ()
e. Maintained Age Appro Trajectory 23 16 24
d. Changed Traj Age Appro 15 23 13
c. Changed Traj Closer to Age Appropriate 32 34 37
b. Same Trajectory -Progress 28 21 25
a. Flat Trajectory No Prog. 2 6 1
27
Questions to ask

• Do the data make sense?
• Am I surprised? Do I believe the data? Believe
  some of the data? All of the data?
• If the data are reasonable (or when they become
  reasonable), what might they tell us?

28
Validity

• Validity refers to the use of the information
• Does evidence and theory support the
  interpretation of the data for the proposed use?
• Or
• Are you justified in reaching the inference you
  are reaching based on the data?
• Standards for Educational and Psychological
  Testing (1999) by American Educational Research
  Association, American Psychological Association,
  National Council on Measurement in Education

29
The validity of your data is questionable if

• ?

30
The validity of your data is questionable if

• .not all providers are not knowledgeable about
  in the COS process
• not all providers are careful with the COS
  process
• the data look strange
• etc.

31
Many steps to ensuring quality data
Before Good data collection/Training Good data system and data entry procedures
During Ongoing supervision of implementation Feedback to implementers Refresher training
After Review of COSF records Data analyses for validity checks