Reliability in Two Types of Alternate Assessment for Students with Significant Cognitive Disabilitie

1
Reliability in Two Types of Alternate Assessment
for Students with Significant Cognitive
Disabilities
2
PURPOSE

• Determine if trained teachers administer the
  assessments according the test directions.
• Discover if the assessments can be scored in a
  reliable way.
• Determine if test items demonstrate recommended
  levels of consistency supporting the inference
  that students would perform in the same way given
  a comparable opportunity.

3
The extent to which the measures resulting from a
test are the result of the characteristics of
those being measured.
Reliability

• Rudner Schafer (2001)

4
BACKGROUNDColorado EAG Pilot Assessments

• Performance Task Assessments
• Instructionally Embedded Assessments
• N175 teachers
• N176 students

5
DATA SOURCES

• Administration training
• Scoring of video clips
• Volunteer teachers videotaped administration of
  entire PT of day 3 of IEA
• Scoring of assessments
• PT teacher scored
• IEA rater scored

6
Reliability Characteristics

• Interrater Reliability
• Scoring agreement
• Fidelity
• Administration procedural conformity
• Internal Consistency
• Item cohesiveness in measuring a single construct

7
PROCEDURESCalculating Interrater Reliability

• Performance Task
• Percentage of agreement between teacher scores
  and independent rater
• (number of agreements/number of scores x 100)
• Instructionally Embedded Assessment
• Percentage agreement between paired raters on
  evidences of student performance
• (number of agreements/number of scores x 100)

8
Practice Scoring Clips N174
Percent
9
Percent Agreement Between Teacher/Outside Scorer
Percent
Percent
Percent
10
Percent Agreement Between Raters
Percent
Percent
11
Calculating Fidelity

• Performance Task
• Percentage of steps teacher followed and/or
  provided
• Set up procedures
• Adapted materials
• Wait time
• Scaffolding procedures

(number of steps following procedures/number of
steps observed x 100)
12
Observations Science 112 ELA75 Math123
Percent
13
Calculating Fidelity

• Instructionally Embedded Assessment
• Percentage of steps teacher followed and/or
  provided
• Task initiation
• Task completion
• Adapted materials
• Wait time
• Prompting procedures

(number of steps following procedures/number of
steps observed x 100)
14
Observations Science 24 ELA16 Math30
Percent
15
Calculating Internal Consistency

• Performance Task Assessment
• Alpha Coefficient
• Item to Total Correlations with Item Removed
• P-value based on level of scaffolding

16

1
17
Note Values are reported as average of
item-total correlations/reliability coefficients
18
P-value
Item
19
pvalue
Item
20
P-value
Item
21
CONCLUSIONS

• More studies are needed to support the
  generalization of findings
• Data for the PT show that most teachers
  demonstrate scoring accuracy and procedural
  fidelity.
• Scoring agreement for the IEA is inconclusive,
  although greater when the adjacent score is
  considered.
• There is an issue of task completion for the IEA.
• PT Cronbachs alpha correlations are high, even
  when adjusted for item total correlation
• PT p-values demonstrate range of difficulty on
  items within each test

22
Further Conclusions about Reliability

• There is more to reliability than interrater
  reliability.
• Training and practice of scoring is necessary to
  understand and achieve scoring accuracy.
• Fidelity to administration procedures is critical
  for standardization and comparability of scoring.
  Scripted procedures are an essential part of the
  assessment we presented here.