Valid, Reliable

1
Valid, Reliable Efficient

• A Psychometric Evaluation of
• Flash Word Recognition and
• NSSI Passage Reading Measures

2
Kathleen J. BrownMatthew K. FieldsUniversity of
UtahR. Darrell Morris Appalachian State
University
3
Impetus for Current Study

• Need for valid, reliable, efficient instruments
  to determine instructional reading level
• Flaws with current instruments
• DRA no rate time consuming
• DIBELS screen only
• IRIs psychometric evaluation often weak or
  missing

4
Impetus for Current Study

• Growing use of Flash selected graded passages
  a.k.a. NSSI (Virginia/ASU effect)
• Initial psychometric evaluations positive
• (Frye, 2004 Frye Trathen, 2004 Frye Trathen,
  Olson, Schlagal, 2002 Palmer, Trathen, Olson,
  Schlagal, 2002)

5
Theoretical Framework
(Anastasi, 1988 APA, 1985)
6
Methods

• 4 schools
• 2 Title 1 1 public, 1 parochial
• 2 non-Title 1 both public mixed SES
• 192 students in G2-G5 in March, 2006
• Rank ordered DIBELS or QRI, then sampled 12
  students per grade 4 high, 4 average, 4 poor to
  achieve a representative distribution for testing

7
Methods

• 135 minutes of assessment in 3 sessions
• Presentation order counterbalanced
• Flash item selection counterbalanced
• 9 on data team 4 hours protocol training
• Manual flash interrater differences n.s.

8
Alternate Form Reliability

• measure of temporal stability for scores
• measure of consistency of response for scores

9
Alternate Form Reliability

• To what extent are NSSI A passage scores
  equivalent to NSSI B passage scores?
• To what extent are computer Flash scores
  equivalent to manual Flash scores?

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Results Alternate Form Reliability
p lt .01
14
Content Validity

• provides evidence that items on test represent a
  specific domain
• provides evidence that the format and response
  properties of the test represent the domain

15
Content Validity

• To what extent do the NSSI passages
  reflect/measure expected grade level benchmarks?
• Maybe look at separate means for accuracy, rate,
  comp report those to show
• To what extent does the Flash measure reading
  instructional level?

16
NSSI Reading Level Criteria
17
Performance Levelfor NSSI by Grade
18
Performance Levelfor Flash by Format Criterion
19
Concurrent Validity

• To what extent are Flash scores and NSSI scores
  consistent with scores achieved on a flagship
  standardized reading measure (i.e., the GORT)?

20
(No Transcript)
21
Results Concurrent Validity
p lt .01
22
Average Performance Levelfor NSSI GORT by
Grade Level
23
Conclusions For G2-G5

• NSSI A and NSSI B seem to have high validity for
  identifying students instructional reading
  levels
• NSSI A and NSSI B can be considered equivalent
  forms

24
Conclusions For G2-G5

• Manual Flash and Computer Flash seem to have high
  validity for identifying students instructional
  reading levels when the criterion is set at 85
• Manual Flash and Computer Flash seem to be
  equivalent forms

25
Conclusions For G2-G5

• The GORT does not seem to have high validity for
  identifying students instructional levelsat any
  grade level.
• The GORT over-predicts instructional levelby
  approx. 2 years.
• Note most GORT comp questions are passage
  independent (Keenan Betjemann, 2006)