Diagnostic%20testing%20that%20just%20might%20make%20a%20difference

1
Diagnostic testing that just might make a
difference

• Dylan Wiliam
• Institute of Education, University of London

www.dylanwiliam.net
2
Diagnostic Items in Mathematics and Science
Project rationale

• Traditional testing deals with individuals, but
  teachers mostly deal with groups
• Data-Push vs. Decision-Pull
• Data-push
• Quality control at end of an instructional
  sequence
• Monitoring assessment
• Identifies that remediation is required, but not
  what
• Requires new routines to utilize the information
• Decision-Pull
• Starts with the decisions teacher make daily
• Supports teachers on-the-fly decisions
• If a 30-item test provides useful information on
  an individual, then responses from 30 individuals
  on a single item might provide useful information
  on a class

3
Premises of the DIMS project

• Single, well-developed, questions provide a
  reasonably strong basis for real-time
  instructional decision-making
• Questions with principled and interpretable
  incorrect answers can be used to elicit evidence
  of student thinking
• Such questions, when used with all-student
  response systems, and when followed with
  discussion, can advance student learning
• Such questions can deepen teacher knowledge (both
  content and content-pedagogical)

4
The DIMS project in practice

• Focuses on 4th 8th grade, mathematics science
• Includes a bank of approximately 150 diagnostic
  questions per subject/grade level
• Question bank is a complementary resource for any
  curriculum, not a curriculum replacement
• Items to be used one at a time, within the flow
  of day-to-day lessons
• Focuses on collecting evidence about student
  learning in order to adapt instruction to meet
  the learning needs of students in real time

5
Item development

• Review of state standards
• Review of relevant literature on student
  conceptions
• Identification of significant (mis-)conceptions
• Item construction
• appears to be very difficult for many
  item-writers
• a distractor-stem-key approach worked for some
• Item review and editing
• Expert review
• Piloting
• Final review and editing

6
DIMS meets traditional psychometrics

• Concept-based distractors increase difficulty
• Most IRT models assume
• Monotonicity
• Equivalence of incorrect responses
• Need for analysis of single items makes most
  theories difficult to apply, or irrelevant

7
Semi-dense items(Bart, Post, Lesh Behr, 1994)

• Five properties
• Response interpretability
• Response discrimination
• Rule discrimination
• Exhaustive set usage
• Semi-density

8
Response interpretability
Each response interpretable by at least one
cognitive rule
9
Response discrimination
Each response interpretable by exactly one
cognitive rule
10
Rule discrimination I
Response discrimination uniqueness of cognitive
rules that interpret a response
11
Exhaustive rule set usage
Response discrimination every cognitive rule
interprets at least one response to the item.
12
Semi-density
Exactly one cognitive rule interprets each
response to the item and each cognitive rule
interprets exactly one response
13
Discriminating incorrect cognitive rules (Hart,
1981)

• Version 1
• If ef  8, then efg  
• 9
• 12
• 15
• 8g

• Version 2
• If fg  8, then fgh  
• 9
• 12
• 15
• 16
• 8h

14
Discriminating correct cognitive rules (Bart et
al., 1994)
Ann and Kathy each bought the same kind of bubble
gum at the same store. Ann bought two pieces of
gum for six cents. If Kathy bought eight pieces
of gum, how much did she pay?

• Version 1
• 12
• 24

• Version 2
• 24 cents, because 8324 or 8 pieces 3 cents
  per piece24 cents.
• 24 cents, because 4624.
• 24 cents, because 2/68/x and 2x48 for which
  x24.
• 24 cents, because 2/68/x and 2/6x4/48/24.
• 24 cents, because 2/64/126/188/24.
• 24 cents, became 2/64/128/24.
• 12 cents, because 246 implies that 8412

15
Discriminating between incorrect and correct
cognitive rules

• Version 1
• There are two flights per day from Newtown to
  Oldtown. The first flight leaves Newtown each day
  at 920 and arrives in Oldtown at 1055. The
  second flight from Newtown leaves at 215. At
  what time does the second flight arrive in
  Oldtown? Show your work.

• Version 2
• There are two flights per day from Newtown to
  Oldtown. The first flight leaves Newtown each day
  at 905 and arrives in Oldtown at 1055. The
  second flight from Newtown leaves at 215. At
  what time does the second flight arrive in
  Oldtown? Show your work.

16
Discriminating between incorrect and correct
cognitive rules (2)
17
Conclusion
Correct
Incorrect
18
Conclusion (2)

• For an item to support instructional
  decision-making, the key requirement is that in
  no case do incorrect and correct cognitive rules
  map on to the same response.
• If this property is met, then the semi-density
  properties are less important.
• If this property is not met, then the
  semi-density properties are irrelevant.
• The discovery of new incorrect cognitive rules
  that interpret item keys leads to item improvement

19
Conclusion (3)

• Data on impact on student achievement not yet
  available
• Evidence of impact on
• Teachers practice
• Student engagement
• Student affect

20
Questions?Comments?