How to convince your friends NOT to misuse raw scores

1
How to convince your friends NOT to misuse raw
scores

• Benjamin D. Wright
• Institute for Objective Measurement
• MESA Psychometric Laboratory
• bd-wright_at_uchicago.edu

2
THE TROUBLE WITH RAW SCORES

• The BROKEN BUCKET of Missing Data
• The CRUMBLING CATEGORIES of Lumpy Ratings
• The DIRTY DATA of Unpredictable Responses
• The PERVERSE PRECISION of Extreme Scores
• The RUBBER RULER of Irregular Intervals and
  Squashed Extremes

3
The BROKEN BUCKET of Missing Data

• Compare Two Patients on an 8-item, 7-category
  Functional Independence Measure
• Patient A 2 3 3 m m m m 4 12
• Patient B 1 2 2 2 2 2 2 3 16
• Which Patient is more Able?
• Patient B has the higher score 16 gt 12 on all 8
  items, BUT On the 4 Items A and B have in common
• Patient A 2 3 3 m m m m 4 12
• Patient B 1 2 2 m m m m 3 8
• Patient A has the higher score of 12 gt 8
• Are you sure you want to misuse
  missing-data-leaking raw scores for
  missing-data-impervious measures?

4
The CRUMBLING CATEGORIES of Lumpy Ratings

• Rating Forms Offer Equally Spaced Categories
• 1 . 2 . 3 . 4 . 5 . 6 . 7
• But Raters Reply with Unequally Spaced Responses
• 1 . . . 2 . . 3 4 . 5 . . 6 . . . . . 7
• The Measure Distance of One More Point from
  Category 1 to 2 can be FOUR times BIGGER than The
  Measure Distance of One More Point from Category
  3 to 4 !!
• Are you sure you want to mistake Lumpy Ratings
  for Equal Interval Measures?

5
The DIRTY DATA of Unpredictable Responses

• When Item Responses are Arranged from Easy Items
  to Hard Items you Expect Response Patterns like
• 7 7 6 6 6 5 5 4 46 and 4 4 3 3 2 1 1 1 19
• BUT suppose you get
• 7 7 6 6 15 5 4 41 ? or 4 4 3 3 71 1 1 24
  ?
• What then?
• Raw Scores are Blind to Unpredictable Responses.
• Only Quality Control of Well-Constructed Measures
  Tells you about Response Surprises
• Are you sure you want to suffer
  raw-score-dirty-data blindness instead of
  enjoying data-vigilant measures?

6
The PERVERSE PRECISION of Extreme Scores

• The Statistical Precision of a Raw Score is
  MAXIMUM at exactly the place where the
  Information a Raw Score Provides is MINIMUM
• When a person gets the lowest possible score,
  their raw score precision is perfect. We know
  exactly the score their low ability implies. But
  we have no idea how far Below that Score their
  ability might be!
• It is the same with the highest possible score.
  We know exactly the score their high ability
  implies. But we have no idea how far Above that
  Score their ability might be!
• They are Off-Our-Scale and Our Precision for
  their Unknown Measure is ZERO!
• Are you sure you want to mistake imprecise raw
  scores for precise measures?

7
The RUBBER RULER of Irregular Intervals and
Squashed Extremes

• When our items bunch in clumps of equally
  difficult items then a count of one more right
  answer within a clump implies only a little
  increase in our ability.
• But when we leap ahead and our next right answer
  is in a distinctly harder clump, then we see that
  this one more right implies a large increase in
  our ability.
• As for the ends of the test where one more right
  is from 0 to 1 or from all but one to all.
• Then the implied change in our ability is
  infinite.

8
(No Transcript)
9
WHAT ARE VARIABLES?

• Length and weight may be real variables. But we
  construct their units of measure.
• Inches and ounces are our creations - Our own
  imaginative constructions.
• A variable is an amount of something which we can
  always picture as a distance
• gtFrom Less -------------------------gt To More
• We can arrange to experience evidence of this
  "something".
• But its measurement line and its units of
  measurement are up to us to construct.

10
EVIDENCE OF A VARIABLE?

• The variable and its evidence could be
• length benchmarks exceeded
• health symptoms absent
• ability problems solved
• skill tasks completed
• attitude assertions condoned
• We can arrange to provoke occurrences of evidence
  and count how many pieces occur.
• But these counts are not measures.

11
REQUIREMENTS FOR MEASUREMENT

• Pieces of evidence must be concrete to be
  observed.
• This necessary reality keeps them uneven in size.
  
• To measure we need an even abstraction, a line
  marked out in abstractly equal units.
• Pieces of evidence are unstable. They appear and
  disappear by accident. They are only probable
  signs of the variable which they are designed to
  manifest.
• To measure, we must find a way to connect the
  pieces of evidence we can arrange to observe to
  the probabilities of the measures we want.

12
WHAT IS MEASUREMENT?

• DISTANCE (Length) was our First Variable
• COUNTING Steps and Fingers was our First
  Measuring Operation
• The Trouble with Counting is its UNEQUAL UNITS
• How many apples fill a basket? How many oranges
  squeeze a glass? You may not believe it. But can
  we mix apples and oranges?
• We do it all of the time, by WEIGHING them!

13
CONSTRUCTING MEASURES

• But weighing is a constructed abstraction.There
  are no tangible equal units. We have to invent
  them.
• Equal feet are abstracted from real feet. Equal
  pounds are abstract real weights
• We construct our instrumentation of the
  variables length and weight to approximate units
  equal enough to serve our practical purposes
• We measure so that we can use the past to plan
  and navigate the future. But the future is by
  definition UNCERTAIN

14
HANDLING UNCERTAINTY

• Imagine two batters Smith bats 400 and Jones
  bats 200
• So which one will hit at their next batter-up? No
  way to know ahead of time.
• Even Smith has only a 4 out of 10 record. We
  cant wait to find out.
• So which one shall we send to the plate? Smith's
  odds for a hit are 2/3 Jones' odds for a hit are
  only 1/4
• Smith odds for a hit are 8/3 times better than
  Jones'.
• Even though we know nothing for sure, Does any
  doubt remain as to who to send to bat?
• That's how we handle uncertainty. We use past
  experience to estimate PROBABILITIES and use
  these probabilities to forsee the future.

15
COUNTING ABSTRACT UNITS

• To finish this job we have to construct a
  reproducible transition from counting concrete
  events, like right answers, observed or reported
  symptoms, relative agreements, frequency or
  importance categories to counting abstract units
  of equal size and wide generality.
• How can we do this?

16
INVERSE PROBABILITY

• To deal with the uncertainty we ask Bernoulli,
  Bayes and Laplace and interpret our observation X
  as evidence of its occurence probability Px
• To construct unit equality we ask Campbell,
  Thurstone, Rasch and Luce Tukey and define Px
  to satisfy the equation logPnix/(1-Pnix) Bn
  - Di
• Pnix is the probability of a successful response
  Xni being produced by person n to item i
• Bn is the ability of person n
• Di is the difficulty of item I
• The construction of equal size and hence additive
  units is called CONJOINT ADDITIVITY

17
CONJOINT ADDITIVITY

• For situations where Xni occurs in incremental
  steps such as Xni 0,1,2,3,,,M
• This simple solution generalizes to
• logPnix/Pnix-1 Bn - Di - Fix
• For situations where Xnijk 0,M occurs as the
  result of
• a Rater j rating the performance of
• a Person n on
• a Task k
• This MEASUREMENT MODEL becomes
• logPnijkx/Pnijkx-1 Bn - Di - Cj - Ak - Fix

18
CLOSING THE DEAL

• Raw scores cause problems with
• missing data
• lumpy ratings
• unpredictable responses
• extreme scores
• irregular intervals and squashed extremes
• Rasch measurement provides a solution to these
  problems by
• abstracting units of measurement
• using probabilities to predict futures
• constructing equal-sized intervals