Measurement: Theory and Practice

1
Measurement Theory and Practice
2
Items

• True Score Theory XTE
• Item Response Theory
• Both
• Define construct
• Create many items
• Review items
• Collect data
• Select items, make scale
• Further construct validity

3
Measurement

• Measurement involves abstracting a concrete
  ordinal to an interval.
• Nominal
• Ordinal
• Interval
• Ratio

4
Measurement

• Counting
• How many apples in a pie, why dont 3 apples
  always cost the same
• Observations ordinal, measurement must be
  interval
• Resolution through interval measures (volume,
  weight).
• Spelling test
• Counting number of words correct does not measure
  spelling
• Raw scores are not linear, thus biased. All
  statistics based on raw scores are biased.

5
Measurement (Thorndike)

• Unidimensional
• Linear
• Abstraction
• Invariance
• Sample-Free
• Additive

6
Guttman Scale

• if a person endorses an extreme statement, then
  he/she should endorse less extreme statements on
  the scale
• Rasch
• A person having greater ability should have the
  greater probability of solving any item
• F(P)b/d ability/ item difficulty
• Logit natural log odds for succeding on item

7
Logit Metaphor

• Ability Item Difficulty

8
Nominal to Interval

• Data (yes/no), (always, usually, sometimes)
• Observations Ordinal, Measurement Interval
• Combine 2-parameters
• B the location measure of a person
• D location of item
• Log-Odds

9
Estimating B-D

• Initial Attempt Maximum Liklihood parameters
  with highest liklihood of producing data.
• Log-Odds for items
• Logit Log (P/(1-P) B-D

10
Estimating Liklihood of Data Person

• Item A B C D E Total
• P(M) 1 1 1 0 0 3/5
• P(G) 0 1 1 0 1 3/5
• M1 .5 .27 .12 .95 .98 .015
• M2 .73 .5 .27 .88 .95 .082
• M3 .88 .73 .5 .73 .88 .206
• M4 .95 .88 .73 .5 .73 .222
• M5 .98 .95 .88 .27 .5 .111
• M3.5 .93 .83 .65 .5 .8 2.41

11
G

• Item A B C D E Total
• P(G) 0 1 1 0 1 3/5
• G1 .5 .27 .12 .95 .02 .0003
• G2 .27 .5 .27 .88 .05 .002
• G3 .004
• G4 .05 .88 .73 .5 .27 .00
• G5 .02 .95 .88 .27 .5 .002
• G3.59 .07 .83 .65 .60 .20 .005 .

12
Estimation

• M and G both 3.5, different liklihood curves
• Ability measure same with maximum liklihood
• Good when item difficult is known
• Well-defined curve
• 1 peak curve
• Need fit statistic

13
Estimating Fit

• Joint maximal liklihood (unconditional max
  liklihood)
• Items and Ability estimates simultaneously
• As one estimate moves, the other changes
• Multiple peaks (stuck in minor peaks, miss max
  peak)
• Help from logistic model

14
Curve Fitting Approach

• Know underlying curve for Ability and Items
  (logistic ogive curve)

15
Fit Statistics

• Does model fit the data?
• OUTFIT
• INFIT
• Chi-Square (X(obs) E (exp))2
• __________________________________________
• Variance
• Small when expectation and observation meet
• Square to cancel out and
• Divide by variance to standardize
• If model data then 0
• Greater than 1, data outside model

16
Outfit

• Sum chis for all observations / items (L)
• Expected value 1.0
• Item variance p(success failure)
• P.92 variance .92.08 .07
• BB obs-exp / Total variance
• 3-2.92/ total variance(.85) .09
• Next estimate 3.5 .09 3.59

17
George

• Item A B C D E Total
• P(G) 0 1 1 0 1 3/5
• G(3.59)
• A (0-.93)2/.06
• B (1-.83)2/ .14
• C (1-..65)2 / .22
• D (0-.4)2 / .24
• E (1-.06)2 / .16
• Total 20.21 / 5 4.04 outfit, exp 1.0, no fit
• Chi 20, df 5

18
Mary

• Ability 3.59, same as George so variance same
• A (1-.95)2 / .06
• ABCDE 1.6
• 1.6/5 .325 outfit
• Follows Guttman pattern
• Data is predictable
• Unexpected pattern (G first response)
  .86/.0614.42 (contributes to outfit)

19
INFIT

• Similar calculation
• Add all top terms then divides by all bottom
  terms
• Less sensitive to extremes
• More sensitive to item misfit around person
  ability estimate

20
Applications?

• IRT (Rasch) analysis of drug dependency scale.
• Sample independence
• Classical Test theory statistics?
• Sample dependent
• All items equal