Fit%20of%20Ideal-point%20and%20Dominance%20IRT%20Models%20to%20Simulated%20Data

1
Fit of Ideal-point and Dominance IRT Models to
Simulated Data

• Chenwei Liao and Alan D Mead
• Illinois Institute of Technology

2
Outline

• Background and Objective
• Hypotheses and Methods
• Results
• Discussions

3
Background

• Personality
• Used in personnel selection - Incremental
  validity to predict job performance beyond
  cognitive ability (Barrick Mount, 1991 Ones et
  al, 1993) - Less adverse impact (Feingold, 1994
  Hough, 1996 Ones et al, 1993).
• Model-data-fit
• - Need to calibrate personality traits
• - Use IRT models
• - Degree of fit depends on data structure

4
Background (cont.)

• Item response processes thinking of data
  structure
• IRT models and item response processes
• 1) Traditional dominance IRT models
• - high trait - high probability of endorsing
• 2) Ideal-point IRT models
• - similar item trait high probability of
  endorsing

5
Background (cont.)
Dominance Model IRF - x Theta (trait level) -
y Probability of endorsing
Ideal-point Model IRF - x distance between
person trait and item extremity - y Probability
of endorsing
6
Background (cont.)

• Chernyshenko et al, (2001)
• - Traditional dominance IRT models have failed.
  Suggest to look at item response processes and
  Ideal-point IRT models
• Stark et al. (2006)
• - Ideal-point IRT models as good or better fit
  to personality items than do dominance IRT
  models
• Chernyshenko et al. (2007)
• - Ideal-point IRT method more advantageous than
  dominance IRT and CTT in scale development in
  terms of model-data-fit

7
Limitation of previous studiesand objective of
current study

• Limitation of previous studies
• - Unknown item response processes!
• Objective of current study
• 1) Investigate model-data-fit by utilizing
  simulation with known item response processes
• 2) Test the assumption that the best fit model
  represents data underlying structure of response
  processes

8
Current Study
9
Models

• Dominance
• - Samejimas Graded Response Model (SGRM)
• Ideal Point
• - General Graded Unfolding Model (GGUM).
• Larger sample and longer test were said to be
  related to a better fit (Hulin et al, 1982 De la
  Torre et al, 2006).

10
Hypotheses

• Generating models
• H1 Data generated by an ideal point model will
  be best fit by an ideal-point model and data
  generated by a dominance model will be best fit
  by a dominance model.
• H2 The ideal point model will fit the dominance
  data better than the dominance model will fit the
  ideal-point data.
• H3 The ideal-point model will fit the mixture
  data better than the dominance model.

11
Hypotheses (cont.)

• Sample Sizes
• H4 All models will fit better in larger
  samples.
• H5 The GGUM model will fit relatively worse in
  smaller samples, as compared to simpler,
  dominance models.
• Test Lengths
• H6 The GGUM model will fit relatively worse for
  very short tests, as compared to longer tests.

12
Datasets

• Self-Control Scale from the 16PF
• Procedure1) Calibrate 16PF data to get item
  parameters - SGRM PARSCALE4.1 GGUM
  GGUM2004.2) Generate simulated data - models
  ideal point/dominance/mixed - sample size 300,
  2000 - test length 10, 37 - 50
  replications

13
Model-Data-Fit

• Cross validation ratio each item
  in each condition
• Only singles simulation study assures
  unidimensionality assumption
• Smaller value better fit
• Frequencies of ratios were
  tallied into 6 groups very small (lt1), small
  (1-lt2), medium (2-lt3), moderately large (3-lt4),
  large (4-lt5), very large (gt5).

14
Results overview
Condition Best fitting model
Dominance data generation GGUM
Ideal point data generation GGUM
Mixed data generation GGUM
Small Sample (N300) GGUM
Large Sample (N2000) GGUM
Short Test (n10) GGUM
Long Test (n37) GGUM
15
Results
16
Discussion (1)

• GGUM fits better - Confirm previous findings.
  - However, because regardless of the underlying
  response process, GGUM fits better than SGRM, it
  does not demonstrate that the response process or
  IRF/ORF is non-monotone. The previous assumption
  does not hold true.
• - Possible reason Software (PARSCALE GGUM)
  manifest models differently
• Better fit in small samples, especially for SGRM
  - Explanation chi-square is sensitive to
  sample size

17
Discussion (2)

• Examine similarities of the theta metrics
• - Negative correlation between theta
  estimates from GGUM and those from SGRM

TRUE SGRM GGUM
TRUE 1.000
SGRM 0.928 1.000
GGUM -0.923 -0.995 1.000
18
Discussion (3)

• Scaling issue

GGUM - Reverse the estimate - Add a constant
in scaling
19

• Thanks!