New Insights in Personality Measurement: Application of Ideal Point IRT Models

1
New Insights in Personality Measurement
Application of Ideal Point IRT Models

• Stephen Stark, Oleksandr S. Chernyshenko, Wayne
  C. Lee
• University of Illinois at Urbana-Champaign

2
Overview

• Personality measures and job selection
• Summary of research on fitting IRT models to
  personality data
• Dominance models vs. ideal point models
• Purpose of our research Examine fit of three
  ideal point models to 16PF data
• Results, conclusions, future research

3
Why Use Personality Measures in Job Selection?

• Predict job performance in numerous occupations
• Predict contextual performance (e.g.,
  organizational citizenship behaviors)
• Conscientiousness very important
• Provide incremental validity in job selection
  account for more variance in job performance than
  general cognitive ability alone
• May have little or no adverse impact against
  minority group members

4
Chernyshenko, Stark, Chan, Drasgow, Williams
(1999)Fit Series of IRT Models to Personality
Data

• Fit three traditional IRT Models to 16PF and
  Goldbergs Big Five personality data
• 2PLM
• 3PLM
• Samejimas Graded Response (SGRM ordered
  categories)
• Results Some scales were not fit very well

5
Three-Parameter Logistic Model (3PLM)
5
6
Examples of 16PF Scales having Poor Fit
IM Impression Management
I Sensitivity
7
MFS An Exploratory IRT Model for Discovering the
Shape of IRFs

• Levines Maximum Likelihood Formula Scoring Model
  (MFS)

7
8
Example of IRF for 16PF Sensitivity Scale
Obtained using MFS
9
Why do traditional IRT models fit cognitive
ability data well, but NOT personality data?

• Nature of responding to personality items might
  differ from that of cognitive ability tests
• Maximum vs. Typical performance (Cronbach, 1960)
• Dominance vs. Ideal Point response
  processes(Roberts, Laughlin, Wedell, 1999)

10
Maximum vs. Typical Performance

• Maximum performance can do
• Ex Cognitive ability
• Individual is motivated to respond accurately and
  testing time is limited
• Testing situation restricts behavior greatly
• Typical performance will do
• Ex Personality
• Variability in effort and ample time to respond
• Traditional models may fit well the constrained
  responding to maximum performance tests, but are
  unable to model the complexity of typical
  performance tests.

11
Why dont traditional IRT models fit
personality data well?

• Other factors that may influence fit
• Impression management
• Self-deception
• Choice of reference group
• Interpretation of items

12
Two Types of Response ProcessesExample of
Dominance (cumulative) Model IRFPerson endorses
item if her standing on the latent trait, theta,
is more extreme than that of the item.
Item
Person
13
Two Types of Response Processes Example of
Ideal Point Model IRF
Example of Ideal Point Model IRFPerson endorses
item if her standing on the latent trait, theta,
is near that of the item.
I think that traveling to other countries is
okay. Disagree if either hate to travel or love
to travel
PersonHates
PersonLoves
Item
14
Purpose of our Investigation

• Explore the possibility of modeling responses to
  personality items using ideal point IRT models
• Most personality scales are constructed using
  psychometric procedures that assume a dominance
  response model
• BUT, theoretical and empirical evidence suggests
  that ideal point models might be more appropriate

15
Fitted Three Ideal Point Models to 16PF
Personality Data

• Hyperbolic Cosine Model (HCM)
• Andrich Luo (1993)
• RATEFOLD (1999) computer program
• PARELLA
• Hoijtink (1990)
• RATEFOLD (1999) computer program
• Maximum Likelihood Formula Scoring (MFS) with
  ideal point constraints
• Levine Williams (1999)
• FORSCORE (1999) computer program

16
Data

• Fifth Edition of the 16PF (1993)
• 13,059 examinees provided by IPAT
• 5 non-cognitive scales examined
• Liveliness (F)
• Sensitivity (I)
• Openness to Change (Q1)
• Tension (Q4)
• Impression Management (Z)
• Calibration sample (N 6,530)
• Cross-validation sample (N 6,529)
• MFS only

16
17
Data Preparation

• HCM, PARELLA
• Dichotomized (middle option scored as high)
• MFS
• Dichotomized (middle option scored as high)
• Reverse scored

18
Item 1, Sensitivity
MFS
Parella
HCM
19
Item 4, Openness to Change
MFS
HCM
Parella
20
Chi-square / df Ratios forHCM and PARELLA
21
Chi-square / df Ratios for MFS ModelUsing Ideal
Point Constraints
Good Fit for All Scales
22
HCM Person-Item DistributionsOpenness to
ChangeAll Items are Moderately Extreme
Moderately NegativeItems
Moderately PositiveItems
Person Frequencies
23
Summary of Results

• Sensitivity scale fitted best by all models
• Consistent with results from previous
  investigation (Chernyshenko et al., 1999)
• HCM fit better than PARELLA for all scales
• Although PARELLA appeared to fit some items very
  well, the overall chi-squares for scales were
  much larger than HCM
• MFS provided best fit for all scales
• Small chi-squares obtained for scales that
  previously exhibited poor 3PL model-data fit

24
Conclusions Implications

• Personnel selection
• Scores obtained using ideal point and dominance
  models differ for persons with extreme standing
  on the latent trait
• Changes rank order of high scoring individuals
• Improving model-data fit may lead to more
  accurate
• Scoring of respondents
• Detection of Differential Item Functioning
• Scale construction
• Factor analysis, item-total correlation, and
  reliability coefficients tend to select
  moderately extreme items
• Neutral items excluded because methods assume a
  dominance response process
• Application of ideal point model requires new
  methods

25
Avenues for Future Research

• Ideal point methods of scale construction
• Potential applications include measurement of
  personality, job satisfaction, and attitudes
• Will construct criterion-related validity
  increase?
• To facilitate scale construction from an ideal
  point perspective, research is needed to develop
• More flexible ideal point IRT models
• New methods for selecting items