Kristiann Heesch, MPH

1
Using Item Response Theory to Evaluate a Self
Efficacy Test

• Kristiann Heesch, MPH
• Louise Mâsse, PhD
• University of Texas-Houston

2
Item Response Theory (IRT)

• Analysis is at item level
• Evaluates Likert scale responses
• IRT assumes that scores on a test vary as a
  function of a trait (Ramsay, 1982)
• Whether or not a person endorses an item depends
  on (Santor Ramsay, 1998)
• Amount of trait that the person has
• Effectiveness of item at indicating underlying
  trait
• Population from which person is drawn

3
Purpose of the study

• To conduct an item analysis of a 20-item Likert
  scale using IRT
• The scale measures self-efficacy toward moving
  forward through stages of physical activity
  adoption

4
Protocol

• Study design
• cross sectional
• Reason for the study
• part of the Women On The
• Move, a study to validate physical
  activity surveys
• Recruitment
• Latinas African American
• women 40 years of age
• Data collection
• July 97 March 99
• During one-on-one interview

5
Which model to use?

• Raschs Simple Logistic Model
• Simplest IRT model
• Small sample size (264 participants)
• 1-parameter
• Dichotomous data
• Two choices for Likert scale
• Rating scale model
• Extend Rasch model to multiple-category response
  format
• Can use with Likert-scale items
• Treats intervals between response options as
  constant
• Formats of all items are the same
• Partial credit model
• Similar to rating scale model
• Intervals between response options can vary
• Formats of items can vary

6
CFA structure
7
Model fit

• Examine difficulty of the items

Sample size was 246

Partial Credit model
TABLES OF RESPONSE MODEL PARAMETER
ESTIMATES ----------------------------------------
----------------------------------------
VARIABLES UNWGHTED FIT
WGHTED FIT ---------------
------------- ------------- item
ESTIMATE ERROR MNSQ T MNSQ
T ------------------------------------------------
-------------------------------- 1 1
-1.815 0.095 1.95 8.4 1.13 1.4
2 2 -1.474 0.092
0.98 -0.2 0.96 -0.4 3 3
-1.402 0.092 0.88 -1.4
0.80 -2.4 4 4
-1.539 0.091 1.22 2.3 0.98 -0.2
5 5 -1.364 0.086
0.96 -0.4 0.91 -1.0 6 6
-1.479 0.087 0.68 -4.0
0.74 -2.9 7 7
-1.594 0.091 0.78 -2.6 0.92 -0.8
8 8 -1.270 0.089
0.89 -1.2 0.92 -0.9 9 9
-1.178 0.088 1.03 0.3
0.95 -0.5 10 10
0.227 0.079 3.12 15.3 2.65 12.8
11 11 -0.629 0.082
1.11 1.2 1.15 1.6 12 12
-0.082 0.088 1.23 2.4
1.26 2.8 13 13
0.021 0.089 1.18 1.9 1.20 2.1
14 14 -0.144 0.084
1.20 2.2 1.15 1.7 15 15
-1.188 0.093 0.69 -3.9
0.70 -3.7 16 16
-1.179 0.090 0.59 -5.4 0.61 -5.1
17 17 -1.421 0.091
0.96 -0.5 0.98 -0.2 18 18
-1.509 0.094 0.84 -1.9
0.85 -1.7 19 19
-1.643 0.095 0.81 -2.2 0.82 -2.1
20 20 -2.092 0.098
0.95 -0.6 0.99 -0.1
-------------------------------------------------
-------------------------------  
8
Fit of self-efficacy test model

• Item Infit Statistic
• (Weighted MNSQ)
• Item 10 Find activity partner 2.59
• Item 15 Keep activity enjoyable .67
• Item 16 Maintain activity for 3 months .62
• Total of items not fitting 3/20 15
• High values Item content does not fit within
  scale construct.
• Low values Item content overlaps with content
  of other items
• Item 15 may overlap with finding an activity
  that is enjoyable
• Item 16 may overlap with being active on a
  regular basis and making commitment

9
Item Characteristic Curve One item
10
ICC Item 6 (confidence in finding convenient
place)
11
ICC Item 7 (confidence in finding safe place)
12
ICC Item 10 (confidence in finding activity
partner)
13
Item difficulty
 
item -----------------------------
------- 4
X XX
XX
3 X
XXX XX
XX
2 XXX
XXX XXXX
XXX
XXX 1
XXX XXXXXX
XXXXXXX
XXXXXXXXXX10 0
XXXXXXXX13 XXXXXXXXXX12,14
XXXXXXX
XXXXXXXXX11 XXXXXXXXXX
-1 XXXXX9,15,16
XXXXXX3,5,8,17
XXXXX2,4,6,7,18,19 XXXX1
-2 XXXXX20
XXX
XXXX X
-3 X
X


-4
Each 'X'
represents 1.8 cases  
14
Item difficulty
 
item -----------------------------
------- 4
X XX
XX
3 X
XXX XX
XX
2 XXX
XXX XXXX
XXX
XXX 1
XXX XXXXXX
XXXXXXX
XXXXXXXXXX10 (D) Find activity partner
0 XXXXXXXX13 (A) Work
responsibilities
XXXXXXXXXX12 14(A) Family responsibilities,
holiday XXXXXXX
XXXXXXXXX11 (D) Bad weather
XXXXXXXXXX
-1 XXXXX9 (D) Rearrange schedule
15,16(A) Find way to keep activity
enjoyable maintain activity
for another 3 months
XXXXXX3 (T) Make commitment 5,8
(D) Find activity enjoy, find time to be active
17 (S)Being activity again
XXXXX2,4 (T) Active on regular basis, start
in few weeks 6,7 (D) Find convenient
place, find safe place 18,19(S) Commmit
again, feel comfortable again
XXXX1 (T) Think can be physically active
-2 XXXXX20 (S)
Confortable being active again
XXX
XXXX X
-3 X
X


-4
Each 'X'
represents 1.8 cases  
15
Test information function

• Amount of information about a trait at each trait
  level
• Graph sum of item information functions
• A function of standard error of score
• I(?) 1 / ?2(?)
• Use to estimate reliability function

16
Reliability
17
Reliability of self-efficacy test
Reliability
Standardized self efficacy score
  ?2 1 ?2T 1
1/I(?) ?2T ?2E
18
Reevaluate model

• Re-run analysis treating all items with 4-point
  response patterns
• 2 choices
• Evaluate all items on 4-point scale
• Combined options 1 and 2
• For developmental purposes
• Practical
• To evaluate best possible scale, use best
  response pattern for each item
• Best fitting model
• Use the most information in the data
• Minimize error

19
Conclusion

• With responses on a 5-point Likert scale, our
  self-efficacy test does not fit a Rasch partial
  credit model well.
• Most items function best on a 4-point Likert
  scale.
• 5-items (e.g., large work and family
  responsibilities, bad weather, lack of activity
  buddy, holiday season) are difficult for
  participants with average self-efficacy to
  overcome in order to be physically active.
• Reliability varies but is high across levels of
  self-efficacy for the self-efficacy test.

20
Implications

• IRT allows for the examination of items and
  reliability at each level of a psychosocial
  variable.
• It also shows how each option of a Likert scale
  is functioning.
• This information can help researchers better
  evaluate and develop tests to measure
  psychosocial variables associated with physical
  activity behavior.