Neural Test Theory: A nonparametric test theory using the mechanism of a selforganizing map

1
Neural Test TheoryA nonparametric test theory
using the mechanism of a self-organizing map

• SHOJIMA Kojiro
• The National Center for
• University Entrance Examinations, Japan
• shojima_at_rd.dnc.ac.jp

2
Neural Test Theory (NTT)

• Shojima (2008) IMPS2007 CV, in press.
• Test theory using the mechanism of a
  self-organizing map (SOM Kohonen, 1995)
• Scaling
• Latent scale is ordinal.
• Latent rank
• Number of latent ranks is about 3, 20
• Item Reference Profile
• Test Reference Profile
• Rank Membership Profile
• Equating
• Concurrent calibration

3
Why an Ordinal Scale?

• Two main reasons
• Methodological
• Sociological

4
Methodological Reason

• Psychological variables are continuous
• Reasoning, reading comprehension, ability
• Anxiety, depression, inferiority complex
• Tools do not have high resolution for measuring
  them on a continuous scale
• Tests
• Psychological questionnaires
• Social investigation

5
Weight and Weighing Machine

• Phenomenon (continuous)
• Measure (high reliability)

Weight
6
Ability and Test

• Phenomenon (continuous?)
• Measure (low reliability)

1
2
3
4
Ability
7
Resolution

• Power to detect difference(s)
• Weighing machines
• can detect the difference between two persons of
  almost the same weight.
• can almost correctly array people according to
  their weights on the kilogram scale.
• Tests
• cannot discriminate the difference between two
  persons of nearly equal ability.
• cannot correctly array people according to their
  abilities.
• The most that tests can do is to grade examinees
  into several ranks.

8
Sociological Reason

• Negative aspects of continuous scale
• Students are motivated to get the highest
  possible scores.
• They should not be pushed back and forth by
  unstable continuous scores.
• Positive aspects of ordinal scale
• Ordinal evaluation is more robust than continuous
  scores.
• Sustained endeavor is necessary to go up to the
  next rank.

9
NTT

• ML (RN07-04)
• Fitness (RN07-05)
• Missing (RN07-06)

• Equating (RN07-9)
• Bayes (RN07-15)

10
Statistical Learning of the NTT

• For (t1 t T t t 1)
• U(t)?Randomly sort row vectors of U
• For (h1 h N h h 1)
• Obtain zh(t) from uh(t)
• Select winner rank for uh(t)
• Obtain V(t,h) by updating V(t,h-1)
• V(t,N)?V(t1,0)

Point 1
Point 2
11
Mechanism of Neural Test Theory
Response
Point 1
Point 2
Point 1
Point 2
Latent rank scale
12
Point 1 Winner Rank Selection
Likelihood
ML
Bayes

• The least squares method is also available.

13
Point 2 Reference Matrix Update

• The nodes of the ranks nearer to the winner are
  updated to become closer to the input data
• h tension
• a size of tension
• s region size of learning propagation

14
Analysis Example

• Geography test

15
Item Reference Profile(IRP)
16
IRPs of Items 115 (ML, Q10)
The monotonic increasing constraint can be
imposed on the IRPs in the learning process.
17
IRP of Items 1635 (ML, Q10)
18
IRP index (1) Item Difficulty

• Beta
• Rank stepping over 0.5
• B
• Its value

Kumagai (2007)
19
IRP index (2) Item Discriminancy

• Alpha
• Smaller rank of the neighboring pair with the
  biggest change
• A
• Its value

20
IRP index (3) Item Monotonicity

• Gamma
• Proportion of neighboring pairs with negative
  changes.
• C
• Their sum

21
Item Reference Profile Estimate
IRP indices
22
Can-Do Table (example)
IRP indices
IRP estimates
Ability category and item content
23
Test Reference Profile (TRP)

• Weighted sum of the IRPs
• Expected score of each latent rank

• Weakly ordinal alignment condition
• Satisfied when the TRP is monotonic, but not
  every IRP is monotonic.
• Strongly ordinal alignment condition
• Satisfied when all the IRPs are monotonic. ? TRP
  is monotonic.
• The scale is not ordinal unless at least the weak
  condition is satisfied.

24
Model-Fit Indices
ML, Q10
ML, Q5

• Fit indices are helpful in determining the number
  of latent ranks.

25
Latent Rank Estimation

• Identical to the winner rank selection

Likelihood
ML
Bayes
26
Latent Rank Distribution (LRD)

• LRD is not always flat
• Examinees are classified according to the
  similarity of their response patterns.

27
Stratified Latent Rank Distribution
LRD stratified by sex
LRD stratified by establishment
28
Relationship between Latent Ranks and Scores

• R-S scatter plot
• Spearmans R0.929
• R-Q scatter plot
• Spearmans R0.925

Validity of the NTT scale
29
Rank Membership Profile (RMP)

• Posterior distribution of latent rank to which
  each examinee belongs

RMP
30
RMPs of Examinees 115 (Q10)
31
Extended Models

• Graded Neural Test Model (RN07-03)
• NTT model for ordinal polytomous data
• Nominal Neural Test Model (RN07-21)
• NTT model for nominal polytomous data
• Batch-type NTT Model (RN08-03)
• Continuous Neural Test Model
• Multidimensional Neural Test Model

32
Graded Neural Test Model
Boundary Category Reference Profiles of Items
19Dashed lines are observation ratio profiles
(ORP)
33
Graded Neural Test Model
Boundary Category Reference Profiles of Items
19Dashed lines are observation ratio profiles
(ORP)
34
Nominal Neural Test Model
Item Category Reference Profiles of Items 116
correct choice, x merged category of choices
with selection ratios of less than 10
35
Discussion

• Test standardization theory
• Self-Organizing Map
• Latent scale is ordinal
• IRPs are flexible and nonlinear
• Test editing
• CBT and CAT
• Test equating
• Concurrent calibration
• Application
• Japans National Achievement Test for 6th and 9th
  graders

36

• Website
• http//www.rd.dnc.ac.jp/shojima/ntt/index.htm
• Software
• Neutet
• Developed by Professor Hashimoto (NCUEE)
• Available in Japanese and English versions
• EasyNTT
• Developed by Professor Kumagai (Niigata Univ.)
• Japanese version only