Some Methodological Comments from a Validation Study for a U'S' Military Scholarship Selection Model

1
Some Methodological Comments from a Validation
Study for a U.S. Military Scholarship Selection
Model
Akihito Kamata Gershon Tenenbaum Catherine
Alfano Carla Urena Laura Hasller The Learning
Systems Institute Florida State University,
U.S.A. Presentation at 2004 IMTA, Brussels,
Belgium. October 2004.
2
Problems

• In addition to the routine predictive validity
  study
• It seems that only a few selection variables
  dominate the selection decisions.
• Does the selection give an unfair disadvantage
  for some applicants regarding one particular
  selection variable?
• Does the selection process efficiently select
  students who would choose technical majors?

3
Sample Characteristics

• All applicants in 2000-2002

• Applicants with college performance data

4

• Some Descriptive Statistics

5
Weights for selection variables

• Are all selection variables reasonably affecting
  the selection process?

6

• Correct Classification Rate Based on Stepwise
  Discriminant Analysis

7

• Evaluation of Relative Weights

• Weights to each selection variable were assigned
  as optimal weights to maximize R-squared in
  multiple regression to predict college GPA.
• However, weights are not directly comparable
  because the scale of each variable is different.
• Our solution
• Compute a standardized weight
• WXstandardized WX ? SDX
• Then, relative weights are ratio of weights to
  each other.

8

• Relative Weights for Some Selection Variables

9
Recruiting Potential Technical Majors

• Is the selection process reasonably selecting
  students who are likely to choose technical
  majors?

10

• Technical vs. Non-technical Majors

11

• Predicting Technical Majors

• Logistic regression with Technical major (1
  yes, 0 no) as dependent variable

Logistic Regression 1
Logistic Regression 2
12

• Correlations with College Performance Criteria

13
Selection Bias Related One Variable

• Does the selection give an unfair disadvantage
  for some applicants regarding one particular
  variable?

14

• Predicting Selection by V8

• Logistic regression with Selection (1 yes, 0
  no) as dependent variable

Logistic Regression 1
Logistic Regression 2
Logistic Regression 3
15
Possible Modifications to the Selection Model

• All selection variables are included for the
  computation of Selection Score (Current Model
  Model 1).
• V8 is excluded from the list to compute the
  Selection Score (Model 2).
• V6 is excluded from the list to compute the
  Selection Score (Model 3).
• V8 and V6 are excluded from the list (Model 4).
• V8, V6, and 4 other variables with low predicting
  power are excluded from the list (Model 5).

16

• Correlation between selection scores and college
  performance measures

17
Selected vs. Non-selected Applicants, based on
modified selection models

• Selection scores were recomputed and rescaled to
  the original selection score.
• Then, selection decisions were simulated for each
  applicant.

18

• Mean differences on college performance criteria

19

• Percentage of technical majors, based on modified
  models

20
A Few Remarks

• We also used some information about applicants
  school by linking CCD database.
• Some desirable characteristics of applicants may
  not be positively correlated with performance
  criteria.
• It could be a problem of the choice of criterion,
  or the selection model.
• If a right criterion is used, a multi-stage
  selection process is encouraged.