Knowledge and Skill Tests as Measures of Motivation and Interest

1
Knowledge and Skill Tests as Measures of
Motivation and Interest

• Presentation for the International
• Military Testing Association
• Brussels, October 2004
• Janet Held and William Farmer
• Navy Personnel Research, Studies, and
• Technology

2
ASVAB Standards Review for Navy SEALS

• Navy validates their Armed Services Vocational
  Aptitude Battery composites on a regular basis
• SEAL standards were up for review
• Study conducted with several hypotheses

3
ASVAB Standards Review Hypothesis 1

• ASVAB technical tests, AS, MC, EI, carry with
  them
• some measure of hands on experience with
  technical activities SEALS are engaged in
• interest in engaging in these activities
• compatibility and adaptability with the outdoors
  environment where some of these activities are
  performed
• ASVAB VE and AR tests are too academically
  oriented to predict outcome for the physical and
  stressful SEAL training

4
ASVAB Standards Review Hypothesis 2

• Coding Speed (CS) increments ASVAB in the
  prediction of SEAL training outcome
• CS carries with it some measure of perseverance
• termed persistence in a SEAL training report, or
  vigilance, quickness, and detail orientation
• Coding Speed is now part of the ASVAB composite
  used to select Navy Air Traffic Controllers and
  Operations Specialists, stressful occupations

5
ASVAB Tests, Near Past and Present

• GS General Science
• AR Arithmetic Reasoning
• WK Word Knowledge
• PC Paragraph Comprehension
• MK Mathematics Knowledge
• AS Auto Shop Information
• MC Mechanical Comprehension
• EI Electronics Information
• AO Assembling Objects
• NO Numerical Operations (Eliminated in FY02)
• CS Coding Speed (Designated Navy Special Test in
  FY02)
• VE WKPC

6
SEAL Data Source

• Navy Integrated Training Resources and
  Administrative System (NITRAS) from Pensacola, FL
• Fiscal years 1997 and 1998 were difficult
  recruiting environment (N 419)
• Partial FY04 for cross-validation N 137
• Study participants
• Students who graduated and students who dropped
  on request

7
Analytical Method

• Logistic Binary Regression
• To predict the probability of graduating at
  various cutscores and the significance of the
  equations
• To obtain an estimated of a correlation
• Cutscore analysis
• Empirical and theoretically based Taylor Russell
  Tables to confirm expected improvements in the
  graduation rate from increasing the cutscore

8
Graph Explanations

• The three graphs that follow depict the logistic
  regression curves that were generated for the N
  419 students in the FY97/FY98 dataset.
• In each graph, 3 lines are drawn. One line
  depicts the full range of ASVAB scores found for
  the 419 students. Another line is drawn from that
  data set with truncation at VEAR 110,
  which eliminates the restriction in range problem
  at the lower score levels that occurs from
  multiple uses of those tests prior to and
  including SEAL selection. The last line is drawn
  with truncation on a comparable GSMCEI score
  for research purposes.

9
Current SEAL Composite
.70
.60
.50
.40
.30
.20
Predicted probability
.10
0.00
VEAR
Graph 1
10
Recommended SEAL Composite
Target Score and Outcome
Predicted probability
GSMCEI
Graph 2
11
Optimal composite, but not all Sailors have CS
scores

Predicted probability
280
270
260
250
240
230
220
210
200
190
180
170
160
ASMCEICS
Graph 3
12
Graph Explanations (Continued)

• In Graph 1, truncation on VEAR 110 improves
  the VEAR prediction line, indicating the
  adjustment for restriction in range is effective.
  Truncation on GSMCEI 165 results in a flat
  VEAR line, indicating that all of the relevant
  information is accounted for by GSMCEI.
• In Graph 2, adjusting for restriction in range on
  VEAR with truncation at 110 actually improves
  the prediction line for GSMCEI. This is
  expected because the two composites are
  correlated.
• Notice in Graph 3 that ASMCEICS composite
  really is the best, and that the expected
  predicted probability of graduating exceeds .70
  at very high score levels

13
Cutscore and Correlation Analyses

• LR Naglekerke R-square used to approximate a
  correlation of .25 for GSMCEI
• Empirical expectancy analysis conducted for
  GSMCEI, at high scores to reduce restriction in
  range effects that occur with multiple ASVAB use
• Obtained the graduation rate improvement for the
  sample at cutscores that produced 15 and 5
  qualification rates for a accession population
• Found the Taylor Russell table that produced the
  Accession 15 and 5 rates and graduation rates
  that corresponded to the Empirical table, and the
  associated graduation (employee success) rates
• Correlation was .30, not very far off from .25

14
Conclusion

• Personality and interest measure development
  should be explored for the SEALS to determine if
  they are completely redundant to the ASVAB
  measures supported as predictive in this study,
  or if they can be combined with these tests to
  augment validity, possibly from the conservative
  .25 estimated correlation level to .40 or higher.
  A selection instrument with validity of .40 or
  higher makes it possible to increase graduation
  rates significantly while at the same time
  qualifying an adequate percentage of the
  accession population to become SEAL candidates