Knowledge Tracing and Prediction of Future Trainee Performance Tiffany S. Jastrzembski, Kevin A. Glu

1
Knowledge Tracing and Prediction of Future
Trainee Performance Tiffany S. Jastrzembski,
Kevin A. Gluck, Glenn GunzelmannAir Force
Research Laboratory

2
Research Goals

• Basic Science Aims
• Implement mathematical models based on cognitive
  science empirical research and theory
• Integrate mechanisms to handle spaced versus
  massed learning
• Applied Science Aims
• Predict warfighter performance when regimen of
  training is known
• When will the warfighter achieve
  mission-readiness?
• Prescribe warfighter training schedule to
  optimize performance
• How much training is required when deployment
  date is known?
• Is training timetable reasonable?

3
Approach

• Construct a trainee model based on an existing
  knowledge tracing equation
• Capitalize on existing strengths of model
• Account for dynamics of spaced versus massed
  practice
• Extend model capabilities to have predictive
  power
• Calibrate trainee model parameters from
  performance history
• Extrapolate knowledge state transformation for
  predictive and prescriptive purposes

4
Mathematical Basis of Current Research

• General Performance Equation (Anderson Schunn,
  2000)
• Simple knowledge units are acquired according to
  simple principles
• Performance success relies on frequency, recency,
  and amount of practice
• Performance
• S scaling parameter
• N amount of practice
• c learning rate
• T time since learning
• d decay parameter

5
General Performance Equation Model Fits
Skill Retention for Typewriting Aptitude (Bean,
1912)
Knowledge Acquisition for Logic-Based Facts
(Anderson Fincham, 1994)
6
The Spacing Effect

• Empirical Findings
• Memory benefits accrue with increased durations
  in practice
• Rate of forgetting decreases as time passes
• Distributed practice consistently superior to
  massed practice
• Ex Learning is more durable when practices are
  spaced over a month compared to practice crammed
  into a week
• General Performance Equation Results
• Discrete increments in learning are added at each
  training point
• Unable to account for differences in massed
  versus spaced practice
• Results in better performance for massed practice

7
General Performance Equation Model Fit to
Distributed Data- Glenberg, 1976

• Data spaced at practice intervals of every 2
  and 8 trials
• Model fits all post-hoc
• Correlations quite poor

8
Predictive Performance Equation

• Performance
• S original scaling parameter x improvement in
  training history
• N amount of practice
• c learning rate
• T time since learning
• a activation-based decay parameter
• Calculated from training history
• Based on original decay rate (d) and activation
  level at latest data point
• where m latest activation

9
Predictive Performance Equation Model Fit to
Distributed Data- Glenberg, 1976
R .96 RMSD 1.47

• Captures recency, frequency, and spacing effect
• Correlation very good

10
Challenges to Predictive Validity of Mathematical
Models (Part 1)

• Data Resolution
• Mathematical regularities differ at individual
  operator, team, and aggregate levels of analysis
• Aggregate level data eliminates noise and smooths
  performance curves
• True learning trends may become distorted or
  averaged away
• Learning trajectories of individuals may differ
  vastly from each other and the average

11
Unmanned Air Vehicle Synthetic Task Cognitive
Engineering Research Institute

• Three operators must coordinate over headsets to
  complete missions and take reconnaissance photos
  of ground targets
• Performance is based on the weighted sum of
  penalty scores across team members

12
Predictive Performance Equation Model Fits to
Aggregate Data

• Training Scenario
• Session 1 Baseline
• Five, 40-minute missions
• Optimize model parameters for learning and decay
• Session 2 Predictive test
• Return 10-14 weeks later
• Three, 40-minute missions
• Extrapolate mathematical regularities for
  prediction

R .95 RMSD 10.3
13
Predictive Performance Equation Model Fits to
Individual Team Data
R .91 RMSD 15.6
14
Predictive Performance Equation Model Fits to
Individual Operator Data
R .68 RMSD 62.6
15
Comparison of Human Performance Curves Across
Levels of Data Resolution
16
Challenges to Predictive Validity of Mathematical
Models (Part 2)

• Insufficient Training History
• Model will function at a disadvantage if the
  baseline is inadequate
• Model will extrapolate mathematical regularities
  for future prediction that may be less certain or
  valid
• Model analyses will help reveal minimal training
  history required for effective use

17
Predictive Performance Equation Model Fits as a
Function of Training History

• Detailed training history is necessary to
  baseline model parameters
• Instructors need to collect sufficient training
  logs
• Greater amounts of training history may enhance
  prediction at finer grains of resolution (e.g.
  individual level)

18
Notional Prediction

• Scenario How long will it take to achieve 95
  proficiency under the current regimen of practice?

Desired Proficiency

• Baseline Training Days 1 and 84
• Parameterize equation from baseline performance
• At 3 missions/day, 5 days/week, an additional
  1,120 40-minute missions are required to achieve
  proficiency

19
Potential Predictive Utility in the Military
Domain

• Mission-Readiness
• Reveals warfighter mission-readiness under
  specific distributions of practice
• Accounts for practice spaced apart at any length
  of time
• Equipped with ability to account for extended
  breaks in training
• How much additional training will be needed to
  reach proficiency after taking a 2-month break?
• Probabilistically determines if a maneuver or
  mission will succeed at some future point in time

20
Notional Prescription

• Deployment Scenario How much training must a
  warfighter receive to be mission-ready (95
  proficient) by the end of four weeks?

Desired Proficiency

• Baseline Training Days 1 and 84
• Parameterize equation from baseline performance
• Prediction dictates 24 practice missions/day, 5
  days/week

Deployment
21
Notional Prescription

• Deployment Scenario How much training must a
  warfighter receive to be mission-ready (95
  proficient) by the end of four months?

• Baseline Training Days 1 and 84
• Parameterize equation from baseline performance
• Prediction dictates 5 missions/day, 5 days/week
• 40 fewer practice sessions are needed in this
  spaced scenario

Desired Proficiency
Deployment
22
Potential Prescriptive Utility in the Military
Domain

• Optimal Training Regimen
• Compares training schedules to maximize
  performance
• Assesses how effective each training repetition
  will be
• Optimizes spacing of practice to result in larger
  learning gains
• Logistic Utilities
• Determines if training expectations are realistic
• Predicts whether mission-readiness can be
  attained within a specified period of time
• Provides trainers with reasonable
  mission-readiness timetables

23
Conclusions

• Model may
• Serve as a useful tool for warfighters and
  instructors
• Help determine when a warfighter has achieved
  proficiency
• Help streamline practice schedules to optimize
  learning
• Help determine realistic training timetables for
  warfighters to become mission-ready

24
Thank You!Questions?