Using Bayesian Networks to Predict Test Scores - PowerPoint PPT Presentation

Loading...

PPT – Using Bayesian Networks to Predict Test Scores PowerPoint presentation | free to view - id: 74e443-N2U5N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Using Bayesian Networks to Predict Test Scores

Description:

Using Bayesian Networks to Predict Test Scores by Zach Pardos Neil Heffernan, Advisor * ASSISTment * – PowerPoint PPT presentation

Number of Views:79
Avg rating:3.0/5.0
Slides: 31
Provided by: Zac141
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Using Bayesian Networks to Predict Test Scores


1
Using Bayesian Networks to Predict Test Scores
  • by Zach Pardos
  • Neil Heffernan, Advisor

2
Introduction Overview
  • ASSISTment tutoring system
  • The Task
  • Bayesian networks
  • Platform selection

3
ASSISTment Tutoring System
  • Online tutoring system developed at WPI
  • - Assess student knowledge/learning
  • Assists and prepares students for the MCAS
  • 2nd year of operation
  • Participation includes over
  • 2,000 students
  • With 20 teachers/classes
  • At 6 schools

4
ASSISTment Tutoring System
  • Students attempt to answer top level questions
    based on previous MCAS test questions
  • If the student answers incorrectly or asks for a
    hint they are given supporting questions,
    called scaffolds, or hint text messages
  • All answers and actions are logged on the server

5
The Task
  • To use Bayesian networks to assess students
    knowledge levels in the ASSISTment system and
    predict their performance on the MCAS test.
  • Research topic Compare predictive performance of
    fine-grain vs. coarse-grain skill models.

6
Bayesian Networks
  • "The essence of the Bayesian approach is to
    provide a mathematical rule explaining how you
    should change your existing beliefs in the light
    of new evidence. In other words, it allows
    scientists to combine new data with their
    existing knowledge or expertise.
  • - The Economist (9/30/00)

7
Bayesian Networks
  • New data
  • 2,000 students answering questions online
  • MCAS test results
  • Existing knowledge or expertise
  • Various grain skill models
  • Prof. Neil Heffernan
  • Bayes Rule
  • Where R is a random variable with value r
    and evidence e

8
Platform Selection
  • Bayesian network software choices
  • GeNIe
  • MSBNx
  • BayesiaLab
  • Netica
  • MATLAB with BNT (Bayes Net Toolkit)
  • Java Bayes

9
Platform selection
  • Choice MATLAB with BNT
  • Pros
  • Provides wide selection of inference engines
  • MATLABs robust programming environment
  • Automation
  • Runs on GNU/Linux
  • Existing Perl interface for the many scripts that
    will perform data mining tasks.
  • Cons
  • Little Slow

10
Project Overview
  • The datasets
  • Skill models
  • Parameters
  • Implementation
  • Results

11
The Datasets
  • Student online response data
  • 600 students from 2004-2005
  • Student selection criteria
  • Completed at least 100 items online
  • Completed the 2005 MCAS test
  • 2,568 question items
  • Student state MCAS test scores for 05
  • Used for calculating prediction accuracy
  • No test data used for training/parameter learning

12
Skill Models
  • Skill models describe the skills which are
    related to the online and MCAS questions.
  • Skill models used
  • MCAS1
  • MCAS5
  • MCAS39
  • WPI106

13
Skill Models
  • Skill models used for the MCAS test consisting of
    29 multiple choice questions
  • MCAS1
  • MCAS5

14
Skill Models
  • MCAS39
  • WPI106
  • The MCAS1 is a two layer network with skill nodes
    mapped to question nodes. The other 3 networks
    have a third, intermediary layer of AND nodes.
    This allows all question nodes to have the same
    number of parameters (slip/guess). The AND
    nodes also reflect the notion that a student must
    know all tagged skills to answer the item
    correct.

15
Skill Models
Transfer table for skill models
WPI-106 WPI-39 WPI-5 WPI-1
Equation-concept setting-up-and-solving-equations Patterns-Relations-Algebra The skill of math
Plot Graph modeling-covariation Patterns-Relations-Algebra The skill of math
Slope understanding-line-slope-concept Patterns-Relations-Algebra The skill of math
Similar Triangles understanding-and-applying-congruence-and-similarity Geometry The skill of math
Perimeter Circumference Area using-measurement-formulas-and-techniques Perimeter The skill of math
Equation-Solving
Inequality-solving
X-Y-Graph
Congruence
16
Parameters
  • Parameters were set as a best guess starting
    point.
  • Test model guess parameter is 0.25 because
    questions are multiple choice (out of four)

Original Parameters Online Model Test Model
Skills 0.50 Imported
Guess 0.10 0.25
Slip 0.05 0.05
Learned Parameters Online Model
Skills 0.44
Guess 0.30
Slip 0.38
  • Preliminary learning of parameters using EM
  • on the MCAS1 network indicates a guess of
  • 0.30, slip of 0.38 and prior of 0.44 on the
    skills.
  • These numbers were calculated recently and
  • are not used in our prediction results thus far.

17
Implementation
  • The main routine bn_eval() takes in
  • Name of skill model
  • StudentID
  • BNT object of the skill model bayes net
  • bn_eval() outputs
  • Status messages
  • Predicted score/Actual score/Accuracy
  • Logs prediction and skill assessment data

18
Implementation
  • The evaluation is a 2 stage process
  • Stage 1
  • Bayes skill model for the online data is loaded
  • Students online results are compiled and
    sequenced for the network
  • Student is given credit for all scaffold
    questions relating to a top level item answered
    correctly
  • Results are entered into the network as evidence
  • Marginals on the skill nodes are calculated using
    liklihood_weighting approximate inference .

19
Implementation
  • Stage 2 of evaluation
  • Bayes skill model for the MCAS test is loaded
  • Skill marginals calculated from stage 1 are
    entered into the test model as soft evidence
  • Marginals on the question nodes are calculated
    using jtree (join-tree) exact inference.
  • Test score points are summed by multiplying each
    marginal by 1 and then taking the ceiling of the
    total score.
  • Predicted test score is compared to actual
    student test score.

20
Implementation
  • Example student run using MCAS1 model

21
Implementation
  • Assessed skill marginals using MCAS1

22
Implementation
  • Example student run using MCAS5 model

23
Implementation
  • Assessed skill marginals using MCAS5

24
Implementation
  • Example student run using MCAS39 model

25
Implementation
  • Assessed skill marginals using MCAS39

26
Implementation
  • Example student run using WPI106 model

27
Implementation
  • Assessed skill marginals using WPI106

28
Results
  • Model performance/accuracy results
  • MAD is Mean Average Difference. The test is out
    of 29 points so a MAD score of 4.5 indicates that
    the model on average predicts a score that is 4.5
    points from the actual score.

MODEL MAD (RAW) ERROR
WPI-39 4.500 15.00
WPI-106 4.970 16.57
WPI-5 5.295 17.65
WPI-1 7.700 25.67
29
Future Work
  • Reduce runtime
  • Optimize the number of samples used with
    liklihood_weighting inference for each model.
  • Increase accuracy
  • Learn full parameters in all models
  • Use analysis to improve skill model tagging
  • Experiment with alternative models
  • Combine skill models into a hierarchy
  • Introduce time as a variable (DBNs)

30
References
  • A copy of this presentation as well as our
    initial paper submitted to ITS2006 entitled
    Using Fine-Grained Skill Models to Fit Student
    Performance with Bayesian Networks can be found
    online at
  • http//users.wpi.edu/zpardos/bayes.html
  • Thanks to the WPI-CS department, Neil Heffernan,
    contributors at CMU and the ASSISTment
    developers.
About PowerShow.com