Evaluation of a Hybrid Selfimproving Instructional Planner

1
Evaluation of a Hybrid Self-improving
Instructional Planner

• Jon A. Elorriaga and Isabel Fernández-CastroCompu
  ter Languages and Systems Dept.University of the
  Basque Country 649 Postakutxa, E-20080
  Donostia.e-mail jipelarj_at_si.ehu.es 

2
Contents

• Introduction
• Our approach HSIIP
• Case-based Instructional Planner
• How it works
• Methodology of application
• Evaluation
• Conclusions
• Related and Future Work

3
Introduction

• Self-improving Vs. Adaptive ITSs
• Self-improving ITSs generalise the acquired
  knowledge and use it in future instructional
  sessions (Dillenbourg, 1989)
• Few self-improving systems
• ITSs can learn in each of its modules
• Tutor Module
• Student Model
• Domain Module
• Interface
• By using data
• From the Student Model (ITS)
• Directly from the student
• From the teacher

4
Our Approach HSIIP

• HSIIP Hybrid Self-Improving Instructional
  Planner
• Objectives
• To improve the adaptation ability
• To incorporate a learning ability to existing
  ITSs
• Focus Tutor Module, Instructional Planning
• Learning Techniques
• Case-Based Reasoning
• Learning from memorization
• Statistical learning

5
The HSIIP Approach

• Proposal To incorporate a CBIP into existing
  ITSs
• Aim To enhance the ITS with learning ability
• Result SIITS
• ITSCBIP gt SIITS

6
Case-Based Reasoning
7
Case-Based Instructional Planner
8
CBIP Case Structure

• Application Context
• ltsequence of student related featuresgt
• ltsequence of session related featuresgt
• ltsequence of domain related featuresgt
• Instructional Plan
• ltsequence of plan itemsgt
• Results of the application
• ltsequence of result valuesgt

9
CBIP Detail of Instructional Plan Memory
10
CBIP Generation Component (1)

• Search
• Hierarchical Organisation of the IPM
• Exhaustive and Heuristic Search
• Heuristic function Similarity function
• Similarity Threshold

• Retrieval of Cases
• Matching
• Nearest neighbour matching
• Similarity function

11
CBIP Generation Component (2)

• Adaptation of cases
• Critic-Based Adaptation
• Production System
• Knowledge intensive task

12
CBIP Learning Component (1)

• Revision of Cases
• Evaluation items
• Trace of the learning session
• Two Dimensions
• Educational
• Beliefs of the ITS (Student Model)
• Beliefs of the Learner Interaction
• Computational
• Case Reuse level

13
CBIP Learning Component (2)

• Revision of Cases
• Evaluation items
• Knowledge Acquisition Levels
• Misconceptions
• Student Beliefs about KAL
• Student Beliefs about the session
• Replanning
• ...

• Result Object
• Elementary results
• Collective results
• Normalisation of values (0 .. 1)
• Statistical Learning
• Creation of new cases

14
CBIP Learning Component (3)

• Storage of new Cases
• Find the appropriate GE
• Adapted Search Algorithms
• Generalisation
• On-line
• Off-line
• Thresholds

15
CBIP Heuristic Assessment Component
Assesses some Instructional Planning Objects

• Heuristic Formulae
• Adaptable - Weights
• Assessed objects
• Retrieved cases
• Built Instructional Plan
• Global result of the executed Instructional Plan

• Assessment Factors
• Beliefs of the STI (SM)
• Beliefs of the learner
• Similarity
• Reuse rating
• Adaptation level
• Influence level

16
SHIIP Working (1)
TUTOR MODULE
Estimates
Cases
Session Data
Estimates
Instructional Plan
Cases
17
SHIIP Working (2)
TUTOR MODULE
Estimates
Cases
Session Data
Estimates
NIL
Cases
Instructional Plan
18
HSIIP

• Working
• Initially empty IPM
• Training Phase
• Co-operation Phase
• Kernel of CBIP
• An object-oriented framework that facilitates the
  development of Case-Based Instructional Planners
• Represent explicitly and separately the
  characteristics of the concrete ITS
• Methodology of Application
• Procedure and Guidelines

19
HSIIP Methodology of Application (1)

• Analysis of the ITS (levels, plan-items,
  attributes) Most important task Knowledge
  Engineering
• Adaptation of the framework
• Representing the ITS related knowledge
• Setting of the parameters Thresholds and Weights
• Construction of the adaptation module
• Integration of the CBIP
• Test

20
HSIIP Methodology of Application (2)

• Analysis of the ITS (instructional planning)
• Level structure
• Items of each level
• Attributes
• Useful for retrieving
• Matching
• Indexing
• Useful for evaluating
• Student Model
• Strudent

21
HSIIP Evaluation (1)

• Objective Evaluate the performance of the HSIIP
  in terms of the changes in the students
  knowledge
• Design of the experiments
• Four classical instructional planners (CIP)
• Their corresponding HSIIP
• A Population of simulated students (4 groups)
• To test isolated modules
• To perform a significative number of experiments
  in the same conditions

22
HSIIP Evaluation (2)
23
HSIIP Evaluation (3)
 
24
Conclusions

• HSIIP An hybrid approach to enhance ITSs with
  learning capabilities based on a Case-Based
  Instructional Planner.
• Case-Based Learning
• Statistical Learning
• Learning from Memorization
• Sound performance (combines two planners).
• CBIP kernel A framework for developing
  Case-Based Instructional Planners.
• Generic Module for Instructional Planning
• Adaptable
• Positive evaluation results
• Simulated student a useful tool for formative
  evaluation

25
Related and Further Work

• Related Work
• Tool for interacting with the teacher
  (supervision of plans and results)
• Data System, Student, Teacher
• Further Work
• Experimentation
• More aspects taken into account in revision
• Application to other planning problems

26
Result Object
Object Result (is-a Case-component
Result-object) result-history ltlist of
Elementary-result instancesgt result-average ltCol
lective-result instancegt   Object
Elementary-result (is-a Result-object) evaluation-
item-list ltlist of Evaluation-item
instancesgt learner ltstringgt date ltdategt lea
rner-estimate 0..1 global-estimate 0..1  
Object Collective-result (is-a
Result-object) direct-use-number ltintegergt use-n
umber ltintegergt evaluation-item-list ltlist
of Evaluation-item instancesgt learner-estimate
0..1 global-estimate 0..1   Object
Evaluation-item (is-a Result-object) evaluation-at
tribute ltAttribute instancegt final-value 0..
1 change 0..1 learner-estimate 0..1
27
Elementary Result of a Subplan (ERS)

ERS Elementary Result of a Subplan
ARS Average Result of a Subplan
WBT Weighted Belief of the Tutor (applied to
each individual feature) WBL Weighted Belief
of the Learner (applied to each individual
feature) WOBL Weighted Overall Belief of the
Learner (applied to a subplan) AEIS Average of
the Evaluation Items of a Subplan NEI Number of
Evaluation Items NA Number of Applications of
the case EIj Evaluation Item j S Subplan WX
Weight related to factor X
28
Retrieved Case Estimate (RCE)
RCE Retrieved Case Estimate ARS Average
Result of a Subplan (see figure
6) RCAF Retrieved Case Appropriateness
Factor WOBL Weighted Overall Belief of the
Learner (applied to a case) WSF Weighted
Similarity Factor WRF Weighted Reuse
Factor C Case S (C) Subplan attached to the
case C WX Weight related to factor X
29
Created Plan Estimate (CPE)
CPE Created Plan Estimate NL Number of Levels
in the Instructional Plan CLE Created Level
Estimate PCN Number of Primary
Cases RCE Retrieved Case Estimate (see figure
7) CPF Created Plan Factor WTF Weighted
Stability Factor WIF Weighted Importance
Factor IP Instructional Plan L, Li Levels of
the Instructional Plan C, Ci Cases WX Weight
related to factor X