Building Students Metacognitive Skills through Interactions with Computer-based Teachable Agents

1
Building Students Metacognitive Skills through
Interactions with Computer-based Teachable Agents

• Gautam Biswas
• gautam.biswas_at_vanderbilt.edu
• Dept of EECS ISIS
• Vanderbilt University
• Collaborators Dan Schwartz, Kefyn Catley
• Postdoc, Students (at Vanderbilt) Rod Roscoe,
  John Wagster, Hogyeong Jeong, Nancy Morabito, Jim
  Segedy, Garrett Linn
• Supported by Dept. of Education IES, and NSF
  REESE Awards

2
Goals of our work

• Learn Science through principles that apply
  across domains
• Processes, Entities, Relations, Interdependence,
  and Balance
• Preparation for Future Learning
• Students should become independent learners, even
  when they move away from the computer environment
• Learning for oneself ability to assess ones
  learning progress
• Learning is never a one step process
• Cognitive tasks and Metacognitive strategies

3
Outline of Talk

• Our Approach to Learning by Teaching
• Bettys Brain, a Teachable Agent
• Learning Science by creating Causal Concept Maps
• Assessment through self-other monitoring
• Adaptive Tutoring
• Providing Metacognitive support in support of
  Preparation for future learning
• Experimental Studies
• Current/Future Work

4
Bettys Brain
Additional resources

• Teach

• Mentor Agent
• Mr. Davis

• Betty can explain her answers

• Query

• Quiz

• Text Resources

5
Teachable Agents

• Students teach computer agent using visual
  representations
• Agents performance based on what she is taught
• Students re-teach agent so that they may do
  better, (and in that process they learn)

• Agents only learn what they are taught explicitly
  by student
• No machine learning algorithms drive our agent
• Learning through social interactions
• Shared representations
• Shared responsibility

6
Learning Science

• By creating visual concept map structures
• Entities
• e.g., fish, macroinvertebrates, dissolved oxygen
• Relations
• causal fish consume macroinvertebrates
• increase decrease effects
• Causal Reasoning
• Cause-effect relations extended to chain of
  events
• Fish ? waste ? bacteria ? nutrients ? algae
• Interdependence
• Multiple dependencies everything depends on each
  other

7
Metacognition to aid Learning

• Metacognition describes two component processes
• Ability to monitor ones cognitive activities
• Ability to take appropriate regulatory steps when
  problem is detected
• Implemented as Self-regulated learning strategies
• Involves multiple aspects when learning
• Setting goals
• Planning
• Seeking help
• Monitoring ones own learning
• .

8
Monitoring when Problem Solving

• Self monitoring (cf. to self explanation)
  requires two coordinated processes
• Ability to generate solution steps
• Analyze and correct for discrepancies
• Our approach Self-other monitoring while
  teaching ( learning for oneself)
• Provide support to help students organize their
  own learning
• Betty demonstrates self-regulated learning
  behaviors by example
• Mentor provides additional support and hints

9
Example regulation strategies
Regulation Goal Pattern Description Bettys response
Monitoring by asking Queries Successive quiz requests but no queries asked of Betty in between quizzes Im still unsure of this material and I would like to do well. Mr. Davis said take the quiz only if you think you will do well. (Betty refuses to take quiz)
Monitoring through Explanations Multiple requests for Betty to give an answer but no request for explanation Lets see, you have asked me a lot of questions, but you have not asked for my explanations lately. Please make me explain my answers so you will know if I really understand.
Tracking Progress The most recent quiz score is significantly worse than the previous score I would really like to do better. Please check the resources, teach me, and make sure I understand by asking me questions that are on the quiz. My explanation will help you find out why I am making mistakes in my answers. Also, be sure to check out the new tips from Mr. Davis.
10
Example regulation strategies
Regulation Goal Pattern Description Mentors response
Monitoring through Explanations Multiple requests for Betty to give an answer but no request for explanation Without asking Betty to explain her answers, you may not know whether she really understands the chain of events that you have been trying to teach her. Click on the Explain button to see if she explains her answer correctly.
Tracking Progress The most recent quiz score is significantly worse than the previous score Betty did well on the last quiz. What happened this time? Maybe you should try re-reading some of the resources and asking Betty more questions so that you can make sure she understands the material.
Setting Learning Goals Betty is asked a question that she cannot answer for the second time Ive seen this kind of difficultly with teaching other students in the past. You should look for missing links between concepts or links that are in the wrong direction.
11
Mentor other forms of help

• On-Demand Help Students select which kind of
  helps they need
• Pedagogical examples
• What should I teach Betty?
• How can I be sure that Betty learns what I have
  taught?
• Learning examples
• How do I know that I know enough to teach?
• Domain-content examples
• General What domain content is relevant, chains
  of reasoning
• Specific I need help on the quiz.
• Help after quiz taken Adaptive
• ICS LBT systems where errors have occurred
  in concept map and possible fixes
• SRL groups what to read so as to do generate a
  more correct map

12
Experimental Studies
13
Bettys Brain Experimental Studies

• Fifth-grade students teach and learn about river
  ecosystems in several 45-min. sessions, and
  complete written pre/post tests
• Domain River ecosystems interdependence and
  balance involving (i) Food Chain, (ii)
  Photosynthesis and Respiration, and (iii) Waste
  cycle
• They later participate in a transfer (PFL) phase
  where they learn a new domain (e.g., nitrogen
  cycle on land, or global warming).
• We have compared several versions of the system
• ICS create a map(no teach) with content
  feedback
• LBT teach Betty with content feedback
• T-SRL teach Betty with SRL feedback
• M-SRL create a map (no teaching) with SRL
  feedback

14
Data Analysis

• Performance learning of domain content
• Number of correct concepts links in students
  final concept maps
• Behaviors sequence of activities
• Key student actions are logged
• Edit map (EM)
• Ask query (AQ)
• Request quiz (RQ)
• Access resources (RA)
• Request explanations (RE/CE)
• Betty could sometimes take (QT) or refuse (QD)
  the quiz

15
Results Learning Performance
Study 1 ICS, LBT, and T-SRL (56 students)
Condition Mean (SD) Map Scores Mean (SD) Map Scores
Main Phase Transfer Phase
ICS 22.83 (5.3) 22.65 (13.7)
LBT 25.65 (6.5)c 31.81 (12.0)
T-SRL 31.58 (6.6)a,b 32.56 (9.9)a
a T-SRL gt ICS, p lt .05 b T-SRL gt LBT, p lt .05 c
LBT gt ICS, p lt .05.
Study 2 ICS, M-SRL, and T-SRL (83 students)
Condition Mean (SD) Map Scores Mean (SD) Map Scores
Main Phase Transfer Phase
ICS 35.80 (10.5) 36.56 (13.61)
M-SRL 38.41 (8.55) 39.66(16.41)
T-SRL 41.79 (7.37)a 42.97(15.83)
16
Behavior Analysis

• Roscoe, et al. (2008) ICS, LBT, and T-SRL in
  main study
• Map quality was associated with AQ and RE/CE
  activities
• AQ RE/CE may indicate students attempts to
  regulate their own knowledge

17
Behavior Analysis using HMMs

• Jeong, et al. (2008) ICS, LBT, and T-SRL in main
  and transfer
• Used hidden Markov models (HMMs) to model
  learning patterns
• States hidden, output observer
• Three patterns related to SRL differed by
  condition
• Map Building EM-RA-RQ
• Map Probing AQ-RA
• Map Tracing AQ-RE-CE
• Interpreted models on right

18
Behavior Analysis with HMMs

• Stationary probabilities show the likelihood of
  exhibiting a given state

State ICS LBT T-SRL Transfer ICS Transfer LBT Transfer T-SRL
Map Building 0.72 0.66 0.42 0.73 0.73 0.68
Map Probing 0.24 0.30 0.47 0.25 0.25 0.27
Map Tracing 0.04 0.04 0.11 0.02 0.02 0.05
19
Pre-Post Test Analysis

• Detailed analyses of students written responses
  to examine learning of five river ecosystem
  principles
• balance, interdependence, microscopic entities,
  photosynthesis and cellular respiration,
  pollution
• Learning about microscopic entities (e.g.,
  oxygen, bacteria, and macroinvertebrates) was
  strongest
• Perhaps, because concept map representations make
  normally invisible concepts explicit.

20
Current and Future Work

• Adaptive Tutoring through Interactive
  metacognition
• Betty emulates aspects of self-regulated learner
• Mentor provides additional metacognitive support
  to remind students of important cognitive
  learning tasks and to help organize these tasks
• Further study of self vs self-other monitoring
• Mentor SRL versus Betty SRL
• Increased dose of self-other monitoring
• Front-of-class (FOC) Betty
• Moving TA system into classroom strong links to
  science curriculum
• Adaptive teaching by the classroom teacher(s)
• Learning science
• From concepts and their relations to causal
  reasoning about chain of events
  (interdependence)
• Aggregate Processes and Balance

http//www.teachableagents.org