An Implementation of Vicarious Learning with Deep-Level Reasoning Questions in Middle School and High School Classrooms

1
An Implementation of Vicarious Learning with
Deep-Level Reasoning Questions in Middle School
and High School Classrooms

• Barry Gholson, Art Graesser, and Scotty Craig
• University of Memphis

2
Memphis Systems K12 and College
AutoTutor
iSTART
MetaTutor
ARIES
ALEKS - math
IDRIVE
Tutor Agent
3
What is AutoTutor?

• Art Graesser (PI)
• Zhiqiang Cai
• Patrick Chipman
• Scotty Craig
• Don Franceschetti
• Barry Gholson
• Xiangen Hu
• Tanner Jackson
• Max Louwerse
• Danielle McNamara
• Andrew Olney
• Natalie Person
• Vasile Rus

• Learn by conversation in natural language
• Graesser, A.C., Chipman, P., Haynes, B.C.,
  Olney, A. (2005). AutoTutor An intelligent
  tutoring system with mixed-initiative dialogue.
  IEEE Transactions in Education, 48, 612-618.
• VanLehn, K., Graesser, A.C., Jackson, G.T.,
  Jordan, P., Olney, A., Rose, C.P. (2007). When
  are tutorial dialogues more effective than
  reading? Cognitive Science, 31, 3-62.

4
AutoTutor
5
LEARNING GAINS OF TUTORS(effect sizes)

• .42 Unskilled human tutors
• (Cohen, Kulik, Kulik, 1982)
• .80 AutoTutor (14 experiments)
• (Graesser and colleagues)
• 1.00 Intelligent tutoring systems
  
• PACT (Anderson, Corbett, Aleven, Koedinger)
• Andes, Atlas (VanLehn)
• Diagnoser (Hunt, Minstrell)
• Sherlock (Lesgold)
• (?) Skilled human tutors
• (Bloom, 1987)

6
Is an intelligent interactive tutor really needed?

• Vicarious Learning. Perhaps observing a scripted
  dialogue can be just as effective.
• Deep Questions. Perhaps a dialogue organized
  around deep questions may be just as effective.

7
Why Vicarious Learning?

• Observation is an important learning method
• Recall (Baker-Ward, Hess, Flannagan, 1990)
• Language (Akhtar et al., 2001, Huston Wright,
  1998)
• Cultural norms (Ward, 1971 Metge, 1984)
• Vicarious learning can be as effective as
  interactive learning.
• Human tutoring if observers collaborate (Chi,
  Hausman, Roy, in press Craig, Vanlehn, Chi,
  2007)
• Intelligent tutoring when guided by deep
  questions (Craig et al, 2006)
• Provides a cost effective method that can easily
  be integrated into classrooms.

8
Facts about Deep Questions

• Students and teachers are not inclined to ask
  deep questions (Dillon, 1988 Graesser Person,
  1994).
• Training students to ask deep questions
  facilitates comprehension (Rosenshine, Meister
  Chapman, 1996).
• Vicarious learning is effective when students
  observe animated conversational agents asking
  deep questions (Craig, Gholson, Ventura,
  Graesser, 2000 Craig, et al., 2006 Gholson
  Craig, 2006).

9
Deep-level reasoning questions

• Deep-level reasoning question
• A question that facilitates logical, causal, or
  goal-oriented reasoning
• Example Shallow vs. Deep questions
• What is a type of circulation? (shallow)
• What is required for Systemic Circulation to
  occur? (deep)

10
The Contest

• Interactive computer tutor (Interactive
  condition)
• vs.
• Vicarious learning from dialogue with deep
  reasoning questions (Dialogue condition)
• vs.
• Monologue (Monologue condition)

11
Q-Dialogue versus Monologue
Agent 2 How does the earth's gravity affect the
sun?

• Agent 1 The sun experiences a force of
  gravity due to the earth, which is equal in
  magnitude and opposite in direction to the force
  of gravity on the earth due to the sun.

Agent 2 How does the gravitational force of the
earth affect the sun?
Agent 1 The force of the earth on the sun will
be equal and opposite to the force of the sun on
the earth
12
Laboratory results with multiple choice
dataCraig, Sullins, Witherspoon, Gholson,
(2006). Cognition Instruction.

• College students and computer literacy
• Three Conditions
• Interactive (AutoTutor)
• Yoked vicarious (AutoTutor sessions)
• Q-Dialogue with deep questions

Cohens d effect size
13
Memphis City School Study I

• Middle and high school students in two domains
• Computer literacy Grades 8 10
• Physics Grades 9 11
• Three Conditions
• Interactive (AutoTutor)
• Dialogue (Monologue with deep questions)
• Monologue (AutoTutor Ideal Answers)

14
Impact of condition as a function of prior
knowledge Memphis City School Study I
Cohens d effect size
15
Classroom Research

• Standard classroom teaching
• vs.
• Vicarious learning from dialogue with deep
  reasoning questions
• vs.
• Monologue

16
Overview of biology studyMemphis City School
Study II

• 8th grade biology (circulatory system)
• Day 1
• Pretesting
• Gholson (multiple choice)
• Azevedo (matching, labeling, flow diagram, mental
  model shift)
• Days 2-6
• 30-35 minutes of vicarious dialogue, vicarious
  monologue, or standard classroom instruction
• 10 minutes to answer essay questions
• Day 7
• 15-20 minutes of vicarious or interactive review
• Day 8
• Posttests
• Gholson (multiple choice)
• Azevedo (matching, labeling, flow diagram, mental
  model shift)

17
Azevedo and Gholson test resultsMemphis City
School Study II
Mental model shift
Cohens d effect size
18
Daily essay questions Memphis City School Study
II
Effect size compared to standard classroom
Cohens d effect size
Dialogue vs. standard pedagogy
Monologue vs. standard pedagogy
19
Conclusions

• Vicarious learning is effective when students
  observe animated conversational agents asking
  deep questions.
• Deep-level reasoning questions effect replicates
  in computer literacy and Newtonian Physics
  (8th-11th).
• Vicarious learning is most effective for learners
  with low domain knowledge.
• Vicarious learning transfers to classroom
  settings for daily essays, but not for the
  primarily more shallow one day delayed tests.

20

• ?

21
Memphis City School Study IIDesign
Class Format Conditions
1 vicarious Monologue
1 vicarious Dialogue
2 vicarious Monologue
2 vicarious Dialogue
3 interactive Regular classroom instruction
22
Memphis City School Study II

• Using vicarious learning to teach course content
  at Snowden Middle School
• 8th Graders
• Our first foray into the circulatory system domain

23
Memphis City School Study IIMaterials

• Students in vicarious conditions observe the
  virtual tutoring session via laptop computer in
  the classroom
• Students in the interactive condition receive the
  regular classroom instruction
• 2 Pretests developed by
• Gholson (multiple choice)
• Azevedo (matching, labeling, flow diagram, mental
  model shift)
• 3 Posttests developed by
• Gholson Azevedo (identical to pretest)

24
Memphis City School Study IIProcedure

• Day 1
• Pretesting
• Days 2-6
• 30-35 minutes of vicarious or interactive
  instruction in the circulatory system
• 10 minutes to answer review questions after
  instruction
• Day 7
• 15-20 minutes of vicarious or interactive review
• Day 8
• Posttests (Gholson and Azevedo)

25
Alternative Predictions

• 1. Interactive hypothesis
• Interactive gt Q-Dialog Monolog
• 2. Dialogic hypothesis
• Interactive Q-Dialog gt Monolog
• 3. Deep question hypothesis
• Q-Dialog gt Interactive Monolog

26
Learning Conceptual Physics

• Four conditions
• Read Nothing
• Read Textbook
• AutoTutor
• Human Tutor

27

• What are Deep-Level Reasoning Questions?
• (Graesser and Person,1994)
• LEVEL 1 SIMPLE or SHALLOW
• 1. Verification Is X true or false? Did an
  event occur?
• 2. Disjunctive Is X, Y, or Z the case?
• 3. Concept completion Who? What? When?
  Where?
• 4. Example What is an example or instance of a
  category?).
• LEVEL 2 INTERMEDIATE
• 5. Feature specification What qualitative
  properties does entity X have?
• 6. Quantification What is the
  value of a quantitative variable? How much?
• 6. Definition questions What does X mean?
• 8. Comparison How is X similar to Y? How is X
  different from Y?
• LEVEL 3 COMPLEX or DEEP
• 9. Interpretation What
  concept/claim can be inferred from a pattern of
  data?
• 10. Causal antecedent Why did an event occur?
• 11. Causal consequence What are the consequences
  of an event or state?

28
Learning Environments with Agents developed at
University of Memphis
AutoTutor Understanding science technology
MetaTutor Learning how to learn and think
iSTART Deep reading
SEEK True versus false information on the web
iDRIVE Deep question asking and answering
HURAA Reasoning about research ethics
ARIES Scientific reasoning
iMAP Multi-channel communication
29
Memphis City School Study IResults - Overall
Cohens d
Cohens d effect size
30
Other Collaborations with Agents at University
of Memphis
iDRIVE Question answering in science technology Gholson
MetaTutor Metacognition in science Azevedo
iMAP Multichannel commun ication with maps Louwerse
SEEK Critical stance while exploring web Wiley, Goldman
ARIES Critical Reasoning in science Millis, Britt, Magliano, Wiemer-Hastings
31
Conclusions and summary

• Deep-level question effect - Deep-level question
  dialog improves learning over an interactive
  session, yoked vicarious session, monolog
  session with same content
• (Craig, et al., 2006)
• Effect replicates in computer literacy and
  Newtonian Physics.
• Effect transfers to classroom settings

32
Questions in Newtonian physics

• The sun exerts a gravitational force on the
  earth as the earth moves in its orbit around the
  sun. Does the earth pull equally on the sun?
  Explain why?

33
Expectations and misconceptions in Sun Earth
problem

• EXPECTATIONS
• The sun exerts a gravitational force on the
  earth.
• The earth exerts a gravitational force on the
  sun.
• The two forces are a third-law pair.
• The magnitudes of the two forces are the same.
• MISCONCEPTIONS
• Only the larger object exerts a force.
• The force of earth on sun is less than that of
  the sun on earth.

34
Misconceptions
contact forces exerted after contact ceases
vertical forces might have a non-zero horizontal component
heavier objects fall faster
heavier objects accelerate faster for the same non-gravitational force
air resistance non negligable
freefall means constant velocity
lighter object exerts no force on a larger object
nonzero net force but no acceleration
same force means same acceleration regardless of mass
action and reaction force acts on same body
0 force implies slowing down
0 force implies speeding up
0 force implies 0 velocity
(no autotutor equiv) 0 acceleration implies 0 velocity
action and reaction force do not have same magnitude
After an object is dropped or thrown the only force acting on it is gravity
Gravitational force acts only in the vertical direction
Inanimate object exerts no/less force in interaction
Object that has been hit exerts no/less force in interaction
Accelerations of both objects equal during interaction
Only masses of part of compound body considered
The force acting on a body is dependent on the mass of the body
Action and reaction force have same directions
Acceleration considered relative to accelerated reference frame

35
Force equals mass times acceleration
Pretest Essay Pretest MC Training Posttest Essay Posttest MC
All-or-none Learning X00X X0XXXX0XXX XXX0XXX0XX XX0X0X X0XX0X1XX1 X1X1 XX1XX1XXXX X11XXXXXX1 XX1XXX
Variable Learning X10X X0XXXX0XXX XXX1XXX0XX XX0X0X X0XX1X1XX0 X1X0 XX1XX0XXXX X11XXXXXX0 XX1XXX
No Learning X00X X0XXXX1XXX XXX0XXX0XX XX1X0X X0XX0X1XX0 X1X0 XX0XX0XXXX X10XXXXXX0 XX1XXX
Refresher Learning X00X X0XXXX1XXX XXX1XXX1XX XX1X1X X1XX1X1XX1 X1X1 XX1XX1XXXX X11XXXXXX1 XX1XXX
36
Conceptual Physics(Graesser, Jackson, et al.,
2003)

• Three conditions
• AutoTutor
• Read textbook
• Read nothing

37
Impact of Monolog versus Dialog on recall and
questions in a transfer task (Craig, Gholson,
Ventura, Graesser, 2000)

• Learning about computer literacy with
  conversational agents.
• Monolog on computer literacy content
• Dialog with added deep questions
• Recall of content in training task
• Transfer tasks on new material
• Students instructed to generate questions about
  new computer literacy topics
• Recall of content of new material

38
Impact of Dialog versus Monolog on recall and
questions in a transfer task (Craig, Gholson,
Ventura, Graesser, 2000)
39
Managing One AutoTutor Turn

• Short feedback on the students previous turn
• Advance the dialog by one or more dialog moves
  that are connected by discourse markers
• End turn with a signal that transfers the floor
  to the student
• Question
• Prompting hand gesture
• Head/gaze signal

40
Expectation and Misconception-Tailored Dialog
Pervasive in AutoTutor human tutors

• Tutor asks question that requires explanatory
  reasoning
• Student answers with fragments of information,
  distributed over multiple turns
• Tutor analyzes the fragments of the explanation
• Compares to a list of expected good idea units
• Compares to a list of expected errors and
  misconceptions
• Tutor posts goals performs dialog acts to
  improve explanation
• Fills in missing expected good idea units (one at
  a time)
• Corrects expected errors misconceptions
  (immediately)
• Tutor handles periodic sub-dialogues
• Student questions
• Student meta-communicative acts
  (e.g., What did you say?)

41
Dialog Moves During Steps 2-4

• Positive immediate feedback Yeah Right!
• Neutral immediate feedback Okay Uh huh
• Negative immediate feedback No Not quite
• Pump for more information What else?
• Hint What about the earths gravity?
• Prompt for specific information The earth
  exerts a gravitational force on what?
• Assert The earth exerts a gravitational force
  on the sun.
• Correct The smaller object also exerts a force.
  
• Repeat So, once again,
• Summarize So to recap,
• Answer student question

42
Procedure
Gates-McGinitie reading test Pretest
Interactive, Monologue, or Dialogue instruction
Posttest
43
Memphis City School Study(342 students)

• 2 x 2 x 3 Design

Age Subject Condition Condition Condition
Dialogue Monologue Interactive
8th 9th Computer
8th 9th Physics
10th 11th Computer
10th 11th Physics
44
Multiple Choice Test Results Physics Computer
Literacy
45
How to cover a single expectation

• The earth exerts a gravitational force on the
  sun.
• Who articulates it student, tutor, or both?
• Fuzzy production rules drive dialog moves
• Progressive specificity drives dialog moves
• Hint ? Prompt ? Assertion cycles
• Strategies tailored to student knowledge and
  abilities

46
How does AutoTutor compare to comparison
conditions on tests of deep comprehension?

• 0.80 sigma compared to pretest, doing nothing,
  and reading the textbook
• 0.22 compared to reading relevant textbook
  segments
• 0.07 compared to reading succinct script
• 0.13 compared to AutoTutor delivering speech acts
  in print
• 0.08 compared to humans in computer-mediated
  conversation
• -0.20 compared to AutoTutor enhanced with
  interactive 3D simulation
• ZONE OF PROXIMAL DEVELOPMENT

47
Memphis City School Study IIVicarious Interface
48
Memphis City School Study II

• Question How will the vicarious conditions
  perform next to interaction with a human teacher?