Imaginary Worlds and Grounded (Embodied) Cognition

1
Imaginary Worlds and Grounded (Embodied) Cognition

• John B. Black
• Teachers College
• Columbia University

2
Research with Gordon Bower

• Studied Students Reading and Remembering Stories
• Found That Switches in Point of View Lead to
  Longer Reading Times and Memory Errors Spatial
  Layout
• Found That Actions That Changed The Story World
  Were Remembered Better
• Seemed to be Story World Separate From Symbolic
  Content of Story (Mental Model)

3
Symbolic Content vs Model World

• Johnson-Laird Proposed Mental Models That
  Involved Imagining Models of Logical Expressions
• Kintsch Proposed Situation Models of Textual
  Content Separate from Textbase of Propositional
  Networks
• Philosophers like Frege and Carnap Distinguish
  Between Sense and Reference

4
My Research at TC

• Shifted from Text to Multimedia
• What is the Nature of These Mental Referent or
  Imaginary Worlds?
• How Does One Teach For Imaginary World Level of
  Understanding?

5
World of a Cathedral

• Multimedia System to Learn About St. Johns
  Cathedral (Van Esselstyn and Black)
• Learned a lot from just Floor Plan (survey map
  knowledge) and Text
• Providing Virtual Tour (dynamic route map
  knowledge) only helped when given rationale for
  spatial layouts
• Constructing this Cathedral World needed reasons
  for spatial layout

6
Multimedia Cathedral
7
Research with Maggie Chan

• Crucial Element of Imaginary World Reasoning is
  Functional Relations
• Functional Relations Describe How One Entity
  Changes as a Function of Change in Another
• Deeper Lever Understanding Than Structural Causal
  Relations
• Showed Direct Manipulation Animation Good Way to
  Learn This

8
Roller Coaster Simulation Game

9
Roller Coaster Game Studies

• Snapshots (like Comics) Better than System
  Controlled Animation
• Consistent with Many Studies of Animation in
  Tversky Survey
• Direct Manipulation of one Factor with
  Animation of Other Factors Gave Better Learning
  (Chan and Black)

10
Experimental Design
IV1 Presentation Formats text-only,
text-plus-static diagrams, text-plus-direct-manipu
lation animation IV2 System Complexity
simple (Energy conversion in a playground swing),
moderate (Energy conversion in a roller
coaster ride), complicated (Energy
conversion of pole vaulting) Dependent Measures
DV1 Verbal Recall (summary) DV2
Visual Recall (drawing) DV3 What-if
Scenarios DV4 Whats wrong here? Case
study DV5 Transfer
11
Direct Manipulation AnimationSimple system --- a
playground swing
12
Direct Manipulation Animation Complex system ---
pole-vaulting
13
7th Grade

• What-if scenarios

14
6th Grade
15
Result Varies within Type of Student and System
Complexity

• General form seems to be for simple systems and
  capable students text is sufficient
• Also for capable students and intermediate
• systems text plus diagrams sufficient
• But for complex systems for any students and less
  capable students for any system need direct
  manipulation animation

16
Video Games for Learning

• There is a Lot of Hype About Video Games as
  Learning Environments
• However, Almost No Studies That Show This in
  fact, Some Show Reverse
• But We Have Found Video Games Good as Part of
  Learning Environment But Not By Themselves

17
Video Games as Preparation for Future Learning

• Civilization

18
Preparation for Future Learning Video Game Study

• Compared Experts in Civilization
• Formally Learning New Material
• Civilization Experts Were Better at
• Formally Learning New Material
• Particularly Systems Related
• Thus Video Games May Provide
• Experiences as
• Preparation for Future Learning

19
Reflective Questions with Games

• Studied Business Student Learning From Business
  Simulation Game (from HBS)
• Students Learned Much More If Game Supplemented
  with Reflective Questions
• About Strategies Used and Relation to Background
  Reading
• Army Has Found Learning With Americas Army Game
  Occurs in After Action Reports Which Provide
  Reflection

20
Grounded (Embodied) Cognition

• Currently Think Imaginary Worlds Involves
  Grounded or Embodied Cognition
• This is a Perceptual Simulation to Supplement
  (Provide Mental Referents)
• Recent Neural Imaging Studies Show Perceptual
  Areas of Brain Activated in Symbolic Tasks in
  Addition to Symbolic Areas

21
How To Investigate This?

• Glenberg Text Comprehension
• ARIA
• REAL
• ???

22
ARIA Project Associating Real and Imaginary
Agents

• ARIA Project Provides Embodiment by Having
  Students Work With Robot Simulations on Computer
  Screen Then Construct Robots to Do It In Real
  World
• Also Provides Embodiment By Having Students Act
  Out Using There Own Bodies What they Want Robots
  to Do
• Being Done as After School Program at The School
  at Columbia U.

23
REALReflective Agent Learning environment

• Provides Embodiment by Allowing Learner to Input
  Knowledge Representations in Head of Agents
• Then See How Agent with Knowledge
• Behaves in Virtual World
• Various Depictions of Results Reflects
• What is Good and Bad About Knowledge
• Then Try Again

24
REAL Business
25
REAL Business (Design Mode)
26
REAL Business (Game/Reflection Mode)
27
Constructing Dynamic Imaginary Worlds

• Virtual Worlds with Animated Agents can be
  Effective for Learning if
• Spatially Meaningful
• Causally Transparent
• Directly Manipulate Factors and See
• Animated Results
• Gets Internalized as Imaginary World
• Augmented Reality Promises to Give
• Same Benefits Better Integrated with
• Real World