What do we Know about the Effectiveness of Instructional Strategies in Computer Games

1
What do we Know about the Effectiveness of
Instructional Strategies in Computer Games

• Harold F. ONeil
• and
• Joan (Yuan-Chuang) Lang
• University of Southern California/CRESST

AERA v.4 New York March 2008
2
CRESST Model of Learning
Content Understanding
Collaboration/Teamwork
Learning
Problem Solving
Communication
Self-Regulation
3
Instructional Strategy

• Definition
• Prescribed sequences and methods of instruction
  to achieve a learning objective
• Source
• Spector, J.M., Merrill, M.D., van Merrienboer,
  J., Driscoll, M.P. (2008). Handbook of
  Research on Educational Communications and
  Technology. New York Taylor Francis Group.
• Macro vs Micro Instructional Strategies
• - Do games work? (macro)
• - What strategies within games work? (micro)

4
Macro Instructional Strategies Issues
If in class
If out of class

• Integrate into recreation/family time
• Increased requirement for effective micro
  instructional strategies (feedback, worked
  examples)
• Student training need (e.g., self-monitoring)
• Flexible
• Fantasy role

• Integrate into Syllabus
• Instructor training
• accuracy re standards
• Limited role for fantasy

5
Macro Instructional Strategies
If in class
If out of class

• Motivation provided by student/game
• Game needs to be fun
• Good application
• Refresher training prevent skill decay
• Available time in minutes/hours

• Motivation provided by instructor
• Game can be less fun
• Good application
• Initial acquisition training
• Available time in hours/days/weeks

6
Research Questions

• Will adding effective instructional strategies to
  commercial off the shelf games improve problem
  solving?
• Trade-off between development and selection

7
Do Games Teach? Check Validity of Micro
Instructional Strategy

• Embedded in game
• Usually inductive discovery approach
• Usually doesnt result in learning (Kirschner, P.
  A., Sweller, J., Clark, R. E. 2006. Why minimal
  guidance during instruction does not work An
  analysis of the failure of constructivist,
  discovery, problem-based, experiential, and
  inquiry-based learning. Educational Psychologist,
  41, 75-86.)
• What Works in Distance Learning (ONeil, 2005)
• Good instructional practices that can be applied
  to games

8
Selection of Game for Research

• Off-the shelf games lacking learning objectives
  and assessment of learning
• Use wrap around instructional assessment
  strategies as no access to source code

9
Common Methodology

• Participants
• Young adults selected to have no experience of
  playing SafeCracker but game players
• Measures
• Knowledge mapper
• Retention and transfer questions analogous to
  Mayers
• Trait self-regulation questionnaire

10
Domain Specific Problem-Solving Strategy Measures

• Retention Question
• List how you solved the puzzles in the rooms.
  
  
• Transfer Question
• List some ways to improve the fun or challenge
  of the game.
• Modifications of previous researchers (Mayer,
• 2001 Mayer Moreno, 1998 Moreno Mayer,
  2004)

11
Knowledge Mapper
12
Measurement of Self-Regulation

• Trait self-regulation questionnaire (ONeil
  Herl, 1998).
• planning
• self-checking
• self-efficacy
• effort

13
Study I, II, III

• Study I Without effective instructional
  strategies.
• Study II With worked examples.
• Study III With just-in time worked examples

14
Study IDiscussion/Implications

• There was an increase in problem-solving. But it
  was small.
• Existing instructional strategies (discovery
  learning) in the game were not effective.
• More research on a game designed with effective
  research-based instructional strategies
• Worked examples (Danny Shen)
• Pictorial aids (Richard Wainess)
• Just-in-Time Worked Examples (Joan Lang)
• After-Action Review

15
Study II and Study III Worked Examples

• Worked examples are procedures that focus
  attention on problem states and associated
  operators (i.e., solution steps), enabling
  students to induce generalized solutions or
  schemas (Sweller, 1998).
• Many researchers investigated the efficacy of
  using worked examples in classroom and
  computer-based instruction and provided evidence
  of the effectiveness of worked examples
  instruction (Cooper Sweller, 1987 Mayer
  Mautone, 2002 Ward Sweller, 1990).
• No research used worked examples in a game-based
  environment.

16
Study Worked Example Discussion/Implications

• The worked example group significantly improved
  more than the control group in content
  understanding and problem-solving strategies.
  However, the improvement was small.
• This study provided evidence that using worked
  examples could be one of the good instructional
  methods to facilitate adults problem solving
  with a commercial off-the-shelf computer game.

17
Just in Time Worked examples

• What worked examples can do?
• Facilitate learning of problem solving in the
  computer game SafeCracker
• Contribute to superior content understanding in
  problem solving (when presented just-in-time)
• May contribute to superior retention skill
  (partial evidence) but not transfer skill

18
What Are Continuing RD Issues?

• Can we leverage game technology for training?
• Embedded instructional and assessment strategies
• Wrap-around instructional and assessment
  strategies

19
Walk Issues

• Analytically, would you predict that commercial
  off-the-shelf games should teach?
• No
• What support and guidance would help training
  game developers to do a better job?
• Alignment with What Works in Distance Learning
• Instructional strategies that could work
• Wrap-around or embedded instructional and
  assessment strategies
• Worked examples
• feedback

20
CRESST Web Site
http//www.cresst.org or any search engine type
CRESST
honeil_at_usc.edu
21
What Is a Game?

• A computer game consists of four key components
• Settings that are real or imaginary
• Roles or agendas for the participants
• Rules (real life vs. imaginative)
• Scoring, recording, monitoring, or other kinds of
  systematic measurement
• Motivation comes from challenge, complexity,
  fantasy

22
What Works in Distance Learning

• Develop a core set of research-based DL
  guidelines and lessons
• Instructional Design (Clark)
• Multimedia (Mayer)
• Learning Strategies (Dembo et al.)
• Assessment (Baker et al.)
• Management Strategies (Kazlauskas)
• Self-Regulation and Motivation (ONeil et al.)
• Team Training and Assessment (Cannon Bowers)
• Team Motivation (Clark)

ONeil, H. F. (Ed.). (2005). What works in
distance learning Guidelines. Greenwich, CT
Information Age Publishing Inc.
23
Sample of a Worked Example
Search for the code related to the Liberty Safe
Find a Note from switchsafe in the toolbox
Click the Note from switchsafe
The Note from switchsafe will show on the screen
Notice that some codes are missing
Click the Note to close it
24
The Specification of What We Are Teaching Is
Essential

• From goal/objective of teaching leadership,
  situational awareness, decision making, tactical
  problem solving
• The instructional strategies follow
• Nature of feedback, timing of feedback, take-home
  packages, instructor training, homework
  assignments, etc.
• The type of assessment follows
• Different assessment measures, after-action
  reviews

25
SafeCracker

• Puzzle-solving game
• Example of problem solving
• No special background knowledge, motor skills, or
  extraordinary visual-spatial ability required
• Adult-oriented
• Single-player game
• Pacing controlled by players
• Not popular

26
Results Content UnderstandingBold as
hypothesized

• The JIT worked example (M 7.97) group
  demonstrated significantly better (p .01)
  content understanding than the worked example
  group (M 6.63) and the control group (M 6.51)
• The worked example group did not demonstrate
  significantly better content understanding than
  the control group

27
Results Problem Solving Strategies

• Retention skill
• The worked example group (M 2.72, SD .72)
  demonstrated significantly greater retention
  skill than the control group (M 2.03, SD .88)
  by opening significantly more worked-example-illus
  trated safes (p .01)
• Transfer skill
• No difference among groups