Research Issues in Developing Games for Learning and Assessment

1
Research Issues in Developing Games for Learning
and Assessment
Gregory K.W.K. Chung
California Educational Research Association
(CERA) San Francisco, CA November 19, 2009
2
Overview

• Project overview
• Why study games for learning?
• Tensions along the way
• Some design variables
• Study results
• Conclusion and next steps

3
Project Overview

• Center for Advanced Technology in Schools (CATS)
• USC Game Innovation Lab
• RD focused on games and simulations for learning
  and assessment
• Content focus is pre-algebra (rational numbers,
  solving equations, functions)
• Target population is underprepared students
• Systematic testing of features (instructional
  variations, game-based) before full-scale
  implementation

4
Why Study Games for Learning?

• If you build it, they will play (and learn) ...
• Given Students choose to spend hours playing
  games
• Idea Lets put academic content in games
• Magic Students will play the game, be engaged in
  the game, and will learn the stuff
• fait accompli
• Recall scantron (1950s), word processors (1980s),
  calculators (1980s), OPAC (1980s), Web (1990s)
  ...
• Its going to happen with or without RD, so
  lets figure out ways to shape the process

5
Why Study Games for Learning?

• Help determine the relationship among
• Different instructional design variables AND
• Different game design variables AND
• Different types of learning outcomes AND
• Different types of students AND
• Different types of game outcomes

6
Tensions Games for Learning Math

• game lt--gt learning
• fun lt--gt math
• play time lt--gt efficiency
• choose to play lt--gt have to play
• pure math lt--gt applied math
• basic skills lt---gt 21st century skills
• simple tasks lt--gt complex tasks
• unobtrusive measures (embedded) lt---gt obtrusive
  measures (external)

6
7
The RD Challenge

• Math outcomes
• Skills
• Conceptual understanding
• Game outcomes
• Game level
• Gaminess

• Instruction
• Tutorial
• Feedback
• Core mechanics
• Must use math
• Motivational elements
• Bling

7
8
Game Design Variables

• Instruction
• Game mechanics
• Conceptual
• Procedural
• Core mechanics
• Part of game
• Motivation
• Bling

• Feedback
• Type
• Timing
• Precision
• Impasse-driven
• In-game Assessment
• Scoring
• Performance sensing

8
9
Outcome Variables

• Math outcomes
• Skills
• Conceptual understanding
• Game outcomes
• Student perception of gaminess
• Flow
• Game level

9
10
Prototype Gamelet
11
Game Design Requirements

• The Outcome
• Conceptual and computational fluency with
  rational numbers (fractions)
• The Math
• Idea of unit and fractional parts
• Additive operations
• Denominator ? no. of pieces in 1 unit
• Numerator ? no. of pieces
• Equivalence
• The Challenge How to do math without killing the
  game

12
Prototype Game Design

• Genre
• Puzzleneed to figure out how to navigate from
  start to end points
• Game and Learning Mechanics
• Jumping/bouncing from point to point
• Adding coils to go from point to point
• Only allowed to add pieces of the same fractional
  size (i.e., common denominator)
• Need to convert among equivalent units (2/2 3/3
  4/4)

13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Study
17
Research Study

• Research Question
• To what extent do different kinds of feedback
  affect understanding of fractions (i.e., unit),
  game performance, and perception of game play?
• Design
• 2 conditions that varied feedback
• Gamey Minimal math instruction
• Mathy Emphasized math concepts related to unit

18
Sample

• Sample
• N 137
• 9th (30) 10th (18), 11th (31), 12th (15)
• Amount of weekly game play
• 0hr (21) 1-2hr (40) 3-6hr (19) gt 6hr (23)
• Math achievement
• Self-reported grades As and Bs (55), Cs
  (31), Ds and Fs (13)
• Math pretest M 6.34, SD 3.39, Min. 0, Max.
  11

19
Measures

• Math outcome
• Pretest, posttest
• Game outcome
• Last level reached, perception of game
• Game process measures
• Time, correct fraction additions, incorrect
  fraction additions
• Background

20
Results

• Did we build a game?
• Did students learn from the game?
• Was there an effect of type of feedback on
• Learning?
• Game performance?
• Game perception?

21
Did we build a game?
22
Yes
23
Results
24
Results
25
Results
26
Did students learn from the game?
27
It depends
28
Did students learn from the game?

• No overall effects of game play on math posttest
  scores
• Not surprisingsample was composed of high and
  low performers
• However, our target grouplow math
  performersappeared to profit from game play
• Low performers posttest scores (M 3.08, SD
  2.04) were significantly higher than their
  pretest scores (M 2.55, SD 1.22). t (48)
  2.0, p .05, d 0.32.

29
Was there an effect oftype of feedback on
learning?
30
No
31
Was there an effect oftype of feedback on game
performance?
32
Yes
33
Was there an effect oftype of feedback on game
performance?

• Students in the mathy condition (vs. the gamey
  condition)
• Appear to have gone further in the game (p .08,
  d 0.31)
• Committed more correct additions (p .003, d
  0.49)
• Committed fewer incorrect additions (p .007, d
  0.48)

34
Was there an effect oftype of feedback on game
perception?
35
Probably
36
Was there an effect oftype of feedback on game
performance?

• Students in the mathy condition (vs. the gamey
  condition)
• Perceived the game as more game-like (p .08)
• Were more willing to use the game as part of
  school work (p .06)
• Agreed more with the statement that the game
  helped them understand math (p .003, d 0.54)

37
Summary

• Did we build a game? (YES)
• Did students learn from the game? (ONLY LOW
  PERFORMERS)
• Was there an effect of type of feedback on
• Learning? (NO)
• Game performance? (YES)
• Game perception? (PROBABLY)

38
Conclusion and Next Steps

• Beginning to understand conditions under which
  mathification may not hurt game play
• Speculate that math instruction helped students
  progress in game
• Impasse-driven instruction
• Results promising for the development of a game
  that includes math content while preserving game
  aspect
• Need stronger instructional intervention
• Building tutorial, just-in-time feedback

39
Backup
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)