Problem Solving Assessment in Games and Simulation Environments

1
Problem Solving Assessment in Games and
Simulation Environments Richard E.
Mayer University of California, Santa Barbara
Collaborators Eva Baker Roxana Moreno Gayle
Dow Harold F. ONeil, Jr. Jennifer
Dyck William Prothero Anne Fay Jill
Quilici James Lester Valerie Sims Patricia
Mautone Hiller Spires Sarah Mayer William
Vilberg
Supported by National Science Foundation, Andrew
Mellon Foundation
2
Problem Solving Assessment in Games and
Simulation Environments
Introduction What Is the Rationale for Games and
Simulations? What Are Instructional Games and
Simulations?What Is Problem-Solving Assessment
in Games and Simulations? Example 1 Assessment
of Problem Solving in Computer Programming Example
2 Assessment of Problem Solving in Playing
Educational Games Example 3 Assessment of
Problem Solving in the Bunny Game Example 4
Assessment of Problem Solving in the
Design-A-Plant Game Example 5 Assessment of
Problem Solving in the Profile Game Example 6
Assessment of Problem Solving in the Dr. Phyz
Simulation Example 7 Assessment of Problem
Solving in an Aircraft Simulation Conclusion
3
What Is the Rationale for Games and Simulations?

• Bransford, Brown, Cocking (1999)
• Visual modes of presentation technologies can
  help people visualize difficult-to-understand
  conceptsso students are able to work with
  visualizationsoftware
• Interactive modes of presentation new
  technologies are interactive so it is easier to
  create environments in which students learn by
  doing

4
What Are Instructional Games and Simulations?

• An instructional simulation
• Is a multimedia environment that models a
  to-be-learned system.
• Allows for user interaction by responding in
  rule-based ways to user input.
• Is intended to promote learner understanding of
  the to-be-learned system.

An instructional game 4. Also involves one or
more players competing to achieve some goal which
is accomplished through understanding the subject
matter.
5
What Is Problem-Solving Assessment in Games and
Simulations?
Types of
Knowledge Name Definition Example Facts basic
elements magnets are made of
metal Concepts models or principles how an
electric motor works Procedures step-by-step
processes how to add numbers Strategies general
methods how to break a problem into parts
6
Criteria for Problem-Solving Assessments Interact
ivity criterion involves interaction between the
learner and the computer-based system Multimedia
criterion involves verbal and non-verbal forms
of representation (such as graphics) Problem-solvi
ng criterion involves solving problems

• Seven Games and Simulations for Assessing the
  Learners Knowledge
• Computer programming
• Educational games
• Bunny game
• Design-a-plant game
• Profile game
• Dr. Phyz simulation
• Aircraft simulation

7
Programming in Logo
RT 90
FD 5
8
Purpose To determine the cognitive consequences
of learning Logo or BASIC programming Test Know
ledge Description Score(s) Map
test Strategic Given a neighborhood map and
list Number correct of locations draw a path
connecting them. Given a neighborhood map
showing Number correct a path connecting
various locations, give verbal
directions. Command Conceptual Given a Logo
command, draw Number correct,
prediction what happens when it is
executed. misconception type
Problem Conceptual Given a word problem,
select a Number correct translation correspon
ding equation. Following Strategic Given a
list of verbal commands, Number
correct procedures describe the final
output. From Fay Mayer (1987), Mayer Fay
(1987), Mayer, Dyck, Vilberg (1986)
9
Command prediction problem
RT 90
10
Problem translation A car rental service charges
20 dollars a day and 15 cents a mile to rent a
car. Find the expression for total cost C, in
dollars, of renting a car for D days to travel M
miles. a. C 20D 0.15M b. C 15D
0.20M c. C 20D 15 M d. C 0.15D 20M
11

• Following procedures
• Put 5 in Box A.
• Put 4 in Box B
• Add the number in Box A and the number in Box B,
  put the result in Box C.
• Add the number in Box A and the number in Box C,
  put the result in Box A.
• Write down the numbers from Box A, B, and C.
• What is the output of this program?
• a. 5, 4, 9
• b. 14, 4, 9
• c. 14, 9, 9
• d. 9, 4, 9
• e. none of the above

12
Purpose To determine the cognitive consequences
of playing educational computer
games. Test Knowledge Description Score(s) Sen
tence Conceptual Given a sentence from a
word Number correct translation problem,
select the corresponding number
sentence. Necessary Conceptual Given a word
problem, select Number correct numbers the
numbers that are needed to solve the
problem. Necessary Strategic Given a word
problem, select Number correct operations the
arithmetic operations that are needed to
solve the problem. Puzzle Strategic Given a
new game, make Number of excess tank each
necessary move. moves, type of
strategy Mental Strategic Judge
whether two shapes Response time, rotation are
same or different. best fitting model Form
Strategic Judge whether shapes Response
time board fit together.
From Mayer et al. (1997), Mayer et al. (1999),
Sims Mayer (2002)
13
Sentence translation Ann and Rose have 20 books
altogether. Which number sentence is correct? a.
Anns books Roses books 20 b. Anns
books 20 Roses books c. Anns books
Roses books 20 d. Anns books Roses
books
14
Necessary numbers Which numbers are needed to
solve this problem? A package of 3 toys costs 88
cents. Richie bought 2 packages. How many toys
did he buy? a. 3, 88, 2 b. 3, 88 c. 88,
2 d. 3, 2 Necessary operations What should you
do for this problem? The 200 children at River
View School are going on a bus trip. Each bus
holds 50 children. How many buses are
needed? a. divide, then add b. subtract
only c. multiply only d. divide only
15
Purpose To assess knowledge of addition and
subtraction of signed numbers in the bunny
game. Test Knowledge Description Score(s) Bunn
y Conceptual Given an arithmetic Number
correct, game problem, use virtual misconcepti
on type joystick to simulate movement of
bunny along number line. Arithmetic Strateg
ic, Given an arithmetic Number
correct, solving conceptual problem, type
in misconception type correct answer. From
Moreno Mayer (1999)
16
Bunny game
4 - -5 ___
-9 -8 -7 -6 -5 -4 -3 -2 -1 0
1 2 3 4 5 6 7 8 9
JUMP FORWARD FACE LEFT
FACE RIGHT JUMP BACK
17
Purpose To assess knowledge of how plants grow
in the design-a-plant game. Test Knowledge Descr
iption Score(s) Design-a- conceptual Select
roots, stem, and leaves Number correct Plant
game for a plant to grow in a specified
environment. Given a plant with
specified Number correct roots, stem, and
leaves, describe the environment it
is best suited for. From Moreno et al.
(2001)
18
Design-a-plant game Design a plant to live in
an environment with low sunlight. Select
leaves, stem, and roots. Describe the
environment that would be best for a plant with
thick, small, think-skinned leaves think, long,
and dark bark stem and branching, shallow, and
thin roots.
19
Purpose To assess knowledge of geology sonar
detection in the profile game. Test Knowledge De
scription Score(s) Profile game strategic Detec
t geological features Number correct, in a
geology simulation. solution time From Mayer,
Mautone, Prothero (2002)
20
Section of earth
Draw a line here
Profile line
See its profile here
21
Purpose Assess knowledge of how an electric
motor works in the Dr. Phyz simulation. Test Kno
wledge Description Score(s) Trouble- conceptual
List possible reasons Number of
shooting for motor to not correct
answers problem work when turned
on. Redesign conceptual Describe how to
change Number of problem a motor for a
certain correct answers purpose. Principle
conceptual Explain why an event Number of
problem occurs. correct answers
Prediction conceptual. Tell what would
happen Number of problem for a specific set
of actions. correct answers Action conceptual T
ell actions to take to make Number
of problem the motor accomplish a correct
answers specific goal. From Mayer et al.
(2003)
22
Troubleshooting problem Suppose you switch on an
electric motor but nothing happens. What could
have gone wrong? Redesign problem What could
you do to increase the speed of the electric
motor, that is, to make the wire loop rotate more
rapidly? Principle problem Why does the wire
loop move? Prediction problem What happens if
you connect the negative terminal to the red wire
and the positive terminal to the yellow
wire? Action problem What can you do to
reverse the movement of an electric motor, that
is, to make the wire loop rotate in the opposite
direction?
23
Purpose To assess knowledge of an aircraft fuel
system in the aircraft simulation. Test Knowledg
e Description Score(s) Trouble- conceptual Lis
t possible reasons for Number correct shooting
fuel system to malfunction. problem Redesign con
ceptual Describe how to change the Number
correct problem fuel system for a
certain purpose Prescription conceptual Tell
what would happen for Number correct problem a
specific set of actions From ONeil et al.
(2000)
24
Troubleshooting problem Suppose you try to
transfer fuel from one tank to another but
nothing happens. What could have gone
wrong? Redesign problem What could be done to
make fuel transfer more reliable, that is, to
minimize the chances of something going wrong
when fuel is transferred from one tank to the
next? Prediction problem What would happen if
the ejector pump stopped working?
25
Conclusion Problem-solving performance in any
domain depends on the learners knowledge, which
is measurable. Conceptual and strategic
knowledge can be assessed. Assess conceptual
knowledge by asking the learner to Specify what
happens when a command is executed Translate a
word problem into an equation Tell what numbers
are needed to solve a word problem Ask a learner
to role play what happens for a numerical
computation Answering open-ended questions
involving troubleshooting, redesign,
predicting, explaining, and prescribing Assess
strategic knowledge by asking the learner
to Generate or follow directions Describe the
output of a procedure Specify the operations
needed to solve a word problem Learn to play a
new game Make judgments Solve new problems