Latent Problem Solving Analysis (LPSA): A computational theory of representation in complex, dynamic problem-solving tasks

1
Latent Problem Solving Analysis (LPSA) A
computational theory of representation in
complex, dynamic problem-solving tasks
José Quesada
2
Complex problem solving (CPS) definition

• dynamic, because early actions determine the
  environment in which subsequent decision must be
  made, and features of the task environment may
  change independently of the solvers actions
• time-dependent, because decisions must be made at
  the correct moment in relation to environmental
  demands and
• complex, in the sense that most variables are
  not related to each other in one-to-one manner

3

• Despite 10 years of research in the area, there
  is neither a clearly formulated specific theory
  nor is there an agreement on how to proceed with
  respect to the research philosophy. Even worse,
  no stable phenomena have been observed
• (Funke, 1992, p. 25)

4
"How similar are two participant's solutions?"

• For CPS there is no common, explicit theory to
  explain why a complex, dynamic situation is
  similar to any other situation or how two slices
  of performance taken from a problem solving task
  can possibly be compared quantitatively
• This lack of formalized, analytical models is
  slowing down the development of theory in the
  field

5
Example of a complex, dynamic task Firechief
(Omodei and Wearing 1995)
6
(No Transcript)
7
Problems with the classic 'problem space
approach!

• Most of the theories about cognitive skill
  acquisition and procedural learning are based in
  two principles
• The problem space hypothesis
• Representation of procedures as productions

8
Problems with the classic 'problem space
approach!

• The problem with the generation of the problem
  space
• The utility of the state space representation for
  tasks with inner dynamics is reduced because in
  most CPS environments it is not possible to undo
  the actions, and prepare a different strategy

9
Problems with the classic 'problem space
approach!

• The classic problem solving theory used mainly
  verbal protocols as data. However, TALK ALOUD
  INTERFERES PERFORMANCE IN COMPLEX DYNAMIC TASKS
  (Dickson, McLennan Omodei, 2000)
• Independence (or very short-term dependences) of
  actions/states is assumed in some of the methods
  for representing performance. That is, the
  features that represent performance are local

10
Objectives of the dissertation

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

11
LPSA as a theory of representation in CPS tasks

1. Applications Automatic landing technique
   assessment

1. Expertise effects of amount of practice
2. Expertise effects of amount of environmental
   structure

1. human similarity judgments
2. Strategy changes

12
What LPSA is and how it relates to other theories
13
Latent Problem Solving Analysis (LPSA)

• Assumptions
• Similarity-based theory of representation
• The problem space is a vector space
• It can be generated from experience automatically
  (corpus-based)
• Search and movement in this problem space
  consists of vector operations

14
LSA
LPSA
The problem space is a metric space, where states
and trials are represented as vectors
15
Approaches to complexity The ant and the beach
parable (Simon, 1967,1981)
16
Approaches to complexity The ant and the beach
parable (Simon, 1967,1981)
17
Approaches to complexity The ant and the beach
parable (Simon, 1967,1981)
18
Approaches to complexity The ant and the beach
parable (Simon, 1967,1981)
?
19
Latent Problem Solving Analysis (LPSA)

• Unsupervised learning
• Empirical adjustment of a problem space
• Definition of a productivity mechanism and a
  similarity measure.
• LPSA addition and cosine.

20
Latent Problem Solving Analysis (LPSA)

• m(trial) fm(sa1), m(sa2),.. m(san), context
• Simplifying assumptionsm(trial1) m(sa11)
  m(sa21) .. m(san1) m(trial2) m(sa12)
  m(sa22) .. m(san2). m(trialk) m(sa1k)
  m(sa2k) .. m(sank)
• Where sa is a state or action

21
Latent Problem Solving Analysis (LPSA)

• Complexity reduction Reducing the number of
  dimensions in the space reduces the noise

22
LPSA solutions for the problems with the classic
'problem space approach

• The problem with the generation of the problem
  space
• The utility of the state space representation for
  tasks with inner dynamics is reduced because in
  most CPS environments it is not possible to undo
  the actions, and prepare a different strategy

LPSA proposes a mechanism to generate
automatically the problem space
LPSA does not propose a specific solution for
this problem, but it enables the experimenter to
represent very different complex problem solving
tasks using a common formalism that could
implement, as additional assumptions, the
irreversibility of some actions
23
LPSA solutions for the problems with the classic
'problem space approach

• The classic problem solving theory used mainly
  verbal protocols as data. However, TALK ALOUD
  INTERFERES PERFORMANCE IN COMPLEX DYNAMIC TASKS
  (Dickson, McLennan Omodei, 2000)
• Independence (or very short-term dependences) of
  actions/states is assumed in some of the methods
  for representing performance. That is, the
  features that represent performance are local

LPSA uses log files and human judgments as data,
but not concurrent verbal protocols
LPSA does not assume independence or short
dependences between states/actions. Indeed, it
uses the dependences of all of them
simultaneously to derive the problem space. The
features that represent performance are global
24
Theoretical surroundings of Latent Problem
Solving Analysis
25
(No Transcript)
26

1. Encoding processes
2. Processes of internal transformation
3. Decoding processes

27
LPSA applied to model human judgments
28
Actions Move_4_Copter_11_4_11_9_Forest_
1 Move_4_Copter_11_4_11_9_Forest_ 2
Move_2_Truck_4_11_17_7_Clearing_ 3
Drop_Water_4_Copter_11_9_Forest___ 4
Move_3_Copter_8_6_10_11_Forest_ 5
Move_1_Truck_4_14_18_10_Forest_ 6
Drop_Water_3_Copter_10_11_Forest___ 7
Move_4_Copter_11_9_21_8_Dam_ 8 Move_3_Copter_10_11
_12_14_Dam_ 9 Control_Fire_2_Truck_17_7_Clearing__
_ 10 Control_Fire_1_Truck_18_10_Forest___ 11
Move_4_Copter_21_8_12_10_Clearing_ ( . . . )
Participants trials
29
Firechief corpus

• Data from the experiments described in
  experiments 1 and 2 in Quesada et al. (2000), and
  Canas et al. (2003).
• Total 3441 trials, 75.575 different actions
• The first 300 dimensions where used

30
Trial 1
Trial 2
Trial 3
log files containing series of actions
Action 1
Action 2
57000 actions 3400 log files
actions
31
Human Judgment correlation

• if LPSA captures similarity between complex
  problem solving performances in a meaningful way,
  any person with experience on the task could be
  used as a validation
• To test our assertions about LPSA, we recruited
  15 persons and exposed them to the same amount of
  practice as our experimental participants, so
  they could learn the constraints of the task.

32
Human Judgment correlation

• Replay trials, with different similarities
• People watched a randomly ordered series of
  trials, in a different order for each
  participant, which were selected as a function of
  the LPSA cosines

33
Human Judgment correlation
FULL-SCREEN REPLAY OF THE TRIAL SELECTED, 8 TIMES
FASTER THAN NORMAL SPEED
34
Human Judgment correlation Results
Correlation .948
Human Judgment
LPSA
35
Human Judgment correlation Discussion

• Applied LSA is an automatic way of generating a
  problem space and compare slices of performance
  in complex tasks. It scales up very well and does
  not depend on a-priori task analyses
• Theoretical LSA proposes that problem spaces are
  metric spaces that are derived from experience.
  Actions or States that are functionally related
  are represented in similar regions of the space.
  In this sense, problem solving is unified with
  theories of object recognition and semantics.

36
LPSA as a theory of expertise in problem solving
37

• Ebbinghaus approach manipulating previous
  knowledge by eliminating it. Random assignment of
  participants to groups.
• Chase and Simon approach (expert novice),
  manipulating previous knowledge by pre -
  selecting participants (no random assignment of
  participants to groups)
• Move complexity to the lab, and manipulate
  previous knowledge (exactly amount of practice
  and experience for all participants)

38

• Ebbinghaus approach manipulating previous
  knowledge by eliminating it. Random assignment of
  participants to groups.
• Chase and Simon approach (expert novice),
  manipulating previous knowledge by pre -
  selecting participants (no random assignment of
  participants to groups)
• Move complexity to the lab, and manipulate
  previous knowledge (exactly amount of practice
  and experience for all participants)

39
Move complexity to the lab

• To simulate expertise environments in labs, we
  need tasks more complex than the standard ones
• More representative
• Long learning curve
• Interesting enough to keep the motivation for a
  long period of time

40
(No Transcript)
41
DURESS

• Christoffersen Hunter, Vicente (1996, 1997,
  1998) 6-month long longitudinal experiment using
  Duress II. 225 trials, with different goals
  values. Every participant received exactly the
  same kind of trials.
• However, analysis mostly qualitative. Not without
  a good reason

42
Example DURESS protocol
34 variables, governed by mass and energy
conservation laws
43
Trial 1
Trial 2
Trial 3
log files containing series of States
State 1
State 2
57000 States 1151 log files
States
44
Current theories of expertise

• Constraint Attunement Hypothesis (CAH)
• Vicente and Wang (1998)

• Long Term Working Memory (LTWM)
• Ericsson and Kintsch (1995)

• EPAM IV
• (e.g., Gobet, Richman, Staszewski and Simon,
  1997)

45
Current theories of expertise

• Constraint Attunement Hypothesis (CAH)
• Vicente and Wang (1998)

• Long Term Working Memory (LTWM)
• Ericsson and Kintsch (1995)

• EPAM IV
• (e.g., Gobet, Richman, Staszewski and Simon,
  1997)

PRODUCT THEORY
PROCESS THEORIES
46
LTWM (Ericsson and Kintsch, 1995)

• STM accounts for working memory in unfamiliar
  activities but does not appear to provide
  sufficient storage capacity for working memory in
  skilled complex activities (p.220)
• LTWM is acquired in particular domains to meet
  specific demands imposed by a given activity on
  storage and retrieval. LTWM is task specific.

47
LTWM (Ericsson and Kintsch, 1995)

• Intense practice in a domain creates retrieval
  structures associations between the current
  context and some parts of LTM that can be
  retrieved almost immediately without effort
  (example SF and digits).
• LTWM permits rapid and reliable reinstantiation
  of a context after interruption without a
  decrease in performance.

48
CAH (Vicente and Wang, 1998)

• Contrary to what process theories maintain,
  Constrain Attunement Hypothesis (CAH) does not
  commit to a particular psychological mechanism to
  explain the phenomenon of expertise.
• How should one represent the constrains that the
  environment (i.e., the problem domain) places on
  expertise?
• Under what conditions will there be an expertise
  advantage?
• What factors determine how large the advantage
  can be?

49
CAH (Vicente and Wang, 1998)

• Describing the constraints in the environment is
  the task of an expertise theory.

50
CAH (Vicente and Wang, 1998) the Abstraction
Hierarchy
Overall system goals (how much water each
reservoir is outputting, and at which temperature)
FUNCTIONAL
'D1','D2','T1','T2'
conservation of mass and energy for each
reservoir (how much mass energy is entering and
leaving the reservoir).
'MI1', 'MO1', 'EI1', 'EO1', 'M1', 'E1',
ABSTRACT
'FA','FA1','FA2','HTR1
GENERALIZED
Flows and storage of heat
PHYSICAL
Settings of valves, pumps, and heaters
'PA','PB','VA','VA1','VA2,
Continuum of abstraction, means- ends
relationship between levels
51
LTWM vs. CAH

• LTWM claims that the magnitude of expertise
  effects is related to the level of attained
  skill and to the amount of relevant prior
  experience
• CAH argues that this claim is incomplete.
  Expertise effects in memory recall are also
  determined by the amount of structure in the
  domain (and by active attunement to that
  structure)
• LPSA is sensible both to relevant previous
  practice and to amount of structure in the
  domain

52
3/4
1/4
?
53
3/4
1/4
?
54
(No Transcript)
55
Predictions

• Only huge amounts of experience with the system
  would enable the actor (human or model) to make
  accurate predictions of the last quarter of the
  trial
• Sparse practice should clearly lead to poor
  prediction
• Only structured environments should show the
  expertise advantage. Following CAH, the expert
  (human or model) should not do well in a
  completely unstructured environment

56
Expertise results Three years of experience with
DURESS
Average cosine between the fourth quarter of a
target trial and the fourth quarter of the 10
nearest Neighbors When the three first quarters
are used to retrieve the neighbors
57
Expertise results Six months of experience with
DURESS
Average cosine between the fourth quarter of a
target trial and the fourth quarter of the 10
nearest Neighbors When the three first quarters
are used to retrieve the neighbors
58
Expertise results Three year of experience in a
DURESS with no constraints (random states)
Average cosine between the fourth quarter of a
target trial and the fourth quarter of the 10
nearest Neighbors When the three first quarters
are used to retrieve the neighbors
59
Expertise results Discussion

• LPSA can explain both LTWM and CAH main
  assertions
• LTWM claims that the magnitude of expertise
  effects is related to the level of attained skill
  and to the amount of relevant prior experience
• CAH claims that expertise effects in memory
  recall are also determined by the amount of
  structure in the domain (and by active attunement
  to that structure)
• Better yet, LPSA proposes both processes and
  representational structures

60
Automatic Landing Technique Assessment using
Latent Problem Solving Analysis (LPSA)
61
The problem

• There is currently no methodology to
  automatically assess landing technique in a
  commercial aircraft or a flying simulator.
  Instructors are a significant cost for training
  and evaluation of pilots, and the use of
  instructors also incorporates a subjective
  component that may vary from pilot to pilot.
• The advantages of automatic landing technique
  evaluation are many
• Reduced cost of the evaluation.
• Increased objectivity in the evaluation.
• Decrease the influence of the instructor.
• Perfect Test-retest reliability.
• It is always available and can be triggered by
  the trainee at will.
• The model can rate as many landings as time
  enables, etc.

62
A solution Latent Problem Solving Analysis (LPSA)

• In this application of LPSA to landing technique
  evaluation, we assume that an expert uses her
  past knowledge to emit landing ratings by
  comparing the current situation to the past ones,
  and generates an expanded representation of the
  environment by composing the past situations that
  are most similar to the current one.

63
(No Transcript)
64
Complex, dynamic tasks are intractable when
considered as a whole
65
Complex, dynamic tasks are intractable when
considered as a whole

• We need to perform complexity reduction, in a
  mostly automatic way
• The triangulation technique
• Dimensionality reduction (LPSA)

66
The triangulation technique
67
Complexity reduction (I) variable selection
using differently informed experts
Vertical acceleration
Two experts graded the same landings, with
different information the reduced information
expert selected a set of variables and plotted
them in a computer screen. The complete
information expert sat in the copilot seat, and
has access to all the variables, exactly as the
pilot. We trained the model with only the
variables selected by the reduced information
expert.
Radio altitude
Thrust
The reduced information expert plotted the
variables that he believed were sufficient to
rate the landing
The complete information expert had access to all
possible variables (visual, proprioceptive, etc)
68
Complexity reduction (II) Using SVD, the
problem space is a vector space
69
Using the LPSA problem space any known landing is
represented as a vector. We can approximate
humans ratings by retrieving from memory the
nearest neighbors of the vector formed for any
new landing, and averaging the neighbors
ratings.
Model selection

1. Number of dimensions (100, 150, 200, 250, 300,
   350, and maximum dimensionality, 400)
2. and the number of nearest neighbors used to
   estimate the landing ratings (from 1 to 10).
3. The model with the best fit used 200 dimensions
   and 5 nearest neighbors.

70
Results
71
Results no-constraints corpus
72
Objectives of the dissertation
Conclusions

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

73
Conclusions

• LPSA reduces to a minimum the task/specific, a
  priori assumptions
• Generalizability
• Wide variety of systems
• systems are described in terms of nominal
  (discrete) or continuous variables
• Actions or states as units

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

74
Conclusions

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

The problem space hypothesis all intelligent
behavior takes place in a problem space (Newell,
1980)
Problem space
But the question of where the problem space came
from in the first place remains unanswered. The
generation of the problem space is considered
intelligent behavior and thus, takes place as
well in a problem space!
75
Conclusions

• Integration between molecular and molar levels
• Explains both the amount of experience and
  amount of environmental structure effects,
  characteristic of LTWM and CAH simultaneously
• Explains both processes and representations.
  Happy marriage of process and product theories of
  expertise
• Well-specified. In LTWMs original formulation
  the retrieval structures were under-specified. In
  LPSA, the basic mechanisms postulated are defined
  computationally. In CAHs original formulation,
  the representation of the environmental
  constraints (its most central assertion) where
  under-specified too. LPSA proposes an automatic
  mechanism to represent the statistical
  regularities of the environment

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

76
Conclusions

• GENERALITY the fact that the same mechanism,
  with the very same underlying assumptions, can be
  used for language and Problem Solving is
  interesting per-se In LTWM, the retrieval
  structures for chess are different compared to
  the ones proposed for text comprehension In CAH,
  two AH for two different tasks are different too
  In LPSA, any space for any task is a vector
  space.

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

77
Conclusions

• LPSA is not limited to laboratory tasks
• Complex tasks can be explained without recurring
  to constructs such as problem solving, mental
  models or reasoning
• The triangulation technique
• Practical advantages
• Reduced cost of the evaluation.
• Increased objectivity in the evaluation.
• Decrease the influence of the instructor.
• Perfect Test-retest reliability.
• It is always available and can be triggered by
  the trainee at will.

• Solve methodological problems on microworld
  performance assessment
• Propose an alternative theory of problem solving
  and representation
• Present LPSA as a theory of expertise
• Develop a landing technique automatic assessment
  system

78
-END-

• Acknowledgments
• Tom Landauer
• Kim Vicente
• John Hajdukiewicz
• Anders Ericsson
• Simon Dennis
• Alex Rutten
• Adri Marksman
• Nancy Mann
• Yumiko Abe
• Melanie Haupt

• Funding
• Grant EIA 0121201 from the National Science
  Foundation
• European Community - Access to Research
  Infrastructure action of the Improving Human
  Potential Program under contract number
  HPRI-CT-1999-00105 with the National Aerospace
  Laboratory, NLR

79
(No Transcript)
80
Three examples of performance

• 8 first actions in a trial

2
1
RELATED
NON RELATED
3
81
1
0
CONTROL FIRE
1
2
3
4
5
6
7
8
9
10
11
CONTROL FIRE
12
13
14
15
1
2
3
4
5
6
7
8
0
10
11
9
12
13
14
15
16
17
18
19
20
21
22
23
24
82
2
0
1
2
3
4
5
6
7
8
DROP WATER
9
10
11
CONTROL FIRE
12
CONTROL FIRE
13
14
15
1
2
3
4
5
6
7
8
0
10
11
9
12
13
14
15
16
17
18
19
20
21
22
23
24
83
3
CONTROL FIRE
0
1
2
CONTROL FIRE
3
4
5
6
7
8
9
10
11
12
13
14
15
1
2
3
4
5
6
7
8
0
10
11
9
12
13
14
15
16
17
18
19
20
21
22
23
24
84
(No Transcript)
85
Possible way of comparison Exact matching of
actions

• Exact matching count the number of common
  actions in two files. The higher this number, the
  more similar they are

86
(No Transcript)
87
(No Transcript)
88
Possible way of comparison Transitions between
actions

• count the number transitions between actions in
  two files. Create matrices, and correlate them

89
(No Transcript)
90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
Possible way of comparison Transitions between
actions
94
(No Transcript)
95
(No Transcript)