Measuring Team Shared Understanding: Using Analysis-Constructed Shared Mental Model Methodology

1
Measuring Team Shared Understanding Using
Analysis-Constructed Shared Mental Model
Methodology

• Tristan E. Johnson
• Learning Systems Institute, Florida State
  University, Tallahassee, FL, USA
• tjohnson_at_lsi.fsu.edu

International Workshop and Mini-conference
on Extending Cognitive Load Theory and
Instructional Design to the Development of Expert
Performance August 29-30, 2005 Open University
of the Netherlands
2
Background

• Team Performance
• Team Cognition
• Link between SMM and Team Performance
• Shared Understanding and Shared Mental Models

Development of SMM and its relation to team
performance
3
Team Cognition
Elaborated view of team cognition including team
interactions and SMM development
4
Shared Knowledge Types

1. Task Knowledgedomain specific
2. Team Knowledge5 factors

5
Team Knowledge Factors

• Team Knowledge
• Knowledge about team members and tasks that they
  need to perform
• Teammates knowledge, Task knowledge
• Team Skills
• Abilities associated with successful job
  performance
• Communication skills, Interpersonal skills,
  Leadership skills, Skills to deal with conflict
  and team cohesion
• Team Attitudes
• Internal state that influences team members
  choices or decision to act in a certain way under
  particular circumstances
• Shared belief, Shared value
• Team Dynamics
• Combination of dynamic processes of team
  coordination and team cohesion
• Team coordination, Team cohesion
• Team Environment
• External conditions affecting the foundation of
  the team mental model
• Technology, Organization, Synchrony Geographic
  dispersion

6
Measuring Task Knowledge

• Measuring Shared Understandingmeasuring concept
  relatedness
• Card sorting, cognitive interviewing, MDS,
  Pathfinder, surveys, casual maps (Langan-Fox,
  Code, Langfield-Smith, 2000 Trochim, 1989)
• Concept Mapping
• Statistical analysis
• Descriptive analysis
• Analysis Constructed - Shared Mental Model
  (AC-SMM)

7
SMM Elicitation Techniques

1. TmC-SMMWhole team elicitation (1 map)
2. AC-SMMIndividual elicitation with aggregation (n
   maps)

SMM i desired shared mental model state TmC-SMM
involves team negotiation and interaction SMM?
altered team shared mental model state AC-SMM
retains the initial ICMM state
8
AC-SMM Methodology Rationale

• Knowledge Elicitation
• Process allows simultaneous consideration of
  concepts
• Reflection and changes during elicitation
• Analysis
• Allows for explication of implicit
  relationshipsconsidering 1) logic, 2) structure,
  and 3) spatial orientation
• Relatedness
• Specific to three levels
• Concepts
• Links
• Clusters
• Appropriate for studying shared understanding in
  applied settings

9
AC-SMM Methodology Overview

• Instrument Design
• Structured/Semi-Structured/Unstructured
• Task Analysis (Generate Concepts)
• Data Collection
• Guided Practice
• Individually Constructed Mental Model (ICMM)
  Elicitation
• Data Analysis
• Phase I ICMM Analysis/Coding
• Relatedness at concepts, links, clusters levels
• Allows for explication of implicit relationships
• Implicit coding has logic and spatial or logic
  and structural support
• Phase II Shared Analysis
• Determine sharedness levelnumber or percentage
  of team members
• Phase III AC-SMM Construction
• Generates SMM

10
Phase I ICMM Analysis

• Factor 1 Concepts
• Explicit individual nodes
• Factor 2 Links
• Two concepts joined explicitly connector or
  implicitly
• Factor 3 Clusters
• Two or more connectors explicitly bridging three
  or more concepts
• May have implicit connections with evidence
• Combination of clustersSub- and Super- clusters
• Factor 4 Emphasis and Sequence
• Explicit notation of node emphasis or node order

11
ICMM CodingLinks
12
ICMM CodingClusters
13
ICMM CodingEmphasis and Sequence
14
ICMM Coding Example
15
Phase II Shared Analysis

1. Determine Sharedness Level CriterionNumber or
   Percentage
2. Shared Data Used for AC-SMM Construction

16
Phase III AC-SMM Construction
17
Research

• General Research Focus
• What task knowledge is shared?
• How does shared understanding change over time?
• What are the patterns of change?
• What is the affect of task performance on the
  shared understanding of the team?

18
Data Collection Timeline
19
Concepts
20
Team Profiles Findings
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
Shared Data Findings, Team 1 Only
26
(No Transcript)
27
(No Transcript)
28
Shared Data Summary Per Team
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
Cross Case Findings
38
(No Transcript)
39
(No Transcript)
40
Pre, Mid, Post Analysis
41
ACSMM Scores
42
General Findings

1. Similarity among ICMMs tends to increase as does
   the number of clustered concepts, the tendency is
   for the number of concepts used to decrease.
2. ICMMs were becoming more structured and more
   representative of the team task
3. These ideas are not yet proven.
4. We have designed a set of studies to try and
   validate our hypothesis
5. This work is intended to not only learn about
   teams that work in the various settings, but to
   validate the AC-SMM analysis model

43
Summary

1. Richer description of shared understanding in
   teams
2. AC-SMMs compared over time to determine change in
   shared understanding
3. Lacks weighted measures and precise distances
   between concepts, but future work will include
   descriptive statistics of the key factors
4. Lack of prepositional descriptors
5. As we become more precise and descriptive we can
   utilize this new knowledge to better explain and
   understand team cognition
6. Facilitate team training with intent to improve
   team performance outcomes

44
Thanks for your attention.Questions?
45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
Findings and Extra Slides
57
Findings Across Pre, Mid, Post
58
Participants Context

• Participants
• Personnel Qualification Standards (PQS) Team
• Team Task
• Team Members
• Team Member Roles
• Context
• U.S. Navy Training Center, Pensacola, FL
• Face to face workshop
• Equipment

59
Findings From AC-SMM Analysis

• What is shared?
• Does shared understanding change over time?

60
Secondary Analysis

• Sequence
• Where concepts were placed within each ICMM
• Focus on key concepts
• Team member roles
• Sections of PQS book
• Referencing
• Questions
• Links
• Relationships between concepts without
  directionality
• Clusters
• Relationships between concepts without
  directionality
• Commonalities between related clusters of concepts

61
Secondary Analysis

• Links - Relationships between concepts without
  directionality
• Clusters - Relationships between concepts without
  directionality and commonalities between related
  clusters of concepts

Example of Cluster B,D, B,E, B,I with
Secondary Clusters B,D,E,I and D,E,I without
Related Links D,E, D,I, and E,I
62
All Data, Shared by 2
63
All Data, Shared by 50
64
Follow-up Analysis

• Sequence
• Started with original data submitted by each team
  member
• Where concepts were placed within each ICMM
• Focus on key concepts
• Team member roles (concepts D, E, I)
• Sections of PQS book (concepts J, K, L)
• Referencing (concepts H, O, S)
• Questions (concepts N, Q, T)

65
All Data, Shared by 2Secondary Analysis
66
All Data, Shared by 50Secondary Analysis
67
Complete Datasets, Shared by 50
68
Complete Datasets, Shared by 2Secondary
Analysis
69
Complete Datasets, Shared by 50Secondary
Analysis
70
(No Transcript)