Dimensions of Cognitive Research in Health Informatics: Recent Advances and Promissory Notes

1
Dimensions of Cognitive Research in Health
Informatics Recent Advances and Promissory Notes
David R. Kaufman Cognition and Development,
Graduate School of Education, UC
Berkeley davek_at_socrates.berkeley.edu October 25,
1999
2
Outline

• Issues and Theoretical Assumptions
• Computer-Based Patient Record Systems
• Usability and Skill Acquisition
• Computer-Mediated Collaborative Design
• Grounding and Communication
• Advances and Promissory Notes

3
Framing Assumptions

• Emerging Systems Technologies will Have a
  Profound Effect on all Facets of Health Care
• Great Promise for Achieving Significantly Better
  Outcomes
• Affording New Possibilities for Patient Care
• Profound Change in the Medical Workplace
• Impressive Technological Solutions Often do not
  Result in Improved Health Care Delivery
• Meet with Widespread Resistance and Frustration
• Yield Suboptimal Outcomes
• Gulf Between Technological Artifacts and End-Users

4
More Framing Assumptions

• Health Informatics is more than the Thin
  Intersection of Allied Health Disciplines and
  Computing
• Cognitive Science Methods and Theories
• Provide a Set of Valuable Tools
• Corpus of Research in Medical Cognition
• Informatics is a Fertile Domain to Test, Refine,
  and Extend Cognitive Theories
• Health Informatics is a Local Science of Design

5
Medical Cognition

• Studies of Cognitive Processes Perception,
  Comprehension, Reasoning, Decision Making, and
  Problem Solving
• Medical Practice or Representative Experimental
  Tasks
• Theories and Methods from Cognitive Science.
• Intersecting Disciplines Medical Education,
  Medical Informatics, Artificial Intelligence in
  Medicine, Epidemiology, Public Health, and
  Humanities in Medicine

Theoretical Basic Science
Applied Research
6
Local Science of Design

• Seeks to Explain Aspects of a Domain
• Rather than to Derive a set of Unifying
  Principles
• Landscape of Phenomena is Too Complex and Diverse
  to be Reduced to a Single set of Universal
  Principles.
• Reflect Current State of Progress OR
• Inherent Attribute of Discipline
• Local Sciences
• Generate Principles with a Specific Range of
  Applicability,
• Focus Attention on Issues and Details of
  Importance
• Allow Us to Derive Meaningful Predictions.
• Pluralistic Methods
• Case Studies and Experiments have Similar
  Privileged Status

7
Levels of Analysis
System
Individual/System
Collaborative Use of Technology Multiple
Individuals
Integrated Clinical Care System Patients,
Physicians, Nurses, and Other Personnel
Institutions, Communities, Societies, and Cultures
8
Computer-Based Patient Record Systems

• Computer Devices for Inputting, Representing,
  Accessing, and Managing Patient Care Records
• Extensions to Physicians Knowledge Base and
  Decision-Making Capabilities
• Promise to Exert a Profound Effect on Medical
  Practice and Health Care Delivery
• Implementation and Integration in Medical
  Practice has proven to be Difficult
• Obstacles to Achieve Widespread usability
• Cope with the Scale, Detail, and Complexity of
  Information Required for Clinical Medicine
• Nonintuitive and Ill-conceived Interfaces
• Requires High-level Competency to Use in
  Real-time
• Steep Learning Curves

9
Information Technology

• Computer-Based Systems used in the Creation,
  Storage, and Transfer of Information, Knowledge,
  and Services.
• Bibliographic Retrieval Systems, ATM Machines,
  PalmPilots, Hierarchical Menu-Driven Phone
  Answering Machines, and Weight Watcher Calorie
  Counter Calculators.
• Cognitive Artifacts
• Augment Human Memory Systems
• Extend Communication and Exchange of Knowledge
• Enhance Reasoning and Decision-making Processes
• Do Not Merely Augment Cognition, but Transform in
  Significant Ways.

10
High Expectations!
11
Goals of Research Program

• (1) Characterize process of Skill Acquisition in
  Learning to Master CPR Programs and use
  Effectively in Clinical Practice
• (2) Develop and Refine Cognitive Methodologies
  for Assessing Usability
• (3) Develop Measures of Assessment and Benchmarks
  for Competency
• (4) Cognitive Models of Performance in using CPRs
  for Diverse Clinical Tasks in Diverse Settings

12
More Goals

• (5) Characterize Effects Of CPR Systems on
  Reasoning and Decision Making
• (6) Identify Features Of Systems that Enhance
  Usability and Learnability
• (7) Develop Training Methods for Effective Use of
  Medical Information Systems
• (8) Collaborative use of Technology

13
A Case Study of CPR System

• Usability Evaluation
• Complex Skill Acquisition
• Kushniruk, A. W., Kaufman, D.R., Patel, V.L.,
  Lévesque, Y., Lottin, P. (1996) Assessment of a
  Computerized Patient Record System A Cognitive
  Approach to Evaluating an Emerging Medical
  Technology. M.D. Computing, 13, 406-415
• Kaufman, D.R., Kushniruk, A.W., Patel, V.L.
  (1999) A Cognitive Analysis Of Computer-based
  Patient Record Systems Effects Of Technology On
  Medical Reasoning. Proceedings of the Third
  Ergocon Conference.

14
Cognitive Approach toUsability and Performance
Assessment

• Convergent Approaches
• Usability Inspection
• Cognitive Walkthrough/Scenario-Based
• Usability Testing
• Video Analysis/Think-Aloud Protocols
• Users at Varying Levels of Expertise
• In-Depth Cognitive Task Analysis

15
Cognitive Walkthrough

• Methodology for Assessing Usability of a System
• (Polson, Lewis, Reiman, Wharton 1992 94)
• Criteria Focuses on Cognitive Processes Needed to
  Perform Task
• Identifying Sequences of Actions and Subgoals to
  Successfully Complete a Task.
• Assigning Causes to Usability Problems.
• Characterize Sources for Possible Goal-Action
  Mismatches.

16
Walkthrough Process

• Hand Simulation of Users Cognitive Processes for
  Successfully Executing an Action Sequence to
  Complete a Task
• Overall Objectives
• Can a user with a certain degree of Knowledge
  perform the Tasks that the System is intended to
  Support?
• Can a User Learn to Perform what is Unknown?

17
Walkthrough Procedure

• Step through Each Action and Specify
• Goal Structure for Each Step
• Behavior of the Interface and its Effect on the
  User
• Actions that Could be Difficult to Execute
• Source of Potential Problems,

18
Illustration of a Walkthrough

• Multimedia Program on Compact Disk Interactive
  (CD-I) Media
• Tutorial on Coronary Heart Disease and
  Hypercholesterolemia
• For General Practitioners and Family Physicians.
• The tutorial consists of 4 modules
• A risk analysis module which enables the user to
  learn about different risk factors
• Diagnostic criteria
• Differential diagnoses
• Lipids tutorial containing detailed biochemical
  explanations.
• Hypermedia tutorial makes use of the full range
  of sound, text, graphics, and full motion
  capabilities

19
16 Possible User Actions

• 1. Start Program
• 2. Get Help
• 3. Change Settings (i.e., From English to French,
  units of measure).
• 4. Exit Program
• 5. Select Explanation
• 6. Change the Value of a Variable.
• Increase/Decrease Value
• Set Value from ON to OFF
• 7. Observe Effect on Risk For MI
• 8. Go to another module
• 9. Return to previous screen/module.

20
Results of Walkthrough Analysis

• The Tutorial Allowed For 16 Possible Actions
• The Goal to Change a Single Program Setting
  Necessitated
• 7 Subgoals
• 11 Actions
• 6 Transitions between Screens
• Inefficient Far too Many Actions to enact a
  simple change.
• Too many transitions-likely to Confuse and
  Frustrate Users.

21
DCI CPR System

• Developpement Purkinje Inc.
• Fully integrated Patient Management System.
• Operates on a Portable or Desktop computer
• Microsoft Windows operating system
• Record information
• Patients History, Physical Examination, and
  Laboratory Results.
• Patients differential diagnosis
• The ordering of tests
• The prescription of medication
• Provides Supporting Reference Information
• Detect Drug Interactions and Provide Explanations
  on Interactions.

22
Main CPR Display
23
Using DCI System

• Physician first selects a Patient record from
  Appointment list.
• To enter the medical findings, Physician then
  Determines the Context of the Visit.
• For example, if the patient presents with a
  specific complaint (e.g., chest pain),
• Physician would select a Clinical Note Template
  (CNT) for chest pain,
• Displays a selection of Medical Findings and
  Observations on the Computer Screen.
• Data is displayed in a scrollable window
• Icons on the top of the screen
• List Commands for Accessing Appointment List
• Selecting a Patient by Name, Accessing Lab
  results...

24
Cognitive Dimensions
Information Gathering
Problem Representation
Diagnostic Reasoning
25
Subset of User Actions

• 1. Select Appointment
• 2. Choose Clinical Note Template
• 3. Go to Section using Button on Left Hand Panel
• 4. Highlight Category/Finding/Attribute
• 5. Scroll (up or down)
• 6. Search use Keyword Match/Completion
• 7. Select Keyword Match
• 8. Select Delete
• 9. Select Filter
• 10. Select Additional Clinical Note Template

26
Excerpt from a Cognitive Walkthrough

• Context Patient Presents with a medical problem
• Task Diagnostic Reasoning
• Goal Structure and Action Sequence
• Goal Represent chief complaint
• Chief Complaint Patient complains that she has
  been feeling pretty tired for the past 6 months
• Subgoal 1 Characterize Patients Observation of
  Tiredness
• Action 1 Open Top-Level Category General
  Condition
• System Response Displays Findings Organized by
  Category
• Subgoal 2 Enter Finding Fatigue
• Subgoal 3 Locate Finding
• Action 2 Open Category/Select Finding
  "Fatigue"
• Subgoal 4 Describe Severity/Quality
• Subgoal 5 Translate quality "Pretty Tired" into
  Quantity indicating Severity
• Action 3 Enter severity Three

27
CPR Walkthrough

• Goal to Represent Chief Complaint of Patient
  complains that she has been feeling pretty tired
  for the past 6 months Required
• 7 Subgoals
• 7 Actions
• Potential Problems
• Initial Unfamiliarity Difficulty Locating a
  Finding.
• Translating Qualitative descriptors Pretty
  Tired Into Quantity (severity of 3 on a Scale of
  5).
• Quality to Quantity is a Common Conceptual
  Problem in Health Informatics.
• User May Neglect (Or may not know how) to Change
  the Units of Measure (years to months).
• CW provides a model of the Cognitive Processes
  involved in such a task.

28
Action Sequence in CPR Walkthrough

• Action 1 Open Top-Level Category General
  Condition
• Action 2 Open Category/Select Finding "Fatigue"
• Action 3 Enter severity Three
• Action 4 Select Quality Lack of energy
• Action 5 Selects chronology duration of 6
  months
• Action 6 Click on Years
• Action 7 Enters value of 6 months

29
Usability Testing

• Objective
• Cognitive Assessment of a CPR System, Examining
• Aspects of the Usability of the CPR
• Learning Involved in Mastering the System
• Use of Strategies by Subjects at Varying Levels
  of Experience
• How System Affects Physicians Clinical Reasoning
• Subjects
• The Study Involved In-depth Testing of Two
  Subjects
• Experienced Subject GP Used System for 6 Months
• Novice Subject GP Had Not Previously Used System

30
Methods

• Novice Subject
• 1. Initially tested without using system
• 2. Two hour training session.
• 3. First experimental session - 3 cases given
  (two text based, one interactive).
• 4. Underwent second experimental session - 3
  cases (same text based and a second interactive
  case).
• Experienced Subject
• Similar procedure (did not undergo training
  session) -both text cases and interactive cases
  used
• During sessions
• Subjects were instructed to think-aloud as they
  used the CPR
• The subjects interaction with the CPR was
  videotaped
• The computer screens were captured

31
Cognitive Analysis of Video-Based Data

• Audiotape transcribed and Transferred to a log
  file for use with CVideo
• CVideo program is used for Semi-Automated
  Analysis and Coding of Data
• Coding scheme Partially based on Cognitive
  Walkthrough
• Goals, actions, and inferences pertaining to
  component processes of information-gathering,
  problem representation and diagnostic reasoning
• User Errors and Problems

32
Measures of User Performance

• Accuracy of Representation -
• Reflected by match between the findings and
  qualifiers in the text and those entered into the
  CPR
• Measures pertaining to Goals and Actions needed
  to complete various sections of the problem
• Measures of time to complete various sections of
  the problem (e.g., Search Time)
• Type and Frequency of User Problems

33
Cognitive Dimensions
Information Gathering
Problem Representation
Diagnostic Reasoning
34
Narrative of Data Gathering Process Experienced
User/Interactive
35
Excerpt from Interactive Condition

• In the following excerpt, the same subject
  attempts to indicate that the patient is thin
• 001729 GOAL Enter Finding
• Im trying to figure out where I put down that
  he is thin and/or oriental
• SUBGOAL Locate Matching Descriptor/Finding
• General condition not good, I couldnt put that
  down
• SUBGOAL Locate Matching Category
• PROBLEM Finding Match
• Looks sick.. It doesnt really say that. It is
  hard, the general impression is things you put
  down for history dont always categorize that
  easily I guess
• 001806 Other, Ill see what they have in
  Other
• ACTION Selects/Opens Category Other
• OK, here we are, that is where it is

36
Excerpt from Text Condition

• 001838 OK, lets go to the eyes, Ill close
  up general condition
• ACTION Selects/Closes Category on Screen
• Thats not bad
• 001904 General inspection, here we are
• ACTION Selects/ Opens Category on the Screen
• GOAL Enter Finding
• Exophthalmos, mild
• 001925 ACTION Selects/Opens Finding on Screen
• GOAL Qualify Finding Severity
• So I guess I will put that for a plus
• ACTION Select/Qualify Finding Severity Level 1
• 001937 Positive lid lag
• GOAL Enter Finding
• Lid lag, here we are
• ACTION Selects/Opens Category on Screen
• ACTION Select/Enter Finding Lid Lag

37
Novice Performance Across 2 Sessions (Text
Condition)
38
Problems and Errors
39
Process of Skill Acquisition

• Component Processes appear to be Additive for
  Novice Users
• Perceptual-Motoric (PM) Problem Representation
  (PR) Information (IG) Gathering Diagnostic
  Reasoning (DR)
• High Cognitive Load/Burden on Working Memory
• IG and DR are Co-extensive for Experienced
  physicians
• System Initially Gets in the Way
• Skilled Users
• Component processes Highly Integrated
• PM effect is minimal/Facilitates
  Structuring/Coordinating Encounter
• PR process appears to sometime drive the
  Doctor-Patient encounter and IG DR Process
• Screen-Driven Strategy
• Changes in Goal-Action Pairings

40
Information Gathering Strategies

• Hypothesis Driven
• Goals and Questions Guided by Diagnostic
  Hypotheses
• Screen Driven
• Information-Gathering and Diagnostic Reasoning is
  Guided by Findings Presented on the Screen
• CPR is implicitly or explicitly Structuring the
  Interview
• Problem Representation process is prioritized
• Advantages
• Thorough Investigations
• Alleviates Burden on Working Memory allowing
  physician to focus on diagnostic reasoning
  process
• Disadvantages
• Can Induce Complacency
• Lead Physician down Garden-Paths

41
Summary of Results

• Results
• Across Sessions, Subjects Developed more accurate
  representation
• Experienced User an initial advantage in Accuracy
  of Representation over the Novice
• The novice subject substantially improved the
  accuracy of the problem representation
• 2 Primary Information Gathering Strategies
• Hypothesis-Driven Strategy Requests for
  Information were Guided by Physicians Hypothesis
  Independent of the Screen Displays.
• Screen-Driven Guided in their information-gatheri
  ng by the ordered sequence of information on the
  computer screen
• Novice Shift from Hypothesis-Driven strategies
  to Screen-Driven .

42
Conclusions

• Learning to use Information technologies engages
  multiple component processes
• Somewhat Distinct Goal-action Hierarchies
• Component Processes appear to have an additive
  effect on working memory in early stages
• Technologies can have a Profound Transforming
  Effect on Task performance
• Need to understand the productive and
  counterproductive impact
• Current and Future Directions
• Analyze other Information Technologies
• Real-World Settings (i.e., Hospital clinics)
• Wider Range of Tasks (Therapy and Management)
• Shared Information Systems

43
Advances

• Theoretical and Methodological Tools
• Progress in Characterizing Dimensions of
  Usability/Learnability
• Effects With Technology
• Changes in Performance While Using a System
• Effects of Technology
• Enduring Cognitive Changes even in Absence of
  System.
• Positive
• More Structured Approach to Data Collection
• Negative
• Rigidity or Deskilling

44
Promissory Notes

• Characterize and Prescribe Effective Use of
  Technology as Embedded in the Ebb and Flow of
  Daily Work.
• Information Technologies as Used by Diverse
  Personnel/Sharing of Information.
• Effective Training
• Many Many other Challenges for Cognitive Research
  in Health Informatics

45
What was left Unsaid

• Speaker A Excuse me, I need to make a phone
  call.
• Im looking for a phone,
• Preferably some place close by
• Can you help me?
• Speaker B Theres a gas station just around the
  corner.
• I Understand your request
• Im willing to help
• The gas station has a pay phone.
• I expect that you will understand my intentions
• Speaker A Thats great! Thank you very much.
• I understand your communication
• Im acknowledging your help and terminating the
  conversation

46
The Alternate Conversation

• Speaker A Excuse me, I need to make a phone
  call.
• Speaker B Thats just fine. Please keep me
  informed about your other needs as they arise.
• ..
• Speaker A Excuse me, I need to make a phone
  call.
• Speaker B Theres a gas station just around the
  corner.
• Speaker A No No! Why are you telling me this?
  Im not looking for gas. I dont even own a car.
  Do you know where I might find a pay phone?
• Speaker B Yes, I do know where to find a pay
  phone. Thanks for asking.

47
Grounding

• Grounding Development of a shared understanding
  of the goals of the exchange and the content of
  the communication
• The act of conveying a message
• An indication that the message has been
  understood.
• Casual conversation between acquaintances has
  different criteria for efficient communication as
  compared to interactions among scientists working
  together on a research project.
• Different media have different constraints

48

Sociometric Analysis of Email Communication
Patterns
Stanford
Columbia
InterMed Central
DSG
MGH
49
Email Communication Patterns Episode 1
Vocabulary Server and Guidelines (Jan-Feb 95)
Stanford
Columbia
MGH
50
Email Communication Patterns Episode 2
Guidelines Representation (Feb-Mar 96)
Stanford
Columbia
MGH
DSG
51
Exemplars of Research in Medical Cognition

• Differences between students and physicians at
  varying levels of expertise in clinical reasoning
  tasks
• Diagnosis
• Therapy
• Patient Monitoring and Management
• Diagnostic Reasoning in Perceptual Domains
• Comprehension of Medical Texts
• Differences between students in Problem-Based
  Learning and Conventional Medical Curricula
• Cognitive Objectives for Pre-Clinical Curricula
• Cognitive Assessment of Student Performance

52
More Exemplars

• Dynamic Decision Making in Intensive Care
  Medicine
• Effective use of computerized patient record
  Systems
• Process of Learning
• Impact on Clinical Reasoning
• Patient Models of Health and Disease
• Cultural Belief Systems and Health Practices
• Indian mothers beliefs about childhood
  Nutritional disorders
• Role of biomedical knowledge in clinical
  reasoning
• Mental models of the regulation of cardiac output
  and venous return

53
Dimensions of Learning

• Learning by Doing/Skill Acquisition
• Focus on individual learning through practice,
  emphasis on procedural skills
• Conceptual Change
• Meaning Learning in Understanding Concepts
• Apprenticeship Learning
• Guided learning through a process of mentorship
  in a practice setting
• Situated and Collaborative Learning
• Learning through a process of interactive
  discourse and participation in activities
• Case-Based or Exemplar-Based Learning
• Learning through exposure to cases or problems

54
Conclusions

• Information and Other Technologies are not merely
  Tools to Expedite, Facilitate and Enable the
  Execution of Tasks
• The have Profound and Enduring Consequences
• Need to Develop new Kinds of Literacy and
  Competencies.
• Generative Skills that Extend Beyond Training
  with a Particular System on a Given Task.

55
(No Transcript)
56
Research Settings

• Computer-Based Patient Record Systems Purkinje
  DCI system
• Medical Information Systems in Outpatient Clinics
  (Columbia-Presbyterian Health Center)
• Computer-Supported Collaborative Design
  (InterMed)
• Patient Clinical Information Systems - PatCIS
  (Columbia-Presbyterian Health Center)
• Integrated CPR Systems in Clinical Settings
  Metabolic Day Center at Royal Victoria Hospital.

57
Dependent Measures

• Goal A deliberate intent to achieve a specific
  end, as indicated by verbal comments or a pattern
  of actions.
• Action The unit of behavioral activity
  involving a subject's interaction with the CPR.
• Findings Observations used to identify clinical
  manifestations of disease corresponds to the
  basic unit of information in the CPR system.
• Qualifiers Terms used to describe Findings
  (e.g., indicating severity).
• Requests for Information Physician's inquiries
  from the 'patient'.
• Problems Difficulties pertaining to the use of
  the CPR.
• Total Findings Qualifiers A count of the
  findings and qualifiers recorded in the CPR.

58
Dependent Measures

• Matching Findings in Text (FT) The number of
  findings recorded in the CPR that correspond to
  the model representation of the findings in the
  text.
• Percentage Degree of Match The percentage of
  correspondence between the findings and
  qualifiers entered into the CPR by a subject and
  the model representation of the findings in the
  text.
• Percent Search Time Percentage of Time devoted
  to locating findings.
• Relevant Findings Qualifiers obtained /
  Requests for Information () The ratio of
  percent relevant findings qualifiers obtained
  to the total number of requests made by the
  physician
• Actions/ Total Findings Qualifiers The ratio
  of actions to data entries in the CPR.

59
Technology Effects

• Effects With Technology
• Changes in Performance While Using a System
• Effects of Technology
• Enduring Cognitive Changes even in Absence of
  System.
• Positive
• More Structured Approach to Data Collection
• Negative
• Rigidity or Deskilling

60
Basic Functionality of CPR Systems

• Integrated View of Patient Data
• Access to Various Data Sources across Time
• Access to Knowledge Resources
• Clinical Guidelines, Administrative Knowledge
• Physician Order Entry and Clinician Data Entry.
• Lab test, Prescriptions
• Integrated Communications Support
• Link to Shared Patient record Facilitates Overall
  Coordination of Care.
• Clinical Decision Support
• Drug Interaction Checking, Laboratory Alerts

61
The Conversation

• Context A Meeting between 2 Strangers on a
  Street.
• Speaker A Excuse me, I need to make a phone
  call.
• Speaker B Oh, Theres a gas station just around
  the corner.
• Speaker A Thats great! Thank you very much.

62
Task Conditions

• Two Different Conditions
• Text Condition
• Subject Presented With Written Case Descriptions
• Clinical History, Physical Examination, and
  Laboratory Investigation)
• Subjects Asked to Enter the Information Into the
  CPR As Accurately As Possible
• Interactive Condition
• Subject Conducts a Clinical Interview With an
  Experimenter Who Plays the Patient.
• Subjects Are Asked to Use the CPR to Record the
  Information As They Conduct the Interview