Title: Critical Thinking in Medical Education: Assessing What We Mean and What We Know
1Critical Thinking in Medical EducationAssessing
What We Mean and What We Know
- Ed Krupat, PhD
- Director, Center for Evaluation
- Medical Education Grand Rounds
- Dec. 5, 2008
2Components of Talks that I Have Liked
- 1. A bit of context or framing
- 2. A touch of theoretical/conceptual background
- 3. A good portion of research and data
- 4. A bounty of concern for practical application
- 5. A strong dose of provocative discussion
3Context The Origins of These Efforts
- Derek Bok--to--Richard Hersh
- Evaluation of programs
- Confusion-to-clarity-to-confusion-to-???
- Why is critical thinking so important
- In life
- In medicine
- Desire to be data-driven
4Conceptual Analysis Just What Is Critical
Thinking??
- Overlapping concepts
- Analytic reasoning
- Problem solving
- Decision making
- Clinical/diagnostic reasoning/judgement
- Habits of mind
- Meta-cognition
- Adaptive expertise
5Some DefinitionsCritical thinking is
- ...the intellectually disciplined process of
actively and skillfully conceptualizing,
applying, analyzing, synthesizing, and/or
evaluating information gathered from, or
generated by, observation, experience,
reflection, reasoning, or communication as a
guide to belief and action. (Scriven, 1996) - the art of thinking about your thinking while
you are thinking in order to make your thinking
better more clear, more accurate, and more
defensible. (Paul et al, 1989)
6Blooms Taxonomy
7Research Data
- Two projects being conducted simultaneously
- 1.Qualitative Do physicians agree about just
what critical thinking is? - 2.Quantitative Can we assess critical thinking
among medical students? -
8What is Critical Thinking The Responses of
Physicians
- Survey of practicing MD faculty at 5 medical
schools - HMS, UCSF, Case Western, U Mass, Baylor (total
n73) - Convenience sample
9Two Tasks
- Define critical thinking
- Think of a clinical scenario in which critical
thinking was important - describe it
- state what a good critical thinker would do or
say in that situation - state what a poor critical thinker would do or
say - state how the outcome would differ if one or the
other would have been the physician involved
10Coding of the Definitions. 1
- Definition specifically included
- Collection of information/data 27
- Making sense of information 96
- Utilization for decision making 74
- Utilization for action 14
- Specific ties to medical context 43
- Necessity of building upon knowledge base 27
11Coding of the Definitions. 2
- Critical thinking characterized as
- A process
- A skill or ability
- A disposition
12Critical Thinking as a Process
- a process of reflective reasoning that uses
objective evidence, a deliberate weighing of
options and alternatives, and clinical judgement
and experience to guide decision making. - the process by which one is able to rationally
acknowledge different choices, processes and
outcomes in the clinical encounter. - a process in which problems are being analyzed
from different angles and connected to
pre-existing knowledge before any conclusions are
being drawn - 44 of all responses
13Critical Thinking as a Skill or Ability
- the ability to think through a problem using
reasoning. Also the ability to judge the
credibility of sources. - the ability to rigorously weigh the validity of
evidence and then to effectively synthesize this
evidence to reach a clinical decision. - the ability to effectively problem solve using
known data or under conditions of uncertainty. - 50 of all responses
14Critical Thinking as a Disposition
- careful attention to what you know, vigilance
for what you do not, and the courage to question
both of the above categories. - thinking about an topic, issue, or challenge in
a way that sets aside my immediate gut
response,so that I can be open and reflective to
other possible ways of viewing the challenge - thinking deeply, keenly, flexibly, openly,
reflectively, with an awareness of self and
others, with attention to what is known and
unknown, and with humility. - 10 of all responses
15Breakdown by Specialty
16Clinical Situations An Interim Report
- Looking for
- What situations, choices, challenges
differentiate critical and non-critical thinkers - What is it that critical thinkers do or say
- What is it that non-critical thinkers do or say
- What outcomes differ when critical thinking is or
is not in evidence - Bottom line
- What are the key differences in the thoughts,
words, and actions of critical thinkers
17Clinical Contexts and Tasks
- Not surprising
- Diagnosis and treatment
- Surprising
- Collaboration and interpersonal issues
- How to protect patients rights and autonomy
- How to deal with difficult patient requests
- How to provide patient with appropriate options
18The Differences
- Critical thinkers
- Do more data gathering
- Avoid premature conclusions
- See inconsistencies in information
- Utilize knowledge more extensively and explicitly
to make decisions - Are aware of limitations and doubts
- Monitor and evaluate their own decisions
- Involve patients more fully
- Provide patients with options
19 A Few Conclusions, Many Questions
- Physicians are not necessarily all speaking the
same language when they discuss critical thinking
with one another - Is critical thinking a skill, a process, or a way
of looking at the world? - Are skill and disposition both necessary?
- Do you teach/encourage/foster skills in the same
way as dispositions??? - Do you assess skills in the same way that you
assess dispositions? - Where does building upon a knowledge base come in?
20How do we assess critical thinking?
- Collegiate Learning Assessment (CLA)
- Used in over 200 colleges and universities
- Assesses higher order outcomes
- Students work on 90 minute performance tasks
- System has been devised to generate reliable
scores - Question Is this a valid measure of critical
thinking for medical students?
21Catfish One of the Performance Tasks Used
- Grotesquely mutated catfish has been found in the
local lake that supplies the towns water - You will serve on the mayors advisory panel
- You are provided with 6 documents to read
- Newspaper article
- Editorial by environmental activist
- Radio interview with a biologist from a nearby
college - State report on water testing from lake
- Area map
- Journal article about similar discoveries
22Task
- Open-ended written questions ask students to
- Identify main hypotheses to explain phenomenon
- Identify strengths and weakness of each
- State and defend most likely explanation
- Suggest course of action
23Scoring
- Evaluation of evidence
- What is relevant, what is valuable
- Analysis and synthesis of evidence
- Connections, inconsistencies, flaws in reasoning
- Conclusion drawing
- Acknowledging alternatives and options
- Presentation of arguments
- Concise, evidence-based, logically structured
24Research Design
- Recruit multiple schools
- Recruit and compare students at two (or more)
points in time - Entering Year 1 students took two tasks
- Students at end of Year 3 took one task
- Look for associations between CLA scores and
other performance indicators - MCAT scores
- Gender
- Age
- English as first language
- Step scores
- Compare scores between first and third year
students
25Analysis Problems
- Among year 1 students
- scores on the two tasks not as highly correlated
as would be expected - Brain Boost task always given first
- Year 1 students spend consistently less time and
do consistently worse on Catfish (always second
task) than on Brain Boost - Among Year 3 students--who only took one task,
either Brain Boost or Catfish--students take more
time and perform better than on Brain Boost - Small ns
26Findings 1.
- Medical school students have good critical
thinking skills at baseline - Across the 3 schools, year 1s
- fell at the 87th percentile among graduating
seniors nationally - were 1.25 standard deviations above the national
mean - Is that good news or, with selectivity of the
medical schools, should we expect to be even
higher??
27Table 1. CLA performance by task, year, and school
plt.01
28Table 2. Correlations among tests and
demographic variables
Significance codes plt.05, plt.01,
plt.001 Note Significance not adjusted for
multiple significance tests
29Findings to Come
- What is the correlation between CLA scores and
- Step II CK
- Step II CS
- HMS Comprehensive Exam
30What have we learned?
- Critical thinking has many dimensions, and
physicians may be talking past one another if
they dont share meaning when they speak. - Critical thinking can manifest itself in how to
deal with patients, not just in generating
diagnoses and treatment plans. - Critical thinking involves being
- Sensitive
- Inclined
- Able
- (Perkins Ritchhart
2002)
31What have we learned? 2
- Medical students enter school with solid critical
thinking skills - Critical thinking is somewhat, but not greatly,
associated with other measured outcomes - It is still unclear the extent to which medical
education fosters or hinders critical thinking - We have more questions at the end of this process
than we did than when it began, but the questions
are - more sophisticated
- more tied to possible action
32Some Big Questions
- If we think that critical thinking is so
important, why dont we explicitly design the
curriculum to assure that we teach it? - If we wanted to do so, how and when would it be
taught? - Should measures of critical thinking, as a skill
or disposition, be built into our admissions
screening process? - Should critical thinking be a core competency
that is assessed throughout the curriculum? as a
graduation requirement? As a requirement for
licensure?