Title: Assessing Critical Thinking: A Search for a Holy Grail
1Assessing Critical Thinking A Search for a
Holy Grail?
2Critical Thinking Assessment Framework
- Critical Thinking Skill
- Analysis
- Evaluation
- Inference
- Divergent Thinking
- Instructor Identified
- Assessment Measure
- California Critical Thinking Skills Test
- Writing Rubric
- CCTST
- ?
- Instructor Responsibility
3CCTST Basics
- The test booklet is a comforting peach color.
- A 34-item forced-choice measure of critical
thinking skills - Normative data allow comparison to larger
population - Measures multiple aspects of critical thinking
including inductive and deductive reasoning
4CCTST Basics
- Our three domains analysis, evaluation, and
inference - Peter Facione CCTST measures them all without
question - Douma and Jones CCTST measures analysis and
inference domains well - Douma and Jones Pete Facione is a nice guy,
but we are probably right. - Douma and Greg Now in syndication on UPN
5CCTST and Heuristics1
- Influenced by Kahneman and Tverskys decision
making research and theory - Heuristic
- a problem solving method that provides a
reasonable, probably correct decision - selecting among actions
- multi-attribute alternatives
- A very practical outcome of critical thinking
6Heuristics and Decision Making
- Heuristics and thinking errors
- Availability heuristic
- Representative heuristic
- Base-rate fallacy
7Heuristics and Decision Making
- Availability heuristic
- If a word of three letters or more is sampled
from an English text, is it more likely that the
word starts with r or has r as the third
letter?
8Heuristics and Decision Making
- Representative Heuristic
- Linda is 31-years old, single, outspoken and
very bright. She majored in philosophy. As a
student, she was deeply concerned with issues of
discrimination and social justice, and also
participated in anti-nuclear demonstrations. - Considering this description, which of the
following statements is most likely? - A Linda is a bank teller.
- B Linda is a bank teller involved in the
feminist movement.
9Heuristics and Decision Making
- Base rates
- A taxi-cab was involved in a hit-and-run
accident. Two cab companies, the Green and the
Blue, operate in the city. You are given the
following data - Eighty-five percent of the cabs in the city are
Green while 15 percent are Blue. - In court, a witness identified the cab in
question as Blue.
10Heuristics and Decision Making
- The court tested the ability of the witness to
identify cabs under a variety of conditions.
When presented with a series of pictures of cabs
in which half were blue and half were green, the
witness correctly identified the cabs in 80 of
cases. - What was the probability that the cab involved
in the accident was Blue rather than Green?
11Heuristics and Student Thinking
- These errors are how typically people think.
- Bad news This is how people typically think.
- Good news Adults seem to naturally use
heuristics when making decisions. - Challenge 1 Get students to use more valid
heuristics. - Challenge 2 Measure students use of valid
heuristics.
12CCTST and Heuristics
- Even as a special interest reporter and
weatherman in Indiana, I realized the
importance of assessing critical thinking. You
there, the third person from the left in the
second row, what do you wonder about the CCTST?
13Critical Thinking Dispositions
- Alas, a topic only an academician could love
14Critical Thinking Dispositions
- Suppose that you, taking a page from Super Dave
Osbornes playbook, decided to teach your
students every critical thinking skill known to
humankind - every law of logic,
- every handy heuristic,
- every rule of rhetoric,
- every competency a student needs in order to ace
the CCTST - (which the Faciones insist has never been done)
Photo from http//www.tommcmahon.net/2004/04/bob_e
instein_ak.html
15Critical Thinking Dispositions
- would your CT course float?
- Im not convinced.
- (Paul, you agree?)
- First, like Super Dave, you would hurt yourself.
- Second, even if your students can think
critically, they may not order their intellectual
behavior so that they do think critically.
Photo from http//www.tommcmahon.net/2004/04/bob_e
instein_ak.html
16Critical Thinking Dispositions
- Is our ultimate goal to develop students who can
think critically, or students who do think
critically? - These characteristics are not identical
- in recognition of this, the good folks at Insight
Assessment have developed the CCTDI, as a
companion to the CCTST, to measure student
development toward the latter (greater?) end.
17Critical Thinking Dispositions
- The CCTDI measures intellectual behavior along
seven scales - Truthseeking the intellectual integrity to
follow reason and evidence wherever it may lead - Open-Mindedness tolerant of divergent views,
self-monitoring for possible bias - Analyticity demanding application of reason and
evidence, inclined to anticipate consequences - Systematicity valuing organization, persistence
even in complex problems - CT Self-Confidence trusting ones own reasoning
- Inquisitiveness eager to acquire knowledge,
even when applications of the knowledge are not
immediately apparent - Maturity of Judgement prudence in making,
suspending, or revising judgement awareness of
circumstances that require closure even in
absence of complete knowledge - http//www.insightassessment.com/test-cctdi2.html
18Critical Thinking Dispositions
- The CCTDI measures intellectual behavior along
seven scales - These categories were selected to correspond to
the attributes of critical thinkers identified in
a 1990 report from the American Philosophical
Association - The ideal critical thinker is habitually
inquisitive, well-informed, trustful of reason,
open-minded, flexible, fair-minded in evaluation,
honest in facing personal biases, prudent in
making judgements, willing to reconsider, clear
about issues, orderly in complex matters,
diligent in seeking relevant information,
reasonable in selection of criteria, focused in
inquiry, and persistent in seeking results which
are as precise as the subject and the
circumstances of inquiry permit. - (P. Facione, Delphi Report, 1990, p. 3)
19Critical Thinking Dispositions
- Is this anything?
- Consider the following
- The CCTDI is normed 4, with descriptive and
quantitative classifications
20Critical Thinking Dispositions
- Is this anything?
- Consider the following
- The CCTDI has been tested for reliability 2,
with Cronbachs Alpha .90 for overall disposition
and a range of .72 to .80 for each of the seven
subscales.
21Critical Thinking Dispositions
- Is this anything?
- Consider the following
- In prior studies, r 2 values for the relationship
between CT skills and CT dispositions were
consistently below 0.16, with r 2 0.04 for the
overall relationship between CCTST and CCTDI
scores 2.
22Critical Thinking Dispositions
- Is this anything?
- Consider the following
- While the CCTST focuses largely on a particular
set of critical thinking skills (its no surprise
that r .708 for CCTST vs GRE Analytic), the
CCTDI may allow us insight into other critical
thinking characteristics. (Nancy, heres a
partial answer to your question.)
23Critical Thinking Dispositions
- Fast-Breaking News
- A new and improved dispositional inventory, the
CM3, is now available. Preliminary studies on
groups of adolescents suggest that the CM3 may
provide greater reliability and factor stability
than the CCTDI 3.
24Critical Thinking Dispositions
- Fast-Breaking News
- The CM3 classifies CT disposition according to
four factors 3 - Learning Orientation disposed toward increasing
ones knowledge, values learning process, seeks
challenges - Creative Problem Solving solves problems with
original ideas, engages in puzzles, desires to
understand underlying function of objects - Mental Focus diligent, systematic, organized,
clear-headed - Cognitive Integrity interacts with different
viewpoints for the sake of seeking truth or
making best decision, intellectually curious,
fair-minded
25Critical Thinking Dispositions
- Two Guys and aWorkshop Recommend
- If you are teaching a CT course, you might be
well advised to consider using skills and
disposition tests in tandem as a means of course
assessment. (CCTSTCCTDI or TERCM3)
26Course-Level Assessment of CT
- What are the domains of critical thinking
that we assessing? - What measures must you use as CT course to assess
these domains? - Are you limited to using these measures to assess
CT?
27- Excellent answer, my friend. You are a lot
better than those suckers at State U. Now,
please accept this toast on a stick as a reward
for your superior insight and generally pleasant
demeanor.
28Critical Thinking Assessment Framework
- Critical Thinking Skill
- Analysis
- Evaluation
- Inference
- Divergent Thinking
- Instructor Identified
- Assessment Measure
- California Critical Thinking Skills Test
- Writing Rubric
- CCTST
- ?
- Instructor Responsibility
29Assessing CT in Your Course
- Ideas and examples
- Start
- Build on what you already do to teach and assess
critical thinking - Start
30University and General Education Level Assessment
- A Modest Proposal
- Administer the CCTSTCCTDI (or TERCM3) to a
random sample of freshmen on Assessment Day, and
readminister to the same sample (those
persisting, matched pairs design) on Assessment
Day in the junior year.
31University and General Education Level Assessment
- A Modest Proposal
- This design would ostensibly allow us to measure
and assess change in CT skills and dispositions,
and would allow us to control for potential
confounding (general change in maturity and
intellectual skills/habits through first to years
of college), as some fall semester
juniorshopefullyhave not yet completed a CT
course.
32University and General Education Level Assessment
- A Modest Proposal
- By classifying data according to number and type
of CT courses taken by junior year, we couldin
principlediscern the impact of the designated CT
course model on student CT skills and
disposition.
33University and General Education Level Assessment
- A Modest Proposal
- Warning This is not official policy or
processjust an idea at this point. - Lots of people far more important than Jones and
Douma need to weigh in on this first.
34An Exercise in Understanding and Evaluating an
Argument in a Biased Context
- http//www.crispinsartwell.com/meds/math.htm
- In light of Dr. Sartwells article, should USF
require all of its students to study mathematics,
regardless of their major or career plans? - (assignment used in various forms in MAT151)
35Sample CT Proposal and Course Syllabus
MAT151 Proposal MAT151 Syllabus (Fall
2002) MAT151 Assessment Report to APC MAT151
Final Report to Bush Foundation
36If you need hep, here I am
Keith Jones Jason Douma at your service
37Critical Thinking Dispositions
- List of References
- 1 American Philosophical Association, The
Delphi Report Research Findings and
recommendations prepared for the committee on
pre-college philosophy, Critical Thinking A
Statement of Expert Consensus for Purposes of
Educational Assessment and Instruction, P.
Facione (project director), 1990. - 2 Facione, P., N. Facione, and C. A. Giancarlo,
The Disposition Toward Critical Thinking Its
Character, Measurement, and Relationship to
Critical Thinking Skill, Informal Logic (2000),
20(1), 61-84. - 3 Giancarlo, C.A., S. Blohm and T. Urdan,
Assessing Secondary Students Dispositions
Toward Critical Thinking Development of the
California Measure of Mental Motivation,
Educational and Psychological Measurement (2004),
64(2), 347-364. - 4 Giancarlo, C.A. and P. Facione, A Look
Across Four Years at the Disposition Toward
Critical Thinking Among Undergraduate Students,
The Journal of General Education (2001), 50(1),
29-55.
38End Note
- 1. The material on heuristics was adapted from
Gilhooly, K. J. (1996). Thinking Directed,
undirected, and creative. San Diego Academic
Press.
39CT Assessment Resources
- http//www.criticalthinking.org/
- This is the Foundation for Critical Thinking
(Richard Pauls group). Like a corporate
conglomerate, they have merged with several other
groups and changed some of their names
(International Congress on Critical Thinking is
now International Conference of Critical
Thinking). Still, I believe this group is one
of the most listened-to voices in the critical
thinking community. - http//innumeracy.com/criticalthink.htm
- This is a nifty collection of information that is
consonant with both the CCTST perspective as well
as Richard Pauls group (indeed it includes links
to both). The emphasis here, as the innumeracy
title suggests, is toward the quantitative,
deductive, and analytic however, I believe some
of this material is universal. This is actually
the site that received my praises in this
afternoons conversation. The alleged New
Mexico site (see mea culpa below) is one of the
dozens of links from this site which I had
followed a few weeks ago. -
40CT Assessment Resources
- http//admin.santafe.cc.fl.us/acres/ct/tutorial/
- See how another school is using the CCTST.
- http//www.usiouxfalls.edu/faculty/Development/Cri
tical20Thinking20Proposal20Guidelines.doc - Okay, you probably could have accessed this link
yourself, but just for convenience heres the
document that outlines current APC policy for
approving designated CT courses.